[SOLVED] Apparently I broke my nftables

Kernels & Hardware, configuring network, installing services

[SOLVED] Apparently I broke my nftables

Postby MediumRar » 2020-03-30 14:04

Hi! :)

It seems I somehow broke my nftables during the last few days on my Debian Buster remote Server. Everything has been working really well up to this point and now all of the sudden systemd always fails upon starting my nftables.service:

Code: Select all
root@XXXXXX:~# systemctl start  nftables.service
Job for nftables.service failed because the control process exited with error code.
See "systemctl status nftables.service" and "journalctl -xe" for details.

Code: Select all
Mar 30 16:15:30 XXXXXX systemd[1]: Starting nftables...
-- Subject: A start job for unit nftables.service has begun execution
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit nftables.service has begun execution.
--
-- The job identifier is 572.
Mar 30 16:15:30 XXXXXX nft[1794]: /etc/nftables.conf:3:1-14: Error: Could not process rule: Operation not supported
Mar 30 16:15:30 XXXXXX nft[1794]: flush ruleset
Mar 30 16:15:30 XXXXXX nft[1794]: ^^^^^^^^^^^^^^
Mar 30 16:15:30 XXXXXX nft[1794]: /etc/nftables.conf:5:1-2: Error: Could not process rule: Operation not supported
Mar 30 16:15:30 XXXXXX nft[1794]: table inet filter {
Mar 30 16:15:30 XXXXXX nft[1794]: ^^
Mar 30 16:15:30 XXXXXX nft[1794]: /etc/nftables.conf:6:15-19: Error: Could not process rule: Operation not supported
Mar 30 16:15:30 XXXXXX nft[1794]:         chain input {
Mar 30 16:15:30 XXXXXX nft[1794]:                      ^^^^^
Mar 30 16:15:30 XXXXXX nft[1794]: /etc/nftables.conf:9:15-21: Error: Could not process rule: Operation not supported
Mar 30 16:15:30 XXXXXX nft[1794]:         chain forward {
Mar 30 16:15:30 XXXXXX nft[1794]:                      ^^^^^^^
Mar 30 16:15:30 XXXXXX nft[1794]: /etc/nftables.conf:12:15-20: Error: Could not process rule: Operation not supported
Mar 30 16:15:30 XXXXXX nft[1794]:         chain output {
Mar 30 16:15:30 XXXXXX nft[1794]:                      ^^^^^^
Mar 30 16:15:30 XXXXXX systemd[1]: nftables.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- An ExecStart= process belonging to unit nftables.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Mar 30 16:15:30 XXXXXX systemd[1]: nftables.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- The unit nftables.service has entered the 'failed' state with result 'exit-code'.
Mar 30 16:15:30 XXXXXX systemd[1]: Failed to start nftables.
-- Subject: A start job for unit nftables.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- A start job for unit nftables.service has finished with a failure.
--
-- The job identifier is 572 and the job result is failed


And this is with the very very basic configuration of nftables.

Code: Select all
#!/usr/sbin/nft -f
 
flush ruleset

table inet filter {
    chain input {
        type filter hook input priority 0;
    }
    chain forward {
        type filter hook forward priority 0;
    }
    chain output {
        type filter hook output priority 0;
    }
}


I'm using this one now, since I first thought that there is something wrong with my actual config, which doesn't seem to be the case.
My initial Setup, looked like the following:
- iptables installed and running without blocking anything
- fail2ban inserting iptables rules to block specific IPs
- nftables basically only allowing input on port 22, 80 and 443

Although https://wiki.debian.org/nftables says, that this is discouraged according to https://wiki.nftables.org/wiki-nftables ... leshooting there should not necesserelly be an issue with my setup. I can confirm that, it worked like a charm for weeks. It was only today, when I noticed that nftables is not able to startup anymore after I wanted to add a new port to accept.

This is actually my very first time asking for help online regarding stuff like this, since I can barely find any help on issues with nftables online.

Okay. So what have I tried to solve the issue?
- Well first of all I reset my config, as mentioned above, to the bare minimum -> Nothing
- Second of all I removed my custom .service file from /etc/systemd/system
-- I created a python script that initiates a server socket and communicates over a certain port to a remote client
- I shut down fail2ban
- funnily enough I did not find an option to stop iptables :oops: (Except for the update-alternatives thingy in the Wiki mentioned below)
- I did however, more or less, try everything mentioned in the Debian Wiki (https://wiki.debian.org/nftables)
- And then finally tried to reinstall nftables from scratch

Well... Nothing....

I also tried to type in some rules manually, only to find out that I'm ending up with the same result:
Code: Select all
root@XXXXXX:/etc/apt/sources.list.d# nft list ruleset
root@XXXXXX:/etc/apt/sources.list.d# nft flush ruleset
Error: Could not process rule: Operation not supported
flush ruleset
^^^^^^^^^^^^^^
root@XXXXXX:/etc/apt/sources.list.d#


And yes, I also did try to switch it off and back on again. I am desperate at this point :D

Im running on:
deb http://ftp.debian.org/debian buster main contrib non-free
deb http://ftp.debian.org/debian buster-updates main contrib non-free
deb http://security.debian.org buster/updates main contrib non-free
with no additional sources and everything up to date

I really don't know what to do anymore tbh. I really hope you guys can help me with this. If you need any more Infos, please just let me know :)

KR!
Last edited by MediumRar on 2020-04-17 22:14, edited 2 times in total.
User avatar
MediumRar
 
Posts: 2
Joined: 2020-03-30 12:56

Re: Apparently I broke my nftables

Postby Head_on_a_Stick » 2020-03-30 16:56

MediumRar wrote:Debian Buster remote Server

Is that running in a container or something? See https://serverfault.com/questions/98678 ... y-commands
Black Lives Matter

Debian buster-backports ISO image: for new hardware support
User avatar
Head_on_a_Stick
 
Posts: 12744
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: Apparently I broke my nftables

Postby reinob » 2020-04-01 14:34

@MediumRar,

To me it looks like you're not loading the nftables modules?
Check with "lsmod" if you have nf_tables (and probably a bunch of others). My server has:

Code: Select all
# lsmod | grep nft
nft_masq_ipv4          16384  1
nft_masq               16384  1 nft_masq_ipv4
nft_chain_nat_ipv4     16384  2
nf_nat_ipv4            16384  2 nft_chain_nat_ipv4,nft_masq_ipv4
nft_log                16384  3
nft_reject_inet        16384  2
nf_reject_ipv4         16384  1 nft_reject_inet
nf_reject_ipv6         16384  1 nft_reject_inet
nft_reject             16384  1 nft_reject_inet
nft_ct                 20480  4
nf_conntrack          172032  5 nf_nat,nft_ct,nf_nat_ipv4,nft_masq,nft_masq_ipv4
nf_tables             143360  206 nft_ct,nft_log,nft_chain_nat_ipv4,nft_reject_inet,nft_masq,nft_masq_ipv4,nf_tables_set,nft_reject


If you don't have them, check your boot options (cat /proc/cmdline and see if you have something funny there, or in /etc/modprobe.d).

You could also just "journalctl -b" and review everything.
reinob
 
Posts: 847
Joined: 2014-06-30 11:42

Re: Apparently I broke my nftables

Postby MediumRar » 2020-04-14 07:29

Hi again,

sorry for taking so long, but this Problem was now resolved with the Help of my SP. Apparently they had a bug in their host kernel which caused this issue. I asked them for a reference in case anybody else experiences similar issues and there you go:

Mon Nov 25 2019 Konstantin Khorenko khorenko@virtuozzo.com [3.10.0-1062.4.2.vz7.116.5]
ms/netfilter: nft_compat: use-after-free when deleting targets (Pablo Neira Ayuso) [PSBM-99656]
ms/netfilter: nf_tables: fix use-after-free when deleting compat expressions (Florian Westphal) [PSBM-99656]
ms/netfilter: nft_compat: fix crash when related match/target module is removed (Liping Zhang) [PSBM-99656]

For further reference:
Yes this bug caused my nftable modules not to be loaded properly "lsmod | grep nft" didn't show anything, whereas now it does.

Thank you guys very much for your help!
- Can be closed -
User avatar
MediumRar
 
Posts: 2
Joined: 2020-03-30 12:56


Return to System configuration

Who is online

Users browsing this forum: No registered users and 17 guests

fashionable