iptables... nftables... iptables-nft... help me understand!

If none of the more specific forums is the right place to ask

iptables... nftables... iptables-nft... help me understand!

Postby rudepeople » 2020-02-22 20:25

I have a pair of servers running a company website. I'm just the sys admin so I don't mess much with the production software but the devs are using ngix, docker, php, mysql, and a TON of other random apps to cobble togeter the company website (I'm not a webdev and I don;t care to be).

The prod environment is running on debian 10 (buster) on two servers (prod and staging). both have two networks configured (public and private) and out of the box, they running iptables-nft.

recently we ran into issues where docker will randomly change the firewall rules. while the devs are using docker, we have all agreed that docker should not be allowed to actually communicate over the public network. we have firewall rules to prevent this but docker is very insistent on bypassing these rules. After a little digging it seems docker is using iptables commands to achieve its goal of world domination. the solution seems simple, purge iptables and install something docker isn't setup to control... I would think it better to stop docker from changing the rules... but I guess we don't know how to do that.

I'm aware that buster ships with nftables, but it's NOT nftables... its iptables-nft... which is confusing me... is it nftables? or iptables? because it only respects iptables rules and ignores the nftables config files. I decided it wasn't worth the headache so I installed nftables properly and was about to purge iptables when I ran into a problem... docker-ce depends on iptables... because of course it does.

So I decided to leave iptables where it is, disable it, and enable nftables and be done with it... however, now I'm confused because I cant find the firewalld service or any kind of systemd service running iptables or iptables-nft... ANYWHERE! how the heck is this stock firewall running?!? the only thing I could find was a service called netfilter-persistent but disabling that doesnt bring down the firewall... so how is iptables (iptables-nft) running? is there an easy way to strip it from the system without docker noticing?

for now I've just cleared the iptables rules and applied nft rules (which appears to be working)... but it's only a matter of time before docker updates the iptables rules again!
rudepeople
 
Posts: 7
Joined: 2009-03-28 19:14

Re: iptables... nftables... iptables-nft... help me understa

Postby Head_on_a_Stick » 2020-02-22 20:33

rudepeople wrote:I'm just the sys admin

Hmm...

For Debian buster's iptables setup see https://www.debian.org/releases/stable/ ... l#nftables

If you want more specific help then repost with a clear and concise description of the problem. Or wait for somebody with the patience to wade through your crap.
User avatar
Head_on_a_Stick
 
Posts: 11964
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: iptables... nftables... iptables-nft... help me understa

Postby rudepeople » 2020-02-22 20:42

If you want more specific help then repost with a clear and concise description of the problem. Or wait for somebody with the patience to wade through your crap.


Alright, allow me to elaborate:

how do I properly disable iptables-nft in favor of nftables (as described both in your link and this one), considering I can't remove iptables-nft without also removing a necessary application.
rudepeople
 
Posts: 7
Joined: 2009-03-28 19:14

Re: iptables... nftables... iptables-nft... help me understa

Postby Head_on_a_Stick » 2020-02-22 20:51

Code: Select all
# update-alternatives --config iptables

As clearly stated in the release notes.
User avatar
Head_on_a_Stick
 
Posts: 11964
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: iptables... nftables... iptables-nft... help me understa

Postby rudepeople » 2020-02-22 21:03

Code: Select all
 # update-alternatives --config iptables

As clearly stated in the release notes.


I truely wish it was that simple... that only allows me to select between iptables-nft or iptables-legacy. I need to halt/kill/dev/null/murder/death/kill iptables and iptables-nft, not switch it to iptables-legacy. and no, there is no 3rd option for nftables.

Unless there's another option I'm overlooking or isn't present in my selection list? This is the output:
Code: Select all
$ sudo update-alternatives --config iptables
There are 2 choices for the alternative iptables (providing /usr/sbin/iptables).

  Selection    Path                       Priority   Status
------------------------------------------------------------
* 0            /usr/sbin/iptables-nft      20        auto mode
  1            /usr/sbin/iptables-legacy   10        manual mode
  2            /usr/sbin/iptables-nft      20        manual mode

Press <enter> to keep the current choice[*], or type selection number:


I may be the admin, but I don't claiming to know what I'm doing!

EDIT: I realize you may be telling me to simply set the default to iptables-nft... the trouble is, iptables-nft is built as an in-between path to update to nftables... it respects iptables style rules and accepts input from iptables-save and iptables-restore while ignoring nft commands and config files. Basically, iptables-nft might as well BE iptables in our environment.
rudepeople
 
Posts: 7
Joined: 2009-03-28 19:14

Re: iptables... nftables... iptables-nft... help me understa

Postby Head_on_a_Stick » 2020-02-22 21:14

rudepeople wrote:I need to halt/kill/dev/null/murder/death/kill iptables and iptables-nft

Well iptables won't be running unless you've configured it.

Check the output of
Code: Select all
# iptables-save

If it's blank then there are no rules loaded (which is the default configuration).

I use pure nftables myself:
Code: Select all
empty@E485:~ $ systemctl status nftables --no-p
● nftables.service - nftables
   Loaded: loaded (/lib/systemd/system/nftables.service; enabled; vendor preset: enabled)
   Active: active (exited) since Sat 2020-02-22 17:25:55 GMT; 4h 34min ago
     Docs: man:nft(8)
           http://wiki.nftables.org
  Process: 298 ExecStart=/usr/sbin/nft -f /etc/nftables.conf (code=exited, status=0/SUCCESS)
 Main PID: 298 (code=exited, status=0/SUCCESS)

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
empty@E485:~ $ sudo nft list ruleset
table inet filter {
        chain input {
                type filter hook input priority 0; policy accept;
                iif "lo" accept
                ct state established,related accept
                ip6 nexthdr ipv6-icmp icmpv6 type { nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert } accept
                counter packets 358 bytes 25154 drop
        }
}
table ip filter {
        chain INPUT {
                type filter hook input priority 0; policy accept;
        }

        chain FORWARD {
                type filter hook forward priority 0; policy accept;
        }

        chain OUTPUT {
                type filter hook output priority 0; policy accept;
        }
}
empty@E485:~ $
User avatar
Head_on_a_Stick
 
Posts: 11964
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: iptables... nftables... iptables-nft... help me understa

Postby rudepeople » 2020-02-22 21:24

If it's blank then there are no rules loaded (which is the default configuration).


Yep, and right now it is blank... the trouble is, docker periodically execs the following:
Code: Select all
iptables -A DOCKER-USER -j RETURN
iptables -A DOCKER-INGRESS -p tcp -m tcp --dport 8081 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 8081 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m tcp --dport 1337 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 1337 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m tcp --dport 7070 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 7070 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m tcp --dport 3000 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 3000 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m tcp --dport 9000 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 9000 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m tcp --dport 3300 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 3300 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m tcp --dport 4006 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 4006 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m tcp --dport 30001 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 30001 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m tcp --dport 30000 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 30000 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m tcp --dport 27017 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 27017 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m tcp --dport 4000 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 4000 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m tcp --dport 9080 -j ACCEPT
iptables -A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 9080 -j ACCEPT
iptables -A DOCKER-INGRESS -j RETURN


I've looked into killing docker's ability to do this, but it would require the docker devs pulling their heads out of their respective butts and accepting that people don't WANT gaping holes in their firewalls!
You'll notice those are blanket accept rules. and they supersede our deny and drop rules!

I use pure nftables myself:

Me too! just... not on THESE servers... friken docker...
rudepeople
 
Posts: 7
Joined: 2009-03-28 19:14

Re: iptables... nftables... iptables-nft... help me understa

Postby Head_on_a_Stick » 2020-02-22 21:39

rudepeople wrote:docker periodically execs the following

What system is running in the container? Is it Alpine or something?
User avatar
Head_on_a_Stick
 
Posts: 11964
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: iptables... nftables... iptables-nft... help me understa

Postby tynman » 2020-02-23 15:27

The concern is about some piece of software - presumably Docker - adding firewall rules to custom chains named DOCKER-USER and DOCKER-INGRESS. The thing I would want to see is what rule, if any, in the "default chains" (i.e., INPUT, OUTPUT and FORWARD) references those two chains.

It isn't unusual for a piece of software to add a rule to the default INPUT chain to allow client software running on remote computers to connect to the software running on the server. But that should only occur during the install process. I agree, I don't think software should be modifying firewall rules in the default chains on an on-going basis without some good reason. But updating rules in their own custom chains like this is less of cause for concern. Without knowing the reason, it seems stupid, but ??
tynman
 
Posts: 124
Joined: 2016-05-03 19:48
Location: British Columbia, Canada

Re: iptables... nftables... iptables-nft... help me understa

Postby rudepeople » 2020-02-28 14:19

Head_on_a_Stick wrote:What system is running in the container? Is it Alpine or something?


I don't know, this is the devs thing. I just manage the server itself.

tynman wrote:The concern is about some piece of software - presumably Docker - adding firewall rules to custom chains named DOCKER-USER and DOCKER-INGRESS. The thing I would want to see is what rule, if any, in the "default chains" (i.e., INPUT, OUTPUT and FORWARD) references those two chains.

It isn't unusual for a piece of software to add a rule to the default INPUT chain to allow client software running on remote computers to connect to the software running on the server. But that should only occur during the install process. I agree, I don't think software should be modifying firewall rules in the default chains on an on-going basis without some good reason. But updating rules in their own custom chains like this is less of cause for concern. Without knowing the reason, it seems stupid, but ??


this is a nice article about the consern in question written by jeekajoo (link)
in reading his document, you're right, they're custom chains NOT part of the main set. so yes in theory I can block the traffic in the main sequence rules. that said, we've noticed that adding our own rules to counter docker on the public network don't work.... at all.

For example, Docker adds this:
Code: Select all
-A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 8081 -j ACCEPT


so I add this:
Code: Select all
-A FORWARD -p tcp -i enp2s0f0 --dport 8081 -j DROP


my hope is to at least stop docker from communicating outside the private network (if I cant break it's firewall editing ways, I should at least be able to deny it's ability to alter which network it communicates over).
the servers have two nics each, the switch they connect to has two vlans; one private, one public. the private lan is closed to the internet so the devs can set their apps to communicate without a firewall safely. but the public vlan is PUBLIC. that side MUST have firewalls up and secure. but the rule I added to drop docker traffic on the wan port simply doesn't change a damn thing. docker is still very visible to the internet.
rudepeople
 
Posts: 7
Joined: 2009-03-28 19:14

Re: iptables... nftables... iptables-nft... help me understa

Postby tynman » 2020-02-28 16:59

In the link you provided, it mentions being able to override Docker's default behavior - to tell it NOT to update the firewall - by adding
Code: Select all
--iptables=false

to the docker service file. Did you try that?
tynman
 
Posts: 124
Joined: 2016-05-03 19:48
Location: British Columbia, Canada

Re: iptables... nftables... iptables-nft... help me understa

Postby rudepeople » 2020-03-02 22:41

tynman wrote:In the link you provided, it mentions being able to override Docker's default behavior - to tell it NOT to update the firewall - by adding
Code: Select all
--iptables=false

to the docker service file. Did you try that?

Yes. and it works... but it also breaks some of our containers. It looks like we're just going to have to bite the bullet and get an external firewall appliance. I'll probably just throw in a wrap with pfsence on it.
rudepeople
 
Posts: 7
Joined: 2009-03-28 19:14

Re: iptables... nftables... iptables-nft... help me understa

Postby reinob » 2020-03-03 11:29

rudepeople wrote:
tynman wrote:In the link you provided, it mentions being able to override Docker's default behavior - to tell it NOT to update the firewall - by adding
Code: Select all
--iptables=false

to the docker service file. Did you try that?

Yes. and it works... but it also breaks some of our containers. It looks like we're just going to have to bite the bullet and get an external firewall appliance. I'll probably just throw in a wrap with pfsence on it.


Would it be possible to define, yourself, whatever rules are needed at /etc/nftables.conf (i.e. using nft and nft syntax), so that your containers continue to work even if docker is not allowed to touch iptables anymore?

I ran into a similar issue today (toy project, no problem), and noticed that even with "--iptables=false" docker, when starting, will presumably invoke iptables (which is alternative'd to iptables-nft), which creates a "table ip nat" and "table ip filter" (my config had only a "table inet filter"), so it's not even respecting the iptables=false flag.

I'm considering replacing iptables with nop scripts, to make sure the nft configuration is what I want. I will test it someday (soon?).

Keep us posted if you figure out a nice way of making docker behave with debian.

(Rant: I hate it when debian changes something, but only partially. Like systemd when lots of startup scripts were still using /etc/init.d, or now nftables but keeping a messy iptables compatibility..)
reinob
 
Posts: 777
Joined: 2014-06-30 11:42


Return to General Questions

Who is online

Users browsing this forum: No registered users and 3 guests

fashionable