Having a firewall is something that’s necessary, in my opinion, for every server. Not only for those with a Public IP Address, for any server. Not only for IPv4, for any IP version.
Running a firewall and managing it adds overhead to the server administration and most people either ignore it, or use their provider’s firewall. Amazon and Google have done a great work in pushing people to use them in their clouds, but often that’s not enough.
In order to increase the cost for an attacker, some people go one step further, “defense in depth” they call it, and set up a firewall on the actual server, too. That means an issue in the provider firewall (if there’s such a thing at all), won’t easily affect them, assuming the rules of course are correct.
I am one of these people. I try to set up a firewall on the server (or any other intermediary routers) in addition to the one that may be available by the provider. Make no mistake: I know it won’t save me from everything, but it’s a good start.
Debian, the operating system in almost all of my servers, has a very powerful tool included: iptables. Although not as beginner friendly as others, and with many wrappers (like ufw), iptables is all you need to run a firewall on the server.
The past few years however, a new piece of software, called Docker, became really popular, and currently has millions of installations for either development, testing, or production. In my opinion, Docker is a very useful and powerful tool that solves a lot of problems if used properly. Now what “properly” is, I leave to you.
For this reason, I am currently using Docker in as many places as I can, and of course, as many servers as needed. I am not running to “dockerize everything” and run “100% on containers” like some others, but if I were to calculate the percentage of services I run on it, it would be pretty high.
However, when installing Docker using the official documentation as a guide, you end up with some pretty ugly iptables rules: docker creates a virtual interface, docker0, as well as a new interface for every container that is running. After running a few containers, you end up with your iptables rules looking like the following:
Chain INPUT (policy ACCEPT 99M packets, 99G bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 99M packets, 99G bytes)
pkts bytes target prot opt in out source destination
99M 99G DOCKER-ISOLATION all -- * * 0.0.0.0/0 0.0.0.0/0
99M 99G DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0
99M 99G ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
99M 99G ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0
99M 99G ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/0
Chain OUTPUT (policy ACCEPT 99M packets, 99G bytes)
pkts bytes target prot opt in out source destination
Chain DOCKER (1 references)
pkts bytes target prot opt in out source destination
1 1 ACCEPT tcp -- !docker0 docker0 0.0.0.0/0 172.17.0.2 tcp dpt:80
1 1 ACCEPT tcp -- !docker0 docker0 0.0.0.0/0 172.17.0.3 tcp dpt:80
1 1 ACCEPT tcp -- !docker0 docker0 0.0.0.0/0 172.17.0.4 tcp dpt:80
1 1 ACCEPT tcp -- !docker0 docker0 0.0.0.0/0 172.17.0.5 tcp dpt:80
Chain DOCKER-ISOLATION (1 references)
pkts bytes target prot opt in out source destination
99M 99G RETURN all -- * * 0.0.0.0/0 0.0.0.0/0
Moreover, every time you start, stop, or restart a container, these rules will change. This is pretty bad if you want to manage your firewall on the host. Every time you make a change and block a port, the tiniest change in docker can revert it.
This behavior from docker adds an additional obstacle and excuse not to set up a host firewall. It does not prevent one from doing it, it just adds complexity.
For this reason, I decided to publish this tutorial/guide in hope it will help you not only work around the problem, but also, maybe, convince you how easy it is to run a firewall on your host.
The first thing we have to do is prevent docker from adding arbitrary entries, or in any other way touching, our firewall ruleset. If you just want to set up a firewall and don’t have docker, you can skip this section.
Debian, at least in its current version, 8 / jessie, uses systemd. This guide is therefore based on that. Navigate to /etc/systemd/system/ and create a directory named docker.service.d. Now, open the file /etc/systemd/system/docker.service.d/noiptables.conf and add the following content:
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// --iptables=false
The above will ensure you launch the docker daemon with the --iptables=false flag, preventing docker from modifying the iptables.
Now, if you restart docker, the changes will take effect. However, existing entries in the ruleset will remain. Just to be sure, reboot the server instead of trying to remove everything manually with iptables -F calls.
Now, check if you see any iptables rules left. If everything went right, you should not be able to see any rules: iptables -L -n -v.
Congratulations! Now your installation of docker does not pollute iptables with rules. At this point we will set up a proper firewall and also see the rules we have to enter for docker to work.
Adding rules manually to iptables using the iptables command, for many reasons, does not persist. That means in a reboot you will lose all changes made. To solve this problem, Debian includes the iptables-persistent package. Go ahead and install it:
apt-get install iptables-persistent
After running this, you will be prompted to save your IPv4, and then your IPv6 rules to two files, /etc/iptables/rules.v4 and /etc/iptables/rules.v6 respectively. Agree on both.
At this point, there is a firewall running on the host but it has no rules. As you probably guessed, we can add and remove rules by modifying the two files above. Keep in mind that both versions of IP, v4 and v6 are important. There’s no point closing the MongoDB port in IPv4 and then allow anyone to access it without problem over IPv6.
Let’s open the first file, /etc/iptables/rules.v4 now, to add some rules. As you can see, in the beginning of the file there is a comment, starting with #. You can add comments to any part of the file by prepending # to them.
Depending on your firewall configuration, you will see various rules there. If you do not know what these rules are and want to start from scratch, backup your file and then delete everything between :OUTPUT and COMMIT.
Now go ahead and allow all traffic in the lo interface, and the 127.0.0.0/8 network by adding this between the two lines above:
# Allow localhost
-A INPUT -i lo -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
Then, add the following rule to allow ICMP (ping) traffic:
# ICMP
-A INPUT -p icmp -j ACCEPT
Now go ahead and add the configuration necessary for docker to work:
# Docker
-A FORWARD -i docker0 -o eth0 -j ACCEPT
-A FORWARD -i eth0 -o docker0 -j ACCEPT
Here you need to replace eth0 with the interface you have to access the Internet (or some other network). If you have multiple interfaces, add more lines. You should end up with two lines per interface. If you don’t know which interface you have to connect to the Internet, run ifconfig.
Now go ahead and add three sections to this configuration file:
# Incoming
-A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
# Outgoing
# Routing
Inside these three sections you will add your firewall configuration. The first section, Incoming, will handle all the connections sent to your server. The second section, Outgoing, will handle all the traffic originating from your server. The final section, Routing, will handle connections going through your server.
As you can see, the first section is already including your first actual rule. This rule allows incoming connections if you specifically asked for them. That means that every time you make an HTTP request (outgoing), your firewall will automatically allow all the traffic that comes back as a response (incoming) for this HTTP Request. However, if someone contacts your server at port tcp/80, even during this HTTP request you sent, it will not allow that connection.
Now let’s go over the three sections again and see an example configuration. At this point you should modify this configuration to match your setup. There are plenty of resources online on how to write iptables rules.
In the incoming section, we need to block all incoming traffic except for the services we set up in purpose. For this reason, we’ll add the following rules, that will block all incoming traffic except SSH, and HTTP(S):
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -j ACCEPT
-A INPUT -j DROP
As you can probably tell, iptables evaluates all rules and stops evaluating them as soon as the first match is found. Similarly, you can add tcp and udp ports for your services to have them accept connections.
In the outgoing section, we can either allow all traffic, or allow all traffic except some IP Addresses and/or ports. The following configuration will allow all traffic except SMTP, and IP Addresses in the 10.0.0.0/8 subnet:
-A OUTPUT --dport 25 -j DROP
-A OUTPUT -d 10.0.0.0/8 -j DROP
-A OUTPUT -j ACCEPT
Finally, in the routing section, unless this computer needs to route traffic, we should just drop all traffic:
-A FORWARD -j DROP
The above is good to have for completion, despite the fact that the relevant options in sysctl are disabled.
At this point, it is recommended to configure the rules.v6 file on your own for practice. Most of the configuration should stay the same really, unless you have a different server behavior when using IPv6.
Now since this is the beginning, and a bad firewall configuration can lock you out of the server, I recommend adding your IP Address (v4 or v6) in the Incoming and Outgoing sections, so you will not be blocked:
-A INPUT -s 192.0.2.2 -j ACCEPT
-A OUTPUT -d 192.0.2.2 -j ACCEPT
You should use a different IP Address to verify your configuration, and if you’re satisfied and can at least ssh into the server, you can remove these lines.
Now it’s time to load our rules into the firewall. Every time you change a file, it is not loaded to the firewall and you have to do it manually. However, a server reboot will load the new configuration.
To load the new rules into the firewall, simply run:
netfilter-persistent reload
You will see an output similar to this:
run-parts: executing /usr/share/netfilter-persistent/plugins.d/15-ip4tables start
run-parts: executing /usr/share/netfilter-persistent/plugins.d/25-ip6tables start
If not, it means that one or more of your rules failed to be added and the action has been stopped. Note that iptables-persistent will automatically remove the entire previous ruleset and substitute it with the one in the file.
At this point you should be able to launch docker containers normally, as well as have a fully set up firewall on your Debian or Ubuntu server. For additional security points, do set up any provider firewall with similar rules and never run apps that don’t have to be accessed from the Internet on 0.0.0.0 (::). Instead, use 127.0.0.1 (::1) as the listening IP Address.
Please keep in mind that a firewall is only a step towards a secure server. It is something very easy to set up and maintain and after using it for some time it will feel completely natural. Also note that the less things you allow in the firewall while still being able to run the server without issues or hacks the better for you. By using a firewall you can control the exposed attack surface that someone will see.
Thanks for reading, and if you have any questions or recommendations, do let me know in the comments below!
This post is inspired by @jeekajoo’s docker and iptables post.
In the comments, a lot of people mentioned that I forgot to add the rules to provide the containers with Internet access. This is true, and it’s my bad.
In order to give IPv4 Internet Access to all the containers, the server must perform NAT. The same plague that runs at every home.. ;-)
To do that, in the beginning of the rules.v4 file, add the following:
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -o eth0 -j masquerade
COMMIT
And then below it, it must continue with *filter. Of course, you must replace eth0 with your outbound network interface if it is different than eth0.
After you complete that, restart the firewall via netfilter-persistent reload, and you’re good to go!