Fail2ban might be, in my own humble opinion, the most useful software that was made for Linux. If you are managing a Linux server that is opened on the Internet, you should know that at any time, there is a bot (run by someone) somewhere on this planet, trying to get in the server by brute forcing an account. In this post I will talk how Fail2ban works, what you need to run it, give some example on how to use it and how you may improve the security of your Linux server by using it. I will use for the demo the SSH service, which is the most used one and will also explain how you can secure a mail server with all of the services attached to it, POP3, IMAP, SMTP and authentication and how you can also fight against spammer.

I’ve discovered the existence of Fail2ban a couple of years ago when one of the server I was managing, was hacked on a Sunday (there are more people trying to hack on a Sunday compare to any others given days of the week by the way). We had created a test account for testing one of the service that the server was offering 3 days before that and before leaving for the week-end we completely forgot to flush that account, which had for testing purpose, a simple password. Shame on us for that. That account was hacked by a simple brute force attack. We discovered the problem on Monday morning and delete the problematic account by then. When talking to a fellow admin colleague about that story, he quickly mentioned that by using Fail2ban on that server it would have prevented that brute force attack. That is when I discovered one of the most useful software for Linux that ever existed.

Fail2ban is a log parser, it reads, in real time, whatever log file that you have configured it to read. Based on certain condition that will happens in the log, Fail2ban will then do an action. One of the most used feature that people use Fail2ban for is to prevent bot from trying to brute force the SSH service. A bot will connect on the SSH port (22 by default) and will try different password for different account, most of the time, it will be the root account. I know that a lot of of admin will simply change the default port of the SSH service from 22 to whatever in order to prevent bot from trying. It can helps, but it’s not the ultimate solution. So all connection that are made to the SSH service is logged, successful ones and failed ones too.

From this sshd log file, you can see many information, first of all the date and time will be displayed (the date has been removed from this  example), the remote IP address of the bot trying to brute force the account root and that the attempts has failed. All the information that Fail2ban needs is there. So when a condition is met in the log file, Fail2ban will then do an action, it can be anything ; play a sound, start a program, open the cdrom tray if you want. But for the sake of this article the only action that we will look for is to add some rules to the local firewall of the server that has Fail2ban installed. Note also that Fail2ban support a lot of different firewall. I’m currently using it with iptables on Linux and ipfw for FreeBSD. To understand how Fail2ban works for stopping brute force attack, some basic knowledge of the firewall that you are using is necessary.

So how does it work? After you have successfully installed Fail2ban as a service/daemon (make sure that you have python running, Fail2ban needs it and also please note that all the examples that are given in this article are for Fail2ban version 0.8.0), the first thing you need to do before you start playing with the configuration is to copy the config file of /etc/fail2ban/jail.conf to /etc/fail2ban/jail.local – this is to prevent from losing your configuration when you are updating fail2ban – if you update Fail2ban, the jail.conf might be overwritten. All your configuration will be done inside that new file, the jail.local. Fail2ban will start looking for the jail configuration first in the jail.conf and then after that in the jail.local. So like I said, always make your jail’s configuration inside the jail.local only. At the beginning of the jail.local file are some general configuration that can be use globally for your jail’s configuration. These global configuration can be overridden inside any specific jail inside that same file. Most of these configuration are pretty self explanatory and their definition are written beside them inside the jail.local file. You have the ignoreip which Fail2ban use for ignoring all of the jail’s configuration that are activated with the IP address specified. The bantime, in seconds, how many time the offending remote host will be banned, findtime, the time window that the remote host may try before hitting the “maxretry”which in itself, is how many tries that the remote host may have before being banned from the server. So if I have 120 seconds for the findtime and 3 for the maxretry, the remote host has 3 shots for trying to log in under 120 seconds. At the fourth failed attempt under the time of the findtime value, the remote host will be banned for 3600 seconds, if that is the value of bantime.

Now let’s take a look a the jail’s itself. What is a jail? A jail is a set of configuration for a specific log file. You have many jails available by default inside the jail.local file and all of them are deactivated. Let’s take a closer look at the SSH jail in details to understand how it works.

First of all there is the name of the jail between the square brackets, that one being the SSH Iptables. The first line of the jail is if it’s enabled or not. To enable it you simply type true and to disable it you type false instead. The next line is the filter for the log file itself. In this case the filter name is the name of a file located under the /etc/fail2ban/filder.d/ folder. All of the filter’s file have the .conf appended in their name. Let’s open up the file sshd.conf

The filter file is populated with regular expression, simple search pattern to look for in the log file that you use. In the case of our example, the third line of this filter is the one that will match the failed login for the brute force attack. All the lines that you see here are different way of of displaying failed logged in attempt on the SSH service. So when the filter has hit a match in the log file, the jail of Fail2ban is going for the next step, the action itself. Like I said earlier, there is a lot of different things you can do with the actions. For the continuity of our examples, 2 different actions are done here. The first one being, that we add the IP of the remote host trying to login to the iptables firewall and then, we send an email with the WHOIS result of that remote IP address to the admin taking care of that server. All the actions definition are stored in a file that is located under the /etc/fail2ban/action.d/ folder. You can also configure parameters with the action that you will find between the square brackets []. There are a lot of actions created by default with fail2ban and you just need to pick one for your need. If not, you can always create your own action or filter if you are up to it.

A quick word on some restrictions or weird behavior you might encounter with Fail2ban, mostly running on VPS (Virtual Private Server). When you start Fail2ban from the first time, it will need to parse the complete log file that you have enable it to check, so it might take a couple of minutes before Fail2ban can start working (i.e. get to the end of the log file). The bigger the log file size, the longer it will be, of course. There is also some inner knowledge to understand about the “real time” behavior of Fail2ban. The logging that happens inside a log file (or to syslog) is not in real time but has a slight delay, caused by the buffer of the logging process and/or the speed of the hardware of the computer. This result sometimes on the part of Fail2ban that it has successfully detected a failed login attempt, but will not be able to add the remote IP of the host to the firewall because Fail2ban hasn’t got the time to get the remote host IP address. The result is that you get the email warning about the brute force attack but when you check your firewall tables about the IP blocked, it’s empty. To resolve that particular issue, you simply need to add a small random timer to the action file you are using. Where the action start, simply add a sleep waiting time with a perl random timer generation like this before the rest of the action.

actionstart = sleep `perl -e 'print rand(3);'`
iptables -N fail2ban-<name>
iptables -A fail2ban-<name> -j RETURN
...

You can change the number between the parenthesis, it’s the maximum number of seconds before the action is started. In our example, the function is to wait between 0 and 2 seconds. That was for the weird behavior part, for the restriction part, if you cannot use a firewall on your server, for whatever reason, you can use instead the tcp-wrapper action. Instead of putting an IP address to the firewall tables it will be stored inside the hosts.deny file. For more details, check the hostsdeny.conf file under the action.d folder of Fail2ban. You will also find a tcp-wrapper jail inside the jail.local file as an example.

You can also overwrite global configuration for a specific jail by adding a line for that configuration inside the jail you want it for. If your global config give a value of maxrety of 5, you can set for one jail a different value by simply retyping that config inside the jail you want. A quick word also for the recidivism. Just like the word says it, even tough you will ban a specific IP for X number of hours from your server, after that ban period, nothing is preventing the remote host for trying again to brute force your server. One of the recent release of Fail2ban has fixed that by adding a recidive jail in the jail.local file. That jail is pointing to the Fail2ban log itself. Pretty weird uh? It should be the last jail located at the bottom of the file. Let me give an example to explain it. So let say you are banning for 3 hours the remote IP of someone trying to brute force any service that you have enabled with Fail2ban. But after 3 hours, the ban on that IP is lifted/removed and the same remote IP keep trying back. All of that are logged inside the /var/log/fail2ban.log file by default. By enabling that jail you can set a higher ban time for the recidivism remote on your server. The given example below will block the offending remote IP address on the 6th ban in the same day and will then ban that IP from all port of your server for a period of 1 week. You will probably need to adjust these settings to make it work with your global configuration. This jail is one of the most useful from Fail2ban, don’t ignore it.

[recidive]
enabled  = true
filter   = recidive
logpath  = /var/log/fail2ban.log
action   = iptables-allports[name=recidive]
           sendmail-whois-lines[name=recidive, logpath=/var/log/fail2ban.log]
bantime  = 604800  ; 1 week
findtime = 86400   ; 1 day
maxretry = 5

That was for the easy part. Now if you have gone through all of this you might have a good idea of all the possibilities that Fail2ban can give you. I’ve been using Fail2ban for some different services over the years and I’m still impressed with everything I can do with it. Lately I’ve been using it to prevent spamming against one of my emails server that I managed. That email server has a lot of users, virtual domains and we get attacked by spammers from time to time, heavily. Of course, that server has many protection, antivirus, antispam and Fail2ban to protect against brute force attack for the POP3, IMAP and the SMTP service (don’t forget to add the authentication mechanism to the Fail2ban list, whatever you are using, PAM, SASL, MySQL, etc. Simply use the existing filter that comes with Fail2ban for all of these services). But sometimes, all these layers of protection cannot prevent the server from getting a ton of emails that is unwanted. One of the specific attack you can get is your server being swamped by spam that are sent to non-existing email address. Even tough these emails are not delivered to existing inbox, your server is losing processing power to flush the spam and reply to the sender that these emails addresses doesn’t exist. And do I need to mention that the reply are sent back to a probably fake SMTP server? That’s great (really not) and I hate it. To prevent that, Fail2ban to the rescue! By using this filter for your SMTP server, you will ban remote IP address that are trying to flood you with spam :

failregex = reject: RCPT from (.*)\[<HOST>\]: 550 5.1.1
            reject: RCPT from (.*)\[<HOST>\]: 450 4.7.1
            reject: RCPT from (.*)\[<HOST>\]: 554 5.7.1

These 3 lines represent 3 specific error code (user’s mailbox not existing, access denied and relay not permitted). Of course, it will not stop spam that are sent to legitimate mailboxes. With that filter you can then use a jail to make it work and tweak it to your own desire. It’s not the solutions to all of the problems, but it will help your email server. The possibilities of Fail2ban are endless, you can use it to protect your databases, FTP access, your WordPress login page, against spam from the comments system of WordPress or even protect your web server against DDOS attack. The rest is up to your imagination!

Update : For those of you who are interested in stats, someone has registered the default email address of fail2ban for reporting the attack, the email address is fail2ban@mail.com. A lot of people who has installed fail2ban has never change this and since the address is real, that person has trapped all the email and has put a webpage for sharing some interesting statistics. Check out the page here.

Furthermore ; I would also suggest that you take a look at the service of blocklist.de It is a free service for reporting the attack to (registration required). With it you can also download a global list of IPs that are being reported as attackers from all around the world and use it with your fail2ban installation.