Noticing weird narrow spikes in server load graph, I decided to investigate the most recent one – at 03:50 GMT+2 on Nov. 6, 2010.

The reason was simple: someone issued a few hundred HEAD-requests over a 30 second period to a PHP-based web-application.

All the requests were coming from IP 109.169.59.139, which belongs to the IP range of thrustvps.com:

inetnum: 109.169.58.0 – 109.169.59.255
netname: ThrustVPS_1
descr: Thrust::VPS
country: US
admin-c: RF5058-RIPE
tech-c: RF5058-RIPE
status: ASSIGNED PA
mnt-by: RAPIDSWITCH-MNT

However, it is the referrer string which is more interesting: in all those requests, decorated with varying UserAgents and even operating systems, there was only one referrer – www.dewlance.com.

Initially I thought that was a test of a new DoS attack – really, who would issue dozens of HEAD requests to the same page over a few seconds? However, after seeing that “referrer” string, I now think this is a cheap, blatant, poor and ugly SEO performed by dewlance. It relies on some sites displaying a box of ‘recent visitors’, sometimes including their referrer URL as a “page where this visitor came from” – this would give dewlance.com some free link-love. Or maybe dewlance.com expects administrators to investigate log files, notice that referrer string, and happily order some services from dewlance? No way

I’ll file a complaint with thrustvps if I see that kind of misbehaviour again. All that started on Nov. 4, so there’s still hope people behind this dumb SEO implementation will get fired.

Update 1: they do this every 4 hours since November 4, 2010 (Thursday). This results in loads up to 22, with ~50 apache processes struggling for a few CPU cores:

Update 2: some 20 hours after sending report to abuse at thrustvps.com nothing has changed – still a bunch of HEAD requests every 4 hours. I have written a fail2ban filter+rule to ban anything issuing more than about 1 HEAD request per second. If that rule works as expected – I’ll publish it here.

Update 3: the last HEAD request referring to dewlance occurred at 12:23 GMT on November 8, 2010. I have no idea if that was my complaint, or if that “experiment” just ended naturally.

I have been testing fail2ban rule for false-positives, and it now seems OK. However, I haven’t tested for true positives – I do not know if it will actually block extra HEAD requests (it should).

Put the fragment below into your /etc/fail2ban/jail.local (edit logpath to match your apache logs):

[head-dos]
enabled = true
port = http,https
filter = head-dos
logpath = /var/log/apache2/other_vhosts_access.log
maxretry = 8
findtime = 6
#ban for 25 hours
bantime = 90000
action = %(action_mwl)s

I recommend leaving action as specified for a few weeks to see if you aren’t blocking legitimate requests.

Also paste the fragment below into /etc/fail2ban/filter.d/head-dos.conf:

# Fail2Ban configuration file
#
# Author: bogdan.org.ua
#

[Definition]

# Option: failregex
# Note: this regex matches *any* HEAD requests; together with a maxretry=8 and findtime=6 (for example)
# this rule should ban anything issuing more than ~1 HEAD request per second.
#
# sample matching entry:
# bogdan.org.ua:80 109.169.59.139 – - [07/Nov/2010:04:38:33 +0200] “HEAD /2009/10/27/search-and-replace-in-a-mysql-table.html HTTP/1.0″ 200 – “http://www.dewlance.com” “Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.2 (KHTML, like Gecko) Chrome/4.0.221.7 Safari/532.2″
#

failregex = ^[^ ]+ -.*”HEAD /.*$

# Option: ignoreregex
# Notes.: regex to ignore. If this regex matches, the line is ignored.
# Values: TEXT
#
ignoreregex =

Update: this rule does work. There were a few false-positives over 2 weeks of testing, so you may need to tune number of requests and time period. After the initial HEAD attacks I’ve seen there were more of these, with other referrer strings – but always a website URL.

Please comment to report improvements/enhancements and problems you’ve encountered with this rule.