Ramblings of Linux openstack & ceph

Solarflare - Dropping Bad Http Request in Hardware or Softrware

| Comments

I ran into a case where a popular website was receiving a large number of undesirable http requests (>20k per second) that the application was unable to reject efficiently with a 400 range status code without adding more web servers.

It was quickly apparent the traffic all originated from single user-Agent that for the purpose of this post we will call ‘BADAGENT’

The first attempt to block this traffic was done using varnish, this worked well but required multiple servers with high end CPU’s to analysis the http headers and drop them - The code used for this is below

1
2
3

if (req.http.User-Agent == 'BADAGENT') {
  return(synth(403,"refused"));
}

The below was also used to try and block based upon the URL not just the User-Agent to test how flexible this was:

1
2
3

if (req.url == '/badrequest.php') {
  return(synth(403,"refused"));
}

After reading good reviews about haproxy Performance, I moved to dropping traffic in haproxy and then sending it back into varnish - This dramatically reduced the CPU usage compared to varnish but did have the undesirable effect of having to proxy traffic to a high end port number when sending traffic back into varnish - The code used for this is below

1
2
3
4
5

listen ipv4.ipv4.ipv4.ipv4:80 ipv4.ipv4.ipv4.ipv4:80 
mode http
acl bad_user_agent hdr_beg(User-Agent) BADAGENT 
reqitarpit . if bad_keys
        default_backend be_webserver

I also found it possible to provide a file containing a line separated list of BADAGENTS as follows

1
2
3
4
5

listen ipv4.ipv4.ipv4.ipv4:80 ipv4.ipv4.ipv4.ipv4:80
mode http
acl bad_user_agent hdr_beg(User-Agent) -f /etc/haproxy/bad.agents 
reqitarpit . if bad_keys
        default_backend be_webserver

It was also possible to use the above to block on url parameters such as http://domain.com/my_api.php?var1=userbob - We could use the below to block on the url key value pair of ‘userbob’

1
2
3
4
5

listen ipv4.ipv4.ipv4.ipv4:80 ipv4.ipv4.ipv4.ipv4:80
mode http
acl bad_keys urlp(key) -f /etc/haproxy/bad.users
reqitarpit . if bad_keys
        default_backend be_webserver

This really stabilised the environment but I really wanted rid of haproxy so using the Solarflare Communications SFC9120 card with the Filter Engine licence It was possible to drop the traffic in the network card before it even hit Linux - This not only saved CPU but also saved bandwith in responding to the http requests. The bandwidth saving was to the tune of > 50TB PCM.

The code used to do this on the card for a User-Agent is below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

set_max_channels 2
set_default_action accept
set_max_objects 5
set_max_miniaddrs 5
start_code
 accept:
 load 1 r0
 stop
 reject:
 load 0 r0
 stop

 start_http:
 test_ip4
 jmp_if_not accept
 test_tcp4 first_frag
 jmp_if_not accept
 load_ip4_dport r2
 test_eq r2 80
 jmp_if_not accept
 test_http_request pl5 "GET " p2 p1
 jmp_if_not accept
 next_http_hdr:
 test_http_header p1 p2 p3
 jmp_if_not accept
 test_case_match p2 0 "user-agent"
 jmp_if_not next_http_hdr
 test_match p3 0 "BADAGENT"
 jmp_if_not accept
 jmp rej

It is also possible to filter on items within the URL with the below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

set_max_channels 8
set_default_action accept
set_max_objects 100
set_max_miniaddrs 5

start_code
        accept:
                load 1 r0
                stop
        reject:
                load 0 r0
                stop
        rst:
                tcp4_reset
                stop

        # p1 should have the rest of the GET request
        check_keys:
                test_search pl5 0 "userbob"
                jmp_if rst
                test_search pl5 0 "userdave"
                jmp_if rst
                stop

        start_http:
                test_ip4
                jmp_if_not accept

                test_tcp4 first_frag
                jmp_if_not accept

                load_ip4_dport r2
                test_eq r2 80
                jmp_if_not accept

                jmp check_keys
end_code

This was the ideally solution - there are many more possible options with the SolarFlare card, full details with more examples can be found in the user guide

Comments