Rate Limiting with Fail2Ban and Nginx — Part II

Api Rate Limiting

In this series of articles, we are discussing rate-limiting with Nginx and fail2ban. This is our 2nd article. If you haven’t read it, read that 1st Article from before this article to get a better understanding.

Let’s Start

In our first article, we discussed how we can put rate limiting on server APIs with fail2ban and Nginx. But that design breaks if we are using a proxy server like AWS CDN or Load Balancer. In this article, we are going to discuss how we can handle that case also.

FUNDA: when an IP not follow our rules, fail2ban write that IP in map file and reload Nginx config files, that file read by Nginx and with some map block configuration Nginx start blocking that IP request. Again when bantime will over fail2ban remove that IP from map file and reload Nginx config file and Nginx starts accept traffic from that IP.

As we discussed we can configure CDN to send a custom header when a request comes via CDN(to identify the request that comes from our proxy server) then our access log pattern can be

log_format custom ‘$remote_addr — $remote_user ”$request” $status $body_bytes_sent $request_time $upstream_response_time “$http_user_agent” "$http_x_cdn_request" “$http_x_forwarded_for” “$request_body” “$connections_active” “$p roxy_host”, “$upstream_addr” “$proxy_add_x_forwarded_for”’

and Fail2ban filter config file regex can be

auth-proxy.local

We can create a new jail that points to above file as a filter and action(below) will add that IP in a map file that reads by Nginx.

nginx-ip-blocking.local

here we are going to write some Nginx config code.

1. map $http_x_forwarded_for $actual_client_ip {"~[\d.\s,]*?([0-9]
   {1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})$" $1;}
2. map $actual_client_ip $blck_list_ip { include blacklisted-
   ips.map; }
3. location = /fail2ban/testing {
       if ($blck_list_ip != "") 
            return 403;
         }
       return 200;
   }

Step 1: Extracting client IP from $http_x_forwarded_for header and store it in $actual_client_ip variable (last IP is actual client IP)

Setup 2: Populate that IP inside inside $blck_list_ip variable if it exists inside blacklisted-ips.map map file( fail2ban write and remove IPs from this map file)

Define step 1 and 2 in your nginx.config file

Step 3: At starting of our location block, write a simple if block to block that IP.

This way we can block our actual client IP if we are using a proxy server to handle requests.

Basically Article1 and Article2 is complete recipe to design a simple rate-limiting.

Thanks for reading!.