Have to ever seen this message while using a website HTTP Error 429 too many requests.?
That is because the website owner implements rate limiting on server side to prevent overuse of APIs. we should put rate limiting on our authentication APIs like login, Send OTP, Verify OTP, etc…
Today there are many software (Burp Suite is best once) to call brute-force attack to break our authentication flow. with the help of rate-limiting, we can restraint an IP to hit our APIs a certain number of times in a specific time window. In this article, we will discuss how to implement Rate-Limiting with Fail2Ban and Nginx.
Basic knowledge of Ip-tables, Nginx is needed to understand this article better.
We are using Nginx as a web server. Our jars running behind Nginx. Client hit request on our Nginx server. Nginx passes it to our jars and again the response from jars will reach the client through Nginx server.
Funda : Fail2ban continously parse nginx access logs. When match count of our regex which we wrote in filter config file against access logs will reach its limit in a blocking time window, fail2ban make that IP entry inside IP table as blocking action and our server start blocking further requests. after bantime period is over fail2ban remove that IP entry from IP table and server start accepting request from that IP Address.
Fail2ban: fail2ban is a real-time log parsing tool. which have 3 components Filter, Jails, Action.
- JAILS: Jail is a config file where we define our rules. in what time window fail2ban should check match count of our filters against our logs file and what action fail2ban should take place when that match count exceeds its limit in our blocking time window.
filter—points to filter file which contains regex.
action —pointed to action file which executes an action based on an event (block, unblock..)
Others parameter are self-explanatory
- FILTERS: Filter is a config file that contains regex to match against access.log file. we can define multiple regexes here, if a single regex match, fail2ban increases its match count by 1. If multiple regexes match against a single request log still, it increases the match count by 1.
Based on your Nginx access.log pattern decide your regex for APIs
- ACTION: Action is a config file that contains actions that execute at different events.
actionstart — actions will execute when fail2ban starts.
actionstop — actions will execute when fail2ban stops.
actionBan — actions will execute when fail2ban finds that an IP not following rules which we define in jail config.
actionunban-- actions will execute when bantime period is over.
Define fail2ban logs path inside
fail2ban.localconfig file which helps you to debug if anything wrong happenslogtarget = /var/log/fail2ban.log
Nginx log format
log_format custom ‘$remote_addr — $remote_user ”$request” $status $body_bytes_sent $request_time $upstream_response_time “$http_user_agent” “$http_x_forwarded_for” “$request_body” “$connections_active” “$p roxy_host”, “$upstream_addr” “$proxy_add_x_forwarded_for”Using fail2ban and IP tables we can block IP address if our client hit request direct on our Nginx server. In this case $remote_addr is client IP and IP Tables can block that IP.
If we are using some proxy server like AWS CDN or Load balancer then the client hits request on CDN and CDN server forward that request to our Nginx Server, in that case, $remote_addr contain our CDN server IP address and CDN send actual client IP inside $http_x_forwarded_for header and we do not want to block our proxy server address. $http_x_forwarded_for header comes inside network packet and IP Tables can’t able to detect client IP inside that packet.
In that case, we need to set up 2 jails and 2 filter config files. One is to match requests which hit our Nginx server directly and the other when request comes via CDN server. (to distinguish these request we can configure CDN to send a specific header when request comes through it).
Will explain how to block request when it comes through our proxy server in Next article.
Thanks for Reading!.
