CODEWITHSUNDEEP
amazon-web-servicesamazon-cloudwatchamazon-wafaws-cloudwatch-log-insights
Brian
Brian

Reputation: 13593

Looking for a better way to count the 5 minute request rate of an API

I'm trying to use AWS CloudWatch Logs Insights to count the number of requests coming from each IP address within 5 minutes of an API.

Here's an example of the data I want to get.

Time Range: 01:05 - 01:10

Request Source Number of Requests
53.240.51.81 314
36.241.227.134 237
201.72.45.51 201

Time Range: 01:10 - 01:15

Request Source Number of Requests
43.142.151.81 334
36.131.237.174 230
161.72.15.52 198

Time Range: 01:15 - 01:20

Request Source Number of Requests
31.132.141.91 334
39.138.217.174 230
191.76.15.42 198

...

And here's my query:

fields @timestamp, @message 
| filter (@message like "/my_api") 
| parse @message '* - * [*] * * * "*" * * * "*" "*" "*"' as remote_addr, remote_user, time_local, http_method, path, http_version, header, status_code, request_length, body_bytes_sent, http_referer, http_user_agent, http_x_forwarded_for
| stats count() as requestCount by http_x_forwarded_for 
| sort requestCount desc

The problem of the query above is that it only counts the number of requests within the time range I specified.

So if I want to get the data during the past 24 hours, I'll have to specify the time range of Logs Insights for 24 * 60 / 5 = 288 times. And run the query for 288 times.

It would be very time-consuming to do the steps I describe above.

The reason I want to get the data is that I want to use AWS WAF to set a rate limit on my API.

Before I apply the rate limit, I need to know the request rate of my API during normal days.

Upvotes: 1

Views: 5127

Answers (1)

Parsifal
Parsifal

Reputation: 4516

Tried out a similar query against my CloudTrail events log, and bin() is what you want:

fields @timestamp, @message
| stats count(*) by eventName, bin(5m)
| sort desc
| limit 20

Upvotes: 3

Related Questions