CODEWITHSUNDEEP
nginxamazon-cloudwatchamazon-cloudwatchlogsaws-cloudwatch-log-insights
Aniket Paul
Aniket Paul

Reputation: 359

AWS log Insigts parse NGINX log

I am trying to use aws log insights to run query on my log group that contains nginx log.

This is my log format that I have setup on my ec2 machine:

log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

Sample NGINX Log:

xx.xx.xx.xx - - [10/Nov/2020:15:28:30 +0530] "POST /xx HTTP/1.1" 200 57 "https://xxxx.in/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36" "-"

I am trying to parse this using log insights with the following code:

parse @message '* - - [*] "* * *" * * "-" "*"' as remote_addr, timestamp, request_type, location, protocol, response_code, body_bytes_sent

I am getting the following error:

Expression is invalid: parse@message'* - - [*] "* * *" * * "-" "*"'asremote_addr,timestamp,request_type,location,protocol,response_code,body_bytes_sent

Any help would be appreiciated

Upvotes: 9

Views: 5749

Answers (2)

ChiChing Shek
ChiChing Shek

Reputation: 41

It would be even better with referrer and user_agent

fields @timestamp, @message
| parse @message '* - - [*] "* * *" * * "*" "*"' as remote_addr, timestamp, request_type, location, protocol, response_code, body_bytes_sent, referrer, user_agent
| display @timestamp, remote_addr, request_type, location, protocol, response_code, body_bytes_sent, referrer, user_agent
| sort @timestamp desc

Upvotes: 4

JackLeo
JackLeo

Reputation: 4740

You're 99% of the way there. All that is missing is matching the number of selectors and output variables, so if you would append user_agent to your list it would start to work.

Sample working query:

fields @message
| parse @message '* - - [*] "* * *" * * "-" "*"' as remote_addr, timestamp, request_type, location, protocol, response_code, body_bytes_sent, user_agent
| display request_type, location

Upvotes: 26

Related Questions