CODEWITHSUNDEEP
apacheamazon-web-servicesloggingamazon-cloudwatchamazon-cloudwatchlogs
Ryan Cromwell
Ryan Cromwell

Reputation: 2613

Extract date time from Apache Combined log format using AWS Logs and Cloudwatch

We're using awslogs to collect Apache Combined formatted logs into Cloudwatch. It's all capturing fine, but we're getting timestamp could not be parsed from message error.

An example log entry:

::ffff:10.0.0.1 - blahblah [17/Aug/2017:20:31:07 +0000] "GET /favicon-16x16.png HTTP/1.1" 304 - "http://blahblah:3000/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"

Our config for this set of log files looks like this, including our datetime_format entry:

[access_logs] log_group_name = cromwell log_stream_name = react-172.31.43.245-access file = /home/admin/aperian-react/log/*access.log datetime_format = "%d/%b/%Y:%H%M:%S %z" multi_line_start_pattern = ::ffff: time_zone = UTC encoding = ascii

As you can see, the datetime is mid-line. This is different from most examples for syslogs, etc. We could change our log format, but we'd prefer not to since they flow into other systems as well.

Upvotes: 1

Views: 1947

Answers (1)

Ryan Cromwell
Ryan Cromwell

Reputation: 2613

Our dateformat_string was missing a colon.😒 😢

datetime_format = "%d/%b/%Y:%H%M:%S %z"  # wrong
datetime_format = "%d/%b/%Y:%H:%M:%S %z" # correct

Upvotes: 4

Related Questions