CODEWITHSUNDEEP
postgresqlamazon-web-servicesamazon-rdsamazon-cloudwatchlogs
Borys
Borys

Reputation: 3044

Multiple log streams in CloudWatch - PostgreSQL RDS

I've got logs publishing from an Amazon RDS Postgres instance to CloudWatch:

Going to CloudWatch Service, I see there are 4 different "streams" which contain similar data from the same time: enter image description here

What's the difference between those 4? I've checked, and in all the files are statements like, UPDATE, SELECT etc. I'm not sure which one should I analyze.

Upvotes: 4

Views: 1369

Answers (2)

Evgeny Savelev
Evgeny Savelev

Reputation: 131

Through experimentation I have found that the solution offered by @krispyjala does not guarantee that the log messages in CloudWatch will be sorted in the same way they are written to the logs on the instance (the logs one can download from the console or through RDS API)

The only viable solution would be for Amazon to enable logging with milliseconds, which is controlled by a RDS parameter log_line_prefix that cannot be changed.

Unfortunately, at this moment there is NO RELIABLE WAY to join multiple CloudWatch streams in to a single stream of messages which preserves the original ordering.

CloudWatch records three bits of information: original message timestamp, the message itself, and the ingestion timestamp. The original message timestamp is only given to a second, so we cannot rely on it. The ingestion timestamp can be up to several days off the original timestamp, so it is subject to random network and processing delays, hence it cannot be relied upon while joining the streams.

Sorry, but there is no solution, until logging with milliseconds is turned on, which is something AWS representatives assured me is in the works (as of Apr 8 2021)

Upvotes: 0

krispyjala
krispyjala

Reputation: 93

Better late than never, but I was told from AWS support that in later versions of PostgreSQL in RDS they write to logs in parallel to increase performance.

I also navigated to the log the way you did, by going directly to CloudWatch, but apparently the correct way to access the log is to go to your RDS instance on AWS console, click on Configuration tab, and on the rightmost column there should be a clickable link (PostgreSQL in my case) under Published Logs/CloudWatch Logs section.

Clicking on that link should take you back to CloudWatch, but now all the log entries from the four parallel streams should now be in order.

Upvotes: 1

Related Questions