How to find distinct count or display distinct log message in cloudwatch

Question

1. Here is my cloudwatch logs to count all the event types. I want only distinct values:

fields @timestamp, @message
| sort @timestamp desc
| filter @message like 'RDMErrors::'
| parse @message "[*] *" as LType, LMessage
| filter LType!= 'Error'
| parse @message 'RDMErrors::*::*::Type3::*:*' as eventType, identfier, name, rdmError
| stats count(*) as eventCount by eventType

Right now I am able to get all the count value of specific eventType. However I am looking to get only distinct count where mismatch values are not same across all of them. For eg:

lets say we have two instances of: eventType is eventA and mismatch value is B.
I get following result:

eventType eventCount
eventA     2

I just want 1 count for that. 

2. On The other hand I also have similar logs but this time I am just trying to display distinct logs which is not working. Here is my cw query:

fields @timestamp, @message
| sort @timestamp desc
| filter @message like 'RDMErrors::Both::'
| parse @message "[*] *" as LType, LMessage
| filter LType= 'Error'
| display LMessage

Any suggestion please?

Answer 1

If I understand your question correctly then you are looking for unique field values which you can achieve using count_distinct(fieldName: LogField) within cloudwatch Insights query syntax, which Returns the number of unique values for the field. If the field has very high cardinality (contains many unique values), the value returned by count_distinct is just an approximation!

Example:

fields eventType, @timestamp
| stats count_distinct(eventType)

Also better to look at CWL_QuerySyntax and a SO thread Count unique values in aws cloudwatch metric.

EDIT:

AWS CloudWatch does not have a feature for showing unique messages only. However, you can use the Log Filter/parse Pattern feature to filter out messages that have already been seen. This can help you identify unique messages in your CloudWatch Logs.

AWS Cloudwatch provides the ability to view unique messages only by using the "Unique Count" metric. This metric shows the total number of unique messages that were sent over a given period of time. To view unique messages only, select this metric and set the period to the desired duration. This metric can be used to ensure that only unique messages are being sent and received by the system.

You can try like below to get uniq messages:

fields @timestamp, @message
| stats count(@message) as UniqueMessageCount by @message
| sort UniqueMessageCount desc

OR

fields @timestamp, @message
| parse @message "Checking" as status
| stats count(status) as UniqueMessageCount by status
| sort UniqueMessageCount desc