CODEWITHSUNDEEP
amazon-cloudwatchamazon-cloudwatchlogsamazon-cloudwatch-metrics
postelrich
postelrich

Reputation: 3506

Count unique values in aws cloudwatch metric

I have a set of cloudwatch logs in json format that contain a username field. How can I write a cloudwatch metric query that counts the number of unique users per month?

Upvotes: 41

Views: 82376

Answers (5)

Anikash Chakraborty
Anikash Chakraborty

Reputation: 56

If I need to see all the distinct counts and not just a number then I do this.

fields @timestamp, @message
| sort @timestamp desc
| stats count_distinct(field_1) as myHeader by field_1

Inspired by @sahil-mahajan's answer

Upvotes: 0

Sahil Mahajan
Sahil Mahajan

Reputation: 213

I think you can achieve that by following query:

Log statement being parsed: "Trying to login user: abc ....."

fields @timestamp, @message
| filter @message like /Trying to login user/
| parse @message "Trying to login user: * and " as user
| sort @timestamp desc
| stats count(*) as loginCount by user | sort loginCount desc

This will print the table in such a way,

# user loginCount
1 user1 10
2 user2 15
......

Upvotes: 19

Leonardo Kuffo
Leonardo Kuffo

Reputation: 1261

Now you can count unique field values using the count_distinct instruction inside CloudWatch Insights queries.

Example:

fields userId, @timestamp
| stats count_distinct(userId)

More info on CloudWatch Insights: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/AnalyzingLogData.html

Upvotes: 45

ImperviousPanda
ImperviousPanda

Reputation: 566

You can now do this! Using CloudWatch Insights.

API: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_StartQuery.html

I am working on a similar problem and my query for this API looks something like:

fields @timestamp, @message
| filter @message like /User ID/
| parse @message "User ID: *" as @userId
| stats count(*) by @userId

To get the User Ids. Right now this returns with a list of them then counts for each one. Getting a total count of unique can either be done after getting the response or probably by playing with the query more.

You can easily play with queries using the CloudWatch Insights page in the AWS Console.

Upvotes: 30

John Rotenstein
John Rotenstein

Reputation: 270224

I don't think you can.

Amazon CloudWatch Logs can scan log files for a specific string (eg "Out of memory"). When it encounters this string, it will increment a metric. You can then create an alarm for "When the number of 'Out of memory' errors exceeds 10 over a 15-minute period".

However, you are seeking to count unique users, which does not translate well into this method.

You could instead use Amazon Athena, which can run SQL queries against data stored in Amazon S3. For examples, see:

Upvotes: 2

Related Questions