CODEWITHSUNDEEP
amazon-cloudwatchaws-cloudwatch-log-insights
Christian
Christian

Reputation: 7852

How to extract data from array in a JSON message using CloudWatch Logs Insights?

I log messages that are JSON objects. The JSON has an array that contains key/value pairs:

{
   ...
   "arr": [{"key": "foo", "value": "bar"}, ...],
   ...
}

Now I want to filter results that contains a specific key and extract the values for a specific key in the array.

I've tried using regex, something like parse @message /.*"key":"my_specific_key","value":(?<value>.*}).*/ which extracts the value but also returns the rest of the message. Also it doesn't filter the results.

How can I filter results and extract the values for a specific key?

Upvotes: 2

Views: 6276

Answers (2)

user1658846
user1658846

Reputation: 11

fields jsonParse(@message) as js
| unnest js.arr into item
| filter item.key = "value"

Upvotes: 0

lynkfox
lynkfox

Reputation: 2400

If in your log entry in the cloudwatch log group they are actually showing up as json, you can just reference the key directly in any place you would a field.

(don't need the @, cloudwatch appends that automatically to all default values)

If you are using python, you can use aws_lambda_powertools to do this as well, in a very slick way (and its an actual aws product)

If they are showing up in your log as a string, then it may be an escaped string and you'll have to match it -exactly- - including spaces and what not. when you parse, you will want to do something like this:

if this is the string of your log message '{"AKey" : "AValue", "Key2" : "Value2"}

parse @message "{\"*\" : \"*\",\"*\" : \"*\"} akey, akey_value, key2, key2_value

then you can filter or count or anything against those variables. parse is specifically a statement to match a pattern and assign the wildcard to a variable, one at a time in order

tho with a complex json, if your above regex works than all you need is a filter statement

field @message
 | pares @message ... your regex as value_var
 | filer value_var /some more regex/

if its not a string in the log entry, but an actual json, you can just reference against the key:

filter a_key ~="some value" (or regex here)

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData-discoverable-fields.html

for more info

Upvotes: 1

Related Questions