CODEWITHSUNDEEP
regexlogstashlogstash-grok
Ishara Dayarathna
Ishara Dayarathna

Reputation: 3601

grok parse optional field pattern doesn't work

I've got a log like this:

ERROR_MESSAGE:Invalid Credentials,THROTTLED_OUT_REASON:API_LIMIT_EXCEEDED

I'm trying to parse it with grok using grok debugger:

ERROR_MESSAGE:%{GREEDYDATA:errorMassage},THROTTLED_OUT_REASON:%{GREEDYDATA:throttledOutReason}

It works, but sometimes the log comes without THROTTLED_OUT_REASON field. 

ERROR_MESSAGE:%{GREEDYDATA:errorMassage}

In that case I tried below code since THROTTLED_OUT_REASON is an optional field. 

ERROR_MESSAGE:%{GREEDYDATA:errorMassage}(,THROTTLED_OUT_REASON:%{GREEDYDATA:throttledOutReason})?

So this should work for both cases. The given output for the log with optional field is:

{
  "errorMassage": [
    [
      "Invalid Credentials,THROTTLED_OUT_REASON:API_LIMIT_EXCEEDED"
    ]
  ],
  "throttledOutReason": [
    [
      null
    ]
  ]
}

But the expected output for the log with optional field:

{
  "errorMassage": [
    [
      "Invalid Credentials"
    ]
  ],
  "throttledOutReason": [
    [
      "API_LIMIT_EXCEEDED"
    ]
  ]
}

expected output for the log without optional field: 

{
  "errorMassage": [
    [
      "Invalid Credentials"
    ]
  ],
  "throttledOutReason": [
    [
      null
    ]
  ]
}

Can anyone suggest a solution which gives correct output for both type of logs?

Upvotes: 0

Views: 290

Answers (2)

Ishara Dayarathna
Ishara Dayarathna

Reputation: 3601

I got the answer using @Skeeve 's idea.

Here it is for anyone who would come up with a similar question:

I've used custom pattern in order to avoid excess eating of GREEDYDATA (for errorMessage field).

ERROR_MESSAGE:(?<errorMassage>([^,]*)?)(,THROTTLED_OUT_REASON:%{GREEDYDATA:throttledOutReason})?

Upvotes: 1

Skeeve
Skeeve

Reputation: 8202

Since you use GREEDYDATA it "eats" as much as it can get in order to fill errormessage.

I do not know GROK enough to tell you what alternative defined patterns there are, but you should be able to use a custom pattern:

ERROR_MESSAGE:(?<errorMassage>.*?),THROTTLED_OUT_REASON:%{GREEDYDATA:throttledOutReason}

Upvotes: 1

Related Questions