CODEWITHSUNDEEP
jsonelasticsearchescapinglogstash
Tony Laidig
Tony Laidig

Reputation: 1078

Why doesn't Elasticsearch Ingest accept a grok pattern that Logstash does?

I have the following grok pattern that works in Logstash and in the Grok debugger in Kibana. 

\[%{TIMESTAMP_ISO8601:req_time}\] %{IP:client_ip} (?:%{IP:forwarded_for}|\(-\)) (?:%{QS:request}|-) %{NUMBER:response_code:int} %{WORD}:%{NUMBER:request_length:int} %{WORD}:%{NUMBER:body_bytes_sent:int} %{WORD}:(?:%{QS:http_referer}|-) %{WORD}:(?:%{QS:http_user_agent}|-) (%{WORD}:(\")?(%{NUMBER:request_time:float})(\")?)?"

I am trying to create a new ingest pipeline via the PUT method, but I get an error that contains:

    "type": "parse_exception",
    "reason": "Failed to parse content to map",
    "caused_by": {
        "type": "i_o_exception",
        "reason": "Unrecognized character escape '[' (code 91)\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@61326735; line: 7, column: 25]"
    }

Upvotes: 1

Views: 1540

Answers (1)

Tony Laidig
Tony Laidig

Reputation: 1078

Elasticsearch requires that grok patterns used in pipelines submitted using the PUT method are properly escaped JSON, while Logstash patterns use different escaping.

That includes preceding brackets with double backslashes (\\[) and double quotes with triple backslashes (\\\"). The working pattern (after running through a JSON escaping tool) is:

\\[%{TIMESTAMP_ISO8601:req_time}\\] %{IP:client_ip} (?:%{IP:forwarded_for}|\\(-\\)) (?:%{QS:request}|-) %{NUMBER:response_code:int} %{WORD}:%{NUMBER:request_length:int} %{WORD}:%{NUMBER:body_bytes_sent:int} %{WORD}:(?:%{QS:http_referer}|-) %{WORD}:(?:%{QS:http_user_agent}|-) (%{WORD}:(\\\")?(%{NUMBER:request_time:float})(\\\")?)?

Upvotes: 1

Related Questions