CODEWITHSUNDEEP
elasticsearchfilebeatgrokenvoyproxy
Mah3ndra
Mah3ndra

Reputation: 83

Grok pattern for envoy proxy logs

[2020-05-05T04:27:54.668Z] "GET /click?ab_tst_bckt=-1&dvc_id=IddFVZ8W--0nuPg9P31T&featured=99&hasOfferId=1120&impressionId=NgnMfW7bYb&page_type=Search&paymentType=cpc&placement=ratetable&position=1&rf=https%3A%2F%2Fwww.ratecity.com.au%2Fcar-loans%2Flow-interest&uuid=1abb33fc-e6cd-45c6-94de-44291dfb2871&vertical=car-loans HTTP/1.1" 200 - "-" "-" 0 11971 17 17 "2001:8003:d576:b100:984a:21b6:d74e:67c7, 2001:8003:d576:b100:984a:21b6:d74e:67c7,54.206.38.58,192.168.73.159" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.113 Safari/537.36" "d3632811-e50e-4686-9985-9d10d78e75ba" "swift.ratecity.com.au" "127.0.0.1:4444" inbound|4444|http|swift-client.ratecity.svc.cluster.local - 192.168.13.129:4444 192.168.73.159:0 -

Is there a grok pattern already available for this envoyproxy log?

I am using debugger on kibana and so far come this far, need help in finishing this up

\[%{GREEDYDATA:timestamp}\] \"%{WORD:method} %{URIPATH:request} ?%{URIPARAM:url_params} %{URIPROTO:protocol}/%{NUMBER:httpversion}" %{NUMBER:response} - "-" "-" %{GREEDYDATA:flags} %{IP:client_ip}(,\\s%{IP})*

Upvotes: 0

Views: 1137

Answers (3)

Anton Bakavets
Anton Bakavets

Reputation: 1

As an example, you may use: https://github.com/nitishm/engarde/blob/e7afe8c5bd50dbb08f44215488d508174b539a3b/pkg/parser/parser.go#L73

I have used IstioProxyAccessLogsPattern for NewRelic Logs parsing.

EnvoyAccessLogsPattern:

\[%{TIMESTAMP_ISO8601:timestamp}\] \"%{DATA:method} (?:%{URIPATH:uri_path}(?:%{URIPARAM:uri_param})?|%{DATA}) %{DATA:protocol}\" %{NUMBER:status_code} %{DATA:response_flags} %{NUMBER:bytes_received} %{NUMBER:bytes_sent} %{NUMBER:duration} (?:%{NUMBER:upstream_service_time}|%{DATA:tcp_service_time}) \"%{DATA:forwarded_for}\" \"%{DATA:user_agent}\" \"%{DATA:request_id}\" \"%{DATA:authority}\" \"%{DATA:upstream_service}\

IstioProxyAccessLogsPattern:

\[%{TIMESTAMP_ISO8601:timestamp}\] \"%{DATA:method} (?:(?:%{URIPATH:uri_path}(?:%{URIPARAM:uri_param})?)|%{DATA}) %{DATA:protocol}\" %{NUMBER:status_code} %{DATA:response_flags} %{DATA:response_details} %{DATA:termination_details} \"%{DATA:upstream_failure_reason}\" %{NUMBER:bytes_received} %{NUMBER:bytes_sent} %{NUMBER:duration} (?:%{NUMBER:upstream_service_time}|%{DATA:tcp_service_time}) \"%{DATA:forwarded_for}\" \"%{DATA:user_agent}\" \"%{DATA:request_id}\" \"%{DATA:authority}\" \"%{DATA:upstream_service}\" %{DATA:upstream_cluster} %{DATA:upstream_local} %{DATA:downstream_local} %{DATA:downstream_remote} %{DATA:requested_server}(?: %{DATA:route_name})?$

Upvotes: 0

manish
manish

Reputation: 331

I have used this one on logstash pipeline config and it is working.

 filter {    grok {  match => { "message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] \"%{DATA:method} (?:%{URIPATH:uri_path}(?:%{URIPARAM:uri_param})?|%{DATA:}) %{DATA:protocol}\" %{DATA:response_code} %{DATA:response_flags} %{DATA:response_code_details} %{DATA:connection_termination_details} %{DATA:upstream_transport_failure_reason} %{DATA:envoy_upstream_service_time} %{NUMBER:bytes_sent} %{NUMBER:bytes_received} %{NUMBER:duration} \"%{IPORHOST:clientIp}\" \"%{DATA:user_agent}\" \"%{UUID:request_id}\" \"%{DATA:authority}\" \"%{DATA:upstream_host}\" %{GREEDYDATA:upstream_cluster} %{DATA:upstream_local_address} %{DATA:downstream_local_address} %{DATA:downstream_remote_address} %{DATA:requested_server_name} %{DATA:route_name}"   }    } }

Upvotes: 0

Mah3ndra
Mah3ndra

Reputation: 83

Here is the Grok pattern you can use to pre-process envoyproxy logs before saving them to elasticsearch indices

"""\[%{GREEDYDATA:istio.timestamp}\] \"%{WORD:istio.method} %{URIPATH:istio.request}(?:%{URIPARAM:istio.url_params}|-)? %{URIPROTO:istio.protocol}/%{NUMBER:istio.httpversion}\" %{NUMBER:istio.response} (?<istio.response_flag>%{DATA}|-)? \"-\" \"-\" %{NUMBER:istio.bytes_received} %{NUMBER:istio.bytes_sent} %{NUMBER:istio.duration} (?<istio.service_time>%{NUMBER}|-)? \"(?<istio.x_forwarded_for>%{IP:istio.client_ip},%{SPACE}.+?(?=")|-)\" \"(?<istio.user_agent>%{DATA}.+?(?=")|-)\"?"""

Upvotes: 1

Related Questions