CODEWITHSUNDEEP
jsonfilterfluentd
Roland H
Roland H

Reputation: 96

fluentd filter regexp with json data

I am trying to filter (keep) only oauth authenticated audit logs but it's not working. When I set the pattern to /.*/ everything appears in Kibana. If I set /oauth/ or /.*oauth.*/ nothing appears. What am I doing wrong?

Line in log file I want to filter:

{"kind":"Event","apiVersion":"audit.k8s.io/v1beta1","metadata":{"creationTimestamp":"2020-07-17T20:06:49Z"},"level":"Metadata","timestamp":"2020-07-17T20:06:49Z","auditID":"cf56d61e-30b3-486c-a513-6bd9e96fb592","stage":"ResponseComplete","requestURI":"/api/v1/namespaces/openshift-logging/pods?limit=500","verb":"list","user":{"username":"user","uid":"388e0232-c5bb-11ea-904d-7a59592b634f","groups":["system:authenticated:oauth","system:authenticated"],"extra":{"scopes.authorization.openshift.io":["user:full"]}},"sourceIPs":["10.0.72.20"],"objectRef":{"resource":"pods","namespace":"openshift-logging","apiVersion":"v1"},"responseStatus":{"metadata":{},"code":200},"requestReceivedTimestamp":"2020-07-17T20:06:49.918391Z","stageTimestamp":"2020-07-17T20:06:49.921475Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"registry-controller\" of ClusterRole \"cluster-admin\" to User \"user\""}}

fluentd config:

<source>
  @type tail
  @id in_tail_audit_logs
  multiline_flush_interval 5s
  path "/var/lib/origin/audit-ocp.log"
  tag "ocp-audit"
  <parse>
    @type "json"
    time_format "%Y-%m-%dT%T.%L%Z"
    time_type string
  </parse>
</source>
<filter ocp-audit>
  @type grep
  <regexp>
    key user.groups
    pattern /oauth/
  </regexp>
</filter>

Upvotes: 0

Views: 2118

Answers (1)

Azeem
Azeem

Reputation: 14637

Try this configuration for grep:

<regexp>
  key user
  pattern /.*groups.*oauth/
</regexp>

Alternatively, you can install and configure fluent-plugin-json like this:

<filter ocp-audit>
  @type json
  @id json_filter

  <check>
    pointer /user/groups/0    # point to 0th index of groups array
    pattern /.*:oauth/
  </check>
</filter>

Upvotes: 1

Related Questions