Elasticsearch: Grok-pipeline not working (Not applying to logs)

Question

i created a pipeline: "ExtractOwaspErrorFields" with a grok processor which extracts fields from the message field. It is working perfectly when testing it in Kibana -> Stack Management -> Ingest Pipelines -> Test Pipeline. I use a real Log-Document by providing _id and _index of a Document. The processor extracts fields from the message fields just as expected. When i add the processor to my filebeat.yml no documents are shown in 'Discovery' at all:

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  username: xxx
  password: xxx
  pipeline: ExtractOwaspErrorFields

My pipeline is configure as follows:

{
  "ExtractOwaspErrorFields" : {
    "processors" : [
      {
        "grok" : {
          "field" : "message",
          "patterns" : [
            "%{OWASP_ERRORLOG}"
          ],
          "pattern_definitions" : {
            "OWASP_ERRORLOG" : "\\[%{HTTPDERROR_DATE:timestamp}\\] \\[:%{LOGLEVEL:loglevel}\\] \\[pid %{POSINT:pid}:tid %{NUMBER:tid}\\]?( \\[client %{IPORHOST:client}:%{POSINT:clientport}\\]) \\[client %{IPORHOST:client_ip}\\] ModSecurity: ?(%{APACHE_ERROR_MESSAGE:error}) \\[file \"%{PATH:matching_rule_file}\"\\] \\[line \"%{POSINT:matching_rule_line}\"\\] \\[id \"%{NUMBER:matching_rule_id}\"\\] \\[msg \"%{DATA:owasp_message_string}\"\\] \\[data \"%{DATA:owasp_message_data}\"\\] \\[severity \"%{WORD:owasp_severity}\"\\] \\[ver \"%{DATA:owasp_version}\"\\] %{GREEDYDATA:tags} \\[hostname \"%{HOSTNAME:hostname}\"\\] \\[uri \"%{URIPATHPARAM:uri}\"\\] \\[unique_id \"%{DATA:unique_id}\"\\]",
            "APACHE_ERROR_MESSAGE" : "( .+?(?= \\[%{WORD} \"))"
          },
          "if" : "ctx?.docker.container.labels.com_docker_stack_namespace == 'modsecurity'",
          "ignore_failure" : true
        }
      }
    ]
  }
}

My sample document for testing the pipeline is:

{
  "docs":
  [
    {
      "_id": "OzAKyoIBILrgz4V8VcpG",
      "_index": "filebeat-7.17.5-2022.08.02-000001",
      "_source": {
        "docker": {
          "container": {
            "labels": {
              "com_docker_stack_namespace": "modsecurity"
            }
          }
        },
        "ecs": {
          "version": "1.12.0"
        },
        "stream": "stderr",
        "message": "[Tue Aug 23 11:30:47.675452 2022] [:error] [pid 226:tid 139758264993536] [client 10.0.1.48:41062] [client 10.0.1.48] ModSecurity: Warning. Pattern match \"(?:^|[\\\\\\\\/])\\\\\\\\.\\\\\\\\.(?:[\\\\\\\\/]|$)\" at ARGS:test. [file \"/etc/modsecurity.d/owasp-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf\"] [line \"71\"] [id \"930110\"] [msg \"Path Traversal Attack (/../)\"] [data \"Matched Data: ../ found within ARGS:test: ../\"] [severity \"CRITICAL\"] [ver \"OWASP_CRS/3.3.2\"] [tag \"modsecurity\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-lfi\"] [tag \"paranoia-level/1\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/255/153/126\"] [hostname \"modsecurity\"] [uri \"/\"] [unique_id \"YwSeR-gNfcPQLAl5gStNfAAAAQE\"]"
      }
    }
  ]
}

Result of Testing the pipeline:

{
  "docs": [
    {
      "doc": {
        "_index": "filebeat-7.17.5-2022.08.02-000001",
        "_type": "_doc",
        "_id": "OzAKyoIBILrgz4V8VcpG",
        "_source": {
          "owasp_severity": "CRITICAL",
          "owasp_message_string": "Path Traversal Attack (/../)",
          "pid": "226",
          "error": " Warning. Pattern match \"(?:^|[\\\\\\\\/])\\\\\\\\.\\\\\\\\.(?:[\\\\\\\\/]|$)\" at ARGS:test.",
          "tid": "139758264993536",
          "clientport": "41062",
          "docker": {
            "container": {
              "labels": {
                "com_docker_stack_namespace": "modsecurity"
              }
            }
          },
          "owasp_version": "OWASP_CRS/3.3.2",
          "hostname": "modsecurity",
          "ecs": {
            "version": "1.12.0"
          },
          "stream": "stderr",
          "client": "10.0.1.48",
          "client_ip": "10.0.1.48",
          "timestamp": "Tue Aug 23 11:30:47.675452 2022",
          "unique_id": "YwSeR-gNfcPQLAl5gStNfAAAAQE",
          "matching_rule_id": "930110",
          "message": "[Tue Aug 23 11:30:47.675452 2022] [:error] [pid 226:tid 139758264993536] [client 10.0.1.48:41062] [client 10.0.1.48] ModSecurity: Warning. Pattern match \"(?:^|[\\\\\\\\/])\\\\\\\\.\\\\\\\\.(?:[\\\\\\\\/]|$)\" at ARGS:test. [file \"/etc/modsecurity.d/owasp-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf\"] [line \"71\"] [id \"930110\"] [msg \"Path Traversal Attack (/../)\"] [data \"Matched Data: ../ found within ARGS:test: ../\"] [severity \"CRITICAL\"] [ver \"OWASP_CRS/3.3.2\"] [tag \"modsecurity\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-lfi\"] [tag \"paranoia-level/1\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/255/153/126\"] [hostname \"waf\"] [uri \"/\"] [unique_id \"YwSeR-gNfcPQLAl5gStNfAAAAQE\"]",
          "uri": "/",
          "owasp_message_data": "Matched Data: ../ found within ARGS:test: ../",
          "tags": "[tag \"modsecurity\"] [tag \"application-multi\"] [tag \"language-multi\"] [tag \"platform-multi\"] [tag \"attack-lfi\"] [tag \"paranoia-level/1\"] [tag \"OWASP_CRS\"] [tag \"capec/1000/255/153/126\"]",
          "loglevel": "error",
          "matching_rule_file": "/etc/modsecurity.d/owasp-crs/rules/REQUEST-930-APPLICATION-ATTACK-LFI.conf",
          "matching_rule_line": "71"
        },
        "_ingest": {
          "timestamp": "2022-08-23T12:08:05.5017157Z"
        }
      }
    }
  ]
}

Can somebody tell me what is wrong with my pipeline or what i missed to do. Why is it working in "Test Pipeline" but not when i use them for real logs in filebeat.yml Thank you very much

Answer 1

Solved it by renaming the pipeline to only contain lowercase characters. Apparently the parsing of filebeat.yml converts the pipeline value to lowercase and is therefore not found on the elastic-side.