Filebeat multiline pattern for PHP stack trace

Question

I am trying to import the PHP FPM logs into an ELK stack. For this I use the filebeat to read the files. Before sending this data to logstash, the multiline log entries should be merged.

For this I built this filebeat configuration:

filebeat.inputs:

  # Each - is an input. Most options can be set at the input level, so
  # you can use different inputs for various configurations.
  # Below are the input specific configurations.

  - type: filestream

    # Change to true to enable this input configuration.
    enabled: true

    # Paths that should be crawled and fetched. Glob based paths.
    paths:
      - '/var/log/app/fpm/*.log'

    multiline.type: pattern
    multiline.pattern: '^\[\d{2}-\w{3}-\d{4} \d{2}:\d{2}:\d{2} [\w/]*\] PHP\s*at.*'
    multiline.negate: false
    multiline.match: after

    processors:
      - add_fields:
          fields.docker.service: "fpm"

But as you can see in the ruby debug output from logstash, the messages were not merged:

{
        "@timestamp" => 2021-08-10T13:54:10.149Z,
        "agent" => {
                 "version" => "7.13.4",
                "hostname" => "3cb76d7d4c7d",
                      "id" => "61dec25e-12ec-4a65-9f1f-ec72a5aa83ee",
            "ephemeral_id" => "631db0d8-60ad-4625-891c-3da09cb0a442",
                    "type" => "filebeat"
        },
             "input" => {
            "type" => "filestream"
        },
               "log" => {
            "offset" => 344,
              "file" => {
                "path" => "/var/log/app/fpm/error.log"
            }
        },
              "tags" => [
            [0] "beats_input_codec_plain_applied",
            [1] "_grokparsefailure"
        ],
            "fields" => {
            "docker" => {
                "service" => "fpm"
            }
        },
          "@version" => "1",
           "message" => "[17-Jun-2021 13:07:56 Europe/Berlin] PHP [WARN] (/var/www/html/Renderer/RendererTranslator.php:92) - unable to translate type integer. It is not a string (/url.php)",
               "ecs" => {
            "version" => "1.8.0"
        }
    }
    {
      "input" => {
            "type" => "filestream"
        },
            "module" => "PHP IES\\ServerException",
               "ecs" => {
            "version" => "1.8.0"
        },
          "@version" => "1",
               "log" => {
            "offset" => 73,
              "file" => {
                "path" => "/var/log/ies/fpm/error.log"
            }
        },
        "@timestamp" => 2021-06-17T11:10:41.000Z,
             "agent" => {
                 "version" => "7.13.4",
                "hostname" => "3cb76d7d4c7d",
                      "id" => "61dec25e-12ec-4a65-9f1f-ec72a5aa83ee",
            "ephemeral_id" => "631db0d8-60ad-4625-891c-3da09cb0a442",
                    "type" => "filebeat"
        },
              "tags" => [
            [0] "beats_input_codec_plain_applied"
        ],
            "fields" => {
            "docker" => {
                "service" => "fpm"
            }
        },
           "message" => "core.login"
    }
    {
        "@timestamp" => 2021-08-10T13:54:10.149Z,
             "agent" => {
                 "version" => "7.13.4",
                "hostname" => "3cb76d7d4c7d",
                      "id" => "61dec25e-12ec-4a65-9f1f-ec72a5aa83ee",
            "ephemeral_id" => "631db0d8-60ad-4625-891c-3da09cb0a442",
                    "type" => "filebeat"
        },
               "ecs" => {
            "version" => "1.8.0"
        },
             "input" => {
            "type" => "filestream"
        },
              "tags" => [
            [0] "beats_input_codec_plain_applied",
            [1] "_grokparsefailure"
        ],
            "fields" => {
            "docker" => {
                "service" => "fpm"
            }
        },
          "@version" => "1",
           "message" => "[17-Jun-2021 13:10:41 Europe/Berlin] PHP     at App\\Module\\ComponentModel\\ComponentModel->doPhase(/var/www/html/Component/Container.php:348)",
               "log" => {
            "offset" => 204,
              "file" => {
                "path" => "/var/log/app/fpm/error.log"
            }
        }
    }

I tested the regular expression with Rubular and it matches the stack trace messages.

What am I doing wrong here?

Answer 1

Instead of adjusting the filebeat configuration, I adjusted the log configuration of the application. Now JSON files are written, which can be easily read with the filebeat. The consideration of the line break is then no longer necessary.

Answer 2

You need to set multiline.negate to true.