CODEWITHSUNDEEP
elasticsearch
Aero Wang
Aero Wang

Reputation: 9247

How to create a pipeline on a field that contains dots with ElasticSearch (w/ no script)

We prefer to use no script in our pipeline because with no script the pipeline can reach QPS of 10000+, whilst added the scripts it drops to 3000+.

When I use /_ingest/pipeline/user_agent to create a pipeline as such:

{
    "description": "Add user agent information",
    "processors": [
        {
            "user_agent": {
                "field": "meta.http.headers.user-agent",
                "ignore_missing": true
            }
        }
    ]
}

What I really want is to allow the pipeline to process data below:

{
    "meta": {
        "http.headers.user-agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 14_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/8.0.2(0x1800022c) NetType/WIFI Language/zh_CN"
    }
}

I also tried to create pipeline as below:

{
    "description": "Add user agent information",
    "processors": [
        {
            "user_agent": {
                "field": "meta.http\\.headers\\.user-agent",
                "ignore_missing": true
            }
        }
    ]
}

OR

{
    "description": "Add user agent information",
    "processors": [
        {
            "user_agent": {
                "field": "[meta][http.headers.user-agent]",
                "ignore_missing": true
            }
        }
    ]
}

They wouldn't work.

When I have pipeline as such:

{
    "description": "Add user agent information",
    "processors": [
        {
            "user_agent": {
                "field": "meta.user-agent",
                "ignore_missing": true
            }
        }
    ]
}

Whilst the data is

{
    "meta": {
        "user-agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 14_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/8.0.2(0x1800022c) NetType/WIFI Language/zh_CN"
    }
}

It works.

Unfortunately our architecture has already set-up to produce data like

{
    "meta": {
        "http.headers.user-agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 14_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/8.0.2(0x1800022c) NetType/WIFI Language/zh_CN"
    }
}

It's pretty much impossible to update all of our apps and services to change the data structure at this point... :frowning: So is it possible to support

{
    "meta": {
        "http.headers.user-agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 14_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/8.0.2(0x1800022c) NetType/WIFI Language/zh_CN"
    }
}

and how?

P.S. The below method almost work but I am not able to change the expanded field back to http.headers.user-agent.

{
    "description": "Add user agent information",
    "processors": [
        {
            "dot_expander": {
                "field": "http.headers.user-agent",
                "path": "meta"
            }
        },
        {
            "user_agent": {
                "field": "meta.http.headers.user-agent",
                "ignore_missing": false
            }
        }
    ]
}

Upvotes: 0

Views: 411

Answers (1)

Val
Val

Reputation: 217464

What you can do in this case is to use the dot_expander processor just before the user-agent processor, like this:

{
    "description": "Add user agent information",
    "processors": [
      {
        "set": {
          "field": "meta2",
          "copy_from": "meta"
        }
      },
      {
        "dot_expander": {
          "field": "http.headers.user-agent",
          "path": "meta2"
        }
      },
      {
        "user_agent": {
          "field": "meta2.http.headers.user-agent",
          "ignore_missing": true
        }
      },
      {
        "remove": {
          "field": ["meta2"]
        }
      }
    ]
  }

Upvotes: 1

Related Questions