CODEWITHSUNDEEP
elasticsearchlogstash
Gibbs
Gibbs

Reputation: 22974

Logstash - specify more than one pipeline

I want different fields to be processed in different way.

I have two pipelines. One is to process boolean values, another one is to convert a string to array.

 output {
    stdout {
        codec => rubydebug
    }
    elasticsearch {
        action => "index"
        hosts => ["127.0.0.1:9200"]
        index => "mini_system"
        document_id => "%{mini_system_key}"
        if [source] == "secure_flag" {
            pipeline => "bool-pipeline"
        } else if "partners" == %{FIELD} {
            pipeline => "partners-pipeline"
        }
    }
}

I am trying to do this. But I am not able to achieve this and couldn't find a reference also.

Sample documents:

key,partners,secure_flag,date_added
5369922730525,"1002300,1009747,12359,2285459",FALSE,2020-03-31T14:00:00Z    
2218100624,,FALSE,2020-03-31T14:00:00Z

here,

"1002300,1009747,12359,2285459" is partners. FALSE is secure_flag.

Partners pipeline:

{
  "description": "Converts \"a,b,c\" to [\"a\", \"b\",\"c\"]",
  "processors" : [
    {
      "split" : {
        "field" : "partners",
        "separator": ",",
        "ignore_missing": true
      }
    }
  ]
}

Upvotes: 0

Views: 68

Answers (1)

Val
Val

Reputation: 217554

You cannot apply logic inside plugin configurations, but you can definitely have several output using if/else logic:

output {
    stdout {
        codec => rubydebug
    }
    if [source] == "secure_flag" {
        elasticsearch {
            action => "index"
            hosts => ["127.0.0.1:9200"]
            index => "mini_system"
            document_id => "%{mini_system_key}"
            pipeline => "bool-pipeline"
        }
    } else if [field_xyz] == "partners" {
        elasticsearch {
            action => "index"
            hosts => ["127.0.0.1:9200"]
            index => "mini_system"
            document_id => "%{mini_system_key}"
            pipeline => "partners-pipeline"
        }
    }
}

UPDATE:

You don't actually need any logic, but simply add both of your processors in the same pipeline:

PUT _ingest/pipeline/mini-pipeline
{
  "processors" : [
    {
      "convert" : {
        "field" : "secure_flag",
        "type": "boolean",
        "ignore_missing": true
      }
    },
    {
      "split" : {
        "field" : "partners",
        "separator": ",",
        "ignore_missing": true
      }
    }
  ]
}

And then simply use this configuration

output {
    stdout {
        codec => rubydebug
    }
    elasticsearch {
        action => "index"
        hosts => ["127.0.0.1:9200"]
        index => "mini_system"
        document_id => "%{mini_system_key}"
        pipeline => "mini-pipeline"
    }
}

Upvotes: 1

Related Questions