CODEWITHSUNDEEP
logstashlogstash-configurationlogstash-jdbc
Mahendra kanoujiya
Mahendra kanoujiya

Reputation: 11

Null value not inserted in Postgresql by Logstash

i am trying to insert data from Elasticsearch to postgresql. it insert data in the postgresql but does not handle Blank value properly. As I am trying to insert black value in postgresql it fills the variable name of the Statement.

it is output in the postgresql: enter image description here it is my config file

input {
  elasticsearch {
  hosts => ["http://localhost:9200"]
  index => "input_data"
}
}
output {
  jdbc {
    connection_string => "jdbc:postgresql://hostname:5432/database"
    username => "username"
    password => "password"
    driver_jar_path => "C:/postgresql-42.5.1.jar"
    driver_class => "org.postgresql.Driver"
    statement => [
      "INSERT INTO data_post_two (inputdata,metric,source_table,output_column_alias,method) VALUES (?, ?, ?, ?, ?)",
      "%{inputdata}",
      "%{metric}",
      "%{source_table}",
      "%{output_column_alias}",
      "%{method}"
         ]
}
}

Upvotes: 0

Views: 191

Answers (1)

Badger
Badger

Reputation: 4072

That is expected. sprintf does not replace the value of %{fieldName} if the field does not exist. You will have to make sure the fields exist. A general way is surprisingly complicated because it has to handle boolean fields (if [field] {} cannot distinguish a boolean false from a non-existent field). Note that at the time of writing the best-practices documented by Elastic have a method that does not work because the copy and add_field have to be in seperate filters or the order of operations is incorrect...

filter {
    mutate {
        # we use a "temporary" field with an arbitrary value
        add_field => { "[@metadata][test_field_check]" => "a null value" }
    }  
    mutate {
        # we copy the field of interest into that field.
        # If the field of interest doesn't exist, copy is not executed.
        copy => { "testField" => "[@metadata][test_field_check]" }
    }
    # if testField didn't exist, our field will have the arbitrary value
    if [@metadata][test_field_check] == "a null value" {
        # logic to execute when [testField] did not exist
        mutate { add_field => { "testField" => "" }}
    }
}

Upvotes: 0

Related Questions