CODEWITHSUNDEEP
csvelasticsearchlogstashelastic-stacklogstash-configuration
Patrick Kusebauch
Patrick Kusebauch

Reputation: 77

Cannot parse CSV file with Logstash

I have an issue with importing a CSV file into ElasticSearch using logstash for further processing in Kibana.

This is my logstash config file:

input {
    file {
        path => ["/absolute_path_to_file/export.csv"]
        start_position => beginning 
        ignore_older => 0 
        sincedb_path => "/dev/null"
    }
}
#filter {
#   csv {
#       columns => [
#           "id",
#           "cislo_smlouvy",
#           "zdroj",
#           "produkt",
#           "sjednani",
#           "datum_odeslani",
#           "cas_odeslani",
#           "pojistovna",
#           "tarif",
#           "pojistnik",
#           "telefon",
#           "predmet_pojisteni",
#           "rz",
#           "pocatek_pojisteni",
#           "rocni_pojistne",
#           "urgence",
#           "stav"
#       ]
#       separator => ";"
#       remove_field => ["message"]
#   }
#}
output {
#   elasticsearch {
#       hosts => "localhost:9200"
#       index => "smlouvy"
#   }
    stdout {
        codec => rubydebug
    }
}

And an excerpt from my CSV file:

"id";"číslo smlouvy";"zdroj";"produkt";"sjednání";"datum odeslaní";"čas odeslání";"pojišťovna";"tarif";"pojistník";"pojistnik telefon";"předmět pojištění";"rz";"počátek";"roční pojistné";"urgence";"stav"
"114951";"6132681255";"SRO";"POV";;"1.6.2016";"12:28";"csob";"csob-2";"BB TEST";"721666333";"Škoda Favorit";"NENÍ";"2.6.2016 00:00";"4657,00";;"TEST"
"114950";;"POV";"POV";"VO Bukvicova";"1.6.2016";"12:16";"csob";"csob-2";"BB BB";"721000111";"BMW X3";"NENÍ";"3.6.2016 00:00";"5550,00";;"TEST"

I am calling this command:
sudo logstash -f /absolute_path_to_file/logstash.conf --vebrose

With the following output: 

starting agent {:level=>:info}
starting pipeline {:id=>"main", :level=>:info}
Settings: Default pipeline workers: 2
Registering file input {:path=>["/absolute_path_to_file/export.csv"], :level=>:info}
Starting pipeline {:id=>"main", :pipeline_workers=>2, :batch_size=>125, :batch_delay=>5, :max_inflight=>250, :level=>:info}
Pipeline main started

After a while of doing nothing, I shut it down: 

^CSIGINT received. Shutting down the agent. {:level=>:warn}
stopping pipeline {:id=>"main"}
Closing inputs {:level=>:info}
Closed inputs {:level=>:info}
Input plugins stopped! Will shutdown filter/output workers. {:level=>:info}
Pipeline main has been shutdown

Possibly relevant version info:

logstash 2.3.2
logstash-input-file (2.2.5)
logstash-filter-csv (2.1.3)
logstash-output-elasticsearch (2.6.2)
logstash-output-stdout (2.0.6)
logstash-codec-rubydebug (2.0.7)

I have read all the documentation I could find and tried to replicate a lot of logstash.conf examples from GitHub, but with no luck. Any help with what I am missing?

Upvotes: 0

Views: 1241

Answers (1)

Patrick Kusebauch
Patrick Kusebauch

Reputation: 77

So I finally found out the problem. It was with the input CSV file.

The CSV file had \r for new line and logstash default is \n.

BTW: You cannot set \r as the delimiter in logstash file input filter config, so I had to convert the CSV file into one with \n

Upvotes: 1

Related Questions