Import JSON-file to Elasticsearch and Kibana via Logstash (Docker ELK stack)

Question

I'm trying to import data that is stored in a JSON-file via Logstash to Elasticsearch/Kibana. I've unsuccessfully tried to resolve the issue by searching.

I'm using the ELK stack with Docker as provided here [git/docker-elk].

---

My logstash.conf currently looks as such:

input {
        tcp {
            port => 5000
        }

        file {
            path => ["/export.json"]
            codec => "json"
            start_position => "beginning"
        }

    }

    filter {
        json {
            source => "message"
        }
    }

    ## Add your filters / logstash plugins configuration here

    output {
        stdout { 
            id => "stdout_test_id"
            codec => json 
        }

        elasticsearch {
            hosts => "elasticsearch:9200"
            index => "logstash-indexname"
        }
    }

---

The JSON-file is formatted as such:

[{fields},{fields},{fields},...]

Full JSON-structure: https://jsoneditoronline.org/?id=3d49813d38e641f6af6bf90e9a6481e3

I'd like to import everything under each bracket as-is into Elasticsearch.

---

Shell output after running docker-compose up:

logstash_1       | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash_1       | [2018-10-24T13:21:54,602][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
logstash_1       | [2018-10-24T13:21:54,612][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
logstash_1       | [2018-10-24T13:21:54,959][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or commandline options are specified
logstash_1       | [2018-10-24T13:21:55,003][INFO ][logstash.agent   ] No persistent UUID file found. Generating new UUID {:uuid=>"4a572899-c7ac-4b41-bcc0-7889983240b4", :path=>"/usr/share/logstash/data/uuid"}
logstash_1       | [2018-10-24T13:21:55,522][INFO ][logstash.runner   ] Starting Logstash {"logstash.version"=>"6.4.0"}
logstash_1       | [2018-10-24T13:21:57,552][INFO ][logstash.pipeline   ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
logstash_1       | [2018-10-24T13:21:58,018][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
logstash_1       | [2018-10-24T13:21:58,035][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://elasticsearch:9200/, :path=>"/"}
logstash_1       | [2018-10-24T13:21:58,272][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://elasticsearch:9200/"}
logstash_1       | [2018-10-24T13:21:58,377][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
logstash_1       | [2018-10-24T13:21:58,381][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
logstash_1       | [2018-10-24T13:21:58,419][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//elasticsearch:9200"]}
logstash_1       | [2018-10-24T13:21:58,478][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
logstash_1       | [2018-10-24T13:21:58,529][INFO ][logstash.inputs.tcp   ] Starting tcp input listener {:address=>"0.0.0.0:5000", :ssl_enable=>"false"}
logstash_1       | [2018-10-24T13:21:58,538][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
logstash_1       | [2018-10-24T13:21:58,683][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
elasticsearch_1  | [2018-10-24T13:21:58,785][WARN ][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index_patterns]
elasticsearch_1  | [2018-10-24T13:21:59,036][WARN ][o.e.d.i.m.MapperService  ] [_default_] mapping is deprecated since it is not useful anymore nowthat indexes cannot have more than one type
elasticsearch_1  | [2018-10-24T13:21:59,041][INFO ][o.e.c.m.MetaDataIndexTemplateService] [riEmfTq] adding template [logstash] for index patterns [logstash-*]
logstash_1       | [2018-10-24T13:21:59,158][INFO ][logstash.inputs.file   ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/usr/share/logstash/data/plugins/inputs/file/.sincedb_1ed00aa8bbe3029ead0818433d122587", :path=>["/export.json"]}
logstash_1       | [2018-10-24T13:21:59,210][INFO ][logstash.pipeline   ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x4b7995b9 sleep>"}
logstash_1       | [2018-10-24T13:21:59,337][INFO ][logstash.agent   ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
logstash_1       | [2018-10-24T13:21:59,357][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections
logstash_1       | [2018-10-24T13:21:59,760][INFO ][logstash.agent   ] Successfully started Logstash API endpoint {:port=>9600}

Answer 1

The problem is that this file contains all documents inside a JSON array wrapped on a single line. Logstash cannot easily read that kind of file.

What I suggest is to transform that file into another one where each JSON document sits on its own line, so that Logstash can consume it easily.

First, run this command (you might have to install the jq utility first):

 cat export.json | jq -c '.[]' > export_lines.json

Then change your file input to

 path => ["/export_lines.json"]

Re-run Logstash and enjoy!