CODEWITHSUNDEEP
jsonelasticsearchlogstashlogstash-file
Prashant Agrawal
Prashant Agrawal

Reputation: 381

Logstash : Parsing fails while parsing Json data received through web service like twitter / facebook etc

I am receiving the Web service data say from Twitter and logging to file and there after I need to send that data to Logstash so as same can be indexed to Elasticsearch.

I am using below config and that is giving jsonparsefailure with exception as

JSON parse failure. Falling back to plain-text {:error=>#> LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a >valid value (number, String, array, object, 'true', 'false' or 'null')

My logstash conf files looks like :

input
    {
        file
        {
            path => ["/mnt/volume2/ELK_Prashant/at/events.json"]
            codec => json
            type => json
        start_position => "beginning"
            sincedb_path => "/dev/null"
        }
    }
    output
    {
        stdout { codec => rubydebug }
    }

And data in events.json can be reference from https://dev.twitter.com/rest/reference/get/search/tweets with some sample as below: events.json

[
{ "location": "LA, CA",
        "follow_request_sent": null,
        "profile_link_color": "0084B4",
        "is_translator": false,
        "id_str": "137238150",
        "entities": {
          "url": {
            "urls": [
              {
                "expanded_url": null,
                "url": ""
              }
            ]
          }
        }
}
]

Upvotes: 1

Views: 2024

Answers (1)

Sumit
Sumit

Reputation: 2270

From your sample events.json file, it is clear that you are using a complete json object as input to logstash file plugin but the plugin by default assumes that each event will be single line and because of that only it is able to detect new events coming in and tracking current position.

So your input file should look like this, where each event is separated by new line character

{"location":"LA, CA","follow_request_sent":null,"profile_link_color":"0084B4","is_translator":false,"id_str":"137238150","entities":{"url":{"urls":[{"expanded_url":null,"url":""}]}}}
{"location":"LA, CA","follow_request_sent":null,"profile_link_color":"0084B4","is_translator":false,"id_str":"137238150","entities":{"url":{"urls":[{"expanded_url":null,"url":""}]}}}
{"location":"LA, CA","follow_request_sent":null,"profile_link_color":"0084B4","is_translator":false,"id_str":"137238150","entities":{"url":{"urls":[{"expanded_url":null,"url":""}]}}}

or you have to use multiline codec or filter in input plugin. More info can be found here.

Upvotes: 1

Related Questions