CODEWITHSUNDEEP
elasticsearchapache-nifi
lordoku
lordoku

Reputation: 1122

Apache NiFi: PutElasticSearchHttp is not working, with blank error

I currently have Elasticsearch version 6.2.2 and Apache Nifi version 1.5.0 running on the same machine. I'm trying to follow the Nifi example located: https://community.hortonworks.com/articles/52856/stream-data-into-hive-like-a-king-using-nifi.html except instead of storing to Hive, I want to store to Elasticsearch.

Initially I tried using the PutElasticsearch5 processor but I was getting the following error on Elasticsearch: 

Received message from unsupported version: [5.0.0] minimal compatible version is: [5.6.0]

When I tried Googling this error message, it seemed like the consensus was to use the PutElasticsearchHttp processor. My Nifi looks like: enter image description here

And the configuration for the PutElasticsearchHttp processor: enter image description here

When the flowfile gets to the PutElasticsearchHttp processor, the following error shows up: enter image description here

PutElasticSearchHttp failed to insert StandardFlowFileRecord into Elasticsearch due to , transferring to failure.

It seems like the reason is blank/null. There also wasn't anything in the Elasticsearch log.

After the ConvertAvroToJson, the data is a JSON array with all of the entries on a single line. Here's a sample value: 

{"City": "Athens", 
 "Edition": 1896, 
 "Sport": "Aquatics", 
 "sub_sport": "Swimming",
 "Athlete": "HAJOS, Alfred", 
 "country": "HUN", 
 "Gender": "Men", 
 "Event": "100m freestyle", 
 "Event_gender": "M", 
 "Medal": "Gold"}

Any ideas on how to debug/solve this problem? Do I need to create anything in Elasticsearch first? Is my configuration correct?

Upvotes: 3

Views: 5460

Answers (3)

Pankaj Singh
Pankaj Singh

Reputation: 71

I had a similar flow, wherein changing the type to _doc did the trick after including splitTojSON.

Upvotes: 0

lordoku
lordoku

Reputation: 1122

I was able to figure it out. After the ConvertAvroToJSON, the flow file was a single line that contained a JSON Array of JSON indices. Since I wanted to store the individual indices I needed a SplitJSON processor. Now my Nifi looks like this:

Nifi flow with SplitJSON

The configuration of the SplitJson looks like this:

Processor Details for SplitJSON

Upvotes: 5

Dario Balinzo
Dario Balinzo

Reputation: 691

The index name cannot contain the / character. Try with a valid index name: e.g. sports.

Upvotes: 0

Related Questions