Getting error when connecting Nifi to Elasticsearch

Question

I have Nifi 1.4.0 and Elasticsearch 5.6.2 installed on a VM instance of Ubuntu 16.04.3 LTS.

My aim is to use Nifi to pull some data from Twitter, have that stored in Elasticsearch and ultimately I'll be playing around with the data in Kabana.

The Problem Everything works in isolation, which is to say that I can add data to Elasticsearch directly from the command line and retrieve it. I can also access Nifi and I am successfully pulling data from twitter and having it passed to the PutElasticsearch processor.

Pic of Nifi pulling data from Twitter then throwing error

The exception being thrown is as follows: PutElasticsearch[id=fbb5f3cf-015e-1000-8321-71b19eef4054] Failed to insert into Elasticsearch due to None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]. More detailed information may be available in the NiFi logs.: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{127.0.0.1}{localhost/127.0.0.1:9300}]]

In my research I have seen a few people sight seeing the same error, but they all claim that changing the port to :9300 solves their problem. As you can see below in the picture of my PutElasticsearch processor config tab that shouldn't be the issue here.

Pic of PutElasticsearch processor config

I'd really appreciate some help, I'm stumped.

Answer 1

I had the same issue with PutElasticsearch and PutElasticseach5 and so I switched to PutElasticsearchHttp. This is a tad bit slower but definitely works - I've tested with ES Ver 6.5.2 and 6.3.1 and so should definitely work for 5.6.2

Couple of things to keep in mind are

1. make sure you use the HTTP port - default would be localhost:9200
2. if your json stream is a single line you might want to split it before passing it to PutElasticsearchHttp. else it will give you a parsing error. see image below:

Splitting Json in NiFi

Hopefully this helps!

Answer 2

Please use Nifi 1.4.0 and elasticsearch-5.6.5 and PutElasticsearch5 as processor. It works for me.

Answer 3

PutElasticsearch uses the native Transport API, which effectively lets the NiFi node become a part of the cluster. As such, it is very sensitive to the version of the cluster vs the version of the transport library included with the Elasticsearch bundle. Currently PutElasticsearch uses the 2.1.0 version of the transport library, which is almost certainly not compatible with a 5.6.2 cluster. I should mention that there is a PutElasticsearch5 processor which uses the 5.0.1 version of the library, but this is probably also not compatible with a 5.6.2 cluster due to the large difference in minor version.

However, there is a set of "Http" versions of the Elasticsearch processors (such as PutElasticsearchHttp and PutElasticsearchHttpRecord) which use the REST API for communicating with the cluster. Since the REST API is much more stable across ES versions, you can usually use these to achieve your goal. The tradeoff is in performance, the Http versions of the processor are somewhat slower due to the extra communication overhead with the cluster.