Unable to start Elasticsearch Sink

Question

I'm trying to get my elasticsearch sink up and running. However I'm getting the following error and am running out of ideas on how to fix it. Any help is appreciated. Here is the error:

2015-02-10 11:43:31,234 ERROR org.apache.flume.lifecycle.LifecycleSupervisor: Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@79a805d2 counterGroup:{ name:null counters:{} } } - Exception follows. java.lang.NoSuchFieldError: LUCENE_3_6 at org.elasticsearch.Version.(Version.java:45) at org.elasticsearch.client.transport.TransportClient.(TransportClient.java:168) at org.elasticsearch.client.transport.TransportClient.(TransportClient.java:123) at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.openClient(ElasticSearchTransportClient.java:201) at org.apache.flume.sink.elasticsearch.client.ElasticSearchTransportClient.(ElasticSearchTransportClient.java:79) at org.apache.flume.sink.elasticsearch.client.ElasticSearchClientFactory.getClient(ElasticSearchClientFactory.java:48) at org.apache.flume.sink.elasticsearch.ElasticSearchSink.start(ElasticSearchSink.java:357) at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46) at org.apache.flume.SinkRunner.start(SinkRunner.java:79) at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745)

I am running CDH 5.3 and elasticsearch 1.4.2

Answer 1

Ok, figured it out after you guys pointed me in the right direction.

Since I'm using CHD 5.3, I needed to copy the elasticsearch 1.4.2 jar into the /user/lib/hadoop dir and not the flume-ng dir. After doing this it worked perfectly.

Thanks everyone for the help!

Answer 2

I suspect that you dropped your elastic search jars into FLUME_HOME/lib

The problem here is due to conflicting versions of Elastic Search in Flume. The libraries you provided aren't the same as the ones used in flume. Flume 1.5 is linked against Elastic Search 4.9.