CODEWITHSUNDEEP
hadoopelasticsearchapache-sparkelasticsearch-2.0
Andriy
Andriy

Reputation: 53

Connect Spark 2.X to ElasticSearch 2.X

I'm working with Spark and ElasticSearch and can't find how to use Spark 2.X with ElasticSearch 2.x. ElasticSearch Spark libs support only Spark 1.6 for ES 2.X and supports Spark 2 for ES 5.alpha What do you use to connect Spark to ElasticSearch?

Upvotes: 1

Views: 1307

Answers (3)

Neil_TW
Neil_TW

Reputation: 97

i am using spark 2.2 read elasticsearch 5.5.2 is work

scalaVersion := "2.11.10"
val spark = "2.2.0"
val es = "5.5.2"
libraryDependencies ++= Seq(
  "org.elasticsearch" % "elasticsearch-spark-20_2.11" % es,
  "org.apache.spark" % "spark-core_2.11" % spark % "provided" ,
  "org.apache.spark" % "spark-sql_2.11" % spark % "provided",
  "org.apache.spark" % "spark-hive_2.11" % spark % "provided"
)
 val sparkSession = SparkSession.builder()
    .config("es.nodes",locahost)
    .config("es.port","9200")
    .appName("ES")
    .master("local[*]")
    .enableHiveSupport()
    .getOrCreate()

   sparkSession.read.format("org.elasticsearch.spark.sql")
  .option("es.query", "?q=id:(123)") 
  .load("es_index/es_type")

Upvotes: 0

Thomas Decaux
Thomas Decaux

Reputation: 22651

Yes it supports it.

You can find org.elasticsearch libraries here: https://mvnrepository.com/artifact/org.elasticsearch

https://mvnrepository.com/artifact/org.elasticsearch/elasticsearch-spark-20

Also, take care to match the same scala version (2.11 or 2.10). For instance, Ambari provides Spark2 - scala 2.11.

Upvotes: 0

Ram Ghadiyaram
Ram Ghadiyaram

Reputation: 29155

Please have a look at this.

Apache Spark support

Apache Spark is computing framework that is not tied to Map/Reduce itself however it does integrate with Hadoop, mainly to HDFS. elasticsearch-hadoop allows Elasticsearch to be used in Spark in two ways: through the dedicated support available since 2.1 or through the Map/Reduce bridge since 2.0. Spark 2.0 is supported in elasticsearch-hadoop since version 5.0

Hope that helps!

Also have a loot at ElasticSearch Spark Error Where user is using spark 2.0 with lower version(Spark 2.0.0 ElasticSearch-Spark 2.3.4) of Elastic search and @Crackerman was able to resolve his issue(other issue not version related stuff)

Upvotes: 1

Related Questions