CODEWITHSUNDEEP
scalaapache-sparkelasticsearch
Aymen Rahal
Aymen Rahal

Reputation: 125

How to save a JSON file in ElasticSearch using Spark?

I am trying to save a JSON file in ElasticSearch but its not working.

This is my code:

import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext
import org.elasticsearch.spark.sql._
import org.apache.spark.SparkConf

object HelloEs {

  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("WriteToES").setMaster("local")
    conf.set("es.index.auto.create", "true")
    val sc = new SparkContext(conf)
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    val sen_p = sqlContext.read.json("/home/Bureau/mydoc/Orange.json")
    sen_p.registerTempTable("sensor_ptable")
    sen_p.saveToEs("sensor/metrics")
  }

}

I am getting also this error:

Exception in thread "main" java.lang.NoSuchMethodError:  org.elasticsearch.spark.sql.package$.sparkDataFrameFunctions(Lorg/apache/spark/sql/Dataset;)Lorg/elasticsearch/spark/sql/package$SparkDataFrameFunctions;
    at learnscala.HelloEs$.main(HelloEs.scala:20)
    at learnscala.HelloEs.main(HelloEs.scala)

Upvotes: 2

Views: 1778

Answers (1)

Lakshman Battini
Lakshman Battini

Reputation: 1912

There are multiple ways to save an RDD / Dataframe to Elastic Search.

Spark Dataframe can be written to Elastic Search using:

df.write.format("org.elasticsearch.spark.sql").mode("append").option("es.resource","<ES_RESOURCE_PATH>").option("es.nodes", "http://<ES_HOST>:9200").save()

RDD can be written to ES using:

import org.elasticsearch.spark.rdd.EsSpark EsSpark.saveToEs(rdd, "<ES_RESOURCE_PATH>")

In your case, modify the code as below:

`

object HelloEs {

  def main(args: Array[String]): Unit = {
    val conf = new SparkConf().setAppName("WriteToES").setMaster("local")
    conf.set("es.index.auto.create", "true")
    val sc = new SparkContext(conf)
    val sqlContext = new org.apache.spark.sql.SQLContext(sc)
    val sen_p = sqlContext.read.json("/home/Bureau/mydoc/Orange.json")
    sen_p.write.format("org.elasticsearch.spark.sql").mode("append").option("es.resource","<ES_RESOURCE_PATH>").option("es.nodes", "http://<ES_HOST>:9200").save()
  }

}

`

Upvotes: 2

Related Questions