CODEWITHSUNDEEP
apache-sparkelasticsearch-hadoop
oaksharks
oaksharks

Reputation: 43

How to read a few columns of elasticsearch by spark?

In the es cluster, it has a large scale data, we used spark to compute data but in the way of elasticsearch-hadoop, followed by https://www.elastic.co/guide/en/elasticsearch/hadoop/current/spark.html

We have to read full columns of an index. Is there anything that help?

Upvotes: 1

Views: 2750

Answers (1)

Paul Back
Paul Back

Reputation: 1319

Yes, you can set config parameter "es.read.field.include" or "es.read.field.exclude" respectively. Full details here. Example assuming Spark 2 or higher.

val sparkSession:SparkSession = SparkSession
  .builder()
  .appName("jobName")
  .config("es.nodes", "elastichostc1n1.example.com")
  .config("es.read.field.include", "foo,bar")
  .getOrCreate()

Upvotes: 6

Related Questions