CODEWITHSUNDEEP
apache-sparkelasticsearchpysparkelasticsearch-hadoop
eugene
eugene

Reputation: 41745

Elasticsearch Spark, how to query multiple times?

I'm on jupyter notebook.

I'd like to use query dsl to prepare initial Dataframe.

I use conf.set("es.query", dsl_query) for that. (https://www.elastic.co/guide/en/elasticsearch/hadoop/current/configuration.html#_querying)

But then, I want to apply different query to prepare another Dataframe, and I can't find a way to apply a new dsl_query without creating new SparkContext

But I didn't find a way to recreate a SparkContext inside jupyter environment either.

I want to run analysis using QueryDSL-1 as baseline then run another anlysis using QueryDSL-2 as another baseline

Is there a way of doing this without creating two notebook?

Upvotes: 0

Views: 607

Answers (1)

eliasah
eliasah

Reputation: 40380

You'll just need to specify the es.query as an option to your DataFrameReader i.e :

spark.read.option("es.query", dsl_query).option("...", "...")

Upvotes: 0

Related Questions