CODEWITHSUNDEEP
apache-sparkignite
Vijay Innamuri
Vijay Innamuri

Reputation: 4372

Apache Spark ownership Vs Ignite Ownership

I have been going through Ignite SharedRDD in Spark and would like to understand the underlying execution layers/ phases involved in the Spark-Ignite job.

In the ScalarSharedRDDExample.scala,

val df = transformedValues.sql("select _val from Integer where _val < 100 and _val > 9 ")

(Who is responsible for what?)

Upvotes: 1

Views: 157

Answers (2)

Evgenii Zhuravlev
Evgenii Zhuravlev

Reputation: 3017

where does the following transformation executes?

SQL executed on Ignite nodes.

How does the spark and ignite divide the ownership of a spark application?

You can read about it here

What are the best practices in building Spark-Ignite applications?

Ignite can provide shared storage, so the state can be passed from one Spark application or job to another.

Ignite can provide SQL with indexing so Spark SQL can be accelerated over 1,000x (spark doesn’t index the data)

When working with files instead of RDDs, the Apache Ignite In-Memory File System (IGFS) can also share state between Spark jobs and applications.

Well, best practices is a too wide question - I think you need to start with Ignite Example and ask a certain question if you will face any problem.

Upvotes: 0

Sachin Thapa
Sachin Thapa

Reputation: 3719

Everything still executes they way it used to be. IgniteRDD is implemented as a view over a distributed Ignite cache, which may be deployed either within the Spark job executing process, or on a Spark worker, or in its own cluster.

As per information on their website, shared state may either exist only during the lifespan of a Spark application (embedded mode), or it may out-survive the Spark application (standalone mode), in which case the state can be shared across multiple Spark.

For more information check official website Shared Apache Spark RDDs

Also read some of the documented use cases which helps to understand better.

Distributed Database Key-Value Store

See code example, IgniteContext is created from sparkContext.

val igniteContext = new IgniteContext(sparkContext, 
    () => new IgniteConfiguration())

Hope this helps ! Cheers !

Upvotes: 1

Related Questions