CODEWITHSUNDEEP
apache-sparkspark-streaming
Soumitra
Soumitra

Reputation: 612

Inter application communication in apache spark streaming

I have two applications : App1 and App2. On a single cluster I have to spawn 5 instances os App1 and 1 instance of App2.

What would be the best way to send data from the 5 App1 instances to the single App2 instance ?

Right now I am using Kafka to send data from one spark application to the spark application but the setup doesn't seem right and I hope there is a better way to do this.

Upvotes: 1

Views: 455

Answers (1)

Daniel Zolnai
Daniel Zolnai

Reputation: 16910

Apache Ignite might be useful to you.

Apache Ignite provides an implementation of Spark RDD abstraction which allows to easily share state in memory across multiple Spark jobs, either within the same application or between different Spark applications.

Upvotes: 1

Related Questions