CODEWITHSUNDEEP
apache-sparkvirtual-machineapache-kafkaspark-streaming
benSlash
benSlash

Reputation: 25

How to use Spark Streaming from an other vm with kafka

I have Spark Streaming on a virtual machine, and I would like to connect it with an other vm which contains kafka . I want Spark to get the data from the kafka machine. Is it possible to do that ?

Thanks

Upvotes: 0

Views: 289

Answers (2)

Ram Parashar
Ram Parashar

Reputation: 21

It's possible and makes perfect sense to have them on separate VM's. That way there is a clear separation of roles.

Upvotes: 0

Manav Garg
Manav Garg

Reputation: 522

Yes, it is definitely possible. In fact, this is the reason why we have distributed systems in place :)

When writing your Spark Streaming program, if you are using Kafka, you will have to create a Kafka config data structure (syntax will vary depending on your programming language and client). In that config structure, you will have to specify the Kafka brokers IP. This would be the IP of your Kafka VM.

You then just need to run Spark Streaming Application on your Spark VM.

Upvotes: 1

Related Questions