CODEWITHSUNDEEP
spring-xd
Madhu Mohan Kommu
Madhu Mohan Kommu

Reputation: 335

Spring XD Distributed Environment

I am working on Spring XD and GemFire XD. I want to understand how Spring XD's distributed environment works. I know spring xd uses either redis or rabittmq as the transport.

I am clear about this, I have install spring xd and rabittmq on one machine. I changed the redis.properties file and added hostnames.

Do I need to install spring xd on all the machines? If so, after installing, how to bring those up.

On the master machine, I will do ./xd-admin and ./xd-container

How do you start up the nodes (spring xd instances/workers) so that they can listen for instructions from xd-admin?

Please help me on this.

Thanks, -Suyodhan

Upvotes: 0

Views: 1226

Answers (3)

Mohammad Abu Shaera
Mohammad Abu Shaera

Reputation: 21

You will install Spring xd one time on one machine, Spring XD will be connected to your hdfs distributed scaled out environment. You need to start the followings: 1. redis or rappitMQ in your case 2. hsqldb server 3. container 4. admin

when you start spring xd, you need to register the name node firstly using the command: hadoop config fs --name hdfs://serverip:8020

then you can use any module defined in spring xd (using stream or batch) by specifying its parameters directly without specifying those in the server.yml file.

Moha.

Upvotes: 0

SathiyaS
SathiyaS

Reputation: 246

Redis is used for analytics as only supported platform. For transport, you need either Redis or Rabbit.

Basically you just need to install Redis and RabbitMQ per their respective documentation. They can be in same or different servers, Ideally you would use their high availability option. For example Redis Sentinal. YOu don't need RabbitMQ unless you want to change the default transport from Redis to Rabbit. Once you install Redis and Rabbit, bring them up and provide their host:port info (and any additional as applicable) to the servers.yml in XD install (in all nodes) and bring up admin and containers. Evrything should work automatically by using zookeeper as the means to manage the distributed runtime.

Upvotes: 1

SathiyaS
SathiyaS

Reputation: 246

If you use Spring XD in distributed mode, I assume you have set up zookeeper as well. (If not check this http://docs.spring.io/spring-xd/docs/1.0.0.M7/reference/html/#_setting_up_zookeeper )

Admin and Container instances register themselves with Zookeeper as they come up. Admin queries zookeeper for available containers and assign tasks like deploying modules. Zookeeper is the trick behind Distributed mode.

Hope this helps.

Upvotes: 0

Related Questions