CODEWITHSUNDEEP
spring-bootspring-xdspring-cloud-streamspring-cloud-configspring-cloud-dataflow
Praveen Kumar K S
Praveen Kumar K S

Reputation: 3074

Spring cloud data flow - Micro services deployment

Team, Currently I am working on spring-xd and using as a runtime container for data analytics and yarn jobs.

My questions are

1) Can I leverage the same environment setup which I used for spring-xd?
2) From the documentation,I read that it can be deployed as micro services, is it using embedded drivers for stream processing? If it is using embedded drivers, can I use it to deploy in a clustered environment with the same infrastructure leveraged for spring-xd?
3) Is there any specific wrappers built for Apache Spark?

My Environment: Spark 1.6.1, Hadoop 2.7.2, zookeeper 3.6.8, redis 3.2, spring-xd-1.3.1

Any help on this specific queries would be highly appreciated.

Upvotes: 3

Views: 561

Answers (1)

Sabby Anandan
Sabby Anandan

Reputation: 5651

Can I leverage the same environment setup which I used for spring-xd?

Spring Cloud Data Flow (SCDF) relies on spring-cloud-deployer SPI and there are implementations for Cloud Foundry, Apache YARN, Apache Mesos and Kubernetes. Given that you're already having an Hadoop cluster with YARN in use, you could provision the YARN implementation of the SCDF-server.

is it using embedded drivers for stream processing?

Not clear what you mean by this. If you're referring to jdbc-drivers; specifically, when using jdbc as a sink application, we do embed OSS-friendly drivers, yes.

If it is using embedded drivers, can I use it to deploy in a clustered environment with the same infrastructure leveraged for spring-xd?

Perhaps answer to #1 covers this. You could leverage the same infrastructure and provision YARN SCDF-server using the Ambari plugin.

Is there any specific wrappers built for Apache Spark?

We have Spark as a client and cluster applications. You can register them in SCDF to build task/batch pipelines.

Upvotes: 4

Related Questions