Using Spring Cloud Data Flow to set up data exchanges in heterogeneous systems scenario

Question

I have a use case involving multiple applications written in different languages (Java, Python, C#, etc.), and I would like to create several streams between these apps. An illustrative example is given below.

I'm trying to find the best way to accomplish that via an orchestration layer that does all behind-the-scene chores including bookkeeping, deployment, messaging queues creation, and binding, etc.

For that, I'm exploring Spring Cloud Data Flow (SCDF). However, SCDF was designed to create data integration pipelines for Spring boot applications particularly. But would it be possible to employ/adapt SCDF in a case where we may have heterogeneous applications written in various languages?

The idea that may come to mind is to use connectors. So my other question is would it be possible to set up communication in such a case with connectors, and if so is there any already developed/provided?

Answer 1

As you point out, SCDF is focused more on the Spring/Java-based programming model through Spring Cloud Stream / Spring Cloud Task, and on top of it, SCDF provides the orchestration layer to deploy such apps and manage them at scale.

What you're attempting to do with SCDF, while it is feasible (example in Python), but doesn't come out of the box with great developer experience.

It is not great because there's isn't any auto-configuration component or frameworks like Spring Cloud Stream for other programming languages (eg: Python or .Net, JS). That means you are expected to do a little more than writing the just business logic in a language of choice, including the wiring and binding of message broker semantics in the application (see Python examples in the link shared above).

All that said, we are building the next-gen SCDF capabilities that will focus more specifically on polyglot support from the ground up. Spring Cloud Stream's binder/binding capabilities are run as a sidecar, and the user container (ie: business logic) interacting with this sidecar can be written and containerized in a programming language of your choice. SCDF-like orchestration layer would sit on top of this data plane, so you can orchestrate, deploy and observe the workloads all in a consistent manner.

Until all this early design thinking and the narrative matures, you can follow the Python recipe to assemble the moving parts on your own for the time being.