CODEWITHSUNDEEP
azureservicebusazure-eventhubspring-cloud-dataflowspring-cloud-dataflow-ui
Alan Krueger
Alan Krueger

Reputation: 4786

How to make Azure services available in Spring Cloud Dataflow UI as sources and sinks?

My current team develops microservices in an Azure environment that using Service Bus and Event Hubs for processing messages via Spring Cloud Stream.

I'm evaluating using Spring Cloud Data Flow to do something like this in a more agile manner, but I'm not seeing either Service Bus or Event Hubs as available sources or sinks in the UI.

How can I access those through the Spring Cloud Data Flow UI?

Upvotes: 0

Views: 52

Answers (1)

onobc
onobc

Reputation: 596

My current team develops microservices in an Azure environment that using Service Bus and Event Hubs for processing messages via Spring Cloud Stream.

Sounds like you are using the Spring Cloud Stream Azure Binder https://learn.microsoft.com/en-us/azure/developer/java/spring-framework/configure-spring-cloud-stream-binder-java-app-with-service-bus?tabs=use-a-service-bus-queue for these Microservices.

but I'm not seeing either Service Bus or Event Hubs as available sources or sinks in the UI.

The streaming apps in Dataflow (soures, processors, sinks) are simply Spring Cloud Stream + Spring Boot applications. As such, whatever binder they are using dictates what messaging middleware they will communicate with. Dataflow does provide a set of out-of-the-box applications but only for the Kafka and Rabbit messaging middleware. Also there is not an Azure Source/Sink provided.

Note that the messaging middleware the app communicates with is based on its binder. However, you can still have source and sinks to other external messaging systems. There is a subtle distinction between having a source/sink to an external system and having an app that is using that messaging system as its bus/binder.

Example: I could have a streaming pipeline w/ apps based on the Rabbit binder but I feed the pipeline w/ a KafkaSource. This KafksSource will pull messages from the external Kafka system and source them into an input queue for the Rabbit Processor to consume.

So an Azure source/sink is different than having apps that are using Azure as the binder/messaging middleware.

How can I access those through the Spring Cloud Data Flow UI?

For an app to be visible in the UI it must first be registered w/ Dataflow. The registration tells Dataflow what type of app it is, what metadata it has, and where it is located (Maven, Docker).

This guide shows how to develop, register, and deploy a Spring Cloud Stream based app w/ Datafllow. https://dataflow.spring.io/docs/stream-developer-guides/streams/. One section of interest is how to register your app w/ Dataflow https://dataflow.spring.io/docs/stream-developer-guides/streams/data-flow-stream/#application-registration.

One point is that you will need to expose your apps in either a Maven repository or a Docker registry and then configure Dataflow to point to it.

Upvotes: 1

Related Questions