CODEWITHSUNDEEP
apache-flink
jonathan
jonathan

Reputation: 35

Synchronous response with Apache Flink

I could not find any answer to my question on the web so far, so I thought its good to ask here. I know that Apache Flink is by design asynchronous, but I was wondering if there is any project or design, which aims to build a synchronous pipeline with Flink.

With synchronous response I mean in e.g. having an API endpoint, where I send my data to, the processing is done by Flink, and the outcome of the processing is given back (in what form ever) in the body of the answer to the API call e.g. a 200

I already looked into RabbitMQ RPC but I was not able to successfully implement it. I'm happy for any direction or suggestion. Thanks, Jon

Upvotes: 0

Views: 602

Answers (1)

Dominik Wosiński
Dominik Wosiński

Reputation: 3864

The closest thing that comes into my mind seems to be deploying Flink job with TcpSource available in Apache Bahir. You could have an HTTP endpoint that would receive some data and call Flink on the specified address then process it and create a response. The problem is that there is only TcpSource available in Bahir, which means You would need to create large part of the code (whole Sink) by yourself.

There can be also other ways of doing that (like trying to assign an id to each message and then waiting for message with that Id to arrive on Kafka and sending it as a response, but seems to be troublesome and error-prone)

The other way would be to make the response asynchronous(I know the question specifically mentions sync response but mentioning that just for sake of completeness)

However, I would like to say that this seems like a misuse of Flink to me. Flink was primary designed to allow real-time computations on multiple nodes, which doesn't seem to be a case here. I would suggest looking into different streaming libraries that are much more lightweight, easier to compose, and can offer the functionality You want out-of-the-box. You may want to take a look at Akka Streams for example.

Upvotes: 1

Related Questions