Async Operation in A Flink Sink using CompletableFuture

Question

Background

Planning to set a up data pipeline using Flink.

The flow looks like this

        Kafka --> Flink Job --> gRPC endpoint

Story so far

1. Blocking implementation is up and running. But that will not scale for high QPS
2. Tried simulating async behavior here

Problem

• For Async Behavior not sure how the behavior would be
• if CompletableFuture is used, per message it will be processed in Async manner, but will the next message be fetched for processing before processing of first is complete ? In other words, there is a way to achieve async processing within a task manager. But what is the behavior of Task manager in fetching next message / tuple ? Will is wait till Async process is complete or will it submit to CompletableFuture / Thread and fetch next message ? Not clear about that
• Will using a custom threadpool cause any issues if not shutdown as the pipeline will be running over a long period ?
• Any other solution to achieve async behavior in Flink sink ?

Answer 1

I would leverage Flink's support for async operators, and have a DiscardingSink, versus trying to implement a custom async sink.

And no, I don't see any reason why having a persistent thread pool would cause problems.