CODEWITHSUNDEEP
apache-flinkflink-streaming
dhinesh
dhinesh

Reputation: 31

How to handle backpressure in flink streaming job?

I am running a streaming flink job which consumes the streaming data from kafka and do some process over the data in flink map function and write the data to the Azure data lake and the elastic search. For map function I used a parallelism of one because I need to process the incoming data one by one over the list of data I maintain as a global variable. Now when I run the job as the flink starts to get the streaming data from kafka it's backpressure becomes high in the map function. Is there any settings or configurations I could do to avoid the backpressure in flink?

enter image description here

Upvotes: 1

Views: 2301

Answers (1)

Eron Wright
Eron Wright

Reputation: 1060

Backpressure on a given operator indicates that the next operator is consuming elements slowly. From your description it would seem that one of the sinks is performing poorly. Consider scaling up the sink, commenting-out a sink for troubleshooting purposes, and/or investigating whether you're hitting an Azure rate limit.

Upvotes: 4

Related Questions