CODEWITHSUNDEEP
python-3.xgoogle-cloud-platformgoogle-cloud-dataflowapache-beamdataflow
deepalneema
deepalneema

Reputation: 37

GCP Dataflow Streaming Job getting Processing stuck in step s03 for at least 05m00s without outputting or completing in state finish

I am getting this error daily in my python dataflow job.

I am using Apache beam 2.15 (same with 2.17 as well) Python 3.7.

enter image description here

2020-01-28 17:08:53.801 GMTError message from worker: Processing stuck in step s03 for at least 10m00s without outputting or completing in state finish at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1693) at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323) at java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1729) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.beam.sdk.util.MoreFutures.get(MoreFutures.java:57) at org.apache.beam.runners.dataflow.worker.fn.control.RegisterAndProcessBundleOperation.finish(RegisterAndProcessBundleOperation.java:285) at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:85) at org.apache.beam.runners.dataflow.worker.fn.control.BeamFnMapTaskExecutor.execute(BeamFnMapTaskExecutor.java:125) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1295) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000(StreamingDataflowWorker.java:149) at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run(StreamingDataflowWorker.java:1028) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

Upvotes: 0

Views: 866

Answers (1)

Pablo
Pablo

Reputation: 11041

It is useful to note that the Processing stuck messages do not indicate that your job is stuck. Rather, they indicate that the Dataflow worker has been performing a single operation for over 5 minutes.

If you see this message every now and then, but your pipeline continues to make progress overall, then you've got nothing to worry about.

If you see this message continuously, and your pipeline stops making any progress at all, then that's an indication of actual stuckness.

Upvotes: 0

Related Questions