CODEWITHSUNDEEP
apache-flinkflink-streaminggoogle-cloud-pubsub
Henrik
Henrik

Reputation: 9915

Flink 1.8, parallelism > 1, source never outputs values

I have a cluster with:

If I set default.parallelism: 4 on a job with the Flink PubSub source, I keep getting this error when starting my "job cluster"/taskmanager:

[analytics-job-cluster-7bd4586ccb-s5hmp job] 2019-05-01 16:22:30,888 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Checkpoint triggering task Source: Custom Source -> Process -> Timestamps/Watermarks -> app_events (1/4) of job 00000000000000000000000000000000 is not in state RUNNING but SCHEDULED instead. Aborting checkpoint.

However, if I point the same job at a bunch of files, it works perfectly. What does this mean?

Upvotes: 1

Views: 665

Answers (1)

Dominik Wosiński
Dominik Wosiński

Reputation: 3864

So, the issue is that You need the numberOfTaskSlots equal to Your parallelism basically. So in this case If You have only 1 TaskManager with only 1 TaskSlot Flink will not be able to start the job properly as there is simply not enough slots for it. If You set the numberOfTaskSlots for the given TaskManager equal to the parallelism, then it should work well.

Upvotes: 1

Related Questions