CODEWITHSUNDEEP
apache-flink
Rohit
Rohit

Reputation: 97

how to increase max parallelism in any condition in flink?

I just want to know is there any way after deployment on prod to increase setMaxParallelism in flink job ?

How to setMaxParallelism to a larger number currently is in on default 128.

Currently, i am getting an error after increasing this number.

Caused by: java.lang.IllegalStateException: Failed to rollback to checkpoint/savepoint Checkpoint Metadata (version=2). 
Max parallelism mismatch between checkpoint/savepoint state and new program. 
Cannot map operator cbc357ccb763df2852fee8c4fc7d55f2 with max parallelism 128 to new program with max parallelism 960. 
This indicates that the program has been changed in a non-compatible way after the checkpoint/savepoint.

so is there any way to increase this on any condition? I don't care about checkpoints/savepoints if i lose some data.

Upvotes: 0

Views: 1017

Answers (1)

David Anderson
David Anderson

Reputation: 43677

You can change the max parallelism, but you won't be able to restart from a checkpoint or savepoint taken when the previous setting was in place.

If you did want to migrate the state, rather than drop it, you could use the State Processor API to modify a savepoint accordingly.

If you have used the CEP library or the Table API you could have state that will be difficult to migrate. But for any state where you establish the state descriptors in your code, or for DataStream windows, you'll be fine.

https://docs.immerok.cloud/docs/how-to-guides/development/migrating-state-away-from-kryo/ and https://github.com/immerok/recipes/tree/main/kryo-migration shows how to use the State Processor API to migrate state away from Kryo. This is a different (and more challenging) scenario, but it does provide a complete, end-to-end example.

Upvotes: 2

Related Questions