CODEWITHSUNDEEP
apache-flink
binit
binit

Reputation: 476

Is there a way to timeout flink, at job level or task level?

I have a job that applies some regex to a large number of strings. This I have achieved by creating a cross join job with List of strings and List of regex as the input.

Normally this runs fine, but once in a while for a certain input and regex pair, the task execution never terminates - mostly due to input being too big / regex not being efficient.

In this case I would prefer the task to get 'timed out', or the job as a whole to get 'timed out' so I know something is wrong and skip processing.

I went through the flink config docs but wasn't able to find out.

I did a workaround creating a future async thread inside task and cancelling it after a certain time to apply the regex, but it seems like an overkill. Hence looking for a better solution.

Upvotes: 0

Views: 472

Answers (1)

Arvid Heise
Arvid Heise

Reputation: 3634

There is no job limit in Flink that can do what you look for. Since that sounds very batch-oriented, I'd also don't think that this is a feature that's actively worked on.

Nevertheless, your solution is actually quite good already. Other solutions depend on your infrastructure. If you trigger the job by using airflow or any other workflow system, I'd imagine that they can cancel tasks after some time. If you run it on K8s or YARN, you may be able to limit the total resource usage. But if you don't use any of it, then your solution is a good safe guard.

Some more ideas: do you really need the slow java Regexes or could you use Re2 or other automaton libraries? Could you add some sanity checks on very large input strings and skip them? Could you simply stop applying the CrossFunction after your time ran out (graceful termination)?

Upvotes: 1

Related Questions