CODEWITHSUNDEEP
hadoop-yarnapache-flinkflink-sql
Longxing Wei
Longxing Wei

Reputation: 171

Flink on yarn use yarn-session or not?

There are two methods to deploy flink applications on yarn. The first one is use yarn-session and all flink applications are deployed in the session. The second method is each flink application deploy on yarn as a yarn application.

My question is what's the difference between these two methods? Which one to choose in product environment?

I can't find any material about this.

I think the first method will save resources since only need one jobmanager(yarn application master). While it is also the disadvantage since the only jobmanager can be the bottleneck while flink applications getting more and more.

Upvotes: 2

Views: 1188

Answers (1)

David Anderson
David Anderson

Reputation: 43717

Both modes have their uses in production environments.

Session mode generally makes sense when you will be running a bunch of short-lived jobs, and want to avoid the overhead of starting up a cluster for each one. On the other hand, there are security implications, as any credentials available to any of the jobs will be accessible to all of the jobs. Cluster-per-job mode may use more resources overall, but is, in some sense, more straightforward.

Upvotes: 2

Related Questions