CODEWITHSUNDEEP
apache-flink
Pankaj
Pankaj

Reputation: 53

In Apache Flink, what is the difference between the Job Manager and the Job Master?

In Apache Flink (e.g. v1.8), what is the difference between the Job Manager and the Job Master?

Job Manager and Job Master seem to be used analogously in the logs.

What is the difference between the Job Manager and the Job Master?

Thanks!

Upvotes: 2

Views: 1280

Answers (2)

ardhani
ardhani

Reputation: 333

The JobManager is the composition of mainly 3 components.

  1. Dispatcher - dispatch the job to the Task Managers
  2. Resource Manager - Allocate the required resource for the job
  3. JobMaster - Supervising, coordinating the Flink Job tasks.

So, JobMaster is part of JobManager. As per docs, a single JobManager is assigned to each individual Flink application, which can contain multiple Flink jobs in it.

For example, a Flink Application with 2 jobs will instantiate 1 JobManger but will contain 2 JobMasters.

Upvotes: 2

Ortomala Lokni
Ortomala Lokni

Reputation: 62506

JobManager and JobMaster have different roles.

For the JobManager, according to the JobManager Data Structures section of the documentation:

During job execution, the JobManager keeps track of distributed tasks, decides when to schedule the next task (or set of tasks), and reacts to finished tasks or execution failures.

The JobManager receives the JobGraph, which is a representation of the data flow consisting of operators (JobVertex) and intermediate results (IntermediateDataSet). Each operator has properties, like the parallelism and the code that it executes. In addition, the JobGraph has a set of attached libraries, that are necessary to execute the code of the operators.

The role of the JobMaster is more limited according to the Javadoc:

JobMaster implementation. The job master is responsible for the execution of a single JobGraph.

Upvotes: 0

Related Questions