CODEWITHSUNDEEP
apache-flink
Tom
Tom

Reputation: 6332

Confused with JobManager and JobMaster

I am new to Flink.

On the internet, I have always seen the concept JobManager, but when I look into the Flink source code(the latest code forked from master branch)

Interface: JobManagerRunner
javadoc: Interface for a runner which executes a {@link JobMaster}.


class: JobMaster
 JobMaster implementation. The job master is responsible for the execution of a single

I would ask whether JobMaster in the code is exactly the concept of JobManager on the internet

Upvotes: 1

Views: 553

Answers (1)

David Anderson
David Anderson

Reputation: 43499

Short answer: the JobManager referred to in the docs comprises the Dispatcher, a cluster-framework-specific ResourceManager, the Blob Server, and a set of JobMasters (one per job). You won't see a JobManager class in the code; while there was once a monolithic JobManager, it was refactored by FLIP-6 into a set of separate components.

Longer answer:

The term JobManager has been defined, redefined, and re-redefined over the past few years.

When FLIP-6 was merged, a renaming was done in some of the docs -- but not everywhere, and not in the code. At this point the term "Flink Master" was introduced to refer to what had been the JobManager, and what the code refers to as the JobMaster began to be referred to as the JobManager in the docs.

With the 1.11 release we decided to abandon the Flink Master terminology, and revert back to calling that the JobManager. The collection of per-job services (i.e., the scheduler and checkpoint coordinator) that we had been calling the JobManager in some of the docs since FLIP-6 (and had always called the JobMaster in the code) is now generally left unnamed in the docs. Some vestiges of the interim naming scheme probably still linger here and there.

Upvotes: 6

Related Questions