Azure Batch and use cases for JobManagerTask

Question

I am currently digging into the Azure Batch service, and I am confused about the proper use of a JobManagerTask...

...and maybe what the overall architecture of an Azure Batch application should look like. I have built the below architecture based on code samples from Microsoft found on Github .

These are my current application components.

App1 - ClusterHead

Creates a job (including an auto pool)
Defines the JobManagerTask
Runs on a workstation

App2 - JobManagerTask

Splits input data into chunks
Pushes chunks (unit of work) onto an input queue
Creates tasks (CloudTask)

App3 - WorkloadRunner

Pulls from the input queue
Executes the task
Pushes to the output queue

Azure Storage Account

Linked to Azure Batch account
Provides input & output queues
Provides a result table

Azure Durable Function

Implements the aggregator pattern by using DurableEntities so that I can access incoming results prematurely.
Gets triggered by messages in the output queue
Aggregates results and writes the entity to Azure Storage table

Questions

Is that proper use of the JobManagerTask?
Why do I want/need the extra binary/application package, that encapsulates the JobManagerTask?
Could someone please give an example of when I should prefer to use a JobManagerTask over creating the Jobs manually?

Thanks in advance!

brklein · Accepted Answer

Your example is an example of a how a JobManagerTask can be used, albeit as you mentioned if the data being generated is all generated by the JobManagerTask and is fixed then it could make sense to just merge it into your ClusterHead as you mentioned. In your case it just depends if you want the split and upload of your data to occur as part of the Job or run it on the workstation.

One area JobManagerTasks excel is if the data coming in is continuous. Basically if you had a bunch of writers to a raw input queue, you could have your JobManagerTask run continuously reading from that queue and splitting the data/creating the tasks.