CODEWITHSUNDEEP
pythonairflowairflow-scheduler
Nitish Sharma
Nitish Sharma

Reputation: 331

Airflow Scheduler creating PID for same dag to generate tasks every time

I am using Local Executor. I have a situation where i have unique dags getting generated for each request id for eg 1.py , 2.py .

1.py assume has two tasks and 2.py has 3 tasks. I would also receive more dags periodically for eg 3.py,4.py etc.

Is there any problem of creating a dag for every new id/request ID.

I have observed that Scheduler keeps giving this log. 

Started a process (PID: 92186) to generate tasks for /Users/nshar141/airflow/dags/3.py - logging into /Users/nshar141/airflow/logs/scheduler/2018-05-07/3.py.log

My question here is why scheduler keeps generating separate PIDs for generating tasks. I tried changing different parameters in the config related to concurrency and parallelism but scheduler seems to be executing that statement everytime for every dag present in dags folder.

I am attaching my dag definition. I want to run dag as soon as it is created. What are the parameters i should give in start_time and scheduler_interval?

dag = DAG('3', description='Sample DAG',schedule_interval=@once,start_date=datetime(2018, 5, 07), catchup=False)

Since i have a need to generate dags dynamically with unique dag id and place it in the dags folder my concern here is scheduler would generate too many process IDS for every dag in the folder which already has been executed.

Image showing scheduler generating PIDs repeatedly

Upvotes: 2

Views: 1615

Answers (1)

Antoine Augusti
Antoine Augusti

Reputation: 1608

Why do you want to create a new DAG for every request? I think that the most appropriate way would be to store requests and have a single DAG perform your logic for multiple requests at the same time, in a batch fashion. You can run your DAG very often if you want.

You seem to want tasks to be executed as soon as possible. If you're interested in near real-time with a lot of throughput, Airflow may not be appropriate and you'll want to use a message queue instead.

Upvotes: 1

Related Questions