CODEWITHSUNDEEP
airflow
Michelle
Michelle

Reputation: 1

Airflow log file Permission denied

I install Airflow 2.1.2 with python 3.6.5 in a linux server and trigger a DAG by run_as user but it is not worked as expected.

I've created a logs folder with chmod -R 777 but when the DAG get start and system create the log file under logs/ with "drwx------ 3 root root" and caused permission denied since only root can access the folder.

is there any solution I can fix it?

here is the error I got

[2021-07-16 07:00:14,334] {base_task_runner.py:135} INFO - Running: ['sudo', '-E', '-H', '-u', 'hdfsprod', 'airflow', 'tasks', 'run', 'hdfs_kinit', 'hdfs_kinit', '2021-07-16T07:00:04.092676+00:00', '--job-id', '19', '--pool', 'default_pool', '--raw', '--subdir', 'DAGS_FOLDER/hdfs_test.py', '--cfg-path', '/tmp/tmp8_t6s5o2', '--error-file', '/tmp/tmpmf3zruty']
[2021-07-16 07:00:15,841] {base_task_runner.py:119} INFO - Job 19: Subtask hdfs_kinit [[34m2021-07-16 07:00:15,839[0m] {[34mdagbag.py:[0m487} INFO[0m - Filling up the DagBag from /opt/data/dags/hdfs_test.py[0m
[2021-07-16 07:00:15,992] {base_task_runner.py:119} INFO - Job 19: Subtask hdfs_kinit Traceback (most recent call last):
[2021-07-16 07:00:15,993] {base_task_runner.py:119} INFO - Job 19: Subtask hdfs_kinit   File "/service/python3/lib/python3.6/pathlib.py", line 1246, in mkdir
[2021-07-16 07:00:15,993] {base_task_runner.py:119} INFO - Job 19: Subtask hdfs_kinit     self._accessor.mkdir(self, mode)
[2021-07-16 07:00:15,993] {base_task_runner.py:119} INFO - Job 19: Subtask hdfs_kinit   File "/service/python3/lib/python3.6/pathlib.py", line 387, in wrapped
[2021-07-16 07:00:15,993] {base_task_runner.py:119} INFO - Job 19: Subtask hdfs_kinit     return strfunc(str(pathobj), *args)
[2021-07-16 07:00:15,993] {base_task_runner.py:119} INFO - Job 19: Subtask hdfs_kinit PermissionError: [Errno 13] Permission denied: '/opt/data/logs/hdfs_kinit/2021-07-16T07:00:04.092676+00:00'
[2021-07-16 07:00:15,994] {base_task_runner.py:119} INFO - Job 19: Subtask hdfs_kinit

Upvotes: 0

Views: 1369

Answers (1)

Jarek Potiuk
Jarek Potiuk

Reputation: 20097

The easiest way to fix it is to make your "run_as" user belong to "root" group and change umask to 002 in the scripts that start Airflow.

https://unix.stackexchange.com/questions/12842/make-all-new-files-in-a-directory-accessible-to-a-group

Upvotes: 1

Related Questions