CODEWITHSUNDEEP
airflow
Jürgen Simon
Jürgen Simon

Reputation: 896

GoogleCloudStorageDownloadOperator "Task exited with return code -6"

I am new to airflow and I am trying something simple with GoogleCloudStorageDownloadOperator:

default_args = {
    'start_date': airflow.utils.dates.days_ago(0),
    'schedule_interval': None,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
    'params': {
        'work_dir': '/tmp'
    }
}

dag = DAG(
    'foo',
    default_args=default_args,
    description='This is foobar',
    schedule_interval=timedelta(weeks=1),
    dagrun_timeout=timedelta(minutes=60))

mock_download = GoogleCloudStorageDownloadOperator(
    task_id='download-foo-from-gcp',
    bucket='foo-data',
    object='{% if (task_instance.pid % 2 == 0) %}foo{% else %}bar{% endif %}/data.tar.gz',
    filename='{{ params.work_dir }}/data.tar.gz',
    google_cloud_storage_conn_id='google_cloud_default',
    dag=dag
)

While I can run this task in PyCharm for example (using airflow test), it fails all the time when triggered from the web interface (scheduled). The error message in the log is completely useless, to say the least:

... 
[2020-01-09 17:04:18,871] {gcs_download_operator.py:86} INFO - Executing download: crunchbase-mock-data, foo/data.tar.gz, /tmp/data.tar.gz
[2020-01-09 17:04:28,751] {logging_mixin.py:112} INFO - [2020-01-09 17:04:28,751] {local_task_job.py:103} INFO - Task exited with return code -6

Can anyone shed any light on this? What the heck is -6 supposed to mean? Is there a way to see a little more details about what happened there?

Upvotes: 1

Views: 1066

Answers (2)

Aneesh Makala
Aneesh Makala

Reputation: 351

I had the same issue in Airflow task running tweepy exits with return code -6.

Are you on Mac OS High Sierra (or above)? If so, refer https://stackoverflow.com/a/52230415/4434664. It solved my issue.

Basically, airflow test/PyCharm merely runs the task in-process, but the scheduler would start a worker process which would call fork(), and apparently, High Sierra introduced some new security changes that's breaking fork() usages in python.

This also caused problems in ansible. Refer https://github.com/ansible/ansible/issues/32499#issuecomment-341578864

Upvotes: 7

Ilya Bystrov
Ilya Bystrov

Reputation: 3056

Can anyone shed any light on this? What the heck is -6 supposed to mean?

There is a contract that

A negative value -N indicates that the child was terminated by signal N (POSIX only).

In your case it means that the process was terminated by SIGABRT (code 6) signal

Is there a way to see a little more details about what happened there?

There is no much background info from your site. In general, try to play with different operators and files. Also, from my perspective Airflow is not well documented. And I recommend to check Airflow sources.

Upvotes: 0

Related Questions