How can I run Tasks using the DockerOperator from within a Kubernetes deployment?

Question

Context

I am running Airflow, and trying to run a proof of concept for a Docker container using Airflow's DockerOperator. I am deploying to Kubernetes (EKS), but not using Kubernetes Executor yet. Given that I am running pods, by using the DockerOperator I will be running (to my understanding) Docker in Docker.

Whenever I run my task, I am receiving the Error: ERROR - Error while fetching server API version. The error happens both on docker-compose as well as EKS (kubernetes).

My current status

Thi is how my airflow Dockerfile looks like:

FROM apache/airflow:1.10.14-python3.8
# Use airflow user for pip installs and other things.
USER root
# Copying Airflow requirements
USER airflow
COPY requirements.txt /tmp/requirements.txt
# Installing requirements. Using airflow user (docs: https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html)
RUN pip install --no-cache-dir -r /tmp/requirements.txt

This is how the dag I am trying to run looks like:

with DAG(
        dag_id='task',
        default_args=dict(
            start_date=days_ago(0)
        ),
        schedule_interval='@daily'
) as dag:

    task_1 = DockerOperator(
        dag=dag,
        task_id='docker_task',
        image='centos:latest',
        api_version="auto",
        docker_url='unix://var/run/docker.sock',
        command='/bin/sleep 30'
    )

This is the stack trace of the error I am getting:

[2020-12-29 14:18:52,601] {taskinstance.py:1150} ERROR - Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/util/retry.py", line 410, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/home/airflow/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 392, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1301, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1250, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.8/http/client.py", line 1010, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.8/http/client.py", line 950, in send
    self.connect()
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/transport/unixconn.py", line 43, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 543, in get
    return self.request('GET', url, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/airflow/.local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
    result = task_copy.execute(context=context)
  File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/docker_operator.py", line 260, in execute
    self.cli = APIClient(
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/home/airflow/.local/lib/python3.8/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

What I have tried

1. Mount the socket into docker compose /var/run/docker.sock:/var/run/docker.sock:ro

First, that gives me a new error to:

ERROR - Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))

Second, how will I be able to mount on Kubernetes? I would guess that complicates things

2. Install docker within my container and try to give privileges to the airflow user like:

FROM apache/airflow:1.10.14-python3.8
# Use airflow user for pip installs and other things.
USER root
# Docker
RUN curl -sSL https://get.docker.com/ | sh  
# Usermod
RUN usermod -aG docker airflow
# Copying Airflow requirements
USER airflow
COPY requirements.txt /tmp/requirements.txt
# Installing requirements. Using airflow user (docs: https://airflow.apache.org/docs/apache-airflow/stable/production-deployment.html)
RUN pip install --no-cache-dir -r /tmp/requirements.txt

But that also did not work.

3. Mounting the socket into the DockerOperator task like:

    task_1 = DockerOperator(
        dag=dag,
        task_id='docker_task',
        image='centos:latest',
        api_version="auto",
        docker_url='unix://var/run/docker.sock',
        command='/bin/sleep 30',
        volumes=['/var/run/docker.sock:/var/run/docker.sock:ro'],
    )

But that also has had no effect

Answer 1

Copying down from comment:

The direct issue is likely that the docker control socket file is owned by something like root:docker on the host, and assuming Airflow isn't running as root (which it shouldn't) then you would need to specifically set it to run with the same gid as the docker group on the host machine (which probably has no matching group inside the container, which could confuse other things).

The real answer is to use the KubernetesPodOperator instead. DockerOperator won't work at all once you stop using Docker on the underlying host which will be happening soon now that dockershim is deprecated.