Reputation: 2483

Python in docker – RuntimeError: can't start new thread

I'm unable to debug one error myself. I'm running python 3.8.12 inside docker image on Fedora release 35 (Thirty Five) and I'm unable to spawn threads from python. It's required for boto3 transfer to run in parallel and it uses concurrent.features to do so.

The simplest example which replicates my issue without any dependencies is (copied from python docs)

import concurrent.futures
import urllib.request

URLS = ['http://www.foxnews.com/',
        'http://www.cnn.com/',
        'http://europe.wsj.com/',
        'http://www.bbc.co.uk/',
        'http://some-made-up-domain.com/']

def load_url(url, timeout):
    with urllib.request.urlopen(url, timeout=timeout) as conn:
        return conn.read()

with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
    future_to_url = {executor.submit(load_url, url, 60): url for url in URLS}
    for future in concurrent.futures.as_completed(future_to_url):
        url = future_to_url[future]
        try:
            data = future.result()
        except Exception as exc:
            pass

sadly output of these lines is

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "<stdin>", line 2, in <dictcomp>
  File "/usr/lib64/python3.8/concurrent/futures/thread.py", line 188, in submit
    self._adjust_thread_count()
  File "/usr/lib64/python3.8/concurrent/futures/thread.py", line 213, in _adjust_thread_count
    t.start()
  File "/usr/lib64/python3.8/threading.py", line 852, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

That's all I have. Is there place where should I look? I've already checked ulimit which says unlimited. I'm kind of despair where to look or what to change to debug this issue.

Upvotes: 36

Answers (7)

Radhakrishnan

Reputation: 1040

pip uses thread to show progress bar. Try to disable it

RUN pip install --progress-bar off -r requirements.txt

Reference : https://forums.docker.com/t/runtimeerror-cant-start-new-thread/138142

Upvotes: 8

Harvindar Singh Garcha

Reputation: 454

In my case I was using

FROM python:3.8

I changed it to

FROM python:3.8-bullseye

And it worked for me.

For more info Build started failing using Python:3.8 Docker image on apt-get update and install with GPG error: bookworm InRelease is not signed

Upvotes: 11

targetXING

Reputation: 111

If you are trying to build this Docker container on AWS with CodeBuild and you are using multistage builds, you might need to enable concurrent builds.

Here are some parameters that you might want to look into: DOCKER_BUILDKIT, DOCKER_CLI_AGGREGATE.

Here is an example: Enabling Docker Buildkit in AWS CodeBuild/CodePipeline

Upvotes: 0

Tycho Chen

Reputation: 126

I also got this problem, and after I added --privileged when do docker run, the problem fixed, hope it can help you.

If you use docker-compose,

web:
  image: an_image-image:1.0
  container_name: my-container
  privileged: true

Upvotes: 7

rishabhc32

Reputation: 311

I was deploying a FastAPI asyncio server container on Ubuntu 18 with Docker version 19.03.12 and was getting the following error:

 File "/usr/local/lib/python3.10/threading.py", line 935, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

My Environment:

Docker Image: python:3.9-slim
Python dependencies: uvicorn==0.24.0.post1, gunicorn==21.2.0, httpx==0.25.1

As mentioned by @Yuri Pozniak in the accepted answer's comment that downgrading to Debian bullseye worked.

I just changed the docker image from python:3.9-slim to python:3.9-slim-bullseye and everything started working fine.

Additionally, I also had to disable pip progress bar in the Dockerfile:

RUN pip config --user set global.progress_bar off

Upvotes: 6

user22724983

Reputation: 441

I had this problem when installing Flask in docker. The progress bar caused this problem, so I solved it by pip install Flask --progress-bar off

Upvotes: 44

Kryštof Řeháček

Reputation: 2483

Solution to this problem was to upgrade docker from version 18.06.1-ce to 20.10.7.

Why?

This is because the default seccomp profile of Docker 20.10.9 is not adjusted to support the clone() syscall wrapper of glibc 2.34 adopted in Ubuntu 21.10 and Fedora 35.

Source: ubuntu:21.10 and fedora:35 do not work on the latest Docker (20.10.9)

Upvotes: 59

Python in docker – RuntimeError: can&#39;t start new thread

Answers (7)

Related Questions

Python in docker – RuntimeError: can't start new thread