Docker: runtime/cgo: pthread_create failed: Resource temporarily unavailable

Question

I wrote a docker-compose.yml which is working on my local ubuntu 18.04 system. I ran the same yml on my ubuntu server 18.04 lts machine and got the errormessage in the title. I tried to increase TasksMax for the process but it didnt work. I also set DefaultTasksMax=infinity in /etc/systemd/system.conf.

Some outputs:

systemctl show -p TasksMax docker:

TasksMax=infinity

systemctl status docker | grep Tasks:

└─50-TasksMax.conf  (This line is not showing up on my local 
                     machine)
Tasks: 53

This is leading me to the point that i still havent made the configs correct.

Here is the Docker output when excecuting docker-compose up -d:

ERROR: for wordpress-cert  Cannot start service wordpress-cert: b'OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \\"process_linux.go:413: running prestart hook 0 caused \\\\\\"error running hook: exit status 2, stdout: , stderr: runtime/cgo: pthread_create failed: Resource temporarily unavailable\\\\\\\\nSIGABRT: abort\\\\\\\\nPC=0x7f5cd5db7e97 m=0 sigcode=18446744073709551610\\\\\\\\n\\\\\\\\ngoroutine 0 [idle]:\\\\\\\\nruntime: unknown pc 0x7f5cd5db7e97\\\\\\\\nstack: frame={sp:0x7ffe59e1f6c0, fp:0x0} stack=[0x7ffe59620d50,0x7ffe59e1fd80)\\\\\\\\n00007ffe59e1f5c0:  [...] ffffffffffffffff  ffffffffffffffff \\\\\\\\nruntime: unknown pc 0x7f5cd5db7e97\\\\\\\\nstack: frame={sp:0x7ffe59e1f6c0, fp:0x0} stack=[0x7ffe59620d50,0x7ffe59e1fd80)\\\\\\\\n00007ffe59e1f5c0:  [...] ffffffffffffffff  ffffffffffffffff \\\\\\\\n\\\\\\\\ngoroutine 1 [runnable, locked to thread]:\\\\\\\\ngithub.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Register(0xc0005e3b00, 0x55922583c640, 0xc000760b90, 0x0, 0x0)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus/registry.go:298 +0x1e6\\\\\\\\ngithub.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc0005e3b00, 0xc00075d000, 0x1, 0x1)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus/registry.go:403 +0x78\\\\\\\\ngithub.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus.MustRegister(...)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus/registry.go:152\\\\\\\\ngithub.com/docker/docker/vendor/github.com/docker/go-metrics.Register(...)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/docker/go-metrics/register.go:8\\\\\\\\ngithub.com/docker/docker/builder/dockerfile.init.0()\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/builder/dockerfile/metrics.go:43 +0x2d3\\\\\\\\n\\\\\\\\ngoroutine 20 [syscall]:\\\\\\\\nos/signal.signal_recv(0x559222e8e556)\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/sigqueue.go:139 +0x9e\\\\\\\\nos/signal.loop()\\\\\\\\n\\\\\\\\t/usr/local/go/src/os/signal/signal_unix.go:23 +0x24\\\\\\\\ncreated by os/signal.init.0\\\\\\\\n\\\\\\\\t/usr/local/go/src/os/signal/signal_unix.go:29 +0x43\\\\\\\\n\\\\\\\\nrax    0x0\\\\\\\\nrbx    0x7f5cd6165840\\\\\\\\nrcx    0xffffffffffffffff\\\\\\\\nrdx    0x0\\\\\\\\nrdi    0x2\\\\\\\\nrsi    0x7ffe59e1f6c0\\\\\\\\nrbp    0x559224b7603d\\\\\\\\nrsp    0x7ffe59e1f6c0\\\\\\\\nr8     0x0\\\\\\\\nr9     0x7ffe59e1f6c0\\\\\\\\nr10    0x8\\\\\\\\nr11    0x246\\\\\\\\nr12    0x5592271bd1f0\\\\\\\\nr13    0x11\\\\\\\\nr14    0x559224b1c640\\\\\\\\nr15    0x0\\\\\\\\nrip    0x7f5cd5db7Creating nextcloud-cert             ... error
unknown'

ERROR: for nextcloud-cert  Cannot start service nextcloud-cert: b'OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \\"process_linux.go:413: running prestart hook 0 caused \\\\\\"error running hook: exit status 2, stdout: , stderr: runtime/cgo: pthread_create failed: Resource temporarily unavailable\\\\\\\\nSIGABRT: abort\\\\\\\\nPC=0x7f2376cbce97 m=0 sigcode=18446744073709551610\\\\\\\\n\\\\\\\\ngoroutine 0 [idle]:\\\\\\\\nruntime: unknown pc 0x7f2376cbce97\\\\\\\\nstack: frame={sp:0x7fff898f4ad0, fp:0x0} stack=[0x7fff890f6070,0x7fff898f50a0)\\\\\\\\n00007fff898f49d0:  [...] 00007fff898f4b40  0000564eca7254a3 <runtime.persistentalloc1+515> [...] \\\\\\\\nruntime: unknown pc 0x7f2376cbce97\\\\\\\\nstack: frame={sp:0x7fff898f4ad0, fp:0x0} stack=[0x7fff890f6070,0x7fff898f50a0)\\\\\\\\n00007fff898f49d0:  [...] \\\\\\\\n\\\\\\\\ngoroutine 1 [running]:\\\\\\\\nruntime.systemstack_switch()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc00006e798 sp=0xc00006e790 pc=0x564eca774450\\\\\\\\nruntime.main()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/proc.go:130 +0x67 fp=0xc00006e7e0 sp=0xc00006e798 pc=0x564eca747cd7\\\\\\\\nruntime.goexit()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00006e7e8 sp=0xc00006e7e0 pc=0x564eca776551\\\\\\\\n\\\\\\\\nrax    0x0\\\\\\\\nrbx    0x7f237706a840\\\\\\\\nrcx    0xffffffffffffffff\\\Creating wordpress-it-mueller      ... error
rsp    0x7fff898f4ad0\\\\\\\\nr8     0x0\\\\\\\\nr9     0x7fff898f4ad0\\\\\\\\nr10    0x8\\\\\\\\nr11    0x246\\\\\\\\nr12    0x564ed04ff990\\\\\\\\nr13    0x11\\\\\\\\nr14    0x564ecc404640\\\\\\\\nr15    0x0\\\\\\\\nrip    0x7f2376cbce97\\\\\\\\nrflags 0x246\\\\\\\\ncs     0x33\\\\\\\\nfs     0x0\\\\\\\\ngs  Creating wordpress-cert-it-mueller ... done

ERROR: for wordpress-it-mueller  Cannot start service wordpress-it-mueller: b'OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \\"process_linux.go:413: running prestart hook 0 caused \\\\\\"error running hook: exit status 2, stdout: , stderr: runtime/cgo: pthread_create failed: Resource temporarily unavailable\\\\\\\\nSIGABRT: abort\\\\\\\\nPC=0x7f3d844b2e97 m=0 sigcode=18446744073709551610\\\\\\\\n\\\\\\\\ngoroutine 0 [idle]:\\\\\\\\nruntime: unknown pc 0x7f3d844b2e97\\\\\\\\nstack: frame={sp:0x7ffeff965720, fp:0x0} stack=[0x7ffeff166cc0,0x7ffeff965cf0)\\\\\\\\n00007ffeff965620:  [...] ffffffffffffffff  ffffffffffffffff \\\\\\\\nruntime: unknown pc 0x7f3d844b2e97\\\\\\\\nstack: frame={sp:0x7ffeff965720, fp:0x0} stack=[0x7ffeff166cc0,0x7ffeff965cf0)\\\\\\\\n00007ffeff965620:   [...] ffffffffffffffff  ffffffffffffffff \\\\\\\\n\\\\\\\\ngoroutine 1 [running]:\\\\\\\\nruntime.systemstack_switch()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc00006e798 sp=0xc00006e790 pc=0x55a7e9792450\\\\\\\\nruntime.main()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/proc.go:130 +0x67 fp=0xc00006e7e0 sp=0xc00006e798 pc=0x55a7e9765cd7\\\\\\\\nruntime.goexit()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00006e7e8 sp=0xc00006e7e0 pc=0x55a7e9794551\\\\\\\\n\\\\\\\\nrax    0x0\\\\\\\\nrbx    0x7f3d84860840\\\\\\\\nrcx    0xffffffffffffffff\\\\\\\\nrdx    0x0\\\\\\\\nrdi    0x2\\\\\\\\nrsi    0x7ffeff965720\\\\\\\\nrbp    0x55a7eb47c03d\\\\\\\\nrsp    0x7ffeff965720\\\\\\\\nr8     0x0\\\\\\\\Creating nextcloud                  ... done
    0x11\\\\\\\\nr14    0x55a7eb422640\\\\\\\\nr15    0x0\\\\\\\\nrip    0x7f3d844b2e97\\\\\\\\nrflags 0x246\\\\\\\\ncs     0x33\\\\\\\\nfs     0x0\\\\\\\\ngs     0x0\\\\\\\\n\\\\\\"\\"": unknown'

ERROR: for wordpress-cert  Cannot start service wordpress-cert: b'OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \\"process_linux.go:413: running prestart hook 0 caused \\\\\\"error running hook: exit status 2, stdout: , stderr: runtime/cgo: pthread_create failed: Resource temporarily unavailable\\\\\\\\nSIGABRT: abort\\\\\\\\nPC=0x7f5cd5db7e97 m=0 sigcode=18446744073709551610\\\\\\\\n\\\\\\\\ngoroutine 0 [idle]:\\\\\\\\nruntime: unknown pc 0x7f5cd5db7e97\\\\\\\\nstack: frame={sp:0x7ffe59e1f6c0, fp:0x0} stack=[0x7ffe59620d50,0x7ffe59e1fd80)\\\\\\\\n00007ffe59e1f5c0:  [...] ffffffffffffffff  ffffffffffffffff \\\\\\\\nruntime: unknown pc 0x7f5cd5db7e97\\\\\\\\nstack: frame={sp:0x7ffe59e1f6c0, fp:0x0} stack=[0x7ffe59620d50,0x7ffe59e1fd80)\\\\\\\\n00007ffe59e1f5c0:  [...]ffffffffffffffff  ffffffffffffffff \\\\\\\\n\\\\\\\\ngoroutine 1 [runnable, locked to thread]:\\\\\\\\ngithub.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).Register(0xc0005e3b00, 0x55922583c640, 0xc000760b90, 0x0, 0x0)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus/registry.go:298 +0x1e6\\\\\\\\ngithub.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus.(*Registry).MustRegister(0xc0005e3b00, 0xc00075d000, 0x1, 0x1)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus/registry.go:403 +0x78\\\\\\\\ngithub.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus.MustRegister(...)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/prometheus/client_golang/prometheus/registry.go:152\\\\\\\\ngithub.com/docker/docker/vendor/github.com/docker/go-metrics.Register(...)\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/vendor/github.com/docker/go-metrics/register.go:8\\\\\\\\ngithub.com/docker/docker/builder/dockerfile.init.0()\\\\\\\\n\\\\\\\\t/go/src/github.com/docker/docker/builder/dockerfile/metrics.go:43 +0x2d3\\\\\\\\n\\\\\\\\ngoroutine 20 [syscall]:\\\\\\\\nos/signal.signal_recv(0x559222e8e556)\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/sigqueue.go:139 +0x9e\\\\\\\\nos/signal.loop()\\\\\\\\n\\\\\\\\t/usr/local/go/src/os/signal/signal_unix.go:23 +0x24\\\\\\\\ncreated by os/signal.init.0\\\\\\\\n\\\\\\\\t/usr/local/go/src/os/signal/signal_unix.go:29 +0x43\\\\\\\\n\\\\\\\\nrax    0x0\\\\\\\\nrbx    0x7f5cd6165840\\\\\\\\nrcx    0xffffffffffffffff\\\\\\\\nrdx    0x0\\\\\\\\nrdi    0x2\\\\\\\\nrsi    0x7ffe59e1f6c0\\\\\\\\nrbp    0x559224b7603d\\\\\\\\nrsp    0x7ffe59e1f6c0\\\\\\\\nr8     0x0\\\\\\\\nr9     0x7ffe59e1f6c0\\\\\\\\nr10    0x8\\\\\\\\nr11    0x246\\\\\\\\nr12    0x5592271bd1f0\\\\\\\\nr13    0x11\\\\\\\\nr14    0x559224b1c640\\\\\\\\nr15    0x0\\\\\\\\nrip    0x7f5cd5db7e97\\\\\\\\nrflags 0x246\\\\\\\\ncs     0x33\\\\\\\\nfs     0x0\\\\\\\\ngs     0x0\\\\\\\\n\\\\\\"\\"": unknown'

ERROR: for nextcloud-cert  Cannot start service nextcloud-cert: b'OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \\"process_linux.go:413: running prestart hook 0 caused \\\\\\"error running hook: exit status 2, stdout: , stderr: runtime/cgo: pthread_create failed: Resource temporarily unavailable\\\\\\\\nSIGABRT: abort\\\\\\\\nPC=0x7f2376cbce97 m=0 sigcode=18446744073709551610\\\\\\\\n\\\\\\\\ngoroutine 0 [idle]:\\\\\\\\nruntime: unknown pc 0x7f2376cbce97\\\\\\\\nstack: frame={sp:0x7fff898f4ad0, fp:0x0} stack=[0x7fff890f6070,0x7fff898f50a0)\\\\\\\\n00007fff898f49d0:  [...]00007fff898f4ae8  00000000ffffffff \\\\\\\\n00007fff898f4af0:  00007fff898f4b40  0000564eca7254a3 <runtime.persistentalloc1+515> \\\\\\\\n00007fff898f4b00:  0000564ece51d0f0  0000000000000000 [...] \\\\\\\\nruntime: unknown pc 0x7f2376cbce97\\\\\\\\nstack: frame={sp:0x7fff898f4ad0, fp:0x0} stack=[0x7fff890f6070,0x7fff898f50a0)\\\\\\\\n00007fff898f49d0:  [...] 00007fff898f4ae8  00000000ffffffff \\\\\\\\n00007fff898f4af0:  00007fff898f4b40  0000564eca7254a3 <runtime.persistentalloc1+515> \\\\\\\\n00007fff898f4b00:  0000564ece51d0f0  0000000000000000 [...] \\\\\\\\n\\\\\\\\ngoroutine 1 [running]:\\\\\\\\nruntime.systemstack_switch()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc00006e798 sp=0xc00006e790 pc=0x564eca774450\\\\\\\\nruntime.main()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/proc.go:130 +0x67 fp=0xc00006e7e0 sp=0xc00006e798 pc=0x564eca747cd7\\\\\\\\nruntime.goexit()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00006e7e8 sp=0xc00006e7e0 pc=0x564eca776551\\\\\\\\n\\\\\\\\nrax    0x0\\\\\\\\nrbx    0x7f237706a840\\\\\\\\nrcx    0xffffffffffffffff\\\\\\\\nrdx    0x0\\\\\\\\nrdi    0x2\\\\\\\\nrsi    0x7fff898f4ad0\\\\\\\\nrbp    0x564ecc45e03d\\\\\\\\nrsp    0x7fff898f4ad0\\\\\\\\nr8     [...]  0x0\\\\\\\\ngs     0x0\\\\\\\\n\\\\\\"\\"": unknown'

ERROR: for wordpress-it-mueller  Cannot start service wordpress-it-mueller: b'OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \\"process_linux.go:413: running prestart hook 0 caused \\\\\\"error running hook: exit status 2, stdout: , stderr: runtime/cgo: pthread_create failed: Resource temporarily unavailable\\\\\\\\nSIGABRT: abort\\\\\\\\nPC=0x7f3d844b2e97 m=0 sigcode=18446744073709551610\\\\\\\\n\\\\\\\\ngoroutine 0 [idle]:\\\\\\\\nruntime: unknown pc 0x7f3d844b2e97\\\\\\\\nstack: frame={sp:0x7ffeff965720, fp:0x0} stack=[0x7ffeff166cc0,0x7ffeff965cf0)\\\\\\\\n00007ffeff965620:[...] ffffffffffffffff  ffffffffffffffff \\\\\\\\nruntime: unknown pc 0x7f3d844b2e97\\\\\\\\nstack: frame={sp:0x7ffeff965720, fp:0x0} stack=[0x7ffeff166cc0,0x7ffeff965cf0)\\\\\\\\n00007ffeff965620:  [...] ffffffffffffffff  ffffffffffffffff \\\\\\\\n\\\\\\\\ngoroutine 1 [running]:\\\\\\\\nruntime.systemstack_switch()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/asm_amd64.s:311 fp=0xc00006e798 sp=0xc00006e790 pc=0x55a7e9792450\\\\\\\\nruntime.main()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/proc.go:130 +0x67 fp=0xc00006e7e0 sp=0xc00006e798 pc=0x55a7e9765cd7\\\\\\\\nruntime.goexit()\\\\\\\\n\\\\\\\\t/usr/local/go/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00006e7e8 sp=0xc00006e7e0 pc=0x55a7e9794551\\\\\\\\n\\\\\\\\nrax    0x0\\\\\\\\nrbx    0x7f3d84860840\\\\\\\\nrcx    0xffffffffffffffff\\\\\\\\nrdx    0x0\\\\\\\\nrdi    0x2\\\\\\\\nrsi    0x7ffeff965720\\\\\\\\nrbp    0x55a7eb47c03d\\\\\\\\nrsp    0x7ffeff965720\\\\\\\\nr8     0x0\\\\\\\\nr9     0x7ffeff965720\\\\\\\\nr10    0x8\\\\\\\\nr11    0x246\\\\\\\\nr12    0x55a7ee582990\\\\\\\\nr13    0x11\\\\\\\\nr14    0x55a7eb422640\\\\\\\\nr15    0x0\\\\\\\\nrip    0x7f3d844b2e97\\\\\\\\nrflags 0x246\\\\\\\\ncs     0x33\\\\\\\\nfs     0x0\\\\\\\\ngs     0x0\\\\\\\\n\\\\\\"\\"": unknown'
ERROR: Encountered errors while bringing up the project.

Any ideas what i might doing wrong are very appriciated!

Answer 1

I have the same issue with grafana-agent and I've added security_opt option to fix it:

version: "3"
services:
  agent:
    image: grafana/agent:latest
    security_opt:
      - seccomp:unconfined
    volumes:
      - ./agent/config:/etc/agent-config
    entrypoint:
      - /bin/agent
      - -config.file=/etc/agent-config/agent.yaml
      - -metrics.wal-directory=/tmp/agent/wal
      - -enable-features=integrations-next
      - -config.expand-env
      - -config.enable-read-api

    ports:
      - "12345:12345"

Answer 2

I also had this problem on a v-server from strato. It seems like they removed the hard limit and allow changing the task limit now: https://www.strato.de/faq/server/prozesse-vs-threads-bei-linux-v-servern/

To solve the error I just added the following line to /etc/systemd/system.conf and restarted

DefaultTasksMax=1000

Answer 3

The problem was that the server provider Strato has a hard process limit for their virtual servers. I configured it correctly, but I did not expect such a restriction. It cost me so much time... You should not go cheap with your provider, I have learned my lesson.

Answer 4

One solution:

Comment in and modify:
/etc/systemd/logind.conf UserTasksMax

then:

sudo systemctl daemon-reload
sudo systemctl daemon-reexec
sudo service systemd-logind restart