CODEWITHSUNDEEP
ubuntudocker-registry
Abhay Chowdary
Abhay Chowdary

Reputation: 341

net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

I am getting this problem while pushing the images to registry.

I'm using Ubuntu server inside Ubuntu docker installed, and inside the docker created docker registry container and I have an issue while while registry is integrating with gitlab.

Both gitlab and registry are on same server.

I have created a registry under same domain of gitlab.

no code

Get https://someurl:8000/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Upvotes: 33

Views: 154384

Answers (16)

Maraboc
Maraboc

Reputation: 11083

If you are behind a company’s proxy, you might experience an error in the following format:

Error response from daemon: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

You have to create a file named http-proxy.conf in the path /etc/systemd/system/docker.service.d if doesn’t exist:

sudo vi /etc/systemd/system/docker.service.d/http-proxy.conf

Then add the following content lines to the file and save it (replace [IP] and [PORT] according to your company’s proxy):

[Service]
Environment="HTTP_PROXY=http://[IP]:[PORT]/"
Environment="HTTPS_PROXY=http://[IP]:[PORT]/"
Environment="NO_PROXY=localhost,127.0.0.0/8,::1"

After saving the file, run the following two commands:

sudo systemctl daemon-reload
sudo service docker restart

Upvotes: 1

Bernd Schubert
Bernd Schubert

Reputation: 1

I didn't find this answer anywhere and I only found the reason by accident, in my case there was an invalid auth key in ${HOME}/.docker/config.json. I don't know myself what put in that key into that file.

Upvotes: 0

Rajendra
Rajendra

Reputation: 1434

In my case it was due to multiple entries of registry server in the host file on Windows machine.

I was having:

192.168.1.23 myregistry.local
10.1.25.40 myregistry.local

out of this one was correct.

But still image download used to continue for sometimes and fail after 30 to 50 % with error:

Error response from daemon: Get "https://myregistry.local:5000/v2/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

resolved by deleting the duplicate entry.

Overall the error is very generic and mostly caused due to the HTTP connection failure in docker client connecting to registry.

Upvotes: 0

Arun Solanki
Arun Solanki

Reputation: 184

Solution which I found for this issue is uninstall docker older version and reinstall

Upvotes: 0

sanket jadhav
sanket jadhav

Reputation: 61

(For private company networks only)

Got the following error intermittently:

net/http request canceled (client.timeout exceeded while awaiting headers)

because DNS was configured in the /etc/docker/daemon.json and the request was failing every time through the public IP i.e. 8.8.8.8. We added another company nameserver and the issue was resolved.

From:

{ "dns" : [ "114.114.114.114" , "8.8.8.8" ] }

To:

{ "dns" : [ "114.114.114.114" , "110.110.110.110", "8.8.8.8" ] }

Upvotes: 4

Payam Soudachi
Payam Soudachi

Reputation: 489

I was using wsl in Windows 10 and faced this issue. I know it's not reasonable but everything was ok before changing the Windows password (but I didn't restart the PC). After restarting the PC and running wsl I could pull the image without this error.

Upvotes: 0

Timofey Bugaevsky
Timofey Bugaevsky

Reputation: 184

In my case it was IPv6 that caused DNS instability on my GitLab server. Disabling IPv6 on the server helped:

cat <<EOF >> /etc/sysctl.conf
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1
EOF

Apply settings. sudo sysctl --system

Upvotes: 2

Fateh Singh
Fateh Singh

Reputation: 49

if in the case of network changes:

ERROR: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Solution:

docker network prune

Upvotes: 0

seregamorph
seregamorph

Reputation: 472

I had similar failure, but for default docker repository:

ERROR: Get "https://registry-1.docker.io/v2/": net/http: request
canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)

and

> [internal] load metadata for docker.io/library/openjdk:8-jre:
------
failed to solve: rpc error: code = Unknown desc = failed to solve
with frontend dockerfile.v0: failed to create LLB definition: failed
to do request: Head "https://registry-1.docker.io/v2/library/openjdk/manifests/8-jre":
unable to connect to 52.55.168.20:443. Do you need an HTTP proxy?

Finally I've found this record in my /etc/hosts:

52.55.168.20  registry-1.docker.io

No idea which way it was added (may be it was Docker Desktop setup; I do not remember to do it intendedly), but just deleting or commenting this line solved my problem.

Upvotes: 1

Alex Kolarski
Alex Kolarski

Reputation: 3425

On Ubuntu 20 I had to open the 5050 port like so: sudo ufw allow 5050/tcp

To see if this is your issue you can run install nmap on gitlab and check to see if port 5050 is open:

apt install nmap
nmap localhost

My results were:

root@gitlab:~# nmap localhost
Starting Nmap 7.80 ( https://nmap.org ) at 2022-07-25 09:04 UTC
Nmap scan report for localhost (127.0.0.1)
Host is up (0.0000070s latency).
Not shown: 988 closed ports
PORT     STATE SERVICE
22/tcp   open  ssh
25/tcp   open  smtp
80/tcp   open  http
443/tcp  open  https
3000/tcp open  ppp
5000/tcp open  upnp
5050/tcp open  mmcc
8080/tcp open  http-proxy
8082/tcp open  blackice-alerts
8090/tcp open  opsmessaging
9090/tcp open  zeus-admin
9100/tcp open  jetdirect

So I had port 5050 open.

Then I ran nmap from my local machine

❯ nmap gitlab.example.com
Starting Nmap 7.92 ( https://nmap.org ) at 2022-07-25 11:52 EEST
Nmap scan report for gitlab.example.com (164.92.252.91)
Host is up (0.028s latency).
Not shown: 997 filtered tcp ports (no-response)
PORT    STATE SERVICE
22/tcp  open  ssh
80/tcp  open  http
443/tcp open  https

as you can see we are missing port 5050.

Running sudo ufw allow 5050/tcp on Ubuntu 20 solved the issue for me

Upvotes: 2

Ivan dal Bosco
Ivan dal Bosco

Reputation: 3549

I was getting the same error (on Windows): enter image description here

In my case, it was an issue with the proxy. So I:

  • started px (NTLM would also make it) configured on http://localhost:3128
  • configured the proxy in Docker > Settings > Resources :

enter image description here

This solved the error:

enter image description here

Upvotes: 6

Dat TT
Dat TT

Reputation: 3073

For those who use the docker-compose, particularly on ubuntu:

docker-compose push

That means does not include SUDO. I used: sudo docker-compose push and got the error

Upvotes: 0

Sasi Kumar M
Sasi Kumar M

Reputation: 2630

On Windows OS, Running Linux Containers with WSL2, i followed the following steps,

  1. Ran the command docker logout
  2. Ran the command, docker network prune, so as to remove all the preconfigured settings of the network.
  3. From Docker Settings, Enabled the DNS server configuration with 8.8.8.8
  4. Restarted the Docker
  5. Now executed login command with registry to login, docker login {registry}

Upvotes: 0

Mohi Rostami
Mohi Rostami

Reputation: 1074

This error encountered due to the destination unreachability problem. It means that the port might be filtered by the local or external firewall or as another answer, the DNS setting of the machine might be the trouble.

Therefore, make sure you're connecting to the indeed destination. My problem was the firewall of the service provider that I got the server, after I created a ssh tunnel, I've logged in as local.

Follow the procedure to check what's the problem.

$ sudo docker login -u mrt https://gitlab.mydomain.com:5050

Error response from daemon: Get https://gitlab.mydomain.com:5050: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Check the connection:

$ nmap -p 5050 gitlab.mydomain.com

Starting Nmap 7.80 ( https://nmap.org ) at 2020-10-12 14:15 +0330
Nmap scan report for gitlab.mydomain.com (x.x.x.x)
Host is up (0.057s latency).
rDNS record for x.x.x.x: x.x.x.x

PORT     STATE     SERVICE
5050/tcp filtered  mmcc

Nmap done: 1 IP address (1 host up) scanned in 0.27 seconds

Create an ssh tunnel:

$ ssh -p 26 -L 5050:gitlab.mydomain.com:5050 [email protected]

Then:

$ sudo docker login -u mrt https://localhost:5050

WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

Upvotes: 8

Aziz
Aziz

Reputation: 1614

Deleting all other DNS server settings except 8.8.8.8 resolved the issue for me.

On mac OS:

enter image description here

Upvotes: 0

Stephane
Stephane

Reputation: 12750

I was getting a similar error message:

Get http://localhost:5000/v2/: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

when doing a docker push command on my localhost:

docker push localhost:5000/redis:3.0.7

I saw an interesting point there and tried it.

After I edited the /etc/hosts file and commented out the line:

#::1    localhost ip6-localhost ip6-loopback

the issue was gone.

Upvotes: 23

Related Questions