CODEWITHSUNDEEP
dockerdocker-registrydockerhub
luqo33
luqo33

Reputation: 8361

Docker push - net/http: TLS handshake timeout

I've deployed a private docker image registry on an AWS EC2 Ubuntu 14.04 instance. The registry is secured using Let's Encrypt certificate.

Unfortunately, I'm getting net/http: TLS handshake timeout for docker push operations that take longer than 300s:

This is the output of the time'd command:

[luqo33@home-pc containers]$ time docker push <my-registry-domain:5000>/nginx                                                    
The push refers to a repository [<my-registry-domain:5000>/nginx]
dda5a806f0b0: Layer already exists
ec35cfccb7f7: Layer already exists
94c1a232bb3f: Layer already exists
6d6b9812c8ae: Layer already exists
695da0025de6: Retrying in 1 second
fe4c16cbf7a4: Pushing [================================================>  ]   119 MB/123 MB
net/http: TLS handshake timeout

real    5m0.847s
user    0m0.097s
sys     0m0.017s

Logs of the regsitry:2 container do not show any errors - other than the notification that there was an unexpected EOF while receiving data. I can also push images that take less than 5min to push without problems.

I'm suspecting that it's a system setting at blame as the timeout happens always once the operations goes beyond 300 seconds. There isn't any load balancer or other proxy. <my-registry-domain:5000> points directly at the server IP.

How can I further investigate and possible remedy this situation?

EDIT

The same happens when I push images to other server providers (DigitalOcean), AWS ECS registry or even Docker Hub! I find it hard to believe that the Docker client would have a built-in handshake timeout of 300s.

I'm thinking that perhaps I should start look for the solution at the network level - with my hardware (wi-fi router) or my ISP.

Anybody has a clue what is happening here?

Upvotes: 15

Views: 28565

Answers (5)

DroidT
DroidT

Reputation: 3293

I got the same error because I still had the containers running. After I shut them down with Ctrl + C in the active terminal session and docker compose down everything worked as expected.

Upvotes: 0

Archimedes Trajano
Archimedes Trajano

Reputation: 41290

Just adding a separate answer for those who may be dealing with it on a managed build environment like Azure Devops based on https://github.com/actions/virtual-environments/issues/2152#issuecomment-736325518

Change the setting as follows:

sudo sed -i 's/ }/, \"max-concurrent-uploads\": 1 }/' /etc/docker/daemon.json
sudo systemctl restart docker

Upvotes: 2

James Shapiro
James Shapiro

Reputation: 6226

This is a primitive solution, but one thing that worked for me was commenting out some of the later layers and then uncommenting / pushing new images with one layer added at a time until the whole image succeeded.

Upvotes: 0

user35-02900
user35-02900

Reputation: 71

Disable VPN for any AWS services, including ECS. I had it on auto-enable, and received inside ecs-cli (docker compose to ecs):

ERRO[0042] Error describing service                      error="RequestError: send request failed\ncaused by: Post https://ecs.us-east-1.amazonaws.com/: net/http: TLS handshake timeout" service=1routing
FATA[0042] RequestError: send request failed
caused by: Post https://ecs.us-east-1.amazonaws.com/: net/http: TLS handshake timeout

check your internet connection/routing.

Upvotes: 0

Fetrarij
Fetrarij

Reputation: 7326

I got the same issue, this issue is may be from your internet connection, I solved it by decrementing the concurrency uploads (downloads for get) to 1 in dockerd. By using these args:

--max-concurrent-downloads  (default: 3)    Set the max concurrent downloads for each pull

--max-concurrent-uploads    (default: 5)    Set the max concurrent uploads for each push

uploading 5 images in same time may result a timeout if you have a low bandwidth.

https://docs.docker.com/engine/reference/commandline/dockerd/

Upvotes: 13

Related Questions