CODEWITHSUNDEEP
dockerdocker-registrydocker-watchtower
Dave
Dave

Reputation: 1

Pulling docker image from private registry using watchtower to multiple VPS takes too long

We have ~100 VPS machines where we use containrrr/watchtower to pull updated docker images registry.mydomain.com/image:latest). Our private registry is located on one dedicated server, it usually takes up to 2 hours to propagate the image on all machines. Image weights actually 4.42GB. I think that network is overloaded if every watchtower tries to pull the image at the same time.

What do you think about using something like Dragonfly (P2P) text there? I need an non-kubernetes solution for now. Or maybe loadbalancer/smartdns with replicated docker image.

Upvotes: 0

Views: 266

Answers (1)

Vad1mo
Vad1mo

Reputation: 5543

Adding Dragonfly or any other solution will add more problems and complexity. Besides that, you need to add more nodes anyway if your network is the bottleneck.

Doing quick math with 1 Gbit/s line or theoretically 125 MiB/s.

440 GiB/ 125 MiB/s data would be transferred in 3604.48 seconds or ~1 hour. So something between 1 Gbit/s and 800 MiB/s sounds about right to me. 500 MiB/s is a bit low IMO.

  1. Are you certain that the diff between the old and new image is 4.4 GiB, usually the last layer (where the app should be) changes more often and is smaller. So in total and compressed, you should see 1/10 data traffic.
  2. 4.4 GiB is without compression, so data in transit should be smaller.
  3. can you use s3 with your registry (e.g. minio) this would help to scale straightforward, as blobs will be served from s3 directly.
  4. Use a HA Registry across 2-3 nodes. Harbor can do that if s3 isn't an option. This way you would get 2 or 3x throughput.

Upvotes: 0

Related Questions