Docker. ERROR: Uploading artifacts to coordinator. timeout awaiting response headers

Question

I try to prepare large artifact file (about 1 GB). But I stuck with next error:

...
Stopping my_mysql ... done
$ docker run --rm --volumes-from my_mysql carinamarina/backup backup --source /var/lib/mysql/ --stdout --zip > data/db-data-volume.tar.gz
Unable to find image 'carinamarina/backup:latest' locally
latest: Pulling from carinamarina/backup
ee54741ab35b: Pulling fs layer
a548ccba3eed: Pulling fs layer
f9a81fa3bb17: Pulling fs layer
00d43d18903a: Pulling fs layer
bee2159ca4d3: Pulling fs layer
3a4a2afad82b: Pulling fs layer
bee2159ca4d3: Waiting
3a4a2afad82b: Waiting
00d43d18903a: Waiting
a548ccba3eed: Verifying Checksum
a548ccba3eed: Download complete
f9a81fa3bb17: Verifying Checksum
f9a81fa3bb17: Download complete
ee54741ab35b: Verifying Checksum
ee54741ab35b: Download complete
ee54741ab35b: Pull complete
00d43d18903a: Verifying Checksum
00d43d18903a: Download complete
bee2159ca4d3: Verifying Checksum
bee2159ca4d3: Download complete
3a4a2afad82b: Verifying Checksum
3a4a2afad82b: Download complete
a548ccba3eed: Pull complete
f9a81fa3bb17: Pull complete
00d43d18903a: Pull complete
bee2159ca4d3: Pull complete
3a4a2afad82b: Pull complete
Digest: sha256:aa7de94eb992c54024c3b5a181e8ba0fc61986f26310b723774f01ebcae6788a
Status: Downloaded newer image for carinamarina/backup:latest
Uploading artifacts...
data/db-data-volume.tar.gz: found 1 matching files 
ERROR: Uploading artifacts to coordinator... error  error=couldn't execute POST against https://gitlab.my.ru/api/v4/jobs/3764/artifacts?expire_in=1+week: Post https://gitlab.my.ru/api/v4/jobs/3764/artifacts?expire_in=1+week: net/http: timeout awaiting response headers id=3764 token=M9Xe65yo
WARNING: Retrying...                               
WARNING: Uploading artifacts to coordinator... failed  id=3764 responseStatus=504 Gateway Time-out status=504 Gateway Time-out token=M9Xe65yo
WARNING: Retrying...    

ERROR: Uploading artifacts to coordinator... error error=couldn't execute POST against https://gitlab.my.ru/api/v4/jobs/3764/artifacts?expire_in=1+week: Post https://gitlab.my.ru/api/v4/jobs/3764/artifacts?expire_in=1+week: net/http: timeout awaiting response headers id=3764 token=M9Xe65yo

My .gitlab-ci.yml:

.backup: &backup_definition
  before_script:
    - apk add --no-cache py-pip
    - pip install docker-compose
    - mkdir -p ~/.ssh
    - chmod 700 ~/.ssh
    - echo "$SSH_KNOWN_HOSTS" > ~/.ssh/known_hosts
    - chmod 644 ~/.ssh/known_hosts
    - echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
    - chmod 600 ~/.ssh/id_rsa
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN registry.my.ru
  script:
    - cp env/${ENV_NAME}.env .env
    - docker-compose up -d mysql
    - docker-compose exec -T mysql /download.sh
    - docker-compose exec -T mysql /wait-for-it.sh mysql:3306 -- echo "MySql started"
    - docker-compose exec -T mysql /restore.sh
    - docker-compose stop mysql
    - docker run --rm --volumes-from my_mysql carinamarina/backup backup --source /var/lib/mysql/ --stdout --zip > data/db-data-volume.tar.gz
  artifacts:
    paths:
      - data/db-data-volume.tar.gz
    expire_in: 1 week
  tags:
    - tests-runner
  image: docker:latest
  services:
    - name: docker:dind
  when: manual

backup-my2:
  <<: *backup_definition
  variables:
    ENV_NAME: "my2"

backup-my2-repo:
  <<: *backup_definition
  variables:
    ENV_NAME: "my2_repo"

Small artifacts are process well. I have problem just with large artifacts. How can I increase timeout?

Answer 1

If you use the golang net/http class library and http.client uses transport, please check if your transport has the ResponseHeaderTimeout parameter set. If it is set, it may be caused by this parameter. The request execution time of the http call exceeds the time set by ResponseHeaderTimeout

Solution 1. http.Client.Transport.ResponseHeaderTimeout parameter 2. Optimize back-end service processing capabilities