Gitlab-runner failed to remove permission denied

Question

I'm setting up a CI/CD pipeline with Gitlab. I've installed gitlab-runner on a Digital Ocean Ubuntu 18.04 droplet and gave permissions in /etc/sudoers to the gitlab-runner as:

gitlab-runner ALL=(ALL:ALL)ALL

The first commit to the associated repository correctly build the docker-compose (the app itself is Django+postgres), but following commits are not able to clean previous builds and fail:

Running with gitlab-runner 12.8.0 (1b659122)
on ubuntu-s-4vcpu-8gb-fra1-01 52WypZsE
Using Shell executor...
00:00
Running on ubuntu-s-4vcpu-8gb-fra1-01...
00:00
Fetching changes with git depth set to 50...
00:01
 Reinitialized existing Git repository in /home/gitlab-runner/builds/52WypZsE/0/lorePieri/djangocicd/.git/
 From https://gitlab.com/lorePieri/djangocicd
  * [new ref]         refs/pipelines/120533457 -> refs/pipelines/120533457
    0072002..bd28ba4  develop                  -> origin/develop
 Checking out bd28ba46 as develop...
 warning: failed to remove app/staticfiles/admin/img/selector-icons.svg: Permission denied
 warning: failed to remove app/staticfiles/admin/img/search.svg: Permission denied
 warning: failed to remove app/staticfiles/admin/img/icon-alert.svg: Permission denied
 warning: failed to remove app/staticfiles/admin/img/tooltag-arrowright.svg: Permission denied
 warning: failed to remove app/staticfiles/admin/img/icon-unknown-alt.svg: Permission denied

This is the relevant portion of the .gitlab-ci.yml file:

image: docker:latest
services:
  - docker:dind

stages:
  - test
  - deploy_staging
  - deploy_production

step-test:
  stage: test
  before_script:
    - export DYNAMIC_ENV_VAR=DEVELOP
  only:
    - develop
  tags:
    - develop
  script:
    - echo running tests in $DYNAMIC_ENV_VAR
    - sudo apt-get install -y python-pip
    - sudo pip install docker-compose
    - sudo docker image prune -f
    - sudo docker-compose -f docker-compose.yml build --no-cache
    - sudo docker-compose -f docker-compose.yml up -d
    - echo do tests now
    - sudo docker-compose exec -T web python3 -m coverage run --source='.' manage.py test

...

What I've tried:

usermod -aG docker gitlab-runner
sudo service docker restart

Answer 1

gitlab-runner error:

Running with gitlab-runner 14.10.1 (f761588f)
  on XXX Core Edition i8pfD5N9
Preparing the "shell" executor
00:00
Using Shell executor...
Preparing environment
00:00
Running on devserver...
Getting source from Git repository
00:00
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /home/gitlab-runner/builds/i8pfD5N9/0/xxx/xxx/.git/
Checking out 897db0d2 as dev...
warning: failed to remove xxx/bin/Release/net6.0/xxx.dll: Permission denied

It is obvious that there is a permission issue. I checked the permissions of xxx/bin:

ls -l /home/gitlab-runner/builds/i8pfD5N9/0/xxx/xxx/bin

The result is:

total 4
drwxr-xr-x 3 root root 4096 Sep 15 09:12 Release

Normally, the user here should be gitlab-runner, not root.

I realized that this might be caused by my use of sudo, so I removed all the sudo in the script.

Then I changed the user of runner i8pfD5N9 to root to ensure the highest user privileges. Later I realized that this step might not be necessary.

nano /etc/gitlab-runner/config.toml

[[runners]]
  name = "xxx"
  url = "http://192.168.0.2/"
  token = "i8pfD5N9SNp-qQ8oRUN4"
  executor = "shell"
  user = "root"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]

After making the modifications, another error occurred:

Reinitialized existing Git repository in /home/gitlab-runner/builds/i8pfD5N9/0/xxx/xxx/.git/
fatal: detected dubious ownership in repository at '/home/gitlab-runner/builds/i8pfD5N9/0/xxx/xxx'
To add an exception for this directory, call:
    git config --global --add safe.directory /home/gitlab-runner/builds/i8pfD5N9/0/xxx/xxx
ERROR: Job failed: exit status 1

I followed the prompt and executed:

git config --global --add safe.directory /home/gitlab-runner/builds/i8pfD5N9/0/xxx/xxx

Finally, the problem was resolved!

Answer 2

Ran into the same issue while running a robotframework docker job, that should write its output back the runner's filesystem. The problem was that they were owned by root and thus couldn't be updated by subsequent processes anymore.

The answer is given here

Just add -u ${UID} to your docker/docker-compose run command and the output files are owned by the current user running the docker job.

Answer 3

I have had the exact same problem. Therefore I will explain how it was resolved in details.

Try finding your config.toml file and run the gitlab-runner command with root privileges, since permission denied is a very common UNIX-based operating systems error.

After finding the location of config.toml pass it:

sudo gitlab-runner run --config <absolute_location_of_config_toml>

P.S. You can find all config.toml file easily using locate config.toml command. Make sure you have already installed by executing sudo apt-get install mlocate

1. After facing to permission denied error, I have tried using sudo gitlab-runner run instead of gitlab-runner, but it has its own problem:

   ERROR: Failed to load config stat /etc/gitlab-runner/config.toml: no such 
   file or directory  builds=0
   

   while executing gitlab-runner without root permissions doesn't have any config file problem.

2. Try implementing the ways and solutions as @Grumbanks and @vlad-Mazurkov mentioned. But they didn't work properly.

Answer 4

The best solution for me was adding

pre_clone_script = "sudo chown -R gitlab-runner:gitlab-runner ."

into /etc/gitlab-runner/config.toml Even if you won't have permissions after a previous job it'll set correct permissions before cleaning up the workdir and cloning the repo.

Answer 5

It MAY be because you write a file in cloned out codebase. What I do is simply create another directory outside of gitlab-runner directory:

WORKSPACE_DIR="/home/abcd_USER/a/b"
rm -rf $WORKSPACE_DIR
mkdir -p $WORKSPACE_DIR
cd $WORKSPACE_DIR
ls -la
git clone ..................
AND DO whatever 

I never faced the issue again.

Answer 6

I would recommend setting a GIT_STRATEGY to none in the afflicted job.