GitLab Omnibus Docker boot-loop (previously working) fix?

Question

So after finally getting GitLab set up and using it for the day, I decide to attempt to enable the GitLab Registry. However, after running gitlab-ctl reconfigure the system enters a boot-loop (every boot lasts for 10-15s).

Now I've been searching everywhere for ways to sort this out, end the boot-loop or get the logs that detail why the container keeps rebooting.

I'm running the Omnibus Docker image of GitLab (latest version) on a Digital Ocean droplet VPS. GitLab is configured to not use https as I use a reverse proxy (apache) to connect (this connection is over https).

Reverse Proxy configuration Browser => example.com:443 (https) => localhost:8888 (http)

Docker ps output

c13e26a20f7d        gitlab/gitlab-ce:latest       "/assets/wrapper"        24 hours ago        Up 2 seconds (health: starting)   0.0.0.0:2222->22/tcp, 0.0.0.0:8888->80/tcp, 0.0.0.0:4444->443/tcp   gitlab

What I have tried (with no success):

• Restoring a snapshot of a previously working configuration.
• Restoring the file permissions using the gitlab-ctl command.
• Find any type of hint in the available log files (only access logs.

Any suggestions and/or how I would go about fixing this issue would be greatly appreciated!

Answer 1

i had the same problem. I checked the newest logs:

sudo ls -l /srv/gitlab/logs/reconfigure/

in the file I found errors while running GitLab

sudo more /srv/gitlab/logs/reconfigure/1648382802.log

File content

[2022-03-27T12:08:10+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/M1tQ-4PY-MUmYyTdrsvgZTbPg1KGg-j4rjZCkPG0Bqk] owner changed to 0 [2022-03-27T12:08:10+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/M1tQ-4PY-MUmYyTdrsvgZTbPg1KGg-j4rjZCkPG0Bqk] group changed to 0 [2022-03-27T12:08:10+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/M1tQ-4PY-MUmYyTdrsvgZTbPg1KGg-j4rjZCkPG0Bqk] mode changed to 644 [2022-03-27T12:08:12+00:00] INFO: file[/var/opt/gitlab/nginx/www/.well-known/acme-challenge/M1tQ-4PY-MUmYyTdrsvgZTbPg1KGg-j4rjZCkPG0Bqk] deleted file at /var/opt/gitla b/nginx/www/.well-known/acme-challenge/M1tQ-4PY-MUmYyTdrsvgZTbPg1KGg-j4rjZCkPG0Bqk [2022-03-27T12:08:12+00:00] INFO: Running queued delayed notifications before re-raising exception [2022-03-27T12:08:12+00:00] INFO: Running queued delayed notifications before re-raising exception [2022-03-27T12:08:12+00:00] INFO: Running queued delayed notifications before re-raising exception [2022-03-27T12:08:12+00:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending run action to execute[clear the gitlab-rails cache] ( delayed) [2022-03-27T12:08:56+00:00] INFO: execute[clear the gitlab-rails cache] ran successfully [2022-03-27T12:08:56+00:00] INFO: env_dir[/opt/gitlab/etc/gitlab-workhorse/env] sending restart action to runit_service[gitlab-workhorse] (delayed) [2022-03-27T12:08:56+00:00] INFO: env_dir[/opt/gitlab/etc/registry/env] sending restart action to runit_service[registry] (delayed) [2022-03-27T12:08:57+00:00] ERROR: Running exception handlers [2022-03-27T12:08:57+00:00] ERROR: Exception handlers complete [2022-03-27T12:08:57+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out [2022-03-27T12:08:57+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report [2022-03-27T12:08:57+00:00] FATAL: RuntimeError: letsencrypt_certificate[example.com] (letsencrypt::http_authorization line 6) had an error: RuntimeError: acm e_certificate[staging] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/letsencrypt/resources/certificate.rb line 41) had an error: RuntimeError: ruby_block[create cert ificate for example.com] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/acme/resources/certificate.rb line 108) had an error: RuntimeError: [gitlab.example.com] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/2021548548/xDXgmw, status: invalid, e rror: {"type"=>"urn:ietf:params:acme:error:unauthorized", "detail"=>"Invalid response from http://gitlab.example.com/.well-known/acme-challenge/M1tQ-4PY-MUmYyTdrsvgZ TbPg1KGg-j4rjZCkPG0Bqk [IP]: 404", "status"=>403}} ]

In my case, it was a wrong configuration in LetsEncrypt. I corrected the GitLab configuration file:

sudo vi /srv/gitlab/config/gitlab.rb

Answer 2

Fix: Shutting down the Docker Container, inspecting the reconfiguration logs (located in the logs/reconfiguration directory inside the installation folder) and making adjustments to the configuration file based on the errors apparent in the log.

Fixing the boot-loop (step by step)

NOTE: If you are not running as root, I recommend temporarily running as root (using sudo su) to avoid Permission denied errors.

After manually stopping the Docker Container I was able to view the full log located at /srv/gitlab/logs/reconfigure (with the latest timestamp)

Excerpt from the latest reconfiguration log:

[2018-05-19T00:24:50+00:00] INFO: *** Chef 13.6.4 ***
[2018-05-19T00:24:50+00:00] INFO: Platform: x86_64-linux
[2018-05-19T00:24:50+00:00] INFO: Chef-client pid: 25
[2018-05-19T00:24:50+00:00] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...
[2018-05-19T00:24:52+00:00] WARN: Plugin Network: unable to detect ipaddress
[2018-05-19T00:24:52+00:00] INFO: Setting the run_list to ["recipe[gitlab]"] from CLI options
[2018-05-19T00:24:52+00:00] INFO: Run List is [recipe[gitlab]]
[2018-05-19T00:24:52+00:00] INFO: Run List expands to [gitlab]
[2018-05-19T00:24:52+00:00] INFO: Starting Chef Run for [REDACTED]
[2018-05-19T00:24:52+00:00] INFO: Running start handlers
[2018-05-19T00:24:52+00:00] INFO: Start handlers complete.
[2018-05-19T00:24:54+00:00] INFO: Loading cookbooks [gitlab@0.0.1, package@0.1.0, postgresql@0.1.0, registry@0.1.0, mattermost@0.1.0, consul@0.0.0, gitaly@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, runit@0.14.2, acme@3.1.0, crond@0.1.0, compat_resource@12.19.0]
[2018-05-19T00:24:56+00:00] WARN: Runtime directory '/run' is not a tmpfs.

Error messages are a good start, but won't reveal the underlying issue...

[2018-05-19T00:24:56+00:00] ERROR: Running exception handlers
[2018-05-19T00:24:56+00:00] ERROR: Exception handlers complete

Now the last 3 lines of the log revealed the problem

[2018-05-19T00:24:56+00:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/chef-stacktrace.out
[2018-05-19T00:24:56+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2018-05-19T00:24:56+00:00] FATAL: RuntimeError: Unsupported GitLab Registry external URL path: /gitlab/registry

Now after seeing the very last line, I see that my configuration is invalid and is the cause of the fatal issue.

To fix the issue, I edited the configuration while the docker container was in an unbooted state (configuration file located at /srv/gitlab/config/gitlab.rb). After fixing the configuration, which in my case was to comment out all GitLab Registry configuration options (since I decided to wait with testing this feature for now).

Diff for configuration

################################################################################
## Container Registry settings
##! Docs: https://docs.gitlab.com/ce/administration/container_registry.html
################################################################################

- registry_external_url 'http://[REDACTED]:4567/gitlab/registry'
+ # registry_external_url 'http://[REDACTED]:4567/gitlab/registry'

### Settings used by GitLab application
- gitlab_rails['registry_enabled'] = true
+ # gitlab_rails['registry_enabled'] = true
- gitlab_rails['registry_host'] = "[REDACTED]"
+ # gitlab_rails['registry_host'] = "[REDACTED]"
- gitlab_rails['registry_port'] = "4567"
+ # gitlab_rails['registry_port'] = "4567"
- gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"
+ # gitlab_rails['registry_path'] = "/var/opt/gitlab/gitlab-rails/shared/registry"

I ran the following commands to boot the container and apply the new configuration.

docker start gitlab
sudo docker exec gitlab gitlab-ctl reconfigure

Promising results right away as the Container did not reboot right after I issued the command.

Starting Chef Client, version 13.6.4
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
  - gitlab (0.0.1)
  - package (0.1.0)
  - registry (0.1.0)
  - postgresql (0.1.0)
  - letsencrypt (0.1.0)
  - mattermost (0.1.0)
  - runit (0.14.2)
  - nginx (0.1.0)
  - gitaly (0.1.0)
  - consul (0.0.0)
  - acme (3.1.0)
  - crond (0.1.0)
  - compat_resource (12.19.0)
Installing Cookbook Gems:
...

That's it! Now everything works as it should. Turned out user error was the cause of the issue.