gitlab-ce docker - "PG::ConnectionBad: could not connect to server: No such file or directory"

Question

I've been using gitlab-ce for some time and now after updating it by running docker-compose pull it fails to start with this error.

git_1          | Recipe: gitlab::database_migrations
git_1          |   * bash[migrate gitlab-rails database] action run
git_1          |     [execute] rake aborted!
git_1          |               PG::ConnectionBad: could not connect to server: No such file or directory
git_1          |                Is the server running locally and accepting
git_1          |                connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
git_1          |               /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:48:in `block (3 levels) in <top (required)>'
git_1          |               /opt/gitlab/embedded/bin/bundle:23:in `load'
git_1          |               /opt/gitlab/embedded/bin/bundle:23:in `<main>'
git_1          |               Tasks: TOP => gitlab:db:configure
git_1          |               (See full trace by running task with --trace)
git_1          |     
git_1          |     ================================================================================
git_1          |     Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
git_1          |     ================================================================================
git_1          |     
git_1          |     Mixlib::ShellOut::ShellCommandFailed
git_1          |     ------------------------------------
git_1          |     Expected process to exit with [0], but received '1'
git_1          |     ---- Begin output of "bash"  "/tmp/chef-script20200629-25-idzu10" ----
git_1          |     STDOUT: rake aborted!
git_1          |     PG::ConnectionBad: could not connect to server: No such file or directory
git_1          |        Is the server running locally and accepting
git_1          |        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
git_1          |     /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:48:in `block (3 levels) in <top (required)>'
git_1          |     /opt/gitlab/embedded/bin/bundle:23:in `load'
git_1          |     /opt/gitlab/embedded/bin/bundle:23:in `<main>'
git_1          |     Tasks: TOP => gitlab:db:configure
git_1          |     (See full trace by running task with --trace)
git_1          |     STDERR: 
git_1          |     ---- End output of "bash"  "/tmp/chef-script20200629-25-idzu10" ----
git_1          |     Ran "bash"  "/tmp/chef-script20200629-25-idzu10" returned 1
git_1          |     

There is no postresql in my docker-compose - there never was and it was fine.

In postresql log that was in docker/git files i have found this:

FATAL: database files are incompatible with server 
DETAIL: The data directory was initialized by PostgreSQL version 10, which is not compatible with this version 11.7.

How can i migrate git data to new version of postresql? Is it really part of update gitlab-ce process? Usually it was only docker pull and it was working fine.

Answer 1

I had nearly the same problem, I think. Upgraded my gitlab-ce docker container from 12.9.10 to 13.1.2.

After that, there was the same error as you described. Since gitlab has its own (possibly modified?) postgresql in its docker container, I tried to migrate the database from version 10 to 11 using default prostresql installations.

Using docker-compose to start the container will start gitlab in it. Because of the database failure, the container restart every minute. We don´t want this for our migration work. So we start it manually and access the bash in it.

My way to start it manually:

docker run --rm \
--hostname gitlab.example.org \
--publish 8143:443 --publish 8180:80 \
--env GITLAB_OMNIBUS_CONFIG="external_url 'https://gitlab.example.org'; letsencrypt['enabled'] = false" \
--name gitlab_custom \
--volume /srv/gitlab/config:/etc/gitlab \
--volume /srv/gitlab/logs:/var/log/gitlab \
--volume /srv/gitlab/data:/var/opt/gitlab \
-it gitlab/gitlab-ce:latest /bin/bash

First of all run the default entry script to check that it really doesn't work!

/assets/wrapper

If there is a failure in the Recipe: gitlab::database_migrations section and at least the error message Error executing action 'run' on resource 'bash[migrate gitlab-rails database]', continue to migrate the database.

We need to install the default (old and new) PostgreSQL versions.

apt update
apt install postgresql-10 postgresql-11

Create the destination folder for the migrated database and update the directory owner to gitlab-psql. We assume, the 'old' (version 10) database data is located at /var/opt/gitlab/postgresql/data.

mkdir -p /tmp/11/data
chown -R gitlab-psql /tmp/11 /var/opt/gitlab/postgresql/data

Now we need to initialize the new database structure.

sudo -u gitlab-psql /usr/lib/postgresql/11/bin/initdb -D /tmp/11/data

Take a look at /var/opt/gitlab/postgresql/data/postgresql.conf for the right port (and set it with -p).

Before running the real migration, we should check the access and schema of the database. In case of errors, this command can be run multiple times.

sudo -u gitlab-psql /usr/lib/postgresql/11/bin/pg_upgrade -D /tmp/11/data -d /var/opt/gitlab/postgresql/data/ -v -b /usr/lib/postgresql/10/bin -B /usr/lib/postgresql/11/bin/ -p 5432 --check

If there was an error during the check, you may need to update some other directory permissions.

If the check was successful, do the database migration with the following command. This can take some time, depending on the size of your gitlab instance.

sudo -u gitlab-psql /usr/lib/postgresql/11/bin/pg_upgrade -D /tmp/11/data -d /var/opt/gitlab/postgresql/data/ -v -b /usr/lib/postgresql/10/bin -B /usr/lib/postgresql/11/bin/ -p 5432

Stop all gitlab services, backup the old database and move the migrated data to the original path.

gitlab-ctl stop
rm -f /var/opt/gitlab/postgresql/.s.PGSQL.5432*
mv /var/opt/gitlab/postgresql/data /var/opt/gitlab/postgresql/data_old_10
mv /tmp/11/data /var/opt/gitlab/postgresql/data

Now run the default entry script to finish the built-in gitlab migration.

/assets/wrapper

If it works (some log output appears after loading the gitlab parts and of course the website works), you can press Ctrl+C, and enter exit to leave and remove the current container.

If there were still database errors, ... oops.

If there is an error like this,

PG::ConnectionBad: could not connect to server: No such file or directory
          |                Is the server running locally and accepting
          |                connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?

make sure no postgresql process is running, remove the socket file and try again.

After that, leave the container.

Start your gitlab container the way you start it by default.

Hope that helps 🙂

---

Edit:

The main package repositories in gitlab/gitlab-ce:latest does no longer contain older postgresql versions as 12.

Add the official postgresql repo to install older postgresql versions with apt:

sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt bionic-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | apt-key add -
apt-get update

After that packages like postgresql-10 or postgresql-11 are available again.

Answer 2

Well this wasn't an easy task.

First of all, I started the GitLab container with a bash to investigate what's going on, in my case using docker-compose run --rm gitlab /bin/bash. It's good to know where things are inside the container:

• /assets/wrapper is the usual command of the container, it sets everything up, runs migrations and starts GitLab
• Log files can be found in /var/log/gitlab. The container log didn't contain any information about why postgres couldn't start, but /var/log/gitlab/postgresql/current did.
• The bundled applications (for example postgres) can be found in /opt/gitlab/embedded. For example, to start the bundled Postgres server manually, run /opt/gitlab/embedded/bin/pgctl -D /var/opt/gitlab/postgresql/data start.

Running cat /var/opt/gitlab/postgresql/data/PG_VERSION revealed that my postgres data directory was running on Postgres 9.2. In the GitLab docs there is a Postgres version matrix, where it is shown which GitLab version bundles which Postgres version. Some GitLab versions bundle multiple Postgres versions, and these support migrating between them. In my case, GitLab 9 is the last one to bundle Postgres 9.2 (along with Postgres 9.6). I found out on Docker Hub that the latest version of GitLab 9 is 9.5.10-ce.0, so I changed my docker-compose configuration to start that version.

I recommend deleting all the /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-* files before proceeding, as it seems these are printed out every time GitLab is started, making it seem like lots of errors occurred, even though they are from the past.

I had to try several times to start GitLab 9, each time investigating and fixing the problems that I found in /var/log/gitlab. Eventually starting the container (or running /assets/wrapper by hand) automatically upgraded my Postgres 9.2 database to 9.6 and I could access my GitLab instance again.

From there I continued the recommended upgrade path and started each version and made sure that it was accessible in the browser. Often a version failed to start with a Mailroom timeout, but simply trying a second or third time fixed the issue.

GitLab 9.5.10 upgraded my database from Postgres 9.2 to 9.6 and GitLab 12.10.14 upgraded it to Postgres 11. Looking at the Postgres version matrix, this means that the last GitLab version to support a particular Postgres version makes sure to automatically upgrade the Postgres database to a newer version.

Now I have the latest GitLab 13.8.0 running with Postgres 11. In the future I will make sure to follow the recommended upgrade path.

Answer 3

Looks like i have broke it by trying change docker image versions to follow upgrade recomendations, after it was already upgraded to 13.

For anyone who looks @urzz reply about using postres:10 container to restore db. Someone already did it and nice cheat-sheet available here

For next time its better to stick to recommendations

Answer 4

Did you upgrade your gitlab version?

I encountered the same problem when upgrade gitlab from 12 to 13.

You can try backup postgresql data dir in gitlab like copy it out, and run a postgresql:10 container use the data dir, try to read postgresql data and dump it. If it works, delete the data dir in gitlab and restart gitlab to init postgresql. Then restore data from the dump file in gitlab postgresql.