CODEWITHSUNDEEP
dockergitlab-ce
Adamwolf
Adamwolf

Reputation: 41

Gitlab upgrade issue in Docker

I am trying to upgrade my Gitlab CE which is running in Docker container. I am upgrading from version 11.9.1 to 14.2.1. I am also following the required upgrade path from official Gitlab documentation which is:

11.9.1->11.11.8->12.0.12->12.1.17->12.10.14->13.0.4->13.1.11->13.8.8->13.12.9->14.0.7->14.2.1

The last version that works is 14.0.7, I am also able to run latest 14.1.x version, but during migration to 14.2.x the following error appears, some migrations does not work.

There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20210903-28-4vm0c2" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':       {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_job_artifacts", :column_name=>"id", :job_arguments=>[["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_to_bigint"]]}

Finalize it manualy by running

        sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,'[["id"\, "job_id"]\, ["id_convert_to_bigint"\, "job_id_convert_to_bigint"]]']

For more information, check the documentation

        https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1129:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210706212710_finalize_ci_job_artifacts_bigint_conversion.rb:14:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active':       {:job_class_name=>"CopyColumnUsingBackgroundMigrationJob", :table_name=>"ci_job_artifacts", :column_name=>"id", :job_arguments=>[["id", "job_id"], ["id_convert_to_bigint", "job_id_convert_to_bigint"]]}

Finalize it manualy by running

        sudo gitlab-rake gitlab:background_migrations:finalize[CopyColumnUsingBackgroundMigrationJob,ci_job_artifacts,id,'[["id"\, "job_id"]\, ["id_convert_to_bigint"\, "job_id_convert_to_bigint"]]']

For more information, check the documentation

        https://docs.gitlab.com/ee/user/admin_area/monitoring/background_migrations.html#database-migrations-failing-because-of-batched-background-migration-not-finished
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:1129:in `ensure_batched_background_migration_is_finished'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20210706212710_finalize_ci_job_artifacts_bigint_conversion.rb:14:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:61:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20210706212710 FinalizeCiJobArtifactsBigintConversion: migrating ===========
STDERR:
---- End output of "bash"  "/tmp/chef-script20210903-28-4vm0c2" ----
Ran "bash"  "/tmp/chef-script20210903-28-4vm0c2" returned 1


Running handlers:
Running handlers complete
Chef Infra Client failed. 11 resources updated in 16 seconds
Thank you for using GitLab Docker Image!
Current version: gitlab-ce=14.2.0-ce.0

Configure GitLab for your system by editing /etc/gitlab/gitlab.rb file
And restart this container to reload settings.
To do it use docker exec:

  docker exec -it gitlab editor /etc/gitlab/gitlab.rb
  docker restart gitlab

For a comprehensive list of configuration options please see the Omnibus GitLab readme
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md

If this container fails to start due to permission problems try to fix it by executing:

  docker exec -it gitlab update-permissions
  docker restart gitlab

Cleaning stale PIDs & sockets
Preparing services...
Starting services...
Configuring GitLab...
/opt/gitlab/embedded/bin/runsvdir-start: line 24: ulimit: pending signals: cannot modify limit: Operation not permitted
/opt/gitlab/embedded/bin/runsvdir-start: line 37: /proc/sys/fs/file-max: Read-only file system

I have tried executing migrations by hand and all other fixes that logs propose, but none of them worked. I use Ubuntu 20.04 LTS and Docker version 20.10.7, build 20.10.7-0ubuntu1~20.04.1

Upvotes: 1

Views: 1645

Answers (1)

Adamwolf
Adamwolf

Reputation: 41

Ok, turned out that I had to update to 14.0.5 firstly and wait for some background migration to complete (you can see them in menu->admin->monitoring->background migrations).

Upvotes: 3

Related Questions