PR_END_OF_FILE_ERROR for self-signed SSL certificate in GitLab

Question

Issue

After creating a private and public SSL certificate for a self-hosted GitLab server, and reconfiguring GitLab, I encounter the error:

An error occured during a connection to localhost:80. PR_END_OF_FILE_ERROR

When I try to locally visit the GitLab server.

Verification

To make sure that the issue is with adding the SSL certificates to GitLab, (and not with creating the SSL certificates, nor with Firefox), the following verifications are performed:

• The self-signed root CA is added to Firefox (and Ubuntu for that matter). This is implicitly verified by the next verification.
• I verified the SSL certificate is valid by running a dash app on port 80 and giving it (a different form of) the private and public key of the SSL certificate respectively, and visiting the https://localhost:80 url and seeing it is accessible.

Below is an image that shows the onion domain is reachable over https for dash:

Details

I think the essence of the question boils down to finding the right settings for GitLab and Nginx. This is because verified the SSL certificates work for a Dash app running on https://localhost:80 I do not yet know how to add the public and private SSL certificates to GitLab, nor what the exact nginx settings should be. In particular,

• I am not sure on what port(s) to host GitLab (e.g. 80, 443, 444 and/or a combination of these ports). Specifically, I do not know what the values of --publish "$GITLAB_PORT1" --publish $GITLAB_PORT2" etc. should be, in:

sudo docker run --detach \
      --hostname "$GITLAB_SERVER" \
      --publish "$GITLAB_PORT_1" \
      --publish "$GITLAB_PORT_2" \
      --publish "$GITLAB_PORT_3" \
      --name "$GITLAB_NAME" \
      --restart always \
      --volume "$GITLAB_HOME"/config:/etc/gitlab \
      --volume "$GITLAB_HOME"/logs:/var/log/gitlab \
      --volume "$GITLAB_HOME"/data:/var/opt/gitlab \
      -e GITLAB_ROOT_EMAIL="$GITLAB_ROOT_EMAIL_GLOBAL" \
      -e GITLAB_ROOT_PASSWORD="$gitlab_pwd" \
      -e EXTERNAL_URL="https://127.0.0.1" \
      "$gitlab_package"

• I am not sure what the exact nginx settings should be in the gitlab.rb file, to allow the SSL certificates to function (such that GitLab reachable on https. I tried the following gitlab.rb settings:

## GitLab configuration settings
external_url 'https://localhost'
letsencrypt['enable'] = false
nginx['enable'] = true
nginx['redirect_http_to_https'] = true
nginx['ssl_certificate'] = "/etc/gitlab/ssl/localhost/public_key.pem"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/localhost/private_key.pem"
nginx['listen_port'] = 80
nginx['listen_https'] = true

In particular:

• I do not know whether I should add the root ca certificate to Gitlab.
• I do not know whether I should convert the public and private SSL key for GitLab from a .pem file (which they are when they are generated), into a .crt and .key file or not.
• I do not know whether I should specify the listen port or not, and if yes, whether I should set it to 80 or 443.
• The https examples that I have seen used nginx['listen_https'] = false which confuses me, because I would expect nginx has to listen to https for the GitLab server to be reachable on https.

Question

On which port should I host GitLab and what should the accompanying nginx settings be in the gitlab.rb file to make GitLab reachable at https://localhost:port_nr and https://some_onion.onion:port_nr?

Approach

Based on this video, I have created the following function that:

• is able to convert the private and public SSL certificates for GitLab, from .pem files into the .key and .crt file (for private and public respectively).
• Copies those SSL certificates into the default location where GitLab looks for them.
• Then applies the above gitlab.rb settings.
• and tells Gitlab to reconfigure.
• It has two boolean parameters:
  • include_root_ca_in_gitlab to add the root ca to GitLab (or not), and
  • convert_to_crt_and_key_ext to convert the .pem private and public key into a .crt (for the public key) and a .key (for the private key) respectively.

The instructions to run this script are given in the MWE below.

add_private_and_public_ssl_certs_to_gitlab() {
  local project_name="$1"
  local domain_name="$2"
  local ssl_private_key_filename="$3"
  local ssl_public_key_filename="$4"
  local ca_public_cert_filename="$5"
  local convert_to_crt_and_key_ext="$6"
  local include_root_ca_in_gitlab="$7"

  local ssl_private_key_filepath="certificates/ssl_cert/$project_name/$ssl_private_key_filename"
  local ssl_public_key_filepath="certificates/ssl_cert/$project_name/$ssl_public_key_filename"

  # Assert local private and public certificate exist for service.
  manual_assert_file_exists "$ssl_private_key_filepath"
  manual_assert_file_exists "$ssl_public_key_filepath"
  create_gitlab_ssl_directories

  local ssl_public_key_in_gitlab_filepath
  local ssl_private_key_in_gitlab_filepath

  if [[ "$convert_to_crt_and_key_ext" == "true" ]]; then

    # Convert public .pem into public .crt with:
    openssl x509 -outform der -in "$ssl_public_key_filepath" -out "certificates/ssl_cert/$project_name/$domain_name.crt"
    manual_assert_file_exists "certificates/ssl_cert/$project_name/$domain_name.crt"

    # Convert private .pem into private .key with:
    openssl pkey -in "$ssl_private_key_filepath" -out "certificates/ssl_cert/$project_name/$domain_name.key"
    manual_assert_file_exists "certificates/ssl_cert/$project_name/$domain_name.key"
    # TODO: verify the generated .key is valid with the old public .pem.
    # TODO: verify the generated .key is valid with the new public .crt.

    ssl_public_key_in_gitlab_filepath="/etc/gitlab/ssl/$domain_name.crt"
    ssl_private_key_in_gitlab_filepath="/etc/gitlab/ssl/$domain_name.key"

    # Copy your new certificates into the folder where GitLab looks by default
    # for new SSL certificates.
    sudo cp "certificates/ssl_cert/$project_name/$domain_name.crt" "$ssl_public_key_in_gitlab_filepath"
    sudo cp "certificates/ssl_cert/$project_name/$domain_name.key" "$ssl_private_key_in_gitlab_filepath"

  else
    ssl_public_key_in_gitlab_filepath="/etc/gitlab/ssl/$domain_name/public_key.pem"
    ssl_private_key_in_gitlab_filepath="/etc/gitlab/ssl/$domain_name/private_key.pem"

    sudo mkdir -p "/etc/gitlab/ssl/$domain_name/"
    sudo cp "$ssl_public_key_filepath" "$ssl_public_key_in_gitlab_filepath"
    sudo cp "$ssl_private_key_filepath" "$ssl_private_key_in_gitlab_filepath"
  fi

  manual_assert_file_exists "$ssl_public_key_in_gitlab_filepath"
  manual_assert_file_exists "$ssl_private_key_in_gitlab_filepath"

  # The ~/gitlab/config/gitlab.rb file says:
  ##! Most root CA's are included by default
  # nginx['ssl_client_certificate'] = "/etc/gitlab/ssl/ca.crt"
  # So perhaps also include the self-signed root ca into that dir.
  if [[ "$include_root_ca_in_gitlab" == "true" ]]; then
    manual_assert_file_exists "certificates/root/$ca_public_cert_filename"
    sudo cp "certificates/root/$ca_public_cert_filename" "/etc/gitlab/ssl/ca.crt"
    manual_assert_file_exists "/etc/gitlab/ssl/ca.crt"
  fi

  add_lines_to_gitlab_rb "$domain_name" "$include_root_ca_in_gitlab" "$ssl_public_key_in_gitlab_filepath" "$ssl_private_key_in_gitlab_filepath"

  reconfigure_gitlab_with_new_certs_and_settings

}

create_gitlab_ssl_directories() {
  sudo rm -rf "/etc/gitlab/ssl/*"
  sudo mkdir -p "/etc/gitlab/ssl"
  sudo chmod 755 "/etc/gitlab/ssl"
}

reconfigure_gitlab_with_new_certs_and_settings() {
  # Create a method to get the docker id.
  local docker_container_id
  docker_container_id=$(get_docker_container_id_of_gitlab_server)
  sudo docker exec -i "$docker_container_id" bash -c "gitlab-ctl reconfigure"
}

add_lines_to_gitlab_rb() {
  local domain_name="$1"
  local include_root_ca_in_gitlab="$2"
  local ssl_public_key_in_gitlab_filepath="$3"
  local ssl_private_key_in_gitlab_filepath="$4"

  # Create a copy of the basic gitlab.rb file.
  rm "$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  cp "$GITLAB_RB_TEMPLATE_FILEPATH" "$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  # Verified you only have to add lines (instead of modify) into that basic gitlab.rb.

  if [[ "$domain_name" == "localhost" ]]; then
    echo """external_url 'https://localhost'""" >>"$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  else
    echo "external_url '$domain_name'" >>"$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  fi
  # shellcheck disable=SC2129
  echo """letsencrypt['enable'] = false""" >>"$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"

  echo "nginx['enable'] = true" >>"$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  echo "nginx['redirect_http_to_https'] = true" >>"$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  echo "nginx['ssl_certificate'] = \"$ssl_public_key_in_gitlab_filepath\"" >>"$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  echo "nginx['ssl_certificate_key'] = \"$ssl_private_key_in_gitlab_filepath\"" >>"$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  #echo "nginx['ssl_dhparam'] = \"/etc/gitlab/ssl/dhparams.pem\""  >> "$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  echo "nginx['listen_port'] = 80" >>"$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  echo "nginx['listen_https'] = false" >>"$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  if [[ "$include_root_ca_in_gitlab" == "true" ]]; then
    echo "nginx['ssl_client_certificate'] = \"/etc/gitlab/ssl/ca.crt\"" >>"$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"
  fi

  # TODO: verify the external url is found correctly:
  # sudo cat ~/gitlab/config/gitlab.rb | grep external_url

  tail -15 "$GITLAB_RB_TEMPLATE_DIR""gitlab.rb"

  sudo cp "$GITLAB_RB_TEMPLATE_DIR""gitlab.rb" ~/gitlab/config/gitlab.rb
}

get_docker_container_id_of_gitlab_server() {
  local docker_container_id
  docker_container_id=$(sudo docker ps -aqf "name=gitlab")
  assert_is_non_empty_string "$docker_container_id"

  echo "$docker_container_id"
}

MWE

To reproduce the specified behaviour quickly, I have created the following minimal working example (MWE). To allow you to keep your system clean, I've included a Qemu virtual machine setup as well.

Qemu

One can install qemu and set up an Ubuntu 22 machine with:

# Download the ubuntu-22.04.2-desktop-amd64.iso into this directory.
wget https://releases.ubuntu.com/jammy/ubuntu-22.04.2-desktop-amd64.iso

# Create the image in which the Ubuntu 22.04 VM will be created.
qemu-img create ubuntu22.img 30G

# Create the Ubuntu 22.10 Ubuntu image.
qemu-system-x86_64 \
 --enable-kvm \
 -m 1024 \
 -machine smm=off \
 -cdrom $PWD/ubuntu-22.04.2-desktop-amd64.iso \
 -boot order=d ubuntu22.img

Next one can install and run Ubuntu with:

qemu-system-x86_64 \
 --enable-kvm \
 -m 4096 \
 -machine smm=off \
 -boot order=d ubuntu22_server.img \
 -smp 4 \
 -chardev qemu-vdagent,id=ch1,name=vdagent,clipboard=on \
 -device virtio-serial-pci \
 -device virtserialport,chardev=ch1,id=ch1,name=com.redhat.spice.0

GitLab

The GitLab instance can be installed with:

sudo apt install git -y
mkdir git
cd git
git clone https://github.com/TruCol/Self-host-GitLab-CI-for-GitHub.git
cd Sel*
git checkout https
./install_gitlab.sh \
 --server \
 --gitlab-email somegitlab@email.com \
 --gitlab-password \
 --labprereq \
 --external-url "https://0.0.0.0" \
 --gitlab-server "0.0.0.0"

SSL

The self-signed SSL certificates for the onion domains (and localhost) for GitLab can be created with:

git clone https://github.com/HiveMinds/SSL4Tor.git
cd SSL4Tor
./src/main.sh \
  --1-domain-1-service \
  --delete-onion-domain \
  --services 80:gitlab:443/22:ssh:22 \
  --make-onion-domains \
  --ssl-password somepassword \
  --background-dash \
  --make-ssl-certs \
  --setup-ssh-server \
  --get-onion-domain

Add self-signed SSL cert into GitLab

Next, those SSL-certificates can be automatically added to GitLab, and GitLab is automatically reconfigured with:

./src/main.sh -1d1s --services 80:gitlab:443 --apply-certs

One can check whether or not https://localhost:443 is available or not.

Answer 1

TL;DR

Below is the code I used to successfully run a self-hosted instance of GitLab over https://localhost:443 using self-signed SSL certificates:

GitLab

The GitLab instance can be installed with:

sudo apt install git -y
mkdir git
cd git
git clone https://github.com/TruCol/Self-host-GitLab-CI-for-GitHub.git
cd Sel*
git checkout https
./install_gitlab.sh \
 --server \
 --gitlab-email somegitlab@email.com \
 --gitlab-password \
 --labprereq \
 --external-url "https://0.0.0.0" \
 --gitlab-server "0.0.0.0"

SSL

The self-signed SSL certificates for the onion domains (and localhost) for GitLab can be created with:

git clone https://github.com/HiveMinds/SSL4Tor.git
cd SSL4Tor
./src/main.sh \
  --firefox-to-apt \
  --services 443:gitlab:443/9001:dash:9002/22:ssh:22 \
  --ssl-password somepassword \
  --get-onion-domain

(Note, support for snap Firefox is not included at this time.)

Add self-signed SSL cert into GitLab

Next, those SSL-certificates can be automatically added to GitLab, and GitLab is automatically reconfigured with:

./src/main.sh -1d1s --services 80:gitlab:443 --apply-certs

Next, one should be able to visit GitLab at https://localhost:443

Nginx Settings

The settings below were found to be working nginx settings:

## GitLab configuration settings
external_url 'https://localhost'
letsencrypt['enable'] = false
nginx['enable'] = true
nginx['redirect_http_to_https'] = true
nginx['ssl_certificate'] = "/etc/gitlab/ssl/localhost/public_key.pem"
nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/localhost/private_key.pem"
nginx['listen_port'] = 443
nginx['listen_https'] = true

Note the root ca certificate (neither public nor private cert/key) is not added/given to GitLab

Initial Errors

To explain what I did wrong, and how the accepted answer led me to a working configuration, the following content of that answer is parsed and discussed:

.pem files vs .crt and .key file.

About renaming .pem files to .crt and .key - for unix systems this doesn't really matter, since they're all just regular text files anyway, but for the sake of accuracy, I'd change the extensions.

I assume there indeed should not be a difference between the content of the public.pem and public.crt file, nor between private.pem and private.key. I verified the two files were the same. However, when copying the private.key into the docker, the content got scrambled, I do not exactly know how at this time. In the end, I used the .pem files, because that allowed me to remove the step of converting the .pem files into .crt and .key respectively.

Nginx able to read SSL files?

Did you make sure that the cert files are readable by nginx inside the container? Check the permissions. I got a permission error when I tried to read the content of one of the .pem files within the docker shell. I falsely assumed this meant nginx was not able to read the SSL certs. Later I wrote a method to verify the SSL certs that were copied inside the docker, are valid. At that point, no read error occurred. I did not explicitly verify nginx was able to read the SSL files, as I did not know how.

Which port to make nginx listen to

Listening on port 80 in your config should be fine, there's probably a bit of configuration inside nginx website config file (usually in /etc/nginx/sites-enabled/*) that redirects https traffic from port 80 to 443 automatically.

This was the last bit of information I applied to get it working. I changed the port in the nginx settings in the gitlab.rb file from port 80 to port 443 and then it worked.

Make nginx listen to https

You should definitely enable https in your config. Maybe this will be enough to get your gitlab running on https.

That was True, I did not do that. Making nginx listen to https.

External url name for GitLab

The EXTERNAL_URL envvar inside your docker config is "https://127.0.0.1" - this usually only allows users from localhost to view the website. To allow traffic from "everywhere", change this to "https://0.0.0.0" (check gitlab docker documentation first)

I was unaware of the difference between 0.0.0.0 and 127.0.0.1 thank you for explaining that. Changing that has helped, I expect it will also be valuable when running GitLab as an onion service. I do not yet know how to run GitLab on multiple external urls (neither for the docker run command to start GitLab, nor for the nginx settings.).

How to access GitLab locally

You're trying to access the website through https protocol, but on port 80. In most default configs, this will not work, as port 80 is reserved for http, and port 443 for https traffic.

Yes, after switching the nginx settings from port 80 to port 443, I was able to visit GitLab locally at https://0.0.0.0:443.

Answer 2

A bunch of thoughts:

• About renaming .pem files to .crt and .key - for unix systems this doesn't really matter, since they're all just regular text files anyway, but for the sake of accuracy, I'd change the extensions.

• Did you make sure that the cert files are readable by nginx inside the container? Check the permissions.

• Listening on port 80 in your config should be fine, there's probably a bit of configuration inside nginx website config file (usually in /etc/nginx/sites-enabled/*) that redirects https traffic from port 80 to 443 automatically.

• You should definitely enable https in your config. Maybe this will be enough to get your gitlab running on https.

   nginx['ssl_certificate'] = "/etc/gitlab/ssl/localhost/public_key.crt"
   nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/localhost/private_key.key"
   nginx['listen_port'] = 80
   nginx['listen_https'] = true
  

• The EXTERNAL_URL envvar inside your docker config is "https://127.0.0.1" - this usually only allows users from localhost to view the website. To allow traffic from "everywhere", change this to "https://0.0.0.0" (check gitlab docker documentation first)

• Docker ports 80 and 443 must of course be exposed

• You're trying to access the website through https protocol, but on port 80. In most default configs, this will not work, as port 80 is reserved for http, and port 443 for https traffic.

Try these links:

http://localhost
http://localhost:80
https://localhost
https://localhost:443

• If none of the above solve your error, try accessing the website through another browser, as the error you're mentioning seems to be firefox-specific.