Gitlab pipeline broken after upgrade

Question

I have recently updated my gitlab runner to 16.4.1 and suddenly my pipeline stopped working. I have no clue what causes this state. Before this upgrade, everythign worked fine.

Pipeline is really simple- it just connects to server via SSH and runs script as specific user. I have tried to run this script as that user on the machine and it worked fine. Now it seems that Gitlab completely ignores script part of the pipeline. All variables are properly set and accesible.

my-pipeline:
  stage: deploy
  rules:
    - if: '$PROJECT_IDENTIFIER != null && $CI_PIPELINE_SOURCE == "web"'
      when: always
    - when: never
  before_script:
    - apk add --update --no-cache openssh
    - install -m 600 -D /dev/null ~/.ssh/id_rsa
    - echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
    - ssh-keyscan -H $SSH_HOST > ~/.ssh/known_hosts
  script:
    - echo "start"
    - ssh $SSH_USER@$SSH_HOST "bash /scripts/myScript.sh $PROJECT_IDENTIFIER && exit"
    - echo "stop"
  after_script:
    - rm -rf ~/.ssh

Output is like

Running with gitlab-runner 16.4.1 (XXX) on docker-runner XXX, system ID: XXX
Preparing the "docker" executor
Using Docker executor with image alpine:latest ...
Pulling docker image alpine:latest ...
Using docker image XXX for alpine:latest with digest XXXX ...
Preparing environment
Running on runner XXX
Getting source from Git repository
Fetching changes with git depth set to 20...
Reinitialized existing Git repository in XXX
Checking out 46e2c9a7 as detached HEAD (ref is main)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
Using docker image XXX for alpine:latest with digest alpine@sha256:XXX ...
$ apk add --update --no-cache openssh
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/main/x86_64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.18/community/x86_64/APKINDEX.tar.gz
(1/10) Installing openssh-keygen (9.3_p2-r0)
(2/10) Installing ncurses-terminfo-base (6.4_p20230506-r0)
(3/10) Installing libncursesw (6.4_p20230506-r0)
(4/10) Installing libedit (20221030.3.1-r1)
(5/10) Installing openssh-client-common (9.3_p2-r0)
(6/10) Installing openssh-client-default (9.3_p2-r0)
(7/10) Installing openssh-sftp-server (9.3_p2-r0)
(8/10) Installing openssh-server-common (9.3_p2-r0)
(9/10) Installing openssh-server (9.3_p2-r0)
(10/10) Installing openssh (9.3_p2-r0)
Executing busybox-1.36.1-r2.trigger
OK: 14 MiB in 25 packages
$ install -m 600 -D /dev/null ~/.ssh/id_rsa
$ echo "$SSH_PRIVATE_KEY" | base64 -d > ~/.ssh/id_rsa
$ ssh-keyscan -H $SSH_HOST > ~/.ssh/known_hosts
Running after_script
Running after script...
$ rm -rf ~/.ssh
Cleaning up project directory and file based variables
ERROR: Job failed: exit code 1

Does anyone have a clue why my pipeline is not processing script part at all?

Answer 1

After hours of struggle I gave up and changed the pipeline completely. This is the version that works same as the above one. I hope it might help anyone who will face the same issue as me.

my-pipeline:
  stage: deploy
  rules:
    - if: '$PROJECT_IDENTIFIER != null && $CI_PIPELINE_SOURCE == "web"'
      when: always
    - when: never
  before_script:
    - apk add --no-cache openssh-client bash
    - mkdir -p ~/.ssh
    - echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
    - echo $SSH_PRIVATE_KEY | base64 -d > id_rsa
    - chmod 700 id_rsa
    - mv id_rsa ~/.ssh/id_rsa
  script:
    - ssh $SSH_USER@$SSH_HOST "bash /scripts/myScript.sh $PROJECT_IDENTIFIER"
  after_script:
    - rm -rf ~/.ssh