CODEWITHSUNDEEP
dockersshgitlab-ci
Wintermute
Wintermute

Reputation: 3045

SSH from gitlab-ci docker fails on "SSH2_MSG_KEX_ECDH_REPLY"

I'm trying to SSH from a Docker deployment container running in GitLab CI, to a remote webserver. It did successfully connect, once, the first time I tried. Now it's continually failing: the last two lines of the debug output (the full output is below) are:

debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
Connection closed by ********** port 22

The deploy task first copies the SSH private key from a GitLab CI 'secret variable' (the pubkey is of course in ~/.ssh/authorized_keys on the remote). The full deploy task is defined in gitlab-ci.yml as:

.deploy: &import_deploy
    stage: deploy
    image: **********
    tags: [ek-docker]
    before_script:
        # install openssh & rsync
        - apk update && apk --no-cache add rsync openssh
        # copy private key from GitLab secret variable to container
        - mkdir -p ~/.ssh
        - chmod 700 ~/.ssh
        - eval $(ssh-agent -s)
        - echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
        - echo "$SSH_PRIVATE_KEY" | tr -d '\r' | ssh-add - > /dev/null
    script:
        - ssh -vvv **********@********** 'bash -s' < ./conf/$MY_ENV_NAME/start.sh

    deploy staging:
    <<: *import_deploy
    variables:
        MY_ENV_NAME: "staging"
    only:
        - staging
    environment:
        name: staging

The output logged from SSH is:

OpenSSH_7.7p1, LibreSSL 2.7.4
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: resolve_canonicalize: hostname ********** is address
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to ********** [**********] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_xmss type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.7
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.6
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.6 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to **********:22 as '**********'
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,[email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: [email protected],[email protected],[email protected],[email protected],[email protected],ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected],zlib
debug2: compression stoc: none,[email protected],zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: [email protected],ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: ciphers stoc: [email protected],aes128-ctr,aes192-ctr,aes256-ctr,[email protected],[email protected]
debug2: MACs ctos: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: [email protected],[email protected],[email protected],[email protected],[email protected],[email protected],[email protected],hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,[email protected]
debug2: compression stoc: none,[email protected]
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: [email protected]
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
Connection closed by ********** port 22
ERROR: Job failed: exit code 1

I am way out of my depth here, and Googling suggests two possible causes. The first is something network-based, with changing the... MTU? as the potential fix. That sounds scary, so I'm focussing on the second, a mismatch in key exchange algorithms. It seems plausible: there doesn't seem to be a common value between "KEX algorithms" and "host key algorithms" in the output above. When it did connect that first time, the output was

"Warning: Permanently added '**********' (ECDSA) to the list of known hosts."

I tried specifying the algorithm via ssh -o KexAlgorithms=ecdh-sha2-nistp521 (and a couple other likely-looking candidates) but that just raised an error that the algorithm wasn't supported.

As is now clear, I have NO idea what I'm doing. Can anyone even suggest what might be the cause of the issue, from the SSH output above?

Upvotes: 6

Views: 4704

Answers (1)

Marvin
Marvin

Reputation: 429

I had the same issue I have added this option to ssh and then problem solved:

ssh -o KexAlgorithms=ecdh-sha2-nistp521 192.168.1.100

Upvotes: 9

Related Questions