SSH-Key not recognized "receive packet: type 51" in Gitlab-CE

Question

In my installation of Gitlab Community Edition 14.8.2, clone/push/etc.-actions do not work via SSH. The same actions via HTTPS work as usual. Also SSH actions on gitlab.com work as usual. Multiple users, whose SSH-keys worked a few weeks ago have this problem. Users log in via LDAP. The server runs Ubuntu 20.04.

Gitlab Docs recommend ssh -Tvvv git@example.com. When I do this for my own domain, I get this:

[...]
debug1: Next authentication method: publickey
debug1: Offering public key: /home/user/.ssh/id_ed25519 ED25519 SHA256:a5B[...]
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51

The authentication process ends there. When I do it for gitlab.com the receive packet is 60 instead of 51. And I get successful authentication.

Tried solutions

• The SSH-keys are found in .ssh/authorized_keys on the server and show up in web-GUI. Which differs from descriptions for this bug.
• I created a fresh VM where I followed the guide Gitlab work with SSH. The symptoms didn't change.
• The git config user.name is set to LDAP username, user.email to the corresponding mail address.
• On the client, permissions are set: .ssh/ to rwx------, .ssh/*.pub to rw-r--r-- and .ssh/* to rw-------
• type 51 means the server rejected the client key. So I tried looking at the server logs:
• api_json.log, is mentioned in log section of Gitlab Docs. But to me it seems, it only logs successful actions (I can be wrong though). Also it seems impossible to increase the verbosity of API logging.
• SSH daemon logs only connections on port 22, not the ones by Gitlab
• Adding the ~/.ssh/config form this answer gives: kex_exchange_identification: read: Connection reset by peer. Which seems to tell, the issue is on the server side

I don't know what to do now. As SSH works on gitlab.com, there has to be a solution for this. Could someone help me out to find it?

As I'm not quite sure, what exactly goes wrong, any help to find that out would also be greatly appreciated.

Versions:

GitLab 14.8.2
GitLab Shell 13.23.2
GitLab Workhorse v14.8.2 

Answer 1

I just encountered the issue, and our problem was that the local git account had the wrong home directory in /etc/passwd. This was due to a gitlab.rb SNAFU due to our use of NFS for some of the data.

Because of that the ~.ssh dir was not in the right location, so no keys were available.