AWS CodeCommit - Can clone/pull but can't push (SSH)

Question

I have followed the documented steps to create and clone a CodeCommit repository (as per The AWS documentation).

I have also verified that I am using the correct private and public keys (as per this answer).

My ssh config file is configured correctly (see here).

The user has the AWSCodeCommitFullAccess policy attached (which includes the CodeCommit:GitPush action).

I can successfully clone and pull the repository, however when I attempt to push a commit I receive the following message:

You have successfully authenticated over SSH. You can use Git to interact with AWS CodeCommit.
Connection to git-codecommit.ap-southeast-2.amazonaws.com closed by remote host.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I am using Ubuntu 16.04 and git version 2.7.4

Output of GIT_TRACE_PACKET=true GIT_TRACE=2 GIT_CURL_VERBOSE=1 GIT_SSH_COMMAND="ssh -v" git push:

15:14:19.048714 git.c:369               trace: built-in: git 'push'
15:14:19.049478 run-command.c:369       trace: run_command: 'ssh -v' 'awsgit' 'git receive-pack '\''/v1/repos/hsm'\'''
15:14:19.050040 run-command.c:228       trace: exec: '/bin/sh' '-c' 'ssh -v "$@"' 'ssh -v' 'awsgit' 'git receive-pack '\''/v1/repos/hsm'\'''
OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /home/daniel/.ssh/config
debug1: /home/daniel/.ssh/config line 21: Applying options for awsgit
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to git-codecommit.ap-southeast-2.amazonaws.com [103.8.175.151] port 22.
debug1: Connection established.
debug1: identity file /home/daniel/.ssh/keys/awsgit/awsgit type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/daniel/.ssh/keys/awsgit/awsgit-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
debug1: Remote protocol version 2.0, remote software version AWSCodeCommit     VHVlLCAyNyBKdW4gMjAxNyAwMzoxNDowNSArMDAwMLPfiCbgvY3jqs8ZWuJKQYkz8fFRYb9bCPqRK5nPaegeOk
debug1: no match: AWSCodeCommit VHVlLCAyNyBKdW4gMjAxNyAwMzoxNDowNSArMDAwMLPfiCbgvY3jqs8ZWuJKQYkz8fFRYb9bCPqRK5nPaegeOk5IMVgvTXRVQ1VzQWZCMUc2    aXM1WlFSZS9sOXZCTHY0UE9NUWt0UWJuaVU9
debug1: Authenticating to git-codecommit.ap-southeast-2.amazonaws.com:22 as 'APKAJ...[redacted]'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: hmac-sha2-256 compression: none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<8192<8192) sent
debug1: got SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: got SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: ssh-rsa SHA256:nYp+gHas80HY3...[redacted]
debug1: Host 'git-codecommit.ap-southeast-2.amazonaws.com' is known and matches the RSA host key.
debug1: Found key in /home/daniel/.ssh/known_hosts:1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/daniel/.ssh/keys/awsgit/awsgit
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug1: Authentication succeeded (publickey).
Authenticated to git-codecommit.ap-southeast-2.amazonaws.com ([103.8.175.151]:22).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: network
debug1: Sending environment.
debug1: Sending env LANG = en_NZ.UTF-8
debug1: Sending command: git receive-pack '/v1/repos/hsm'
You have successfully authenticated over SSH. You can use Git to interact with AWS CodeCommit.
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Connection to git-codecommit.ap-southeast-2.amazonaws.com closed by remote host.
Transferred: sent 2960, received 2040 bytes, in 0.1 seconds
Bytes per second: sent 41330.3, received 28484.4
debug1: Exit status -1
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

What am I doing wrong? Thanks for your help!

Answer 1

For whatever reason, it looks like your version of the git client is sending command:

git receive-pack '/v1/repos/hsm'

Running a verbose push myself, I see:

git-receive-pack '/v1/repos/MyRepoName'

Note that your client seems to be sending a space character instead of a hyphen. This suggests to me that maybe there was a bug in the version you are using? I suggest attempting to update to the newest version of the git client.

EDIT: Maybe try forcing your git client to use the correct command. Edit your git config with the command to use for receivepack operations:

git config --edit --global

Then, under your remote (maybe origin?), add:

[remote "origin"]
    receivepack = 'git-receive-pack'

Answer 2

not sure if ssh is the problem but I usually just do ssh-copy-id <user>@<ip>

Answer 3

From what I could gather, this problem has nothing to do with your ssh setup or git itself.

This might be a far fetch but can you check if you have any shell initialization scripts, something like.bashrc, .profile, .bash_login etc.. Those could be the reason why this is failing, try and disable/comment out the files temporarily.

Otherwise, could you provide more debugging info by running

GIT_TRACE_PACKET=true GIT_TRACE=2 GIT_CURL_VERBOSE=1 git push ssh -v