Issue deploying project via capistrano 3 with jenkins

Question

I'm trying to deploy to a server via capistrano 3 from a jenkins job.

"cap environment deploy" works perfectly from my mac with my creds, but under jenkins, with deploy creds, I run into a problem.

What's super strange is that the first few git commands succeed, but when capistrano gets to "git remote update", the server hangs up with a "not authorized" error. I've verified this by ssh'ing to the server and running 'git remote update' from /var/www/repo.

I have key forwarding set up -- below is my Jenkins job:

eval $(ssh-agent)
ssh-add ~/.ssh/keyname.pem
cap staging deploy

I have no idea what's happening inside capistrano that would cause the first git commands to succeeed ('git ls-remote'), but "git remote update" to fail. Yes, the key has been added as a deploy key on github for the repo, and key forwarding is enabled in deploy.rb and via ~/.ssh/config.

Below is an example failing log from 'cap staging deploy', running as the Jenkins user on the Jenkins box:

 INFO [d3e19149] Running /usr/bin/env mkdir -p /tmp/project/ on server.xxxxxyyyy.com
DEBUG [d3e19149] Command: /usr/bin/env mkdir -p /tmp/project/
 INFO [d3e19149] Finished in 0.943 seconds with exit status 0 (successful).
DEBUG Uploading /tmp/project/git-ssh.sh 0.0%
 INFO Uploading /tmp/project/git-ssh.sh 100.0%
 INFO [43533553] Running /usr/bin/env chmod +x /tmp/project/git-ssh.sh on server.xxxxxyyyy.com
DEBUG [43533553] Command: /usr/bin/env chmod +x /tmp/project/git-ssh.sh
 INFO [43533553] Finished in 0.052 seconds with exit status 0 (successful).
DEBUG [40e092d5] Running /usr/bin/env git ls-remote git@github.com:Organization/project.git on server.xxxxxyyyy.com
DEBUG [40e092d5] Command: ( GIT_ASKPASS=/bin/echo GIT_SSH=/tmp/project/git-ssh.sh /usr/bin/env git ls-remote git@github.com:Organization/project.git )
DEBUG [40e092d5]    69a8427388e2958e9b2c67e6048a46cb3a2544a5    HEAD
DEBUG [40e092d5]    69a8427388e2958e9b2c67e6048a46cb3a2544a5    refs/heads/master

DEBUG [40e092d5] Finished in 1.635 seconds with exit status 0 (successful).
 INFO [3d9e0765] Running /usr/bin/env mkdir -pv /var/www/shared /var/www/releases on server.xxxxxyyyy.com
DEBUG [3d9e0765] Command: /usr/bin/env mkdir -pv /var/www/shared /var/www/releases
 INFO [3d9e0765] Finished in 0.046 seconds with exit status 0 (successful).
 INFO [3ef2c63f] Running /usr/bin/env mkdir -pv /var/www/shared/log /var/www/shared/tmp/pids /var/www/shared/tmp/cache /var/www/shared/tmp/sockets /var/www/shared/vendor/bundle /var/www/shared/public/system on server.xxxxxyyyy.com
DEBUG [3ef2c63f] Command: /usr/bin/env mkdir -pv /var/www/shared/log /var/www/shared/tmp/pids /var/www/shared/tmp/cache /var/www/shared/tmp/sockets /var/www/shared/vendor/bundle /var/www/shared/public/system
 INFO [3ef2c63f] Finished in 0.043 seconds with exit status 0 (successful).
 INFO [a0820d89] Running /usr/bin/env mkdir -pv /var/www/shared/config on server.xxxxxyyyy.com
DEBUG [a0820d89] Command: /usr/bin/env mkdir -pv /var/www/shared/config
 INFO [a0820d89] Finished in 0.050 seconds with exit status 0 (successful).
DEBUG [19575759] Running /usr/bin/env [ -f /var/www/shared/config/database.yml ] on server.xxxxxyyyy.com
DEBUG [19575759] Command: [ -f /var/www/shared/config/database.yml ]
DEBUG [19575759] Finished in 0.063 seconds with exit status 0 (successful).
DEBUG [06e522d5] Running /usr/bin/env [ -f /var/www/repo/HEAD ] on server.xxxxxyyyy.com
DEBUG [06e522d5] Command: [ -f /var/www/repo/HEAD ]
DEBUG [06e522d5] Finished in 0.056 seconds with exit status 0 (successful).
 INFO The repository mirror is at /var/www/repo
DEBUG [062dd56a] Running /usr/bin/env if test ! -d /var/www/repo; then echo "Directory does not exist '/var/www/repo'" 1>&2; false; fi on server.xxxxxyyyy.com
DEBUG [062dd56a] Command: if test ! -d /var/www/repo; then echo "Directory does not exist '/var/www/repo'" 1>&2; false; fi
DEBUG [062dd56a] Finished in 0.053 seconds with exit status 0 (successful).
DEBUG [65e9187e] Running /usr/bin/env cd /var/www/repo && git rev-parse --short HEAD on server.xxxxxyyyy.com
DEBUG [65e9187e] Command: cd /var/www/repo && git rev-parse --short HEAD
DEBUG [65e9187e]    69a8427
DEBUG [65e9187e] Finished in 0.080 seconds with exit status 0 (successful).
 INFO [349bd507] Running /usr/bin/env git remote update on server.xxxxxyyyy.com
DEBUG [349bd507] Command: cd /var/www/repo && /usr/bin/env git remote update
DEBUG [349bd507]    Fetching origin
DEBUG [349bd507]    Fetching origin
DEBUG [349bd507]    ERROR: Repository not found.
DEBUG [349bd507]    Fetching origin
DEBUG [349bd507]    fatal: The remote end hung up unexpectedly
DEBUG [349bd507]    Fetching origin
DEBUG [349bd507]    error: Could not fetch origin
cap aborted!

Any ideas?

Thanks,

Andy

Answer 1

I had this same problem when I changed my repo location, but instead of removing the deployment directory:

/www/deployment directory/repo

I ran this command within the /www/deployment directory/repo:

git remote show origin

This showed that the origin was still set to the old repo, and I changed that by running this command:

git remote set-url origin git://new.url.here

(which I learned from this question: Change the URI (URL) for a remote Git repository)

Answer 2

I encountered the same problem with my current deployment. I ran the command cap staging deploy --trace and received the following output:

Command: cd /var/www/deployment directory/repo && (GIT_ASKPASS=/bin/echo GIT_SSH=/tmp/deployment directory/git-ssh.sh /usr/bin/env git remote update )    

DEBUG [520a3953]    Fetching origin   
DEBUG [520a3953]    Fetching origin  
DEBUG [520a3953]    ERROR: Repository not found.
DEBUG [520a3953]    Fetching origin
DEBUG [520a3953]    fatal: The remote end hung up unexpectedly  
DEBUG [520a3953]    Fetching origin
DEBUG [520a3953]    error: Could not fetch origin  
cap aborted!   
SSHKit::Command::Failed: git stdout: Nothing written 
git stderr: Nothing written   

I fixed the error by removing the deployment directory

rm -rf /var/www/deployment directory

I ran cap staging deploy --trace again and fixed unrelated Rails errors and was able to successfully deploy.

In hindsight I could have probably only removed the /var/www/deployment directory/repo and achieved the same results. I'm sure my problem was that I had changed my gitlab project name, and it was being stored in the repo directory of the deployment server. The command: Command: cd /var/www/deployment directory/repo && ( GIT_ASKPASS=/bin/echo GIT_SSH=/tmp/deployment directory/git-ssh.sh /usr/bin/env git remote update ) is probably using the old repository URL stored in some file in the deployment server.