nohup doesn't work when used with double-ampersand (&&) instead of semicolon (;)

Question

I have a script that uses ssh to login to a remote machine, cd to a particular directory, and then start a daemon. The original script looks like this:

ssh server "cd /tmp/path ; nohup java server 0</dev/null 1>server_stdout 2>server_stderr &"

This script appears to work fine. However, it is not robust to the case when the user enters the wrong path so the cd fails. Because of the ;, this command will try to run the nohup command even if the cd fails.

The obvious fix doesn't work:

ssh server "cd /tmp/path && nohup java server 0</dev/null 1>server_stdout 2>server_stderr &"

that is, the SSH command does not return until the server is stopped. Putting nohup in front of the cd instead of in front of the java didn't work.

Can anyone help me fix this? Can you explain why this solution doesn't work? Thanks!

Edit: cbuckley suggests using sh -c, from which I derived:

ssh server "nohup sh -c 'cd /tmp/path && java server 0</dev/null 1>master_stdout 2>master_stderr' 2>/dev/null 1>/dev/null &"

However, now the exit code is always 0 when the cd fails; whereas if I do ssh server cd /failed/path then I get a real exit code. Suggestions?

Answer 1

See Bash's Operator Precedence.

The & is being attached to the whole statement because it has a higher precedence than &&. You don't need ssh to verify this. Just run this in your shell:

$ sleep 100 && echo yay &
[1] 19934

If the & were only attached to the echo yay, then your shell would sleep for 100 seconds and then report the background job. However, the entire sleep 100 && echo yay is backgrounded and you're given the job notification immediately. Running jobs will show it hanging out:

$ sleep 100 && echo yay &
[1] 20124
$ jobs
[1]+  Running                 sleep 100 && echo yay &

You can use parenthesis to create a subshell around echo yay &, giving you what you'd expect:

sleep 100 && ( echo yay & )

This would be similar to using bash -c to run echo yay &:

sleep 100 && bash -c "echo yay &"

Tossing these into an ssh, and we get:

# using parenthesis...
$ ssh localhost "cd / && (nohup sleep 100 >/dev/null </dev/null &)"
$ ps -ef | grep sleep
me 20136     1  0 16:48 ?        00:00:00 sleep 100

# and using `bash -c`
$ ssh localhost "cd / && bash -c 'nohup sleep 100 >/dev/null </dev/null &'"
$ ps -ef | grep sleep
me 20145     1  0 16:48 ?        00:00:00 sleep 100

Applying this to your command, and we get

ssh server "cd /tmp/path && (nohup java server 0</dev/null 1>server_stdout 2>server_stderr &)"

or:

ssh server "cd /tmp/path && bash -c 'nohup java server 0</dev/null 1>server_stdout 2>server_stderr &'"

Also, with regard to your comment on the post,

Right, sh -c always returns 0. E.g., sh -c exit 1 has error code 0"

this is incorrect. Directly from the manpage:

Bash's exit status is the exit status of the last command executed in the script. If no commands are executed, the exit status is 0.

Indeed:

$ bash -c "true ; exit 1"
$ echo $?
1
$ bash -c "false ; exit 22"
$ echo $?
22

Answer 2

Answer roundup:

• Bad: Using sh -c to wrap the entire nohup command doesn't work for my purposes because it doesn't return error codes. (@cbuckley)
• Okay: ssh <server> <cmd1> && ssh <server> <cmd2> works but is much slower (@joachim-nilsson)
• Good: Create a shell script on <server> that runs the commands in succession and returns the correct error code.

The last is what I ended up using. I'd still be interested in learning why the original use-case doesn't work, if someone who understands shell internals can explain it to me!

Answer 3

ssh server "test -d /tmp/path" && ssh server "nohup ... &"