CODEWITHSUNDEEP
linuxbashshell
Geoffrey De Smet
Geoffrey De Smet

Reputation: 27337

Background shell script can't reach directories after ssh logout, even with nohup

I want to run a shell script in the background on a server machine and starts that shell script from an ssh connection. Even though I run the background process script with nohup, the background script fails due to an directory unreachable error as soon as I close my ssh connection (and no sooner).

runInBackground.sh:

#!/bin/bash
...
nohup ./run.sh > /dev/null 2> local/errorLog.txt < /dev/null &

run.sh:

#!/bin/bash
...
while [ true ] ; do
    ...
    cd optaplanner-examples
    mvn exec:exec // calls java process
    cd ..
done

So when I run runInBackground.sh, everything works fine for hours, until I disconnect my ssh connection. As soon as I log out, the errorlog.txt fills up with:

java.io.FileNotFoundException: /home/myUser/server/optaplanner-simple-benchmark-daemon/local/output/
./run.sh: line 64: /home/myUser/server/optaplanner-simple-benchmark-daemon/local/processed/failed_machineReassignmentBenchmarkConfig.xml: No such file or directory
fatal: Could not change back to '(unreachable)/server/optaplanner-simple-benchmark-daemon/local/optaplannerGitClone/optaplanner': No such file or directory
ls: cannot access /home/myUser/server/optaplanner-simple-benchmark-daemon/local/input: No such file or directory
ls: cannot access /home/myUser/server/optaplanner-simple-benchmark-daemon/local/input: No such file or directory
ls: cannot access /home/myUser/server/optaplanner-simple-benchmark-daemon/local/input: No such file or directory
... // 1000+ more of that ls error

(Full source code)

Upvotes: 4

Views: 645

Answers (3)

Sohail
Sohail

Reputation: 4586

I always use the screen utility to run my scripts instead of nohup.

With screen your process will keep running in even your current ssh session times out or gets disconnected.

Use as follows -

apt-get install screen (On Debian based Systems)

OR

yum install screen (On RedHat based Systems)

To run your application and check the output live (provided your script file does not start a background process and it outputs to the stdout and/or stderr 

cd your_app_directory_path
screen ./your_script.sh

Once you are done and want to leave (without stopping the process), use CTRL + A + D to detach the screen.

To check your processes which are run using the screen utility - 

screen -r

to reattach a running process

screen -r <screen id or name>

Hope this was useful.

Upvotes: 1

Chris Gunawardena
Chris Gunawardena

Reputation: 6468

One workaround is to use screen to keep the ssh session open. You can use screen -r to reconnect to the session if you get disconnected.

Upvotes: 0

user3159253
user3159253

Reputation: 17455

well, it's not necessarily an encrypted home directory, but likely it's an auto-mounted home directory (e.g over NFS or so). It's mounted upon session startup, and unmounted upon exit. An encrypted home dir is only one of the possible reasons to use such a technique.

The main question is what's the rule that determines whether a user needs home dir or not. I would expect that it could be an allocated pty. You could test if it's actually true by starting a non-interactive SSH session w/o a pseudo-terminal: ssh -T user@host ls /home/myUser/server. I could expect that in this case you won't get a proper directory listing.

Then I would use a program like screen to prolongate interactive session lifetime beyond SSH session limits.

The server might use some other mechanism to provide the home directory for interactive SSH sessions. E.g. monitor interactive sessions listed in utmp In this case you will need a program that would keep the record as long as you need for your service. Perhaps you could use an automatically re-established SSH session. For example I use the following systemd unit to automatically keep a ssh tunnel from one of my workstations in different private networks:

[Unit]
Description=A tunnel to SOME_HOST
PartOf=sshd.service
Requires=network.service

[Service]
ExecStart=/usr/bin/ssh -N -q -R 2222:localhost:22 SOME_HOST
Restart=on-failure
RestartSec=5
User=tunnel
Group=tunnel

[Install]
WantedBy=sshd.service
WantedBy=network.service

When a failure occurs, systemd automatically restarts the unit and SSH session is re-established.

Upvotes: 6

Related Questions