CODEWITHSUNDEEP
pythonlinuxperldevopsnohup
kumarprd
kumarprd

Reputation: 936

timeout limit for holding exit status from system in perl/python

I have a simple perl script that calls another python script to do the deployment of a server in cloud .

I capture the exit status of the deployment inside perl to take any further action after success/failure setup.

It's like: 

$cmdret = system("python script.py ARG1 ARG2");

Here the python script runs for 3hrs to 7 hrs.

The problem here is that, irrespective of the success or failure return status, the system receive a Signal HUP at this step randomly even if the process is running in backened and breaks the steps further.

So does anyone know, if there is any time limit for holding the return status from the system which leads to sending Hangup Signal?

Inside the python script script.py, pexpect is used execute scripts remotely:

doSsh(User,Passwd,Name,'cd '+OutputDir+';python host-bringup.py setup')
doSsh(User,Passwd,Name,'cd '+OpsHome+'/ops/hlevel;python  dshost.py start')
....

And doSsh is a pexpect subroutine:

def doSsh(user,password,host,command):
    try:
        child =  pexpect.spawn("ssh  -o ServerAliveInterval=100 -n %s@%s '%s'" % (user,host,command),logfile=sys.stdout,timeout=None)
        i = child.expect(['password:', r'\(yes\/no\)',r'.*password for paasusr: ',r'.*[$#] ',pexpect.EOF])
        if i == 0:
                child.sendline(password)
        elif i == 1:
                child.sendline("yes")
                child.expect("password:")
                child.sendline(password)
        data = child.read()
        print data
        child.close()
        return True
    except Exception as error:
        print error
        return False

This first doSsh execution takes ~6 hours and this session is killed after few hours of execution with the message : Signal HUP caught; exitingbut the execution python host-bringup.py setup still runs in the remote host.

So in the local system, the next doSsh never runs and also the rest steps inside the perl script never continue.

Upvotes: 1

Views: 242

Answers (2)

ikegami
ikegami

Reputation: 385735

SIGHUP is sent when the terminal disconnects. When you want to create a process that's not tied to the terminal, you daemonize it.

Note that nohup doesn't deamonize.

$ nohup perl -e'system "ps", "-o", "pid,ppid,sid,cmd"'
nohup: ignoring input and appending output to `nohup.out'

$ cat nohup.out
  PID  PPID   SID CMD
21300 21299 21300 -bash
21504 21300 21300 perl -esystem "ps", "-o", "pid,ppid,sid,cmd"
21505 21504 21300 ps -o pid,ppid,sid,cmd

As you can see,

  1. perl's PPID is that of the program that launched it.
  2. perl's SID is that of the program that launched it.

Since the session hasn't changed, the terminal will send SIGHUP to perl when it disconnects as normal.

That said, nohup changes how perl's handles SIGHUP by causing it to be ignored.

$ perl -e'system "kill", "-HUP", "$$"; print "SIGHUP was ignored\n"'
Hangup

$ echo $?
129

$ nohup perl -e'system "kill", "-HUP", "$$"; print "SIGHUP was ignored\n"'
nohup: ignoring input and appending output to `nohup.out'

$ echo $?
0

$ tail -n 1 nohup.out
SIGHUP was ignored

If perl is killed by the signal, it's because something changed how perl handles SIGHUP.

So, either daemonize the process, or have perl ignore use SIGHUP (e.g. by using nohup). But if you use nohup, don't re-enable the default SIGHUP behaviour!

Upvotes: 2

Doggerel
Doggerel

Reputation: 87

If your goal is to make your perl program ignore the HUP signal, you likely just need to set the HUP entry of the $SIG global signal handler hash:

$SIG{ 'HUP' } = 'IGNORE';

for gory details, see 

perldoc perlipc

Upvotes: 1

Related Questions