CODEWITHSUNDEEP
pythonmultithreading
dede
dede

Reputation: 726

Threading and interpreter shutdown

I have this piece of python code:

def __init__(self):
  self.ip_list=[]
  self.queue=Queue()

  for i in range(5):
    worker=threading.Thread(target=self.__executeCmd, name="executeCmd("+str(i)+")")
    worker.setDaemon(True)
    worker.start()
  self.queue.put(["wget", "-qO-", "http://ipecho.net/plain"])
  self.queue.put(["curl", "http://www.networksecuritytoolkit.org/nst/cgi-bin/ip.cgi"])
  self.queue.put(["curl", "v4.ident.me"])
  self.queue.put(["curl", "ipv4.icanhazip.com"])
  self.queue.put(["curl", "ipv4.ipogre.com"])

def __executeCmd(self):
  cmd=self.queue.get()
  try:
    rc=subprocess.check_output(cmd, stderr=open(os.devnull, 'w')).strip()
  except:
    self.queue.task_done()
    return
  if self.is_valid_ip(rc)==True:
    self.ip_list.append(rc)
  self.queue.task_done()

def waitForIP(self, wait_in_sec):
  cnt=wait_in_sec*10
  while self.ip_list==[]:
    time.sleep(0.1)
    cnt-=1
    if cnt<=0:
      return("")
  return(self.ip_list[0])

Its for querying the external IP address from five URLs and get the response from that one that was delivered first.

But sometimes I get this (and I get it via email because the job was started via crontab):

Exception in thread executeCmd(0) (most likely raised during interpreter shutdown):
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
  File "/usr/lib/python2.7/threading.py", line 505, in run
  File "/home/dede/bin/tunnel_watchdog.py", line 115, in __executeCmd
  File "/usr/lib/python2.7/Queue.py", line 65, in task_done
  File "/usr/lib/python2.7/threading.py", line 296, in notifyAll
<type 'exceptions.TypeError'>: 'NoneType' object is not callable

I think its because the script has ended but a thread was still running and then came out of subprocess.check_output().

Is there a way to avoid this (without waiting for all five URLs delivered their data) ?

Upvotes: 0

Views: 1080

Answers (1)

johntellsall
johntellsall

Reputation: 15170

The project is much simpler than it looks. Here's one implementation using the multiprocessing module.

The function imap_unordered runs the jobs in parallel, and returns the first-completed one first. The outer level function checks the result. If the result is okay, it's printed, then the pool is terminated and the entire program exits. It doesn't wait for the other jobs to complete.

import multiprocessing, re, subprocess, sys

CMD_LIST = [
    ["wget", "-qO-", "http://ipecho.net/plain"],
    ["curl", '-s', "http://www.networksecuritytoolkit.org/nst/cgi-bin/ip.cgi"],
    ["curl", '-s', "v4.ident.me"],
    ["curl", '-s', "ipv4.icanhazip.com"],
    ["curl", '-s', "ipv4.ipogre.com"],
]


ip_pat = re.compile('[0-9.]{7,}')
pool = multiprocessing.Pool(5)
for output in pool.imap_unordered(subprocess.check_output, CMD_LIST):
    print 'output:',output
    m = ip_pat.search(output)
    if m:
        print 'GOT IP:', m.group(0)
        pool.terminate()
        sys.exit(0)

print 'no IP found'

Upvotes: 1

Related Questions