CODEWITHSUNDEEP
pythonmultithreadingpython-2.7python-requestspython-multiprocessing
user7665219
user7665219

Reputation:

requests thread hanging in python

I have a serious issue with python multiprocessing module I am building a vulnerability management tool. It takes a list of linux packages then search for available CVEs per package by calling RedHat API. I am using multiprocessing.dummy to implement thread pool to achieve this work. The problem is that after successful calls for the API, my script is hanging/freezing and I have no way to debug that. Here is a piece of my code 

from multiprocessing.dummy import Pool as ThreadPool 

def check_vulnerability(package):
  done.append(package)
  try:
       time.sleep(1)
       headers = {'User-agent' : 'Mozilla/11.0'}
       with requests_retry_session() as s:
          s.headers.update(headers) 
          print "Processing package "+package+"=>"+str(float(len(done))/len(packages)*100)+"%"
          req = requests.get('https://access.redhat.com/labs/securitydataapi/cve.xml?package='+package, headers = headers, verify=True)
       if req.text != '':
          soup = BeautifulSoup(req.text, 'xml')
       else:
          pass
  except Exception as e:
       #print "#################################################################################"
       #print e
       #print packages"#################################################################################"
  #Some code to process req.text

def main():
   start_time = time.time()
   global packages 
   packages = fetch_packages_rpm() #list of strings ()
   p = ThreadPool(int(results.thread))
   all = p.map(check_vulnerability, packages)

After some successful prints, my script is hanging and no way to debug that (I think thread is waiting for ssl handshake or something like that and it is in waiter.aquire()). Any kind of help is highly appreciated

Upvotes: 2

Views: 1045

Answers (1)

user7665219
user7665219

Reputation:

I found a solution for this issue. The problem was the fact that the thread is still alive waiting SSL handshake with the server. What I did is to use threading instead of thread pool with multiprocessing.dummy. Then, use join() with timeout to terminate thread when the server does not respond by the timeout. 

        for index, package in enumerate(packages):
          t = threading.Thread(target=check_vulnerability, args=(package,))
          threads.append(t)
          t.daemon = True 
          t.start()
          t.join(0.5)
          t.isAlive()

Upvotes: 2

Related Questions