CODEWITHSUNDEEP
pythonmultiprocessing
Kyle Sponable
Kyle Sponable

Reputation: 735

Multiprocess wont execute simultaneously

Ive checked this all sorts of ways, not sure why my various processes wont start and run at the same time there is no error to google as it does work for the most part 

 from multiprocessing import Process, Queue, Event #for threading

 class WebScraper(object):
     def __init__(self, pq, e): 
         self.pdfQueue = pq
         self.e = e 

     def run(self):
         i = 0
         while True:
             print('tweet scraper online')
             i = i + 1    #testing
             if i == 10:  #testing
                 e.set()  #testing TODO remove

             if self.e.is_set():
                 print("killing tweet scraper")
                 break


 class DataParser(object):
      def __init__(self, pq, dq,  e):
          self.pdfQueue = pq
         self.dataQueue = dq
         self.e = e
         self.count = 0


      def run(self):

          while True:
              pdf = self.pdfQueue.get()
              print ('parsing pdf')
              self.e.set()
              if self.e.is_set():
                  print ('killing database parser')
                  break


 class DatabaseWriter(object):
     def __init__(self, dq, e): 
          self.dataQueue = dq
          self.e = e 

     def run(self):

         while True:
              print("database writer")
              if self.e.is_set():
                  print ('killing database writer')
                  break


 if __name__ == '__main__':

      print('\nStarting script')
      e = Event()
      pq = Queue()
      dq = Queue()
      ws = WebScraper(pq,  e)
      dp = DataParser(pq, dq, e)
      dw = DatabaseWriter(dq, e)

      p1 = Process(target=ws.run)
      p1.start()
      p2 = Process(target=dp.run)
      p2.start()
      p3 = Process(target=dw.run)
      p3.start()

      p1.join()
      p2.join()
      p3.join()

The output is as follows it runs quickly with nothing unexpected as you can see both processes 

 Starting script
 tweet scraper online
 tweet scraper online
 tweet scraper online
 tweet scraper online
 tweet scraper online
 tweet scraper online
 tweet scraper online
 tweet scraper online
 tweet scraper online
 tweet scraper online
 killing tweet scraper
 database writer
 killing database writer

If I remove the if statement in WebScraper class WebScraper and DatabaseWriter start but DatabaseParser does not look to be running

Upvotes: 1

Views: 125

Answers (1)

Ryan Villanueva
Ryan Villanueva

Reputation: 116

Your self.pdfQueue.get() is blocking the rest of the method in your DataParser instance. See https://docs.python.org/2/library/queue.html#Queue.Queue.get

Upvotes: 1

Related Questions