CODEWITHSUNDEEP
pythonmultithreadingloopsfor-loopparallel-processing
user1819779
user1819779

Reputation: 21

Multi-threading for embedded loops in python

I have a python code which does some manipulation of data contained in a Shapefile.

Among others stuffs, the code does this:

 xxx=0
 for i in channels:
      ptsi=mat(shapes[i].points)
      xx = ptsi[:,0]
      yy = ptsi[:,1]

      nanx=argwhere(isnan(xx))      
      nany=argwhere(isnan(yy))

      if (nanx == nany and len(nanx) != 0):
          xx[nanx] = []
          yy[nany] = []
      elif (len(nanx) != 0 or len(nany) != 0):
          xx = []
          yy = []

      if (len(xx) > 0):          
       yyy = 0 
       dd = str(i*100/N_channels) + '%\r\r'
      # os.write(1,dd)
       dist = zeros(len(xx)-1)

       dist = dist_min
       for j in channels:
           pts=mat(shapes[j].points)
           xx2 = pts[:,0]
           yy2 = pts[:,1]

           nanx=argwhere(isnan(xx2))          
           nany=argwhere(isnan(yy2))

           if (nanx == nany and len(nanx) != 0):
            xx2[nanx] = []
            yy2[nany] = []
           elif (len(nanx) != 0 or len(nany) != 0):
            xx2 = []
            yy2 = []
           if (len(xx2) > 0):  
            if (i != j): 
               ds = gaa(xx,yy,xx2[0],yy2[0])
               de = gaa(xx,yy,xx2[-1],yy2[-1])
               nande = ~isnan(de)
               nands = ~isnan(ds)
               fe = np.argwhere(de<=dist)
               fs = np.argwhere(ds<=dist)
               nozeroe = array(where(de != 0))
               nozeroe = nozeroe[0]
               nozeros = array(where(ds != 0))
               nozeros = nozeros[0] 
               if(fs.size >0):
                    iis=array(np.argwhere(array(ds==min(ds))))

                    iis = iis.flatten()
                    iis = iis[0]
                    p1 = xxx + iis 
                    p2 = yyy
                    G.add_edge(p2,p1,weight=w_con)

               elif (fe.size > 0):
                    iie=array(np.argwhere(array(de==min(de))))
                    iie = iie.flatten()
                    iie = iie[0]
                    p1 = xxx + iie
                    p2 = yyy  + len(pts) -1
                    G.add_edge(p2,p1,weight=w_con)

           yyy = yyy + len(pts)

      xxx = xxx + len(ptsi)

Basically, the code scans over polylines and search for a minimal distance in order to merge then in a common graph (using Networkx). This part works fine except that this is really slow as I deal with more than 100's of thousand of objects (It takes roughly 20 hours in its current version).

These embedded loops are not efficient, so I'd like to know, if using multithreading could be helpful and if so, how can I modify this very part of the code? I'm fine with CUDA or OpenCL if it can help.

Thanks for any feedback!

Upvotes: 2

Views: 384

Answers (2)

Josh Heitzman
Josh Heitzman

Reputation: 1834

Due to the Global Interpreter Lock, Python code isn't able to fully utilize multiple cores via multi-threading and needs to use multi-processing to be able to fully utilize multiple cores.

Upvotes: 1

pydsigner
pydsigner

Reputation: 2885

More threads will not help; more processes might, but if you can use CUDA, that would probably be a good move. A thread is just a way to allow multiple things to share a processes CPU time, it does not speed up slow code, but will just slow you down more.

Upvotes: 0

Related Questions