CODEWITHSUNDEEP
pythonmultithreadingparallel-processingiteration
Tyranicangel
Tyranicangel

Reputation: 189

Running parallel iterations

I am trying to run a sort of simulations where there are fixed parameters i need to iterate on and find out the combinations which has the least cost.I am using python multiprocessing for this purpose but the time consumed is too high.Is there something wrong with my implementation?Or is there better solution.Thanks in advance

    import multiprocessing
    class Iters(object):
        #parameters for iterations
        iters['cwm']={'min':100,'max':130,'step':5}
        iters['fx']={'min':1.45,'max':1.45,'step':0.01}
        iters['lvt']={'min':106,'max':110,'step':1}
        iters['lvw']={'min':9.2,'max':10,'step':0.1}
        iters['lvk']={'min':3.3,'max':4.3,'step':0.1}
        iters['hvw']={'min':1,'max':2,'step':0.1}
        iters['lvh']={'min':6,'max':7,'step':1}

        def run_mp(self):
            mps=[]
            m=multiprocessing.Manager()
            q=m.list()
            cmain=self.iters['cwm']['min']
            while(cmain<=self.iters['cwm']['max']):
                t2=multiprocessing.Process(target=mp_main,args=(cmain,iters,q))
                mps.append(t2)
                t2.start()
                cmain=cmain+self.iters['cwm']['step']
            for mp in mps:
                mp.join()
            r1=sorted(q,key=lambda x:x['costing'])
            returning=[r1[0],r1[1],r1[2],r1[3],r1[4],r1[5],r1[6],r1[7],r1[8],r1[9],r1[10],r1[11],r1[12],r1[13],r1[14],r1[15],r1[16],r1[17],r1[18],r1[19]]
            self.counter=len(q)
            return returning

    def mp_main(cmain,iters,q):
        fmain=iters['fx']['min']
        while(fmain<=iters['fx']['max']):
            lvtmain=iters['lvt']['min']
            while (lvtmain<=iters['lvt']['max']):
                lvwmain=iters['lvw']['min']
                while (lvwmain<=iters['lvw']['max']):
                    lvkmain=iters['lvk']['min']
                    while (lvkmain<=iters['lvk']['max']):
                        hvwmain=iters['hvw']['min']
                        while (hvwmain<=iters['hvw']['max']):
                            lvhmain=iters['lvh']['min']
                            while (lvhmain<=iters['lvh']['max']):
                                test={'cmain':cmain,'fmain':fmain,'lvtmain':lvtmain,'lvwmain':lvwmain,'lvkmain':lvkmain,'hvwmain':hvwmain,'lvhmain':lvhmain}
                                y=calculations(test,q)
                                lvhmain=lvhmain+iters['lvh']['step']
                            hvwmain=hvwmain+iters['hvw']['step']
                        lvkmain=lvkmain+iters['lvk']['step']
                    lvwmain=lvwmain+iters['lvw']['step']
                lvtmain=lvtmain+iters['lvt']['step']
            fmain=fmain+iters['fx']['step']

    def calculations(test,que):
        #perform huge number of calculations here
        output={}
        output['data']=test
        output['costing']='foo'
        que.append(output)

    x=Iters()
    x.run_thread()

Upvotes: 0

Views: 61

Answers (1)

loopbackbee
loopbackbee

Reputation: 23342

From a theoretical standpoint:

You're iterating every possible combination of 6 different variables. Unless your search space is very small, or you wanted just a very rough solution, there's no way you'll get any meaningful results within reasonable time.

i need to iterate on and find out the combinations which has the least cost

This very much sounds like an optimization problem.

There are many different efficient ways of dealing with these problems, depending on the properties of the function you're trying to optimize. If it has a straighforward "shape" (it's injective), you can use a greedy algorithm such as hill climbing, or gradient descent. If it's more complex, you can try shotgun hill climbing.

There are a lot more complex algorithms, but these are the basic, and will probably help you a lot in this situation.

From a more practical programming standpoint:

You are using very large steps - so large, in fact, that you'll only probe the function 19,200. If this is what you want, it seems very feasible. In fact, if I comment the y=calculations(test,q), this returns instantly on my computer.

As you indicate, there's a "huge number of calculations" there - so maybe that is your real problem, and not the code you're asking for help with.

As to multiprocessing, my honest advise is to not use it until you already have your code executing reasonably fast. Unless you're running a supercomputing cluster (you're not programming a supercomputing cluster in python, are you??), parallel processing will get you speedups of 2-4x. That's absolutely negligible, compared to the gains you get by the kind of algorithmic changes I mentioned.

As an aside, I don't think I've ever seen that many nested loops in my life (excluding code jokes). If don't want to switch to another algorithm, you might want to consider using itertools.product together with numpy.arange

Upvotes: 1

Related Questions