CODEWITHSUNDEEP
pythonpython-itertools
pepoluan
pepoluan

Reputation: 6780

How is it possible that Python's itertools.count does not increase?

I have a code that spins off new multiprocessing Process-es, giving them a serialized name using itertools.count to auto-increment the serial.

This is a glimpse of the code:

class WorkerStation(object):

    def __init__(self, basename, factory):
        # factory is a subclass of multiprocessing.Process
        self.basename = basename
        self.factory = factory
        self.serie = itertools.count(start=1)
        self.workers = []

    def launch_workers(want_count):
        while _count_active() < want_count:
            name = '{0}-{1}'.format(self.basename, self.serie.next())
            new_worker = self.factory(name=name)
            new_worker.start()
            print('Launched ', name)
            self.workers.append(new_worker)

    def _count_active():
        # It is possible as we spun off new workers, some earlier workers 
        # have terminated. The is_alive() also join() these deceased
        # workers.
        self.workers = [w for w in self.workers if w.is_alive()]
        return len(self.workers)

Now if I instantiate the WorkerStation for the first time, e.g.:

station_1 = WorkerStation(basename="st1", factory=WorkerClassOne)
station_1.launch_workers(want_count=5)

I can see the serie to be increasing properly, i.e., I'll see Launched st1-1 up to Launched st1-5

But interestingly, the second time I instantiate this class:

station_2 = WorkerStation(basename="st2", factory=WorkerClassTwo)
station_2.launch_workers(want_count=2)

I'll see serie no longer incrementing, i.e., I'll see TWO lines of Launched st2-1.

How is this possible? How could itertools.count work for the first instantiaton, but no longer work for the second?

Upvotes: 1

Views: 306

Answers (1)

pepoluan
pepoluan

Reputation: 6780

Okay folks, it's apparently a mistake in the code.

I'll paste here (part of) two methods whose interaction caused this grief:

    def maintain_flights(self, target=None, force=False):
        # If target not specified, invoke the auto_count() method to automatically
        # determine the number of workers needed.
        target = target or self.auto_count()
        ... lots of code removed ...

    def rewind(self):
        self.maintain_flights(target=0, force=True)
        self.serie = itertools.count(start=1)

The main part of the program, during certain stages, will invoke the .rewind() method to reduce the number of workers to 0.

As you can see, invoking the method with target=0 will cause the left side of the target or self.auto_count() expression to be false, resulting the code to invoke the self.auto_count() method.

So, instead of ending with zero workers, I got 1 worker.

Afterwards, the self.serie object got 'reset' to start counting at 1.

A bit later in the main code, the main code asks for 2 workers. It already got 1 worker from the botched rewind() attempt, and it added the second worker with serie again starting from 1.

I've replaced the offending code with this line:

target = target if target is not None else self.auto_count()

And now it works splendidly!

Thanks for the assistance, and apologies if I've sent you all on a wild goose-chase.

Upvotes: 1

Related Questions