CODEWITHSUNDEEP
google-app-engine
Joel
Joel

Reputation: 6107

Calling a script after tasks queue is empty

I am running a script which initiates several tasks:

for i in range (0,5):
  taskqueue.add(url='/example', 
                          params={'num': i})

As far as I understand, the tasks are running in parallel. Is there anyway I can tell AppEngine to run a specific task/python file once all the tasks I just inserted to the queue are ALL finished? I thought about sending a flag to the task that was called in the last loop iteration, but if tasks run in parallel it is not guarnteed that once it is finished, the other ones were finished too.

Thanks,

Joel

Upvotes: 5

Views: 375

Answers (1)

Robert Kluin
Robert Kluin

Reputation: 8292

When you initiate the tasks, you know how many there will be. Insert an entity into the datastore with the 'expected' count, or list of tasks. Then use either counters or markers to indicate when a task has run. Roughly, the process might look something like:

The new kinds:

class TaskBatch(db.Model):
    expected_count = db.IntegerProperty()

class TaskMarker(db.Model):
    pass

Then, adjust your calling routine to do something like:

count = 5
taskbatch = TaskBatch(expected_count=count).put()
for i in range(5):
    taskqueue.add(url='/example',
                  params={'num': i, 'batch': str(taskbatch)})

And, at the end of your tasks:

def post(self):
    num = self.request.get('num')
    # do your stuff....

    batch = self.request.get('batch')
    TaskMarker(key_name=num, parent=db.Key(batch))
    taskqueue.add(url='/example/isdone',
                  params={'num': i, 'batch': str(taskbatch)})

And the isdone task could look something like:

def post(self):
    num = self.request.get('num')
    batch_key = db.Key(self.request.get('batch'))
    batch = TaskBatch.get(batch_key)
    marker_keys = [db.Key.from_path('TaskMarker', i, parent=batch)
                     for i in range(batch.expected_count)]
    markers = db.get(marker_keys)
    if all(markers):
       # do your done action.

The exact process will vary slightly depending on the specifics of your usecase, such as how many tasks your inserting.

Upvotes: 4

Related Questions