CODEWITHSUNDEEP
google-app-enginequeuetasktask-queue
ajgisme
ajgisme

Reputation: 1765

Google App Engine - Event when queue finishes

I'm starting to build a bulk upload tool and I'm trying to work out how to accomplish one of the requirements.

The idea is that a user will upload a CSV file and the tool will parse it and send each row of the CSV to the task queue as a task to be run. Then once all the tasks (relating to that specific CSV file) are completed, a summary report will be sent to the user.

I'm using Google App Engine and in the past I've used the standard Task Queue to handle tasks. However, with the standard Task Queue there is no way of knowing when the queue has finished, no event is fired to trigger the report generation so I'm not sure how to achieve this?

I've looked into it more and I understand that Google also offers Google PubSub. This is more sophisticated and seems more suited, but I still can't find out how to trigger and event when a PubSub queue is finished, any ideas?

Upvotes: 2

Views: 376

Answers (3)

minou
minou

Reputation: 16563

Seems that you could use a counter for this. Create an entity with an Integer property that is set to the number of lines of the CSV file. Each task will decrement the counter in a transaction when it finishes processing the row (in a transaction). One task will set the counter to 0, and that task could trigger the event. This might cause too much contention though.

Another possibility could be to have each task create an entity of a specific kind when it finishes processing a row. You can then count the number of these entities to determine when all the rows have been processed.

Upvotes: 2

Jonathan Laliberte
Jonathan Laliberte

Reputation: 2725

I faced a similar problem earlier this week and managed to find a nice workaround for it. What i did was i created an extra column in the table where a task inserts data into. And once a specific task is completed, it updates this 'task_status' column with 'done', otherwise it's left as the default null. Then when the user refreshes the page or goes to a specific URL or you do an AJAX call to query the task status for a specific id in your table, you can see if it is complete or not. 

select * from table where task_status is not null and id = ?;

You can also create a 'tasks' table where you can store relevant columns there instead of modifying existing tables.

Hope this finds you some use.

Upvotes: 0

Dan Cornilescu
Dan Cornilescu

Reputation: 39824

It might be easier to use the The GAE Pipeline API, which would take care of this as a basic portion of its functionality.

There's a nice article explaining it a bit here.

And a related SO question which happens to mention the same reason for moving to this API and has an excellent answer: Google AppEngine Pipelines API

I didn't use it myself yet, but it's just a matter of time :)

It's also possible to implement a scheme to track the related tasks still being active, see Figure out group of tasks completion time using TaskQueue and Datastore.

You can also check the queue (approximate) status, see Get number of tasks in a named queue?

Upvotes: 1

Related Questions