CODEWITHSUNDEEP
google-app-enginegoogle-cloud-platformgoogle-cloud-tasks
Alex
Alex

Reputation: 5276

Bulk Enqueue Google Cloud Tasks

As part of migrating my Google App Engine Standard project from python2 to python3, it looks like I also need to switch from using the Taskqueue API & Library to google-cloud-tasks.

In the taskqueue library I could enqueue upto 100 tasks at a time like this

taskqueue.Queue('default').add([...task objects...])

as well as enqueue tasks asynchronously.

In the new library as well as the new API, it looks like you can only enqueue tasks one at a time

I have an endpoint where it receives a batch with thousands of elements, each of which need to get processed in an individual task. How should I go about this?

Upvotes: 2

Views: 4026

Answers (2)

Artemis Georgakopoulou
Artemis Georgakopoulou

Reputation: 1221

According to the official documentation (reference 1, reference 2) the feature of adding task to queues asynchronously (as this post suggests for adding bulk number of tasks to a queue), is NOT an available feature via Cloud Tasks API. It is available for the users of App Engine SDK though.

However, there is a reference in the documentation regarding adding a large number of Cloud Tasks to a queue via double-injection pattern workaround (this post might seem useful too).

To implement this scenario, you'll need to create a new injector queue, whose single task would contain information to add multiple(100) tasks of the original queue that you're using. On the receiving end of this injector queue would be a service which does the actual addition of the intended tasks to your original queue. Although the addition of tasks in this service will be synchronous and 1-by-1, it will provide an asynchronous interface to your main application to bulk add tasks. In such a way you can overcome the limits of synchronous, 1-by-1 task addition in your main application.

  • Note that the 500/50/5 pattern of task addition to queue is a suggested method, in order to avoid any (queue/target) overloads.

As I did not find any examples of this implementation, I will edit the answer as soon as I find one.

Since you are in a migration process, I figured out that this link would be useful, as it concerns migrating from Task Queue to Cloud Tasks (as you stated you are thinking to do).

Additional information on migrating your code with all the available details you can find here and here, regarding Pull queues to Cloud Pub/Sub Migration and Push queues to Cloud Tasks Migration correspondingly.

Upvotes: 3

Averi Kitsch
Averi Kitsch

Reputation: 921

In order to recreate a batch pull mechanism, you would have to switch to Pub/Sub. Cloud Tasks does not have pull queues. With Pub/Sub you can batch push and batch pull messages.

If you are using a push queue architecture, I would recommend passing those elements as the task payload; however the max task size is limited to 100kb.

Upvotes: -1

Related Questions