CODEWITHSUNDEEP
firebasegoogle-cloud-functions
jean d'arme
jean d'arme

Reputation: 4343

How to multiply Cloud Functions timeout time?

I know that Firebase/Google Cloud Functions can have timeout increased up to 9 minutes, but I have a Pub/Sub function that one request within it needs around 20-30 seconds to complete (document conversion).

async function () {
   // code...
   // const convertedDoc = await convertDocument()
   // ... do something with convertedDoc
}

With 9 minutes of maximum timeout it gives me up to 18 documents I can process.

My question is if after 15th document conversion I would call PubSub function again while finishing the previous invocation will timeout timer start over with new function? Of course, I would need to pass all the data from previous one, but is that a way to do it? Something like recursive PubSub of sorts?

Upvotes: 0

Views: 292

Answers (1)

al-dann
al-dann

Reputation: 2725

If you don't mind, I would suggest a slightly different approach.

Let's divide the whole process into 2 steps.

The first step - a cloud function which "collects" all documents (I mean their id, or reference, or metadata to uniquely distinct one from others) into a list, and then sends a message per document into a PubSub topic. That message contains a unique identifier/handle/hash of the document, so it can be fetch/processed later.

The PubSub topic triggers (push) the second cloud function. This cloud function is deployed with a maximum instances argument of a few dozens (or hundreds) depending on the context and requirements. Thus, many cloud function instances are being executed parallel, but each cloud function instance is triggered by a message with a unique document id.

The cloud function performs the processing you described, and presumably it takes 20 or 30 seconds. As many cloud functions are being executed in parallel, the overall processing time can be less, than if everything is done sequentially.

In addition, you might like to keep the state of a process in a firestore database using a document id as a firestore record id. Thus each record reflects a process of handling of one particular document. By doing that, any possible duplication can be eliminated, and a self-healing process can be organised.

Upvotes: 2

Related Questions