CODEWITHSUNDEEP
python-2.7google-app-enginegoogle-cloud-platformtweepyapp-engine-ndb
Barbara
Barbara

Reputation: 1168

GAE - how to avoid service request timing out after 1 day

As I explained in this post, I'm trying to scrape tweets from Twitter.

I implemented the suggested solution with services, so that the actual heavy lifting happens in the backend.

The problem is that after about one day, I get this error

"Process terminated because the request deadline was exceeded. (Error code 123)"

I guess this is because the manual scaling has the requests timing out after 24 hours.

Is it possible to make it run for more than 24 hours?

Upvotes: 1

Views: 73

Answers (1)

Ying Li
Ying Li

Reputation: 2519

You can't make a single request / task run for more than 24 hours but you can split up your request into different parts and each lasting a day. It's unwise to have a request run indefinitely that's why app engine closes them after a certain time to prevent idling / loopy request that lasts indefinitely.

I would recommend having your task fire a call at the end to trigger the queuing of the next task, that way it's automatic and you don't have to queue a task daily. Make sure there's some cursor or someway for your task to communicate progress so it won't duplicate work.

Upvotes: 1

Related Questions