CODEWITHSUNDEEP
javagoogle-app-engine
cht
cht

Reputation: 367

DeadlineExceededError workarounds

I'm making something like a web spider. It will catch the data i want form the WEB LIST. The amount of the WEB LIST is approximately 20. Each can de done around 1 to 10 seconds.

I know DeadlineExceededError means my request handler didn't finish within 30 seconds. I also know that I can't change the 30 second limit.

So I need to split my job into into small pieces but I don't know how to do that.

Here is my code:

    List<String> ids = agent.getWebList();

    if(ids.iterator().hasNext()) {
        for(String id : ids) {
            Data d = agent.getDetailedDataById(id);
            agent.updateData(d);
        }
    }

Upvotes: 1

Views: 567

Answers (3)

pahan
pahan

Reputation: 2453

If you are working on a web spider the best approach is to use task queues:

Java Task Queue docs

Python Task Queue docs

Upvotes: 3

Sam
Sam

Reputation: 1050

While iterating through your ids, keep a counter. When you catch the DeadLineExceededError, pass the counter as argument to the task which starts processing from that position. This will be done sequentially. You can also split the job and give to several tasks to process at same time.

Upvotes: 2

Chris Farmiloe
Chris Farmiloe

Reputation: 14175

Two DeadlineExceededErrors are raised. You can catch it the first time, and tidy up your state ready to continue again later or spawning a task for the next stage.

You can not catch DeadlineExceededError the second time.

Upvotes: 0

Related Questions