CODEWITHSUNDEEP
pythondjangodjango-celery
Prometheus
Prometheus

Reputation: 33615

Django TimeLimitExceeded error

Today I got an error email from Celery, can someone explain it and possibly how I can fix the timeout issue? It would be very helpful, Thank you.

PS my message seems to have sent despite this error, is that also right?

Error:

Task Request to Process with id 65123935-b190-4718-9ed0-fb863359f27f 
 raised exception:
'TimeLimitExceeded(300.0,)'


Task was called with args: (<Batch: Batch object>,) kwargs: {}.

The contents of the full traceback was:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/billiard/pool.py", line 496, in on_hard_timeout
    raise TimeLimitExceeded(job._timeout)
TimeLimitExceeded: TimeLimitExceeded(300.0,)


--
Just to let you know,
py-celery at w1.ip-10-32-53-113.

Task:

class ProcessRequests(Task):
    name = "Request to Process"
    max_retries = 1
    default_retry_delay = 3

    def run(self, batch):
        # Only run this task on non-scheduled tasks
        if batch.status != "Scheduled":
            q = Contact.objects.filter(contact_owner=batch.user)
            if batch.group == None:
                q = q.filter(id=batch.contact_id)
            else:
                q = q.filter(group=batch.group)

            for e in q:
                msg = Message.objects.create(
                    recipient_number=e.mobile,
                    content=batch.content,
                    sender=e.contact_owner,
                    billee=batch.user,
                    sender_name=batch.sender_name
                )
                gateway = Gateway.objects.get(pk=2)
                msg.send(gateway)

Upvotes: 4

Views: 4498

Answers (4)

ElmoVanKielmo
ElmoVanKielmo

Reputation: 11290

You are sending messages to mobile phones with msg.send(gateway), right?

Inside this method, you are connecting to remote webservice, which actually sends the message, right?

Then, your message might have been sent, but your connection to remote party hasn't been closed, despite webservice should have closed it, after sending message and a response for your request.

The last thing - you haven't done the following before creating socket used to connect with webservice:

import socket
socket.setdefaulttimeout(seconds) # seconds argument is of type float

This way, your task is hanging at the socket for infinite time waiting for webservice to send response and/or close the connection. In fact it would wait forever, if not interrupted by celery timeout.

Default socket timeout value is None as the documentation states http://docs.python.org/2/library/socket.html#socket.setdefaulttimeout

Hope this is helpful.

Upvotes: 1

Michael Anckaert
Michael Anckaert

Reputation: 953

You receive the TimeLimitExceeded exception because your task took longer than 300 seconds to execute. 300 Seconds is the default time a task can run.

As answered by Avichal Badaya, you can configure Celery to allow for longer executing tasks.

You mentioned that your message was still sent but this can happen because your tasks is written for a batch. So either you're trying to send multiple messages or you're only trying to send one message but some how mess up the batch.

Can you show us how you call the task?

Upvotes: 2

Haridas N
Haridas N

Reputation: 549

First you have to know actually what you are doing when you get an message on your task function/method, and its time complexity. If your task takes more than default 300 Second to finish then celery main process will first Send SoftTimeLimitExceed exception that you can catch on your task if we enclose it in try/except block. This exception is actually for cleanup process if any. Right after this exception celery will send you another exception TimeLimitExceed exception and it will kill your worker thread.

If you enabled late ACK setting on celery setting then the same message will be delivered to another worker to process ( This is dangerous since that also going to fail ).

So if you get this exception then this means that your task requires more than 300 Seconds to finish, so adjust the maximum time limit for the worker on the celery settings to suit your requirement.

Regards,

Haridas N.

Upvotes: 1

Avichal Badaya
Avichal Badaya

Reputation: 3619

Can you try modifying celeryd settings . file should mostly be /etc/init.d/celeryd . 

CELERYD_OPTS="--time-limit==3600 -E --loglevel=DEBUG"

Upvotes: 5

Related Questions