CODEWITHSUNDEEP
pythondjangocelery
Valentin Kantor
Valentin Kantor

Reputation: 1844

passing file instance as an argument to the celery task raises "ValueError: I/O operation on closed file"

I need to pass file as argument to the celery task, but the passed file somehow got there closed. It happens just in case I'm executing the task asynchronous way. Is this an expected behavior?

views:

from engine.tasks import s3_upload_handler
def myfunc():
    f = open('/app/uploads/pic.jpg', 'rb')
    s3_file_handler.apply_async(kwargs={"uploaded_file" : f,"file_name" : "test.jpg"})

tasks:

def s3_upload_handler(uploaded_file,file_name):
    ...
    #some code for uploading to s3

traceback:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/app/photohosting/engine/tasks.py", line 34, in s3_upload_handler
    key.set_contents_from_file(uploaded_file)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/key.py", line 1217, in set_contents_from_file
    spos = fp.tell()
ValueError: I/O operation on closed file

flower logs:

kwargs  {
         'file_name': 'test.jpg', 
         'uploaded_file': <closed file '<uninitialized file>', 
          mode '<uninitialized file>' at 0x7f6ab9e75e40>
         }

Upvotes: 6

Views: 5685

Answers (2)

Raghuram Onti Srinivasan
Raghuram Onti Srinivasan

Reputation: 755

Another way of doing this would be to open the file and get a binary blob which you transfer over the wire. Of course if the file is really large then what @Vasily says is better but wont work in case of the worker running on a different m/c (unless your file is on a shared storage).

Upvotes: 0

Spc_555
Spc_555

Reputation: 4121

Yes, of course, the file would get there closed. Asynchronous celery tasks run in a completely separate process (moreover, they can even run on a different machine) and there is no way to pass an open file to it.

You should close the file in the process from where you call the task, and then pass its name and maybe position in file (if you need it) to the task and then reopen it in the task.

Upvotes: 5

Related Questions