Reputation:

Django file upload failing occasionally

I am trying to port my first Django 1.0.2 application to run on OSX/Leopard with Apache + mod_python 3.3.1 + python 2.6.1 (all running in 64-bit mode) and I am experiencing an occasional error when uploading a file that was not present when testing with the Django development server.

The code for the upload is similar to what described in the Django documentation:

class UploadFileForm(forms.Form):
    file  = forms.FileField()
    description = forms.CharField(max_length=100)
    notifygroup = forms.BooleanField(label='Notify Group?', required=False)

def upload_file(request, date, meetingid ):
    print date, meetingid
    if request.method == 'POST':
        print 'before reloading the form...'
        form = UploadFileForm(request.POST, request.FILES)
    print 'after reloading the form'
        if form.is_valid():
            try: 
                handle_uploaded_file(request.FILES['file'], request.REQUEST['date'], request.REQUEST['description'], form.cleaned_data['notifygroup'], meetingid )
            except:
                return render_to_response('uploaded.html', { 'message': 'Error! File not uploaded!' })
            return HttpResponseRedirect('/myapp/uploaded/')
    else:
        form = UploadFileForm()
    return render_to_response('upload.html', {'form': form, 'date':date, 'meetingid':meetingid})

This code normally works correctly, but sometimes (say, once every 10 uploads) and after a fairly long waiting time, it fails with the following error:

IOError at /myapp/upload/2009-01-03/1
Client read error (Timeout?)
Request Method: POST
Request URL:    http://192.168.0.164/myapp/upload/2009-01-03/1
Exception Type: IOError
Exception Value:    
Client read error (Timeout?)
Exception Location: /Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/site-packages/django/http/multipartparser.py in read, line 406
Python Executable:  /usr/sbin/httpd
Python Version: 2.6.1
Python Path:    ['/djangoapps/myapp/', '/djangoapps/', '/Library/Frameworks/Python64.framework/Versions/2.6/lib/python26.zip', '/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6', '/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/plat-darwin', '/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/plat-mac', '/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages', '/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/lib-tk', '/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/lib-old', '/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/lib-dynload', '/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/site-packages']
Server time:    Sun, 4 Jan 2009 22:42:04 +0100

Environment:

Request Method: POST
Request URL: http://192.168.0.164/myapp/upload/2009-01-03/1
Django Version: 1.0.2 final
Python Version: 2.6.1
Installed Applications:
['django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.sites',
 'django.contrib.admin',
 'myapp.application1']
Installed Middleware:
('django.middleware.common.CommonMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware')

Traceback:
File "/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/site-packages/django/core/handlers/base.py" in get_response
  86.                 response = callback(request, *callback_args, **callback_kwargs)
File "/djangoapps/myapp/../myapp/application1/views.py" in upload_file
  137.         form = UploadFileForm(request.POST, request.FILES)
File "/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/site-packages/django/core/handlers/modpython.py" in _get_post
  113.             self._load_post_and_files()
File "/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/site-packages/django/core/handlers/modpython.py" in _load_post_and_files
  87.                 self._post, self._files = self.parse_file_upload(self.META, self._req)
File "/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/site-packages/django/http/__init__.py" in parse_file_upload
  124.         return parser.parse()
File "/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/site-packages/django/http/multipartparser.py" in parse
  134.             for item_type, meta_data, field_stream in Parser(stream, self._boundary):
File "/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/site-packages/django/http/multipartparser.py" in __iter__
  607.         for sub_stream in boundarystream:
File "/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/site-packages/django/http/multipartparser.py" in next
  421.             return LazyStream(BoundaryIter(self._stream, self._boundary))
File "/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/site-packages/django/http/multipartparser.py" in __init__
  447.         unused_char = self._stream.read(1)
File "/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/site-packages/django/http/multipartparser.py" in read
  300.         out = ''.join(parts())
File "/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/site-packages/django/http/multipartparser.py" in parts
  293.                 chunk = self.next()
File "/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/site-packages/django/http/multipartparser.py" in next
  315.             output = self._producer.next()
File "/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/site-packages/django/http/multipartparser.py" in next
  376.             data = self.flo.read(self.chunk_size)
File "/Library/Frameworks/Python64.framework/Versions/2.6/lib/python2.6/site-packages/django/http/multipartparser.py" in read
  406.         return self._file.read(num_bytes)

Exception Type: IOError at /myapp/upload/2009-01-03/1
Exception Value: Client read error (Timeout?)

I tried to run everything using mod_wsgi and no difference.

Does anybody know what am I doing wrong?

Thanks in advance for your help!

ppdo

=====

Updated:

Though I succeeded uploading large files (60+ MB), when it fails it fails with no evident relationship with the size of the upload, i.e. it fails also with 10kB files that have successfully been uploaded before.

Upvotes: 8

Answers (4)

Bryan

Reputation: 771

Long time unanswered here, but having experienced this myself and tried a few things to resolve it. It seems in my case, it happened if someone canceled a download (or lost connection) during an upload.

You can try this yourself to see if this is what is causing the "IOError: request data read error" for you: upload a large enough file that you can unplug your ethernet cable from your computer or router. Wait a bit and see if you get the error. I do everytime!

Upvotes: 0

ppdo

Reputation:

Using mod_wsgi made the problem go away for Firefox.

Limiting my research to an interaction problem between Apache and Safari, I stumbled upon this bug report for Apache https://bugs.webkit.org/show_bug.cgi?id=5760 that describes something very similar to what is happening and it is apparently still open. Reading this gave me the idea to try and disable the keepalive and, though I need to test it more extensively, it seems the problem is gone.

A simple:

BrowserMatch "Safari" nokeepalive

in the Apache configuration did the trick.

Upvotes: 8