Can I convert a django video upload from a form using ffmpeg before storing the video?

Question

I've been stuck for weeks trying to use ffmpeg to convert user uploaded videos to flv. I use heroku to host my website, and store my static and media files on amazon S3 with s3boto. The initial video file will upload fine, however when I retrieve the video and run a celery task (in the same view where the initial video file is uploaded), the new file won't store on S3. I've been trying to get this to work for over a month, with no luck, and really no good resources available for learning how to do this, so I figure maybe if I can get the ffmpeg task to run before storing the video I may be able to get it to work. Unfortunately I'm still not a very advanced at python (or django), so I don't even know if/how this is possible. Anyone have any ideas? I am willing to use any solution at this point no matter how ugly, as long as it successfully takes video uploads and converts to flv using ffmpeg, with the resulting file being stored on S3. It doesn't seem that my situation is very common, because no matter where I look, I cannot find a solution that explains what I should be trying to do. Therefore I will be very appreciative of any guidance. Thanks. My relevant code follows:

#models.py
def content_file_name(instance, filename):
    ext = filename.split('.')[-1]
    new_file_name = "remove%s.%s" % (uuid.uuid4(), ext)
    return '/'.join(['videos', instance.teacher.username, new_file_name])

class BroadcastUpload(models.Model):
    title = models.CharField(max_length=50, verbose_name=_('Title'))
    description = models.TextField(max_length=100, verbose_name=_('Description'))
    teacher = models.ForeignKey(User, null=True, blank=True, related_name='teacher')
    created_date = models.DateTimeField(auto_now_add=True)
    video_upload = models.FileField(upload_to=content_file_name)
    flvfilename = models.CharField(max_length=100, null=True, blank=True)
    videothumbnail = models.CharField(max_length=100, null=True, blank=True)

#tasks.py
@task(name='celeryfiles.tasks.convert_flv')
def convert_flv(video_id):
    video = BroadcastUpload.objects.get(pk=video_id)
    print "ID: %s" % video.id
    id = video.id
    print "VIDEO NAME: %s" % video.video_upload.name
    teacher = video.teacher
    print "TEACHER: %s" % teacher
    filename = video.video_upload
    sourcefile = "%s%s" % (settings.MEDIA_URL, filename)
    vidfilename = "%s_%s.flv" % (teacher, video.id)
    targetfile = "%svideos/flv/%s" % (settings.MEDIA_URL, vidfilename)
    ffmpeg = "ffmpeg -i %s %s" % (sourcefile, vidfilename)
    try:
        ffmpegresult = subprocess.call(ffmpeg)
        #also tried separately with following line:
        #ffmpegresult = commands.getoutput(ffmpeg)
        print "---------------FFMPEG---------------"
        print "FFMPEGRESULT: %s" % ffmpegresult
    except Exception as e:
        ffmpegresult = None
        print("Failed to convert video file %s to %s" % (sourcefile, targetfile))
        print(traceback.format_exc())
    video.flvfilename = vidfilename
    video.save()

@task(name='celeryfiles.tasks.ffmpeg_image')        
def ffmpeg_image(video_id):
    video = BroadcastUpload.objects.get(pk=video_id)
    print "ID: %s" %video.id
    id = video.id
    print "VIDEO NAME: %s" % video.video_upload.name
    teacher = video.teacher
    print "TEACHER: %s" % teacher
    filename = video.video_upload
    sourcefile = "%s%s" % (settings.MEDIA_URL, filename)
    imagefilename = "%s_%s.png" % (teacher, video.id)
    thumbnailfilename = "%svideos/flv/%s" % (settings.MEDIA_URL, thumbnailfilename)
    grabimage = "ffmpeg -y -i %s -vframes 1 -ss 00:00:02 -an -vcodec png -f rawvideo -s 320x240 %s" % (sourcefile, thumbnailfilename)
    try:        
         videothumbnail = subprocess.call(grabimage)
         #also tried separately following line:
         #videothumbnail = commands.getoutput(grabimage)
         print "---------------IMAGE---------------"
         print "VIDEOTHUMBNAIL: %s" % videothumbnail
    except Exception as e:
         videothumbnail = None
         print("Failed to convert video file %s to %s" % (sourcefile, thumbnailfilename))
         print(traceback.format_exc())
    video.videothumbnail = imagefilename
    video.save()

#views.py
def upload_broadcast(request):
    if request.method == 'POST':
        form = BroadcastUploadForm(request.POST, request.FILES)
        if form.is_valid():
            upload=form.save()
            video_id = upload.id
            image_grab = ffmpeg_image.delay(video_id)
            video_conversion = convert_flv.delay(video_id)
            return HttpResponseRedirect('/current_classes/')
    else:
        form = BroadcastUploadForm(initial={'teacher': request.user,})
    return render_to_response('videos/create_video.html', {'form': form,}, context_instance=RequestContext(request))

#settings.py
DEFAULT_FILE_STORAGE = 'myapp.s3utils.MediaRootS3BotoStorage'
DEFAULT_S3_PATH = "media"
STATICFILES_STORAGE = 'myapp.s3utils.StaticRootS3BotoStorage'
STATIC_S3_PATH = "static"
AWS_STORAGE_BUCKET_NAME = 'my_bucket'
CLOUDFRONT_DOMAIN = 'domain.cloudfront.net'
AWS_ACCESS_KEY_ID = 'MY_KEY_ID'
AWS_SECRET_ACCESS_KEY = 'MY_SECRET_KEY'
MEDIA_ROOT = '/%s/' % DEFAULT_S3_PATH
MEDIA_URL = 'http://%s/%s/' % (CLOUDFRONT_DOMAIN, DEFAULT_S3_PATH)
...

#s3utils.py
from storages.backends.s3boto import S3BotoStorage
from django.utils.functional import SimpleLazyObject

StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static')
MediaRootS3BotoStorage  = lambda: S3BotoStorage(location='media')

I can add any other info if needed to help me solve my problem.

Answer 1

I've done some modifications to your code and now it is working. Some issues:

• I've put the call of the tasks inside the save method of the model. You can put it in views if you want, as you did.
• my solution is to transcode to a temporary file and then upload it to AWS S3.

Here is the code:

from django.contrib.auth.models import User

def content_file_name(instance, filename):
    ext = filename.split('.')[-1]
    new_file_name = "remove%s.%s" % (uuid.uuid4(), ext)
    return '/'.join(['videos', instance.teacher.username, new_file_name])

class BroadcastUpload(models.Model):
    title = models.CharField(max_length=50, verbose_name=_('Title'))
    description = models.TextField(max_length=100, verbose_name=_('Description'))
    teacher = models.ForeignKey(User, null=True, blank=True, related_name='teacher')
    created_date = models.DateTimeField(auto_now_add=True)
    video_upload = models.FileField(upload_to=content_file_name)
    #flvfilename = models.CharField(max_length=100, null=True, blank=True)
    video_flv = models.FileField(upload_to='flv', blank=True)
    #videothumbnail = models.CharField(max_length=100, null=True, blank=True)
    video_thumbnail = models.FileField(upload_to='thumbs', blank=True)

    def __unicode__(self):
        return u'%s - %s' % (self.title, self.teacher)

    def save(self, *args, **kwargs):
        # optional parameter to indicate whether perform
        # conversion or not. Defaults to True
        do_conversion = kwargs.pop('do_conversion', True)

        # do something only when the entry is created
        if not self.pk:
            super(BroadcastUpload, self).save(*args, **kwargs)

        # do something every time the entry is updated
        if do_conversion:
            ffmpeg_image.delay(self.pk)
            convert_flv.delay(self.pk)

        # then call the parent save:
        super(BroadcastUpload, self).save(*args, **kwargs)

from django.core.files.uploadedfile import SimpleUploadedFile
import tempfile

@task
def convert_flv(video_id):
    video = BroadcastUpload.objects.get(pk=video_id)
    print "ID: %s" % video.id
    id = video.id
    print "VIDEO NAME: %s" % video.video_upload.name
    teacher = video.teacher
    print "TEACHER: %s" % teacher
    filename = video.video_upload
    #sourcefile = "%s%s" % (settings.MEDIA_URL, filename)
    sourcefile = video.video_upload.url
    # ffmpeg cannot deal with https?
    sourcefile = sourcefile.replace("https","http")
    print "sourcefile: %s" % sourcefile

    # temporary output image
    OUTPUT_VIDEO_EXT = 'flv'
    OUTPUT_VIDEO_CONTENT_TYPE = 'video/flv'    
    f_out = tempfile.NamedTemporaryFile(suffix=".%s"%OUTPUT_VIDEO_EXT, delete=False)
    tmp_output_video = f_out.name

    #ffmpeg = "ffmpeg -i '%s' -qscale 0 -ar 44100 '%s'" % (sourcefile, vidfilename)
    ffmpeg = "ffmpeg -y -i '%s' -qscale 0 -ar 44100 '%s'" % (sourcefile, tmp_output_video)
    print "convert_flv: %s" % ffmpeg
    try:
        ffmpegresult = subprocess.call(ffmpeg, shell=True)
        #also tried separately with following line:
        #ffmpegresult = commands.getoutput(ffmpeg)
        print "---------------FFMPEG---------------"
        print "FFMPEGRESULT: %s" % ffmpegresult
    except Exception as e:
        ffmpegresult = None
        #print("Failed to convert video file %s to %s" % (sourcefile, targetfile))
        print("Failed to convert video file %s to %s" % (sourcefile, tmp_output_video))
        #print(traceback.format_exc())
        print "Error: %s" % e

    #vidfilename = "%s_%s.flv" % (teacher, video.id)
    vidfilename = "%s_%s.%s" % (teacher, video.id, OUTPUT_VIDEO_EXT)
    #targetfile = "%svideos/flv/%s" % (settings.MEDIA_URL, vidfilename)

    # prepare an object with the generated temporary image
    suf = SimpleUploadedFile( 
                             vidfilename,
                             f_out.read(),
                             content_type=OUTPUT_VIDEO_CONTENT_TYPE
                             )

    # upload converted video to S3 and set the name.
    # save set to False to avoid infinite loop
    video.video_flv.save(
                                vidfilename,
                                suf,
                                save=False 
                                )

    # delete temporary output file
    print "[convert_flv] removing temporary file: %s" % tmp_output_video
    os.remove(tmp_output_video)

    #video.flvfilename = vidfilename

    # add do_conversion=False to avoid infinite loop.
    # update_fields is needed in order to not delete video_thumbnail
    # if it did not exist when starting the task
    video.save(do_conversion=False, update_fields=['video_flv'])

@task       
def ffmpeg_image(video_id):
    video = BroadcastUpload.objects.get(pk=video_id)
    print "ID: %s" %video.id
    id = video.id
    print "VIDEO NAME: %s" % video.video_upload.name
    teacher = video.teacher
    print "TEACHER: %s" % teacher
    filename = video.video_upload
    #sourcefile = "%s%s" % (settings.MEDIA_URL, filename)
    sourcefile = video.video_upload.url
    # ffmpeg cannot deal with https?
    sourcefile = sourcefile.replace("https","http")

    # temporary output image
    OUTPUT_IMAGE_EXT = 'png'
    OUTPUT_IMAGE_CONTENT_TYPE = 'image/png'    
    f_out = tempfile.NamedTemporaryFile(suffix=".%s"%OUTPUT_IMAGE_EXT, delete=False)
    tmp_output_image = f_out.name

    #grabimage = "ffmpeg -y -i '%s' -vframes 1 -ss 00:00:02 -an -vcodec png -f rawvideo -s 320x240 '%s'" % (sourcefile, thumbnailfilename)
    grabimage = "ffmpeg -y -i '%s' -vframes 1 -ss 00:00:02 -an -vcodec png -f rawvideo -s 320x240 '%s'" % (sourcefile, tmp_output_image)
    print "ffmpeg_image: %s" % grabimage
    try:        
         videothumbnail = subprocess.call(grabimage, shell=True)
         #also tried separately following line:
         #videothumbnail = commands.getoutput(grabimage)
         print "---------------IMAGE---------------"
         print "VIDEOTHUMBNAIL: %s" % videothumbnail
    except Exception as e:
         videothumbnail = None
         #print("Failed to extract thumbnail from %s to %s" % (sourcefile, thumbnailfilename))
         print("Failed to extract thumbnail from %s to %s" % (sourcefile, tmp_output_image))
         #print(traceback.format_exc())
         print "Error: %s" % e

    #imagefilename = "%s_%s.png" % (teacher, video.id)
    imagefilename = "%s_%s.%s" % (teacher, video.id, OUTPUT_IMAGE_EXT)
    #thumbnailfilename = "%svideos/flv/%s" % (settings.MEDIA_URL, thumbnailfilename)
    #thumbnailfilename = 'thumbnail_image.png'

    # prepare an object with the generated temporary image
    suf = SimpleUploadedFile( 
                             imagefilename,
                             f_out.read(),
                             content_type=OUTPUT_IMAGE_CONTENT_TYPE
                             )

    # upload converted image to S3 and set the name.
    # save set to False to avoid infinite loop

    video.video_thumbnail.save(
                                imagefilename,
                                suf,
                                save=False 
                                )

    # delete temporary output file
    print "[ffmpeg_image] removing temporary file: %s" % tmp_output_image
    os.remove(tmp_output_image)

    #video.videothumbnail = imagefilename

    # add do_conversion=False to avoid infinite loop
    video.save(do_conversion=False, update_fields=['video_thumbnail'])

Answer 2

In order to use subprocess.call with a string containing all arguments, you need to add shell=True:

ffmpegresult = subprocess.call(ffmpeg, shell=True)
...
videothumbnail = subprocess.call(grabimage, shell=True)

Answer 3

hey you are doing a simple mistake

Instead of using settings.Media_url use media_root

Answer 4

not sure whether you'd get help from a similar cloud scenario involving media files uploaded to Cloud ( parse.com ) that need ffmpeg process on arrival with output ( .mp4 ) written back to Cloud ( parse via Curl ).

See paste shellscript that is currently running in a Heroku WEB process that can call out to scripts using CLI.

If you can adapt it so the shellscript runs in a process somewhere that has http access to the inputs and a filesystem where it can write transient files and if you can CURL -X POST an ffmpeg.output from the tmp FS back to S3, then it may be of use to u.