CODEWITHSUNDEEP
djangofacebook-graph-apicelerydjango-celery
Divick
Divick

Reputation: 1273

Parallelizing tasks with django celery

I want to speed up my django view processing where in the view I have to make several web service calls (in my case facebook graph api) which really take quite some time to process (it takes around 15-16 secs just to render a view most of which is spent in getting data from facebook). So instead of

def view(request):
    if new_user
         #facebook graph api call becomes the bottleneck
         profile = facebook.graph_api_call('profile information')

    friends = facebook.graph_api_call('get some more information like friends')

    some lengthy processing of friends
    ...

    facebook.graph_api_call('yet some more information')

    #Finally pass the profile to template to render user details
    render_to_response('mytemplate', { 'profile': profile })

I thought to do:

@task
def get_profile():
    profile = facebook.graph_api_call('profile information')
    return profile

def view(request):
    if new_user
         profile_task = get_profile.delay()

    friends = facebook.graph_api_call('get some more information like friends')

    some really lengthy processing of friends
    ...

    facebook.graph_api_call('yet some more information')

    #Wait for profile task to complete
    profile_task.get()

    #Finally pass the profile to template to render user details
    render_to_response('mytemplate', { 'profile': profile })

This way my processing of friends data can proceed without waiting for the profile information to be retrieved. But if the celery worker(s) are busy then it may not fetch the profile data immediately and hence the rendering of view could take more even more time than the previous approach.

The third approach could be to do everything same as in approach 2 but cancel the task if it has already not been started and make a regular function call instead of launching a task to get profile information.

Just in case someone suggests to get the profile and friend information using a facebook batch request: it is not possible in my case as the code above is just a snippet. I am actually fetching the profile in a middleware when a user first visits my app.

I am not sure which approach out of the 3 above is better. Please suggest if there could be some other way to parallelize the web requests.

Upvotes: 1

Views: 868

Answers (1)

Thierry
Thierry

Reputation: 3155

I'm the author of Django Facebook and I encountered the same problem you have. I think the best solution would be to use eventlet if you want to request the urls in parallel (faster). We also use celery but we mainly use it for running things on the background.

Upvotes: 2

Related Questions