How to upload and process large excel files using Celery in Django?

Question

I am trying to upload and process excel file using Django and DRF with Celery. There is an issue when I am trying to pass the file to my Celery task to be processed in the background, I get a following error:

kombu.exceptions.EncodeError: Object of type InMemoryUploadedFile is not JSON serializable

Here is my view post request handler:

class FileUploadView(generics.CreateAPIView):
    """
    POST: upload file to save data in the database
    """
    parser_classes = [MultiPartParser]
    serializer_class = FileSerializerXLSX

    def post(self, request, format=None):
        """
        Allows to upload file and lets it be handled by pandas
        """

        serialized = FileSerializerXLSX(data=request.data)
        if serialized.is_valid():
            file_obj = request.data['file']
            # file_bytes = file_obj.read()
            print(file_obj)
            import_excel_task.delay(file_obj)
            print("its working")
            return Response(status=204)

        return Response(serialized._errors, status=status.HTTP_400_BAD_REQUEST)

And my celery task:

def import_excel_helper(file_obj):
    df = extract_excel_to_dataframe(file_obj)
    transform_df_to_clientmodel(df)
    transform_df_to_productmodel(df)
    transform_df_to_salesmodel(df)


@shared_task(name="import_excel_task")
def import_excel_task(file_obj):
    """Save excel file in the background"""
    logger.info("Importing excel file")
    import_excel_helper(file_obj)

Any idea what is the way to handle importing Excel files into celery task so that it can be processed by other functions in the background?

Answer 1

As in the error, the body of the request to call a celery task must be JSON serializable since it is the default configuration. Then as documented in kombu:

The primary disadvantage to JSON is that it limits you to the following data types: strings, Unicode, floats, boolean, dictionaries, and lists. Decimals and dates are notably missing.

Let's say this is my excel file.

file.xlsx

Some	Value
Here	:)

Solution 1

Convert the raw bytes of the excel into Base64 string before calling the task so that it can be JSON serialized (since strings are valid data types in a JSON document, raw bytes are not). Then, everything else in the Celery configurations are the same default values.

tasks.py

import base64

import pandas

from celery import Celery

app = Celery('tasks')


@app.task
def add(excel_file_base64):
    excel_file = base64.b64decode(excel_file_base64)
    df = pandas.read_excel(excel_file)
    print("Contents of excel file:", df)

views.py

import base64

from tasks import add


with open("file.xlsx", 'rb') as file:  # Change this to be your <request.data['file']>
    excel_raw_bytes = file.read()
    excel_base64 = base64.b64encode(excel_raw_bytes).decode()
    add.apply_async((excel_base64,))

Output

[2021-08-19 20:40:28,904: INFO/MainProcess] Task tasks.add[d5373444-485d-4c50-8695-be2e68ef1c67] received
[2021-08-19 20:40:29,094: WARNING/ForkPoolWorker-4] Contents of excel file:
[2021-08-19 20:40:29,094: WARNING/ForkPoolWorker-4]  
[2021-08-19 20:40:29,099: WARNING/ForkPoolWorker-4]    Some Value
0  Here    :)
[2021-08-19 20:40:29,099: WARNING/ForkPoolWorker-4] 

[2021-08-19 20:40:29,099: INFO/ForkPoolWorker-4] Task tasks.add[d5373444-485d-4c50-8695-be2e68ef1c67] succeeded in 0.19386404199940444s: None

Solution 2:

This is the harder way. Implement a custom serializer that will handle excel files.

tasks.py

import ast
import base64

import pandas

from celery import Celery
from kombu.serialization import register


def my_custom_excel_encoder(obj):
    """Uncomment this block if you intend to pass it as a Base64 string:
    file_base64 = base64.b64encode(obj[0][0]).decode()
    obj = list(obj)
    obj[0] = [file_base64]
    """
    return str(obj)


def my_custom_excel_decoder(obj):
    obj = ast.literal_eval(obj)
    """Uncomment this block if you passed it as a Base64 string (as commented above in the encoder):
    obj[0][0] = base64.b64decode(obj[0][0])
    """
    return obj


register(
    'my_custom_excel',
    my_custom_excel_encoder,
    my_custom_excel_decoder,
    content_type='application/x-my-custom-excel',
    content_encoding='utf-8',
)


app = Celery('tasks')

app.conf.update(
    accept_content=['json', 'my_custom_excel'],
)


@app.task
def add(excel_file):
    df = pandas.read_excel(excel_file)
    print("Contents of excel file:", df)

views.py

from tasks import add


with open("file.xlsx", 'rb') as excel_file:  # Change this to be your <request.data['file']>
    excel_raw_bytes = excel_file.read()
    add.apply_async((excel_raw_bytes,), serializer='my_custom_excel')

Output

• Same as Solution 1

Solution 3

You might be interested with this documentation of Sending raw data without Serialization