CODEWITHSUNDEEP
pythondjangodjango-rest-framework
Jekson
Jekson

Reputation: 3262

How to create model instances from csv file

There is a task to parse the csv file and create instances in the database based on the received data. On the backend - DRF and at the front - React. The specific feature is that the file processing is not quite hidden. The logic is as follows: There is a button to load the file. The file is loaded and validated, but nothing is created in the database at once. A window appears with a list of saved data (like a table) and in this window there is a new button to confirm by clicking on which the database is already requested.

What I just did: 1. Created a class to download the file (Upload button)

class FileUploadView(APIView):
    parser_classes = ( MultiPartParser, FormParser)
    renderer_classes = [JSONRenderer]

    def put(self, request, format=None):
        if 'file' not in request.data:
            raise ParseError("Empty content")
        f = request.data['file']
        filename = f.name
        if filename.endswith('.csv'):
            file = default_storage.save(filename, f)
            r = csv_file_parser(file)
            status = 204
        else:
            status = 406
            r = "File format error"
        return Response(r, status=status)

In the class, the csv_file_parser function is called, the result of which is json containing all the saved data like this:

{
    "1": {
        "Vendor": "Firstvendortestname",
        "Country": "USA",
         ...
         ...
        "Modules": "Module1",
        " NDA date": "2019-12-24"
    },
    "2": {
        "Vendor": "Secondvendortestname",
        "Country": "Canada",
         ...
         ...
        "Modules": "Module1",
        " NDA date": "2019-12-24"
    }
}

This data will be used to preview the fields from which the model instaces will be created in the base by clicking on the Confirm button.

csv_file_parser function

def csv_file_parser(file):
    result_dict = {}
    with open(file) as csvfile:
        reader = csv.DictReader(csvfile)
        line_count = 1
        for rows in reader:
            for key, value in rows.items():
                if not value:
                    raise ParseError('Missing value in file. Check the {} line'.format(line_count))
            result_dict[line_count] = rows
            line_count += 1


    return result_dict

When the Confirm button is pressed, React passes this data as an argument to a class that works with the database using the POST method. With the implementation of this class, I have difficulties. How to correctly process the received data and record it in the database?

class CsvToDatabase(APIView):

    def post(self, request, format=None):
        data = request.data
        for vendor in data:
            Vendors(
                vendor_name=vendor['Vendor'],
                country=vendor['Country']
            ).save()


        return Response({'received data': request.data})

This code gives error 

TypeError at /api/v1/vendors/from_csv_create/
string indices must be integers

Printing of request.data output

<QueryDict: {'{\n    "1": {\n        "Vendor": "Firstvendortestname",\n        "Country": "USA",\n        "Primary Contact Name": "Jack Jhonson",\n        "Primary Contact Email": "[email protected]",\n        "Secondary Contact Name": "Jack2 Jhonson",\n        "Secondary Contact Email": "[email protected]",\n        "Modules": "Module1, Module2",\n        " NDA date": "2019-12-24"\n    },\n    "2": {\n        "Vendor": "Secondvendortestname",\n        "Country": "Canada",\n        "Primary Contact Name": "Sandra Bullock",\n        "Primary Contact Email": "[email protected]",\n        "Secondary Contact Name": "Sandra Bullock",\n        "Secondary Contact Email": "[email protected]",\n        "Modules": "Module1, Module2",\n        " NDA date": "2019-12-24"\n    }\n}': ['']}>

Maybe I'm using the wrong data format?

And overall, I have a feeling that I'm doing the job the wrong way. I don't use serializers, do I need them here?

Upvotes: 1

Views: 1008

Answers (2)

Dachi Darchiashvili
Dachi Darchiashvili

Reputation: 769

First of all my suggestion is to use bulk create operation instead of creating them one by one. Please follow this documentation link https://docs.djangoproject.com/en/3.0/ref/models/querysets/#bulk-create.

Your problem is caused because you are following data incorrectly in your loop. My advice is to start searching for the problem from the errors. The error clearly says that the BUG is in the data structure.

Now let's look at the request.data, it is not a list containing dicts to loop them like you are doing there. Please see this StackOverflow page for more details: Extracting items out of a QueryDict

Upvotes: 1

neverwalkaloner
neverwalkaloner

Reputation: 47364

You iterating over dict key, but should iterate over items:

   for key, vendor in data.items():
        Vendors(
            vendor_name=vendor['Vendor'],
            country=vendor['Country']
        ).save()

Upvotes: 1

Related Questions