CODEWITHSUNDEEP
pythondjangosqlitedjango-models
polaupa
polaupa

Reputation: 169

Bulk create for models with a foreign key

I have these two models:

class Item(models.Model):
    item_id = models.CharField(max_length=10, primary_key=True)
    item_name = models.CharField(max_length=100)
    description = models.CharField(max_length=500)
    slug = models.CharField(max_length=200)
    price = models.FloatField()
    
    def __str__(self):
        return self.item_name

class Image(models.Model):
    item = models.ForeignKey(Item, on_delete=models.CASCADE)
    image_url = models.CharField(max_length=200)

I have a lot of items, and each Item has some images assigned to it.

To add items in the database, I do it through a bulk create:

    it = (Item(
            item_id = item['id'],
            item_name= item['title'],
            description = item['description'],
            slug = item['slug'],
            price =  item['price']['amount']
            )
        for item in items
    )
    while True:
        batch = list(islice(it, batch_size))
        if not batch:
            break
        Item.objects.bulk_create(batch, batch_size)

This works perfectly fine, ok. But I want to do the same with the images, since I have too many images to do it individually.

For now I do it like this:

for item in items:
    for image in item['images']:
        Item.objects.get(item_id = item['id']).image_set.create(image_url = image['urls']['big'])

But this is too slow in uplaoding all the Images (it may take 30 minutes), so I want to upload in bulk like I do it with Item, but since Image model has Item as a foreign key, I cannot do it the same way. How can I upload Images in a more efficient manner?

I am using SQLite, I know that using another database system would speed up the process, but anyway, I think there is a faster way to upload all the Images.

Thanks.

Upvotes: 4

Views: 4234

Answers (2)

handboy
handboy

Reputation: 1

I have a similar problem using SQLite but in my case, I don't have id generated until save the object. The documentation about bulk-create give a tip:

If the model’s primary key is an AutoField, the primary key attribute can only be retrieved on certain databases (currently PostgreSQL, MariaDB 10.5+, and SQLite 3.35+). On other databases, it will not be set. bulk-create

I change to Postgres and works.

Upvotes: 0

willeM_ Van Onsem
willeM_ Van Onsem

Reputation: 476534

You can make Image objects in bulk as well, with:

images = [
    Image(item_id=item['id'], image_url=image['urls']['big'])
    for item in items
    for image in item['images']
]
Image.objects.bulk_create(images)

Here we thus make use if the item_id=item['id'] to set a value for the ForeignKey later, and then insert these all in bulk in the database.

This works because if you define field named fieldname as a ForeignKey, Django allows you to work with fieldname_id to specify the value for the to field referenced by that ForeignKey.

Upvotes: 4

Related Questions