CODEWITHSUNDEEP
pythondjangodjango-modelsbulkinsert
Adam
Adam

Reputation: 3778

Django `bulk_create` with related objects

I have a Django system that runs billing for thousands of customers on a regular basis. Here are my models:

class Invoice(models.Model):
    balance = models.DecimalField(
        max_digits=6,
        decimal_places=2,
    )

class Transaction(models.Model):
    amount = models.DecimalField(
        max_digits=6,
        decimal_places=2,
    )
    invoice = models.ForeignKey(
        Invoice,
        on_delete=models.CASCADE,
        related_name='invoices',
        null=False
    )

When billing is run, thousands of invoices with tens of transactions each are created using several nested for loops, which triggers an insert for each created record. I could run bulk_create() on the transactions for each individual invoice, but this still results in thousands of calls to bulk_create().

How would one bulk-create thousands of related models so that the relationship is maintained and the database is used in the most efficient way possible?

Notes:

Upvotes: 14

Views: 14236

Answers (3)

Javad
Javad

Reputation: 2098

Another way for this purpose can be like the below code snippet:

from django.utils import timezone
from django.db import transaction

new_invoices = []
new_transactions = []
for sth in sth_else:
    ...
    invoice = Invoice(params)
    new_invoices.append(invoice)

for sth in sth_else:
    ...
    new_transactions.append(transaction)

with transaction.atomic():
    other_invoice_ids = Invoice.objects.values_list('id', flat=True)
    now = timezone.now()
    Invoice.objects.bulk_create(new_invoices)

    new_invoices = Invoice.objects.exclude(id__in=other_invoice_ids).values_list('id', flat=True)
    for invoice_id in new_invoices:
                transaction = Transaction(params, invoice_id=invoice_id)
                new_transactions.append(transaction)

    Transaction.objects.bulk_create(new_transactions)

I write this answer based on this post on another question in the community.

Upvotes: 0

Sagar Adhikari
Sagar Adhikari

Reputation: 1392

You can do it with two bulk create queries, with following method.

new_invoices = []
new_transactions = []
for loop:
    invoice = Invoice(params)
    new_invoices.append(invoice)

    for loop: 
        transaction = Transaction(params)
        transaction.invoice = invoice
        new_transactions.append(transaction)

Invoice.objects.bulk_create(new_invoices)

for each in new_transactions:
    each.invoice_id = each.invoice.id

Transaction.objects.bulk_create(new_transactions)

Upvotes: 4

Ivan
Ivan

Reputation: 6013

You could bulk_create all the Invoice objects, refresh them from the db, so that they all have ids, create the Transaction objects for all the invoices and then also save them with bulk_create. All of this can be done inside a single transaction.atomic context.

Also, specifically for django 1.10 and postrgres, look at this answer.

Upvotes: 14

Related Questions