How to handle bulk create with multiple related models?

Question

Here I have a form that have multiple input values with same name. I want to bulk create objects based on the following template design.

I think the current approach with zipping the list wouldn't work properly if one of the list are unequal.

What will be the better approach ? The front part should be like this as I posted you can check code snippet

   <script>
  $(document).on("click", ".q-name", function () {
    $(this).closest(".untitled").hide();
    $(this).siblings('.q-title').prop('hidden', false);
  });
</script>
<script>
  $(document).on("click", ".addOption", function () {
    option = `<div class="item">
                  <div>
                    <input
                      class="form-control"
                      type="text"
                      name="title"
                      placeholder="Enter title"
                    />
                  </div>
                  <div>
                    <select name="type" class="form-control">
                      Select
                      <option disabled>Select</option>
                      <option value="1">Option1</option>
                      <option value="2">Option2</option>
                    </select>
                  </div>`;
    $(this).closest(".options").prepend(option);
  });

  $(document).on("click", ".newOptionGroup", function () {
    group = `<input type="text" name="q_title" placeholder="model A field" class="form-control q-title"/></div>
              <p>Options of that model (Another model fields)</p>
              <div class="options">
                <div class="item">
                  <div>
                    <input
                      class="form-control"
                      type="text"
                      name="title"
                      placeholder="Enter title"/>
                  </div>
                  <div>
                    <select name="type" class="form-control">
                      Select
                      <option disabled>Select</option>
                      <option value="1">Option1</option>
                      <option value="2">Option2</option>

                    </select>
                  </div>
                </div>

                <div class="last">
                  <button type="button" class="btn btn-icon-only addOption">
                    Add more
                  </button>
                  <div>
                    <div class="custom-control custom-switch">
                      <input
                        name="is_document"
                        type="checkbox"
                        class="custom-control-input"
                        id="customSwitche"
                        value="1"
                      />
                      <label class="custom-control-label" for="customSwitche"
                        >Is File</label
                      >
                    </div>
                  </div>

                  <div></div>
                </div>
              </div>
            </div>
            <div class="option-group-new newOptionGroup">
              <button> Add New group</button>
            </div>
          </div>
          <div class="text-right mt-4">
            <button type="submit" class="btn btn-outline-grey">Submit</button>
          </div>`;

    $(".group-form").append(group);
});
</script>

    <script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>

     <form class="group-form" method="post"> 
                <input
                  type="text"
                  name="q_title"
                  class="form-control q-title"
                  placeholder="model A field name"
                />
              <p>Options of that model (Another model fields)</p>
              </div>
              <div class="options">
                <div class="item">
                  <div>
                    <input
                      class="form-control"
                      type="text"
                      name="title"
                      placeholder="Enter title"
                    />
                  </div>
                  <div>
                    <select name="type" class="form-control">
                      Select
                      <option disabled>Select</option>
                     
                      <option value="1">Option1</option>
                      <option value="2">Option2</option>

                    </select>
                  </div>

                
                </div>

                <div class="last">
                  <button type="button" class="btn btn-icon-only addOption">
                    Add more
                  </button>
                  <div>
                    <div class="custom-control custom-switch">
                      <input
                        name="is_document"
                        type="checkbox"
                        class="custom-control-input"
                        id="customSwitche"
                        value="1"
                      />
                      <label class="custom-control-label" for="customSwitche"
                        >Is File</label
                      >
                    </div>
                  </div>

                  <div></div>
                </div>
              </div>
            </div>
            <div class="option-group-new newOptionGroup">
              <button type="button"> New group</button>
            </div>
          </div>
          <div class="text-right mt-4">
            <button type="submit" class="btn btn-outline-grey">Submit</button>
        
          </div>
        </form>
       </div>

Django views

    ques = Question.objects.get(id=kwargs["q_id"])
    q_title = request.POST.getlist("q_title")
    title = request.POST.getlist("title")
    types = request.POST.getlist("stype")
    is_file = request.POST.getlist("is_file", [0])
    params = zip(q_title, is_file, title, types)
    
    for p in params:

            q = Question.objects.create(
                title=p[0],
                is_file=p[1],
            )
            Option.objects.create(title=p[2], field_type=p[3], question=q)

EDIT:

The question titles and Option titles will be unequal since question can have unlimited options.

For example:

questions = ['q1', 'q2']
options = ['q1_option1', 'q1_option2', 'q2_option1', 'q2_option2', 'q2_option3']

I am not being able to track which option will belongs to particular question.

EDIT2:

ques_titles = ['q1-title', 'q2_title']
is_file = [True, False]

# ques_titles and is_file will be equal.

option_titles = ['q1_option1', 'q1_option2', 'q2-option1', 'q2-option2', 'q3-option3']
types = ['option1_type', 'option2_type', 'option3-type', 'option4-type', 'option5-type3'] 

 #option_titles and types list will be equal

Try1:

With this try while assigning the question fk to the option, only one question gets assigned instead of corresponding question.

 questions = [Question(title=param[0], is_file=param[1]) for param in ques_params]
  ques_objs = Question.objects.bulk_create(questions)
  optn_params = zip(optn_title, types)
  options = []
  for q in ques_objs:
      print(q)
      options.append([Option(title=param[0], field_type=param[1], question=q) for param in optn_params])

  Option.objects.bulk_create(options[0])s)

Answer 1

You might be able to use formset here, but the thing is you would end up with a nesting formsets (because you need multiple questions here), this could make it difficult to style.

I would suggest resolving the question -> options relation problem in the frontend. Here's a snippet using jQuery, after you fill in the form and click "submit", the JSON will be printed in the pre block.

const $questionTemplate = $('#template-question').removeAttr('id');
const $optionTemplate = $('#template-option').removeAttr('id');
const $output = $('#output');

function newOption() {
  // clone the option template
  return $optionTemplate.clone();
}

function newQuestion() {
  // clone the question template, replace the place holder [data-container=options] with 1 Option
  const $questionClone = $questionTemplate.clone();
  $questionClone
    .find('[data-container=options]')
    .replaceWith(newOption());
  return $questionClone;
}

function addQuestion(btnEl) {
  //add a question to before the btnEl
  //You might need to change this to find the right location to add the template suit your design
  $(btnEl).before(newQuestion());
}

function addOption(btnEl) {
  //add a option to before the btnEl
  //You might need to change this to find the right location to add the template suit your design
  $(btnEl).before(newOption());
}

function getValue($el) {
  switch ($el.attr('type')) {
    case 'checkbox':
      // for checkbox type, jquery will give 'on' if checked, we convert it to a boolean here
      return $el.val() === 'on';
    default:
      // otherwise we just take the raw value.
      return $el.val();
  }
}

function elementToObject($el) {
  //this function maps dom to a javascript object (or python dictionary),
  //this implementation only look at one level below the $el
  //You most likely need to change how this function finds the `data-attr` and their values
  /*
  <div>
    <input type="text" data-attr="name1" value="value1">
    <input type="text" data-attr="name2" value="value2">
    <div>
      <!-- NOTE: this won't be included, since we used `.children()`, you might want to use `.find()`
      <input type="text" data-attr="name3" value="value3">
    </div>
  </div>
  becomes
  {
    "name1": "value1",
    "nam2": "value2"
  }
  */
  return Object.fromEntries(
    $el
      .children('[data-attr]')
      .toArray()
      .map((attr) => {
        const $attr = $(attr);
        return [$attr.attr('data-attr'), getValue($attr)];
      })
  );
}

function init() {
  const $form = $('#form');
  $form.click((e) => {
    //binding to the form's click function allows newly added buttons to trigger this even
    //without binding events to the new buttons themselves.
    //all clicks within the form will be captured by this listener
    //but we are only interested in buttons with 'data-action' attributes
    switch ($(e.target).attr('data-action')) {
      case 'add-question':
        return addQuestion(e.target);
      case 'add-option':
        return addOption(e.target);
    }
  });
  $form.submit((e) => {
    e.preventDefault();
    const result = $form
      .find('[data-container=question]')
      .toArray() //find all question elements, turn them into an array
      .map((question) => {
        const $question = $(question);
        const questionObj = elementToObject($question); // serialize it to a object (dictionary)

        questionObj.options = $question // add options to the object (dictionary)
          .find('[data-container=options]')
          .toArray() // find all option elements inside of the question element, turn them into an array
          .map((option) => elementToObject($(option))); // serialize it to a object (dictionary)

        return questionObj;
      });
    $output.text(JSON.stringify(result, null, 4));
  });
  $form.prepend(newQuestion());
}

init();

<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery/3.3.1/jquery.min.js"></script>
<!-- BEGIN templates -->
<!-- these are just here for cloneing -->
<div style="display: none">
  <div
    id="template-question"
    data-container="question"
    style="padding-bottom: 10px"
  >
    <hr />
    Question:
    <input type="text" data-attr="question" placeholder="Enter question" />
    <div data-container="options"></div>
    <button type="button" data-action="add-option">Add more</button>
  </div>

  <div
    id="template-option"
    data-container="options"
    style="padding: 10px 0px 10px 30px"
  >
    Title:
    <input type="text" placeholder="Enter title" data-attr="title" /><br />
    Option:
    <select data-attr="option">
      <option disabled>Select</option>
      <option value="1">Option1</option>
      <option value="2">Option2</option></select
    ><br />
    Is File: <input data-attr="is_file" type="checkbox" />
  </div>
</div>
<!-- END templates -->
<form id="form">
  <button type="button" data-action="add-question">New group</button>
  <button type="submit">Submit</button>
</form>
<pre id="output"></pre>

After we get the data structure we want here, you have several options to send it to the backend:

Option1 (recommended): you can submit via AJAX ($.ajax) request using application/json as content type, and in your view, use json.loads the request body to turn the JSON object into dict, from there you should be able to do your bulk_create easily.

Option2: create another form, with only a hidden textarea, write the result to the textarea, then submit the form, in your view you can do json.loads(request.GET.get('textareaname')) to get the result.

Option3: Use django-restframework to handle the payload. It can handle errors, deserialize data, even create your models for you if you use model serializer. Personally I would go with this option, but you might need to spend some time learning this framework.

Answer 2

I am assuming that you have models as following and same formatted data as used in ques_title and optn_title

model.py

class Question(models.Model):
    title = models.CharField(max_length=255, help_text='The title of the question')
    is_file = models.BooleanField(default=False)


class Option(models.Model):
    title = models.CharField(max_length=255, help_text='The title of the option')
    field_type = models.CharField(max_length=255, help_text='The field type of the option')
    question = models.ForeignKey(Question, on_delete=models.CASCADE)

You have to implement something like this code snippet:

ques_title = ['q1_title', 'q2_title']
is_file = [True, False]
optn_title = ['q1_option1', 'q1_option2', 'q2_option1', 'q2_option2', 'q2_option3']
field_types = ['option1_type', 'option2_type', 'option3_type', 'option4_type', 'option5_type3']

ques_params = zip(ques_title, is_file)
ques_data = question_bulk_create(ques_params=ques_params)
questions = get_option_related_questions(optn_title=optn_title, ques_data=ques_data)
optn_params = zip(optn_title, field_types, questions)
optn_data = option_bulk_create(optn_params=optn_params)

I implement this inside test setup. You can implement that in your views.

Add Functions to Process Data:

common.py

from .models import Question, Option


def question_bulk_create(ques_params):
    questions = [Question(title=param[0], is_file=param[1]) for param in ques_params]
    ques_obj = Question.objects.bulk_create(questions)
    ques_ids = [ques_obj[i].id for i in range(len(ques_obj))]
    ques_data = Question.objects.filter(id__in=ques_ids)

    return ques_data

def option_bulk_create(optn_params):
    options = [Option(title=param[0], field_type=param[1], question=param[2]) for param in optn_params]
    optn_objs = Option.objects.bulk_create(options)
    optn_ids = [optn_objs[i].id for i in range(len(optn_objs))]
    optn_data = Option.objects.filter(id__in=optn_ids)

    return optn_data

def get_option_related_questions(optn_title, ques_data):
    questions = []
    for optn in optn_title:
        ques_str = optn.split('_')[0]
        ques_obj = ques_data.filter(title__icontains=ques_str).first()
        questions.append(ques_obj)
    
    return questions

Complete Test Example Code:

test.py

import json
from django.test import TestCase
from .models import Question, Option
from .common import get_option_related_questions, question_bulk_create, option_bulk_create


class TestTheTestConsumer(TestCase):
    def setUp(self) -> None:
        ques_title = ['q1_title', 'q2_title']
        is_file = [True, False]
        optn_title = ['q1_option1', 'q1_option2', 'q2_option1', 'q2_option2', 'q2_option3']
        field_types = ['option1_type', 'option2_type', 'option3_type', 'option4_type', 'option5_type3']

        ques_params = zip(ques_title, is_file)
        ques_data = question_bulk_create(ques_params=ques_params)
        questions = get_option_related_questions(optn_title=optn_title, ques_data=ques_data)
        optn_params = zip(optn_title, field_types, questions)
        optn_data = option_bulk_create(optn_params=optn_params)

        for data in optn_data:
            print(f'title: {data.title} - field type: {data.field_type} - question title: {data.question.title} - question is file: {data.question.is_file}')

        print ("Question (test.setUp): ", Question.objects.all())
        print ("Option (test.setUp): ", Option.objects.all())

    def test_bulk_create(self):
        all_ques = Question.objects.all().count()
        all_optn = Option.objects.all().count()
        
        self.assertEqual(all_ques, 2)
        self.assertEqual(all_optn, 5)

Test Output:

python manage.py test
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
title: q1_option1 - field type: option1_type - question title: q1_title - question is file: True
title: q1_option2 - field type: option2_type - question title: q1_title - question is file: True
title: q2_option1 - field type: option3_type - question title: q2_title - question is file: False
title: q2_option2 - field type: option4_type - question title: q2_title - question is file: False
title: q2_option3 - field type: option5_type3 - question title: q2_title - question is file: False
Question (test.setUp):  <QuerySet [<Question: Question object (1)>, <Question: Question object (2)>]>
Option (test.setUp):  <QuerySet [<Option: Option object (1)>, <Option: Option object (2)>, <Option: Option object (3)>, <Option: Option object (4)>, <Option: Option object (5)>]>
.
----------------------------------------------------------------------
Ran 1 test in 0.010s

OK
Destroying test database for alias 'default'...