Django 3.1 - "OperationalError: no such table" when using an ORM Class Model before making or applying a migration

Question

TL;DR If I have a Django model for which I create a migration, I can run the migration and create the table in the DB, then I can proceed to reference the model in the code. Now, when I merge the code into master, it will both contain the migration files and the code that references the Django model. When I pull this onto a new machine (e.g. staging/prod), I will want to run the migrations in this new environment so as to have an updated DB. correct? This operation seems impossible though because the code that references the model will want the table to already exist even to just perform the migration itself, otherwise the migration command fails. How to solve this issue?

The Problem

Whenever I create a new model class inside products/models.py (products is the name of my Django app), I cannot reference such model (e.g. Product model) in any method or class before having made and run the migrations. Otherwise, if I reference the new model in the code and then I run any of the following commands/procedures, I receive the OperationalError: no such table error:

1. python manage.py runserver
2. python manage.py makemigrations [appname]
3. python manage.py migrate
4. python manage.py showmigrations
5. Delete all existing migrations and try to regenerate them with python manage.py makemigrations [appname]
6. Delete current db.sqlite3 file; delete all existing migrations and try to regenerate them with python manage.py makemigrations [appname]

The error looks like this:

...
  File "/Users/my_user/Work/django_proj_1/config/urls.py", line 21, in <module>
    path('', include('products.urls')),
...
  File "/Users/my_user/Work/django_proj_1/products/urls.py", line 1, in <module>
    from products.api_views import ProductList3
  File "/Users/my_user/Work/django_proj_1/products/api_views.py", line 7, in <module>
    class ProductList3(ListAPIView):
  File "/Users/my_user/Work/django_proj_1/products/api_views.py", line 8, in ProductList3
    queryset = get_products()
  File "/Users/my_user/Work/django_proj_1/products/data_layer/product_data_layer.py", line 4, in get_products
    all_prods = list(Product.objects.values())
...
  File "/Users/my_user/.local/share/virtualenvs/django_proj_1-a2O6RBaf/lib/python3.8/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: products_product
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The Real Question

This behavior may seem normal, but if you think about it, how would a developer make a release in production where a new model is created and also used in the code simultaneously (given that running migrations will fail if you referenced the created model into the code)?

I found similar questions (which I linked below), but none of them seems actually relevant for a real-world large scale production project. The only 'solution' in the similar questions is to:

1. comment out the code (all functions in the layer/chain) that end up interacting with the new model that does not exist in the DB yet
2. (make and) run the migrations
3. revert the comments

This seems pretty manual and not feasible for real-world production applications. I can understand if the issue was for local development, in which I can

1. create a model,
2. quickly make and run the migrations
3. and then start coding and use that model.

However, how is it possible to go about this in a real-world production scenario? By this I mean, how is it possible to release a single PR of code where I create a model as well as use the model in the code and make sure I can successfully run the migration on the production machine upon release of such code?

More Context of my problem

Let's assume I have the following app in Django called products with, among others, the following files:

...
|_products/  # one of the Django apps in the project
    |
    |_ models.py   # where I defined the Product model
    |_ data_layer/   # a python module to separate all DB logic
    |    |_ __init__.py
    |    |_ product_data_layer.py   # file where I put the function that references the Product model
    |_ api_views.py   # file where I call a function from the data layer
    |_ urls.py   # file where I create an API endpoint that references api_views.py
    |
...

Code inside the above files:

# products/urls.py content

from products.api_views import ProductList3   # THORWS ERROR
from django.urls import path


urlpatterns = [
  path('api/v1/products', ProductList3)
]

# products/api_views.py content

from rest_framework.generics import ListAPIView
from products.serializers import GenericProductSerializer
from products.data_layer.product_data_layer import get_products


class ProductList3(ListAPIView):   # THORWS ERROR
    queryset = get_products()   # THORWS ERROR
    serializer_class = GenericProductSerializer

# products/data_layer/product_data_layer.py content

from products.models import Product

def get_products():
    all_prods = list(Product.objects.values())   # THORWS ERROR (main problem)
    # all the logic I need
    return all_prods

# products/models.py content

from django.db import models

class Product(models.Model):
    product_name = models.CharField(max_length=100)
    product_description = models.TextField('Main Product Description')

    def __str__(self):
        return self.product_name

Similar questions

The following questions are similar questions that I looked at, but besides some dev-process solutions, I found no substantial answer on how to tackle this issue once for all.

1. django migration no such table 1 - unclear as to how not to execute code on import
2. django migration no such table 2 - no answers as of 18/03/2021
3. django migration no such table 3 - does not work in my case

Answer 1

You write in your class declaration:

queryset = get_products()

And this function has this line:

all_prods = list(Product.objects.values())

What happens here? Anything written directly inside a class is executed when the class is being created. If it were a method of the class then that would not be evaluated. So get_products() is called when the class is being created. Next list(Product.objects.values()) is called where calling list on the queryset forces the query to be evaluated. Now obviously you are making migrations and the tables don't exist yet giving you an error.

Firstly this code of yours is bad design even if there are changes to the table Product your view will still only display the products that were gotten in the first query made when the server was started up. This is obviously unintended and not what you want to do.

What is the solution? Obviously your query should reside somewhere where it is actually meant to be. Mostly in Django with classes that get a queryset there is a method named get_queryset that does this task, so you should be overriding that:

class ProductList3(ListAPIView):
    serializer_class = GenericProductSerializer
    
    def get_queryset(self):
        return get_products() # More sensible to write it here.
        # If concerned about overriding something you can use the below which would be equivalent
        # self.queryset = get_products()
        # return super().get_queryset()