Advice on structuring a growing Django project (Models & API)

Question

I’m currently working on improving a Django project that is used internally at my company. The project is growing quickly so I’m trying to make some design choices now before it’s unmanageable to refactor. Right now the project has a two really important models the rest of the data in the database that supports the each application in the project is added into the database through various separate ETL processes. Because of this the majority of data used in the application is queried in each view via SQLAlchemy using a straight up multiline string and passing the data through to the view via the context param when rendering rather than using the Django ORM.

Would there be a distinct advantage in building models for all the tables that are populated via ETL processes so I can start using the Django ORM vs using SQLAlchemy and query strings?

I think it also makes sense to start building an API rather than passing a gigantic amount of information through to a single view via the context param, but I’m unsure of how to structure the API. I’ve read that some people create an entirely separate app named API and make all the views in it return a JsonResponse. I’ve also read others do this same view based API but they simply include an api.py file in each application in their Django project. Others use the Django REST API Framework, which seems simple but is slightly more complicated than just returning JsonResponse via a view. There is really only one place where a users interaction does anything but GET data from the database and that portion of the project uses Django REST Framework to perform CRUD operations. That being said:

Which of these API structures is the most typical, and what do I gain/lose by implementing JsonResponse views as an API vs using the Django REST Framework?

Thanks you in advance for any resources or advice anyone has regarding these questions. Please let me know if I can add any additional context.

Answer 1

I'm usually a huge proponent for DRF. It's simple to implement an easy use case, yet INCREDIBLY powerful for more complex uses.

However, if you are not using Django models for all your data, I think JsonResponse might be easier. Running queries and manual manipulation (especially if it is only a single endpoint) might be the way to go.

Sorry for not weighing in on the other part of the question.

Answer 2

I would second @Av4t4r 's statement that building models for 'legacy' columns enforces:

"a centralized, consistent way of accessing the data, and of course, one less dependency on the project."

Additionally, this also allows you to define methods on your models to provide a consistent way of modifying/checking your data.

You could write a host of validate() functions on a model to check the validity of certain fields, or perhaps a clean() function which does things like 'strip non-digits from the phone_number field' of an instantiated object.

Rather than building these models from scratch, just run:

python manage.py inspectdb > models.py

...from a shell to autogenerate the models for your project's database.

For those of you who landed here that are using SQLAlchemy to define their models:

Checkout sqlacodegen. It will autogenerate SQLAlchemy models from any given database, in any dialect (PostgreSQL, MySQL, etc...) supported by SQLAlchemy.

To install:

pip install sqlacodegen

To run (from bash shell):

sqlacodegen --outfile models.py mysql://username:password@hostname/db_name

This is a great way to bootstrap a Django/Flask project with SQLAlchemy-defined data models. I usually use a combination of etlalchemy and sqlacodegen to get Django/Flask models with a pre-existing database bootstrapped. I use etlalchemy to copy database tables from ANY database type, to the desired local database of my choosing (i.e. SQL Server -> MySQL), and then autogenerate my models with sqlacodegen.

Answer 3

Would there be a distinct advantage in building models for all the tables that are populated via ETL processes so I can start using the Django ORM vs using SQLAlchemy and query strings?

Yes, a centralized, consistent way of accessing the data, and of course, one less dependency on the project.

Which of these API structures is the most typical, and what do I gain/lose by implementing JsonResponse views as an API vs using the Django REST Framework?

In general terms, JSON is used for data, and REST for APIs. You mentioned that Django-REST is already in use, so if there's any tangible benefit from having a REST API, I'd go with it