CODEWITHSUNDEEP
pythondatabasegitlabdbt
zafar
zafar

Reputation: 139

How can we work with multiple DBT projects

can anyone share how to organise multiple projects in dbt , given the best practices. My present abstract architecture hierarchy is as follows

Analytics 
--.dbt/
-----profiles.yml
--projects/
-----project_1/
----------models/
----------dbt_project.yml
-----project_2/
----------models/
----------dbt_project.yml
--tests/
-----projects/
----------project_1/
----------project_2/

To the point of creating models for either projects_1 or project_2 is working perfectly fine.

but the problem comes when i try to run the test (model unit tests) for project_2.when I run it gives the error ' no dbt_project.yml found at expected path in temp/project_1/ ... ' (the unit tests that i am trying to run is for project_2) however, the paths are absolutely correct but the lookup for dbt_project.yml is in wrong directory (in the temp directory). On the side note, some of the project_2 models does depends on few project_1 models.

can anyone share or reference where i can get the help to solve this sort of multi dbt project issue.

Upvotes: 9

Views: 11589

Answers (1)

sgdata
sgdata

Reputation: 2763

Like many things, what you're asking here is two sub questions:

  1. Is it possible?
  2. Is it a good idea?

Is it possible?

While you didn't mention what data platform / dialect you are running on, yes, it is possible. Examples and recommendations here:

Is it a good idea?

I think Jeremy breaks out the main arguments for and against really well in this post reply:

But if you've decided that you are already going to do this and don't want a monorepo and you're most interested in the "best" way to implement this, I'd recommend this post by Stefano:

TLDR: Yes, it's possible(ish). Best suggested methods are managing the generate_schema_name / generate_database_name macros OR have low level models defined in packages that your main projects imports.

Upvotes: 4

Related Questions