CODEWITHSUNDEEP
dbtdagster
Ed Ha
Ed Ha

Reputation: 1

dagster fails to load module after adding another dbt job

I've got a problem which is about to drive me insane since i don't have any clue what causes this issue.

We're using dbt-core with dagster (no dagster+) in a container. Both dbt and dagster are new to us. I've got everything in a single gitlab project which i deploy into out infrastructure. In the dockerfile i run dagster with

CMD ["dagster", "dev", "--log-level", "debug", "--module-name", "orchestration", "--host", "0.0.0.0", "--port", "8080" ]

Everything worked just fine. The module got loaded, i could run my defined asset jobs and schedules. All good, big step for us (migrating from SAS).

So today i added another couple models into our dbt project and also defined another job for our dagster orchestration. I use the same blocks i already use and changed the dbt tag, gave it a new name and passed it back in my definitions.

Suddenly the module can't be loaded anymore and dagster gives me the following error stack trace:

dagster._core.errors.DagsterInvalidConfigError: Invalid default_value for Field.
    Error 1: Received unexpected config entry "dbt_assets" at the root. Expected: "{ }".

  File "/app/lib64/python3.11/site-packages/dagster/_grpc/server.py", line 421, in __init__
    self._loaded_repositories: Optional[LoadedRepositories] = LoadedRepositories(
                                                              ^^^^^^^^^^^^^^^^^^^
  File "/app/lib64/python3.11/site-packages/dagster/_grpc/server.py", line 279, in __init__
    repo_def.load_all_definitions()
  File "/app/lib64/python3.11/site-packages/dagster/_core/definitions/repository_definition/repository_definition.py", line 138, in load_all_definitions
    self._repository_data.load_all_definitions()
  File "/app/lib64/python3.11/site-packages/dagster/_core/definitions/repository_definition/repository_data.py", line 203, in load_all_definitions
    self.get_all_jobs()
  File "/app/lib64/python3.11/site-packages/dagster/_core/definitions/repository_definition/repository_data.py", line 426, in get_all_jobs
    self._all_jobs = self._jobs.get_all_definitions()
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib64/python3.11/site-packages/dagster/_core/definitions/repository_definition/caching_index.py", line 92, in get_all_definitions
    sorted(
  File "/app/lib64/python3.11/site-packages/dagster/_core/definitions/repository_definition/caching_index.py", line 123, in get_definition
    definition = cast(Callable, definition_source)()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib64/python3.11/site-packages/dagster/_core/definitions/repository_definition/repository_data_builder.py", line 138, in resolve_unresolved_job_def
    job_def = unresolved_job_def.resolve(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib64/python3.11/site-packages/dagster/_core/definitions/unresolved_asset_job_definition.py", line 215, in resolve
    return build_asset_job(
           ^^^^^^^^^^^^^^^^
  File "/app/lib64/python3.11/site-packages/dagster/_core/definitions/asset_job.py", line 202, in build_asset_job
    return graph.to_job(
           ^^^^^^^^^^^^^
  File "/app/lib64/python3.11/site-packages/dagster/_core/definitions/graph_definition.py", line 688, in to_job
    return JobDefinition.dagster_internal_init(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib64/python3.11/site-packages/dagster/_core/definitions/job_definition.py", line 286, in dagster_internal_init
    return JobDefinition(
           ^^^^^^^^^^^^^^
  File "/app/lib64/python3.11/site-packages/dagster/_core/definitions/job_definition.py", line 238, in __init__
    self._config_mapping = _config_mapping_with_default_value(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/lib64/python3.11/site-packages/dagster/_core/definitions/job_definition.py", line 1202, in _config_mapping_with_default_value
    updated_fields[name] = Field(
                           ^^^^^^
  File "/app/lib64/python3.11/site-packages/dagster/_config/field.py", line 312, in __init__
    raise DagsterInvalidConfigError(

When i take the new job out of the code it works again, just fine...

The only dagster related files I've got in my project are setup.py, setup.cfg, pyproject.toml and __init__.py, which is in another sub-folder. Here's where i configure my asset jobs etc.

I don't have a single clue where to look next.

Thanks everyone in advance!

Upvotes: 0

Views: 108

Answers (1)

Ed Ha
Ed Ha

Reputation: 1

I got it sorted out by myself. Since the Error Messager were not pointing entirely in the right direction i was looking at the wrong spots. The solution is as simple as it could be.

In dbt i use tags so i can build asset jobs easily afterwards. With dbt_include and dbt_exclude i make sure only the needed models will be run by dbt.

The models i build do basically three different things. 1. A Full-Refresh where everything gets build from scratch. 2. a daily delta. 3. building views to make the new data available for others. The new models from the third section has both tags tags = ['analytic_reporting_full_refresh', 'analytic_reporting_sharing_views']. In my asset definition i included the sharing views and excluded the full_refresh models. Obviously this is not working and dagster crashes because this asset job is basically nothing. So i removed the dbt_exclude and it wokrs just fine.

I hope this will help someone in the same situation.

Upvotes: 0

Related Questions