CODEWITHSUNDEEP
python-3.xdbtdagster
skulden
skulden

Reputation: 423

How to load assets from dbt project? #12893

I'm trying to load assets from my dbt project called "dbtz", but i'm receiving the error

hyprster:dagster_dbt.errors.DagsterDbtCliFatalRuntimeError: Fatal error in the dbt CLI (return code 2): Encountered an error while reading the project:  ERROR: Runtime Error
  Could not find profile named dbtz Encountered an error:
Runtime Error
  Could not run dbt

the dbt run works fine on the command line, but i just can't use it on dagster

i'm trying to execute the following code:

from dagster_dbt import load_assets_from_dbt_project
from dagster._utils import file_relative_path

DBT_PROJECT_PATH = file_relative_path(__file__, "../../dbtz")
DBT_PROFILES = file_relative_path(__file__, "../../dbtz/config/profiles.yml")

dbt_assets = load_assets_from_dbt_project(project_dir=DBT_PROJECT_PATH, profiles_dir=DBT_PROFILES)

the relative path is correct, i can cat the profiles.yml, for example

my profiles.yml is:

dbtz:
  target: dev
  outputs:
    dev:
      dataset: dev
      job_execution_timeout_seconds: 600
      job_retries: 0
      keyfile: ../service_account.json
      location: US
      method: service-account
      priority: interactive
      project: site-hypr
      threads: 1
      type: bigquery
    prod:
      dataset: prod
      job_execution_timeout_seconds: 600
      job_retries: 0
      keyfile: ../service_account.json
      location: US
      method: service-account
      priority: interactive
      project: site-hypr
      threads: 1
      type: bigquery

Upvotes: 0

Views: 1069

Answers (1)

skulden
skulden

Reputation: 423

Sharing here my answer that i did at dagster's github

After a bunch of tries, i understood that for dagster runs the dbt, whe need to pass the dbt-cli as a resource for the job, load_assets_from_dbt_project needs a loaded resource to run the dbt.

so i'm my dagster's init file, i added:

from dagster_dbt import dbt_cli_resource

and updated my Definitions with:

resources = {
        "dbt": dbt_cli_resource.configured(
            {"project_dir": DBT_PROJECT_DIR, "profiles_dir": DBT_PROFILES_DIR}
        )
    }

after that, i put the profiles.yml on a config folder and the following code worked like a charm

from dagster_dbt import load_assets_from_dbt_project
from dagster import (
    file_relative_path,
    define_asset_job,
    AssetSelection,
    ScheduleDefinition
)

DBT_PROJECT_DIR = file_relative_path(__file__, "../../dbtz")
DBT_PROFILES_DIR = file_relative_path(__file__, "../../dbtz/config")

dbt_assets = load_assets_from_dbt_project(DBT_PROJECT_DIR, DBT_PROFILES_DIR, key_prefix = ["dbt_assets"])

dbt_assets_job = define_asset_job(name = 'update_dbt_assets', selection = AssetSelection.groups("dbt_assets"))

dbt_assets_job_schedule = ScheduleDefinition(name="dbt_assets_schedule", job=dbt_assets_job, cron_schedule="0 5 * * *")

Upvotes: 0

Related Questions