CODEWITHSUNDEEP
dagster
pyCthon
pyCthon

Reputation: 12361

Dagster create a single shared asset

Using dagster I'd like to create a set of reusable assets in a single file where I'd like to be able to pass a country parameter and have different schedules for each country/schedule. The only way I know how to do this is via setting up separate asset files for each country and then define the schedule in the __init__.py file for each.

An example asset is below, where all the asset code is the same except for the country variable, I'm not sure how to pass the country from a Config to the shared data_loader object:

country = 'JP'
data_loader= DataLoader(country=country)

@asset(partitions_def=daily_partitions_def)
def my_asset(context):
    dt = context.partition_key
    return RunMyAsset(data_loader=data_loader, country=country).calc(dt)

@asset(partitions_def=daily_partitions_def)
def my_asset2(context):
    dt = context.partition_key
    return RunMyAsset2(data_loader=data_loader, country=country).calc(dt)

Ideally I would like to reuse the above assets, instead of creating separate files with separate assets for every country.

Upvotes: 1

Views: 195

Answers (1)

skulden
skulden

Reputation: 423

You can use static or dynamic partition definitions

https://docs.dagster.io/_apidocs/partitions

from dagster import StaticPartitionsDefinition, asset

oceans_partitions_def = StaticPartitionsDefinition(
    ["arctic", "atlantic", "indian", "pacific", "southern"]
)

@asset(partitions_def=oceans_partitions_defs)
def ml_model_for_each_ocean():
    ...

Upvotes: 2

Related Questions