How to run different dbt commands for the same assets

[alex-gregory-unmind]

Sorry, not entirely sure how to word this, so perhaps an example better illustrates this.

If I'm not mistaken, the asset definitions via the dbt_assets decorator couples the dbt commands and the asset definitions. This make it awkward to say run a full refresh every so often.

Now, I've seen in the documentation this example:

from pathlib import Path

from dagster import AssetExecutionContext, Config
from dagster_dbt import DbtCliResource, dbt_assets

class MyDbtConfig(Config):
    full_refresh: bool

@dbt_assets(manifest=Path("target", "manifest.json"))
def data_dbt_assets(context: AssetExecutionContext, dbt: DbtCliResource, config: MyDbtConfig):
    dbt_build_args = ["build"]
    if config.full_refresh:
        dbt_build_args += ["--full-refresh"]

    yield from dbt.cli(dbt_build_args, context=context).stream()

I assume this does the job when running jobs in the UI, however, I can't workout how I can schedule two jobs that act on the same set of assets, but:

• One runs dbt run
• The other runs dbt run --full-refresh

These models are simply partitioned, i.e. the cursor is derived from the underlying data, not passed as a parameter--I also don't intend to change this for now as it would take considerable time.

One solution, which I dislike, was to bake into the asset definition (the function decorated with the dbt_assets decorator) some conditional that checks the day of the week. This is bad in the sense that if anyone wants to run a job that day, it is always full refresh.

What I really need, is some ability to configure a job. The job can run the same asset selection, but if some context can be passed through to the asset definition, then we could append the --full-refresh to build commands. However, I can't seem to find anything about configuring jobs.

Any help would be appreciated as this feels like a blocker in using the integration at present. Thanks.

[rexledesma]

When using define_asset_job to define a job for a set of assets, you can pass a RunConfig to the config that lets you specify the MyDbtConfig that you want when the job runs.

See a related discussion in #18803 (comment).

[alex-gregory-unmind]

Thanks @rexledesma! Sorry, I didn't see this before, but works great 😄

[Siddharth545]

but then we dont have to pass project dir?