Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

_get_infer_single_to_multi_dimension_deps_result is to restrictive, can't handle identityMapped staticPartitionDefintions #21942

Open
ion-elgreco opened this issue May 17, 2024 · 0 comments
Labels
area: partitions Related to Partitions type: bug Something isn't working

Comments

@ion-elgreco
Copy link
Contributor

Dagster version

1.7.5

What's the issue?

I have two upstream assets, with these partition defintions:

Asset_1:

partitions_def=MultiPartitionsDefinition(
        {
            "month": MonthlyPartitionsDefinition(start_date="202105", fmt="%Y%m", end_offset=1),
            "code": StaticPartitionsDefinition(["FOO"]),
        },
    ),

Asset_2:

    partitions_def=MultiPartitionsDefinition(
        {
            "month": MonthlyPartitionsDefinition(start_date="202105", fmt="%Y%m", end_offset=1),
            "code": StaticPartitionsDefinition(["FOO", "BAR"]),
        },
    ),

Asset 1 and Asset 2 are inputs for Asset 3, where Asset 3 has this single partitionDefinition: StaticPartitionsDefinition(["FOO"])

In my AssetIn I declare each asset_1 and asset_2 to have this mapping: MultiToSingleDimensionPartitionMapping("code"), which results in this error, and that's only because Dagster checks whether the definitions completely match. But with StaticPartitionDefintions it should be possible to also have a subset or no match at all.

dagster._check.CheckError: Failure condition: MultiToSingleDimensionPartitionMapping can only be used when: 
(a) The single dimensional partitions definition is a dimension of the MultiPartitionsDefinition.
(b) The single dimensional partitions definition is a TimeWindowPartitionsDefinition and the MultiPartitionsDefinition has a single time dimension.

If I don't provide a mapping at all, I get even a stranger error:

dagster._core.errors.DagsterInvariantViolationError: Partition key range PartitionKeyRange(start='FOO', end='FOO') in asset_3 depends on invalid partition keys ['FOO'] in upstream asset AssetKey(['feat-partitioning-assets', 'asset_2'])

What did you expect to happen?

Make the multiToSinglePartitionMapping usable

How to reproduce?

No response

Deployment type

None

Deployment details

No response

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

@ion-elgreco ion-elgreco added the type: bug Something isn't working label May 17, 2024
@ion-elgreco ion-elgreco changed the title _get_infer_single_to_multi_dimension_deps_result is to restrictive, can't handle identityMapped staticPartitionDefintions _get_infer_single_to_multi_dimension_deps_result is to restrictive, can't handle identityMapped staticPartitionDefintions May 17, 2024
@garethbrickman garethbrickman added the area: partitions Related to Partitions label May 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: partitions Related to Partitions type: bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants