[dagster-dbt] fix dbt row count behavior on warehouses, test row count code on snowflake & bigquery #21952
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @benpankow and the rest of your teammates on |
benpankow
commented
May 17, 2024
| query_result_table = query_result[1] | ||
| # some adapters do not output the column names, so we need | ||
| # to index by position | ||
| row_count = query_result_table[0][0] |
There was a problem hiding this comment.
Was tested manually before it was changed to index-by-row-name, which evidently works in duckdb but not Snowflake unfortunately
benpankow
changed the title
benpankow
force-pushed
the
benpankow/test-row-count-snowflake
branch
2 times, most recently
from
91f1cac
to
a7f490e
Compare
rexledesma
previously requested changes
May 21, 2024
python_modules/libraries/dagster-dbt/dagster_dbt_tests/core/test_row_count_postprocessing.py
Outdated
Show resolved
Hide resolved
benpankow
force-pushed
the
benpankow/test-row-count-snowflake
branch
from
a7f490e
to
441d6e1
Compare
|
Set up to notify to #eng-buildkite-nightly so that it's targeted in a somewhat generic way and not drowned out (e.g. other teams can start using it + setting up notifs). |
sryza
approved these changes
May 21, 2024
benpankow
changed the title
OwenKephart
pushed a commit
that referenced
this pull request
May 21, 2024
…t code on snowflake & bigquery (#21952) ## Summary Adds Snowflake & BQ tests of new row-count functionality and fixes bug with `Table` return on certain warehouses (incl Snowflake). Added to the new OSS nightly pipeline introduced in #21956 to avoid racking up unnecessary spend. We do already run some tests on these warehouses as part of normal builds for their respective integrations, so maybe it's fine to do the same here - open to either. ## Test Plan Nightly build run: https://buildkite.com/dagster/full-moon-with-face-dagster-nightly/builds/9
nikomancy
pushed a commit
that referenced
this pull request
May 22, 2024
…t code on snowflake & bigquery (#21952) ## Summary Adds Snowflake & BQ tests of new row-count functionality and fixes bug with `Table` return on certain warehouses (incl Snowflake). Added to the new OSS nightly pipeline introduced in #21956 to avoid racking up unnecessary spend. We do already run some tests on these warehouses as part of normal builds for their respective integrations, so maybe it's fine to do the same here - open to either. ## Test Plan Nightly build run: https://buildkite.com/dagster/full-moon-with-face-dagster-nightly/builds/9
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Adds Snowflake & BQ tests of new row-count functionality and fixes bug with
Tablereturn on certain warehouses (incl Snowflake).Added to the new OSS nightly pipeline introduced in #21956 to avoid racking up unnecessary spend. We do already run some tests on these warehouses as part of normal builds for their respective integrations, so maybe it's fine to do the same here - open to either.
Test Plan
Nightly build run:
https://buildkite.com/dagster/full-moon-with-face-dagster-nightly/builds/9