Version 0.11.0

[v0.11.0] - 2024-06-07

The 'Jossy' release

This release is dedicated to our friend and colleague at Datavault who recently left the company. Your hard work, dedication, and significant contributions to this release and to AutomateDV over the years have been invaluable. You will always be remembered and greatly missed! Thank you ❤️

New

dbt Versions

👍 Officially tested on dbt 1.8.x (on all platforms except MS SQLServer, which does not yet support 1.8.x)

All Platforms

• QUALIFY Statement Usage: Improved across all applicable platforms. Previously, the in-line WINDOW + QUALIFY syntax was not used correctly; instead, columns were materialized before applying QUALIFY. This issue has now been resolved, resulting in a noticeable performance improvement.

Google BigQuery & Databricks

• Native Hashing now available , hashes no longer stored as strings! docs

  • Hashes now correctly use the BYTES type on BigQuery (instead of STRING)
  • Hashes now correctly use the BINARY type on Databricks (instead of STRING)

  ⏩ Enable this with the new enable_native_hashes global variable.
  Note this behaviour is opt-in to avoid forcing a breaking change

Fixed

All Platforms

• Fix for multi-column src_pk fields not working with Ghost Records correctly (#232)
• Fixed an issue introduced in 0.10.0 which caused a column not found error during incremental satellite loads (#227)
• Fixed an issue where setting enable_ghost_records to false resulted in the use of local Ghost Records with incorrect values in PITs (#196)
• Fixed an issue where some loading edge-cases would trigger a cartesian JOIN in Effectivity Satellites
• Fixed an edge case for duplicate ghost records during base loads in Satellites

Databricks

• Fixed #219 as a result of the new Native Hashing functionality described above.

Microsoft SQLServer

• Resolved an issue caused by the transition from CONCAT_WS to CONCAT in v0.9.1 (#188)
  • CONCAT_WS is now correctly used on Databricks, MS SQLServer, Snowflake, and Postgres. For Google BigQuery, which does not support CONCAT_WS, CONCAT continues to be used.

Deprecations

PITs & Bridges

We are planning a major overhaul and re-implementation of these macros to address performance issues and ensure compliance with DV 2.0 standards. They are currently not meeting their intended purposes effectively.

Custom Materializations

In version 0.10.0, we introduced new functionality that replaces the vault_insert_by_x materializations (improved satellite loading etc.). These materializations were initially designed to provide an option for rapid iterative development of incremental loading patterns in local environments for development and testing, allowing users to bypass the need for a comprehensive PSA or delta-loading solution. In this release, v0.11.0, we are deprecating the vault_insert_by_x materializations to encourage the use of more robust solutions.

Version 0.10.2

[v0.10.2] - 2024-02-27

New

dbt Versions

👍 Officially tested on dbt 1.7.x

All Platforms

• A new SHA-1 Hashing option is now available for all platforms!* docs

*Not available on Postgres due to requiring pgcrypto extension

Fixed

All Platforms

• Fixed an issue where the payload-exclusion feature was not enabled for payload definitions in Multi-Active Satellites (#217)
• Fixed an issue where in some cases duplicates would be loaded into Satellites (#221)

SQLServer/Postgres

• Fixed multiple SQL code case inconsistencies causing compilation issues in SQLServer (#211, #209)

Notes

• dbt SQLServer 1.7.x has caused the vault_insert_by_x materialisations provided by AutomateDV to behave inconsistently - both materialisations have been temporarily disabled in SQLServer only and will raise an error, and output an error message when used on SQLServer only.

Thanks

@ZacharyMcGuire - Contributing fixes for SQLServer casing issues

Version 0.10.1

[v0.10.1] - 2023-08-28

Fixes

All Platforms

• Fixed the case where repeating a load would cause duplicates when using the Satellites as released in 0.10.0 (#207)
  • Implemented as a new Behaviour Flag for Satellites apply_source_filter
    • Read more in loading best practises
    • Read more on sat() macro Behaviour Flags

Thanks

• With thanks to @cristianivanoff for reporting this (#207) on Slack!

Version 0.10.0

[v0.10.0] - 2023-08-14

New

All Platforms

• Reference Tables (ref_table macro)

  • Tutorial

Improved

All Platforms

• Satellite loading limitations resolved 🎉🎉🎉
  (for standard sat() macro)

  • Migration Guide
  • Updated Loading Guide

Version 0.9.7

[v0.9.7] - 2023-07-19

New

Databricks and Postgres

Databricks and Postgres are now fully supported in v0.9.7!

We have added support for:

• Transactional Links (t_link macro)
• Effectivity Satellites (eff_sat macro)
• Multi-active Satellites (ma_sat macro)
• Extended Tracking Satellites (xts macro)
• Point in Time tables (PITs) (pit macro)
• Bridges (bridge macro)

Fixes

All platforms

Error handling and error messages

• Cases where escape characters are empty now correctly use the platform default instead.
• Cases where too many iterations (>100,000) would occur for custom vault_insert_by_x materialisations on SQLServer now raise an error.

Postgres

• Fixed a hashing length bug (#176)

Version 0.9.6 - The Rebrand

[v0.9.6] - 2023-05-16

The rebrand update! - dbtvault is now AutomateDV

Changes

• Macros are now called using automate_dv instead of dbtvault. e.g. automate_dv.hub(...)
• The default system_record_value is now AUTOMATE_DV_SYSTEM instead of DBTVAULT_SYSTEM

Version 0.9.5

[v0.9.5] - 2023-03-22

Fixes

All platforms

• Added error handling for when the number of iterations in vault_insert_by_x exceeds 100,000 (#175)
• Fixed a regression in PITs where an incorrect join was causing a performance hit and in some cases, incorrect data
• Fixed an issue causing 'LOADING...' log messages to appear when running dbt docs generate or dbt docs serve
• Fixed a bug in the vault_insert_by_period materialisation affecting executions with 'hour' as the period (#178)

SQLServer

• Fixed a minor casing issue in the SQLServer eff_sat macro (#182)

Databricks

• Fixed an issue related to #183 but for MD5 hashing in Databricks

Version 0.9.4

[v0.9.4] - 2023-02-16

This is a minor hotfix update. More bug fixes to come soon! 😄

Fixes

• Binary type not defaulting correctly (Snowflake) (#183)

Version 0.9.3

[v0.9.3] - 2023-01-27

Fixes

• Updated packages.yml for compatibility with dbt-utils 1.0.0

Notes

• Fully tested (and passing) with dbt-utils 1.0.0 and dbt 1.3.2

Thank you to our community

Thank you to all those who were being patient for this release. The delay was due to us wanting to release this with a few other bug fixes and new features. This additional content is being released at a later date so that we could get this dbt-utils fix out to our community sooner.

Version 0.9.2

[v0.9.2] - 2022-12-22

Fixes

• Hotfixes for issues with Ghost Record creation under certain circumstances (#173, #174)