{"payload":{"feedbackUrl":"https://github.com/orgs/community/discussions/53140","repo":{"id":182849188,"defaultBranch":"master","name":"delta","ownerLogin":"delta-io","currentUserCanPush":false,"isFork":false,"isEmpty":false,"createdAt":"2019-04-22T18:56:51.000Z","ownerAvatar":"https://avatars.githubusercontent.com/u/49767398?v=4","public":true,"private":false,"isOrgOwned":true},"refInfo":{"name":"","listCacheKey":"v0:1717116030.0","currentOid":""},"activityList":{"items":[{"before":"085f11718b2a4de2dc72d1cb09139039ba22dab2","after":"e02b65c26105b092909e8909b095987d274c9847","ref":"refs/heads/branch-4.0-preview1","pushedAt":"2024-05-31T22:58:26.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"allisonport-db","name":"Allison Portis","path":"/allisonport-db","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/89107911?s=80&v=4"},"commit":{"message":"[INFRA] Build changes to use the Spark 4.0 preview RC and prepare for Delta 4.0 preview release using Spark 4.0 preview (#3182)\n\n#### Which Delta project/connector is this regarding?\r\n\r\n- [ ] Spark\r\n- [ ] Standalone\r\n- [ ] Flink\r\n- [ ] Kernel\r\n- [X] Other (BUILD)\r\n\r\n## Description\r\n\r\nThis PR makes the initial changes to prepare for the Delta 4.0 preview\r\nrelease on Spark 4.0 preview. This initially updates the branch to use\r\nthe Spark RC3 but will need to be updated once the Spark release is\r\nfinalized.\r\n\r\nI've organized the commits based on what changes they make to hopefully\r\nmake this PR easier to review (I'd recommend reviewing it one commit at\r\na time).\r\n\r\n7d20eab Makes the initial build changes to use the Spark 4.0 preview RC\r\n- Upgrades our version to 4.0.0-SNAPSHOT and updates Mima settings\r\n- Drops Scala 2.12 as it is not supported by Spark 4.0+\r\n  - Makes corresponding necessary changes to Kernel code\r\n- Makes the default sparkVersion to be \"4.0.0-preview1\" and adds the\r\nSpark RC repository as a resolver\r\n  - Also adds settings to make Kernel tests pass with Java 17\r\n- Changes other connector projects other than [Spark, Kernel, storage\r\nprojects] to \"skipReleaseSettings\" as we decided for now to only release\r\nSpark, Kernel and the storage projects (due to dependency complexities\r\nwith Spark and Scala versions)\r\n- Comments out uniform projects as they require other dependencies that\r\nare unavailable\r\n- Comments out Flink as it does not support Scala 2.13\r\n- Updates the CI jobs for Scala and Java versions\r\n- Updates Spark examples to use the Spark RC\r\n\r\n896ef77 Makes fixes for unidoc to work with Kernel & Delta Spark\r\n- Excludes internal Scala code from the source files for javadoc.\r\n- Changes imports in the public classes within kernelDefaults to use the\r\nfull package name inline. Without this, the javadoc compilation fails\r\nsince the internal classes are filtered from the javadoc source files.\r\n- Adds unidoc configurations for Delta Spark with Spark master. \r\n- Note: Without the changes to `sqlDeltaImport` in `build.sbt` the\r\nscaladoc generation fails for Delta Spark with some weird\r\nshimming-related error. I can provide more details if necessary.\r\n\r\nf3fe315 Makes the necessary python changes\r\n- Runs the python tests as part of the `spark_master_test` job\r\n- Upgrades scala versions throughout for `delta-spark` artifact names\r\n- Fixes a mypy error brought on by the mypy version upgrade\r\n- Updates the `delta-spark` python and pyspark versions\r\n\r\nf4587b2 Makes changes so that the integration tests run successfully\r\n- Change scala versions to use 2.13\r\n- Removes uniform integration tests\r\n- Fixes `table_exists.py` which otherwise fails with \"AnalysisException\r\ndoesn't have attribute `desc`\"\r\n\r\nf29496b Fixes failing tests for `sharing` and `storageS3DynamoDB`\r\n- Upgrades the hadoop version in storage related projects to match Spark\r\n- `storageS3DynamoDB` tests were failing due to mismatched hadoop\r\nversions since it relies on delta-spark as a test dependency\r\n- Adds Java 17 test options for `sharing`\r\n\r\n## How was this patch tested?\r\n\r\nAll the CI jobs should pass. Also ran the integration tests using\r\n`--use-local` in a conda environment.","shortMessageHtmlLink":"[INFRA] Build changes to use the Spark 4.0 preview RC and prepare for…"}},{"before":"56ce2126615e9e046d2e0b60a3e4011234d87866","after":"59f8c64c2a2bc49877e8e19913c9249f9f337028","ref":"refs/heads/master","pushedAt":"2024-05-31T21:57:07.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[Spark] Fix replacing clustered table with non-clustered table (#3175)\n\n## Description\r\nFix replacing clustered table with non-clustered table, by creating a\r\ndomain metadata with empty clustering columns.\r\n\r\n## How was this patch tested?\r\nNew UTs.","shortMessageHtmlLink":"[Spark] Fix replacing clustered table with non-clustered table (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2324393909\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3175\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3175/hovercard\" href=\"https://github.com/delta-io/delta/pull/3175\">#3175</a>)"}},{"before":"8cfb11962819edf417a5afb009556797ba217641","after":"56ce2126615e9e046d2e0b60a3e4011234d87866","ref":"refs/heads/master","pushedAt":"2024-05-31T15:27:29.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[Spark] Support in-place migration from unpartitioned table to clustered table (#3174)\n\n## Description\r\nSupport in-place migration from unpartitioned table to clustered table.\r\nIf the table is an unpartitioned table and users run `ALTER TABLE\r\nCLUSTER BY` on it, it will now create a clustered table with\r\nClusteringMetadataDomain.\r\n\r\nResolves #2460 \r\n\r\n## How was this patch tested?\r\nNew UTs.","shortMessageHtmlLink":"[Spark] Support in-place migration from unpartitioned table to cluste…"}},{"before":"085f11718b2a4de2dc72d1cb09139039ba22dab2","after":"8cfb11962819edf417a5afb009556797ba217641","ref":"refs/heads/master","pushedAt":"2024-05-31T15:26:22.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[Spark] Include violating value in DELTA_EXCEED_CHAR_VARCHAR_LIMIT (#3167)\n\n## Description\r\n\r\nThis PR modifies the error message for the\r\n`DELTA_EXCEED_CHAR_VARCHAR_LIMIT` error class to include the value that\r\nviolated the constraint.\r\n\r\n## How was this patch tested?\r\n\r\nModified a test in `DeltaErrorsSuite` and added a test to\r\n`DeltaConstraintsSuite`.\r\n\r\n## Does this PR introduce _any_ user-facing changes?\r\n\r\nYes, the error message for the `DELTA_EXCEED_CHAR_VARCHAR_LIMIT` error\r\nclass is modified to include the value that violated the constraint.","shortMessageHtmlLink":"[Spark] Include violating value in DELTA_EXCEED_CHAR_VARCHAR_LIMIT (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2321627167\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3167\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3167/hovercard\" href=\"https://github.com/delta-io/delta/pull/3167\">#…</a>"}},{"before":null,"after":"085f11718b2a4de2dc72d1cb09139039ba22dab2","ref":"refs/heads/branch-4.0-preview1","pushedAt":"2024-05-31T00:40:30.000Z","pushType":"branch_creation","commitsCount":0,"pusher":{"login":"allisonport-db","name":"Allison Portis","path":"/allisonport-db","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/89107911?s=80&v=4"},"commit":{"message":"[Kernel] Change comparator expression to lazy evaluation (#2853)\n\n## Description\r\nResolves https://github.com/delta-io/delta/issues/2541\r\n\r\n## How was this patch tested?\r\nExisting tests","shortMessageHtmlLink":"[Kernel] Change comparator expression to lazy evaluation (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2227170325\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/2853\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/2853/hovercard\" href=\"https://github.com/delta-io/delta/pull/2853\">#2853</a>)"}},{"before":"39e91af88c838902764743069f2d113ad39a50d3","after":"085f11718b2a4de2dc72d1cb09139039ba22dab2","ref":"refs/heads/master","pushedAt":"2024-05-30T05:38:29.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[Kernel] Change comparator expression to lazy evaluation (#2853)\n\n## Description\r\nResolves https://github.com/delta-io/delta/issues/2541\r\n\r\n## How was this patch tested?\r\nExisting tests","shortMessageHtmlLink":"[Kernel] Change comparator expression to lazy evaluation (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2227170325\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/2853\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/2853/hovercard\" href=\"https://github.com/delta-io/delta/pull/2853\">#2853</a>)"}},{"before":"6421bc52ce7d67e514fc565672cb225fbd5a1216","after":"39e91af88c838902764743069f2d113ad39a50d3","ref":"refs/heads/master","pushedAt":"2024-05-30T04:33:39.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[Kernel] Remove the Reference to Engine in LogReplay (#3165)\n\nRemove the reference to `Engine` in `LogReplay` and get it as an argument to methods on `LogReplay`\r\n\r\nResolves #2641","shortMessageHtmlLink":"[Kernel] Remove the Reference to Engine in LogReplay (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2321580030\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3165\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3165/hovercard\" href=\"https://github.com/delta-io/delta/pull/3165\">#3165</a>)"}},{"before":"6deecf994946b2a17c8a55864ad62ba2d0e0f931","after":"6421bc52ce7d67e514fc565672cb225fbd5a1216","ref":"refs/heads/master","pushedAt":"2024-05-30T00:26:39.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[Spark] Make listDeltaCompactedDeltaCheckpointFilesAndLatestChecksumFile reusable (#3157)\n\n## Description\r\nFactors out the main logic from\r\n`listDeltaCompactedDeltaCheckpointFilesAndLatestChecksumFile` into a new\r\nfunction `listFromFileSystemInternal` which can potentially be reused by\r\nother callers. Also added a new test utility which runs tests with\r\nin-memory-tracking commit owner enabled.\r\n\r\n## How was this patch tested?\r\nExisting tests should cover this.","shortMessageHtmlLink":"[Spark] Make listDeltaCompactedDeltaCheckpointFilesAndLatestChecksumF…"}},{"before":"46c18dfe78fc0d2c491cfaba564bdaea85d7dd3b","after":"6deecf994946b2a17c8a55864ad62ba2d0e0f931","ref":"refs/heads/master","pushedAt":"2024-05-29T21:43:47.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[Spark] Fix license headers for some Spark Connect proto files (#3170)\n\n## Description\r\n- Adjusted the formatting of the license headers of some Spark Connect\r\nproto files.\r\n- Added also the Delta Lake license since even though we copied the\r\nSpark Connect proto files, we modified the `option java_package` of each\r\nproto files in [Spark\r\nConnect](https://github.com/apache/spark/tree/master/connector/connect/common/src/main/protobuf/spark/connect),\r\nsimilar to how it was done\r\n[here](https://github.com/delta-io/delta/blob/master/spark/src/main/scala/org/apache/spark/sql/delta/util/DateFormatter.scala)\r\nby this\r\n[PR](https://github.com/delta-io/delta/commit/e26435bcd787b232c1cf73eb118202971f1e18f1#diff-727ff4ed853adab6032cd9d06895dee225f15dec45fbcb0541676e77f459ef2e).","shortMessageHtmlLink":"[Spark] Fix license headers for some Spark Connect proto files (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2323512204\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3170\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3170/hovercard\" href=\"https://github.com/delta-io/delta/pull/3170\">#3170</a>)"}},{"before":"31f09f05c2e78482b8ed69b5d31c1f222b5770c5","after":"46c18dfe78fc0d2c491cfaba564bdaea85d7dd3b","ref":"refs/heads/master","pushedAt":"2024-05-29T18:27:13.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[Kernel] Few updates to usage guide and javadocs (#3160)\n\n## Description\r\nChanges:\r\n- Fix a few Javadocs in the `Table` interface.\r\n- Update `USER_GUIDE.md` to fix the optional vector usage.","shortMessageHtmlLink":"[Kernel] Few updates to usage guide and javadocs (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2320070669\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3160\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3160/hovercard\" href=\"https://github.com/delta-io/delta/pull/3160\">#3160</a>)"}},{"before":"96411496a0cf9a60416e5ab83ae745ace3aa382a","after":"31f09f05c2e78482b8ed69b5d31c1f222b5770c5","ref":"refs/heads/master","pushedAt":"2024-05-29T15:40:43.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[Docs][4.0] Type Widening documentation (#3162)\n\n## Description\r\nUpdate the type widening documentation to list additional type changes\r\nsupported in Delta 4.0","shortMessageHtmlLink":"[Docs][4.0] Type Widening documentation (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2321269416\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3162\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3162/hovercard\" href=\"https://github.com/delta-io/delta/pull/3162\">#3162</a>)"}},{"before":"5829da49d0c3bb66ac6f26b89e7b9795f8bfdec0","after":"96411496a0cf9a60416e5ab83ae745ace3aa382a","ref":"refs/heads/master","pushedAt":"2024-05-29T14:27:06.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[Spark] Adding Spark Connect Protobuf Messages to the Delta repository (#3168)\n\n## Description\r\nDelta Connect is an extension for Spark Connect that adds support for\r\nthe DeltaTable API. To be able to communicate back to Spark Connect we\r\nneed to have access to Spark Connect's Protobufs.\r\n\r\nSince Buf does not support local dependencies, while Delta and Apache\r\nSpark are two separate repositories, the only solution for Delta Connect\r\nto get access to [Spark Connect's Protobuf\r\nmessages](https://github.com/apache/spark/tree/master/connector/connect/common/src/main/protobuf/spark/connect)\r\nis to have the copies of the messages in the Delta repository.\r\n\r\nThis may go out of sync, and we would need to update it manually from\r\nthe Apache Spark repository, but this should be fine and would not break\r\nanything since Protobufs are backward compatible.\r\n\r\n## How was this patch tested?\r\nN/A.","shortMessageHtmlLink":"[Spark] Adding Spark Connect Protobuf Messages to the Delta repository ("}},{"before":"eb638bba6973a98a13bc543d6d2bde0efd90b300","after":"5829da49d0c3bb66ac6f26b89e7b9795f8bfdec0","ref":"refs/heads/master","pushedAt":"2024-05-29T14:19:45.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[Spark] Set active txn during write and ctas commands (#3163)\n\n## Description\r\nMake WRITE commands call `OptimisticTransaction.withActive` to set as\r\nactive the transaction the transaction that is created by\r\n`startTxnForTableCreation()`.\r\n\r\n## How was this patch tested?\r\nExisting tests.","shortMessageHtmlLink":"[Spark] Set active txn during write and ctas commands (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2321293448\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3163\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3163/hovercard\" href=\"https://github.com/delta-io/delta/pull/3163\">#3163</a>)"}},{"before":"85428ee08cca58f601d9fc45e86b7a28705fee9b","after":"eb638bba6973a98a13bc543d6d2bde0efd90b300","ref":"refs/heads/master","pushedAt":"2024-05-29T14:18:58.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[Spark] Set up Python Protobuf codegen for Delta Connect (#3125)\n\n## Description\r\nAdded the very first protobuf messages for `DeltaTable`, `Clone` and\r\n`Scan`.\r\n\r\nThis is the first PR for Delta Connect, which adds support for Delta's\r\n`DeltaTable` interface to Spark Connect. This is needed to support Delta\r\ntable operations outside of SQL directly on Spark Connect clients.\r\n\r\nThis PR sets up the [Python code generation for the Protobufs of Delta\r\nConnect](https://protobuf.dev/getting-started/pythontutorial/). For this\r\nI created a new Buf workspace and I added a few initial Protobuf\r\nmessages to confirm that everything works. This is the ground work of\r\nthe project, before we move on to setting up the server and client\r\nlibrary.\r\n\r\nWhat we are doing here is similar to the [Spark Connect's protobuf\r\ndevelopment\r\nguide](https://github.com/apache/spark/tree/master/connector/connect).\r\n\r\n\r\n## How was this patch tested?\r\nAdded the `check-delta-connect-codegen-python.py` to the automated\r\ntesting, making sure the Python Protobuf Generated codes stay in sync\r\nwith the proto messages.","shortMessageHtmlLink":"[Spark] Set up Python Protobuf codegen for Delta Connect (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2308480869\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3125\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3125/hovercard\" href=\"https://github.com/delta-io/delta/pull/3125\">#3125</a>)"}},{"before":"6c0137b18eaf0ae93e8d2760381b879ed58794eb","after":"85428ee08cca58f601d9fc45e86b7a28705fee9b","ref":"refs/heads/master","pushedAt":"2024-05-27T17:29:25.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"allisonport-db","name":"Allison Portis","path":"/allisonport-db","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/89107911?s=80&v=4"},"commit":{"message":"Fix concating paths with relative filenames that contain colon ':' (#3153)\n\n#### Which Delta project/connector is this regarding?\r\n\r\n- [x] Spark\r\n- [ ] Standalone\r\n- [ ] Flink\r\n- [ ] Kernel\r\n- [ ] Other (fill in here)\r\n\r\n## Description\r\n\r\nThis PR fixes an issue where the `safeConcatPaths` method throws an\r\nexception when the `relativeChildPath` contains a colon `:`. Such a\r\ncharacter is not allowed in Hadoop paths due to ambiguity (`aa:bb.csv`\r\ncan be interpreted as an absolute path like `aa://bb.csv` where `aa` is\r\nthe scheme), but is allowed in many file systems such as S3. Thus we\r\nneed to handle this case.\r\n\r\nThe fix here is to prepend a `/` so that Hadoop will know that\r\neverything after `/` belongs to the path, not the scheme.\r\n\r\n## How was this patch tested?\r\n\r\nNew tests.\r\n\r\n## Does this PR introduce _any_ user-facing changes?\r\n\r\nNope.","shortMessageHtmlLink":"Fix concating paths with relative filenames that contain colon ':' (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2314744508\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3153\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3153/hovercard\" href=\"https://github.com/delta-io/delta/pull/3153\">#…</a>"}},{"before":"3cd9529b6a9fcc1fd6d72e2574760b1c622e12bb","after":"6c0137b18eaf0ae93e8d2760381b879ed58794eb","ref":"refs/heads/master","pushedAt":"2024-05-27T15:34:36.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"Protocol RFC for collations (#3068)\n\n## Description\r\n\r\nProtocol RFC for adding collation support to Delta\r\n\r\n[Design\r\ndoc](https://docs.google.com/document/d/1cwztlKt7b2hWF6Uu1S895ko6jPfRlP9x-V5POUcXtXk/edit?usp=sharing)","shortMessageHtmlLink":"Protocol RFC for collations (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2284926881\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3068\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3068/hovercard\" href=\"https://github.com/delta-io/delta/pull/3068\">#3068</a>)"}},{"before":"ff5b36fbcc3bb894b9a885eaa05338460c8173d6","after":"3cd9529b6a9fcc1fd6d72e2574760b1c622e12bb","ref":"refs/heads/master","pushedAt":"2024-05-24T18:53:35.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"allisonport-db","name":"Allison Portis","path":"/allisonport-db","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/89107911?s=80&v=4"},"commit":{"message":"[Spark] Add CREATE TABLE LIKE compatibility with user-provided table properties  (#3138)\n\n<!--\r\nThanks for sending a pull request!  Here are some tips for you:\r\n1. If this is your first time, please read our contributor guidelines:\r\nhttps://github.com/delta-io/delta/blob/master/CONTRIBUTING.md\r\n2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]\r\nYour PR title ...'.\r\n  3. Be sure to keep the PR description updated to reflect all changes.\r\n  4. Please write your PR title to summarize what this PR proposes.\r\n5. If possible, provide a concise example to reproduce the issue for a\r\nfaster review.\r\n6. If applicable, include the corresponding issue number in the PR title\r\nand link it in the body.\r\n-->\r\n\r\n#### Which Delta project/connector is this regarding?\r\n<!--\r\nPlease add the component selected below to the beginning of the pull\r\nrequest title\r\nFor example: [Spark] Title of my pull request\r\n-->\r\n\r\n- [x] Spark\r\n- [ ] Standalone\r\n- [ ] Flink\r\n- [ ] Kernel\r\n- [ ] Other (fill in here)\r\n\r\n## Description\r\n\r\n<!--\r\n- Describe what this PR changes.\r\n- Describe why we need the change.\r\n \r\nIf this PR resolves an issue be sure to include \"Resolves #XXX\" to\r\ncorrectly link and close the issue upon merge.\r\n-->\r\n\r\nUser provided properties when performing CREATE LIKE commands were being\r\nignored and only the properties from source table were being added. This\r\nPR adds/overwrites any applicable properties with the user provided\r\nones.\r\n\r\n## How was this patch tested?\r\n\r\n<!--\r\nIf tests were added, say they were added here. Please make sure to test\r\nthe changes thoroughly including negative and positive cases if\r\npossible.\r\nIf the changes were tested in any way other than unit tests, please\r\nclarify how you tested step by step (ideally copy and paste-able, so\r\nthat other reviewers can test and check, and descendants can verify in\r\nthe future).\r\nIf the changes were not tested, please explain why.\r\n-->\r\n\r\nUnit tests were created replicating the customer issue for CREATE LIKE\r\ncommands both originating in Delta tables and other formats.\r\n\r\n## Does this PR introduce _any_ user-facing changes?\r\n\r\n<!--\r\nIf yes, please clarify the previous behavior and the change this PR\r\nproposes - provide the console output, description and/or an example to\r\nshow the behavior difference if possible.\r\nIf possible, please also clarify if this is a user-facing change\r\ncompared to the released Delta Lake versions or within the unreleased\r\nbranches such as master.\r\nIf no, write 'No'.\r\n-->\r\n\r\nNo","shortMessageHtmlLink":"[Spark] Add CREATE TABLE LIKE compatibility with user-provided table …"}},{"before":"bfb5c94aa818495f35ed007bd4566cd2f7fecf42","after":"ff5b36fbcc3bb894b9a885eaa05338460c8173d6","ref":"refs/heads/master","pushedAt":"2024-05-24T16:34:11.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"allisonport-db","name":"Allison Portis","path":"/allisonport-db","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/89107911?s=80&v=4"},"commit":{"message":"[Spark] Allow type widening for all supported type changes with Spark 4.0 (#3024)\n\nThis PR adds shims to ungate the remaining type changes that only work\r\nwith Spark 4.0 / master. Spark 4.0 contains the required changes to\r\nParquet readers to be able to read the data after applying the type\r\nchanges.\r\n\r\n## Description\r\nExtend the list of supported type changes for type widening to include\r\nchanges that can be supported with Spark 4.0:\r\n- (byte, short, int) -> long\r\n- float -> double\r\n- date -> timestampNTZ\r\n- (byte, short, int) -> double\r\n- decimal -> decimal (with increased precision/scale that doesn't cause\r\nprecision loss)\r\n- (byte, short, int, long) -> decimal\r\n\r\nShims are added to support these changes when compiling against Spark\r\n4.0/master and to only allow `byte` -> `short` - > `int` when compiling\r\nagainst Spark 3.5.\r\n\r\n## How was this patch tested?\r\nAdding test cases for the new type changes in the existing type widening\r\ntest suites. The list of supported / unsupported changes covered in\r\ntests differs between Spark 3.5 and Spark 4.0, shims are also provided\r\nto handle this.\r\n\r\n## Does this PR introduce _any_ user-facing changes?\r\nYes: allow using the listed type changes with type widening, either via\r\n`ALTER TABLE CHANGE COLUMN TYPE` or during schema evolution in MERGE and\r\nINSERT.","shortMessageHtmlLink":"[Spark] Allow type widening for all supported type changes with Spark…"}},{"before":"039a29abb4abc72ac5912651679233dc983398d6","after":"bfb5c94aa818495f35ed007bd4566cd2f7fecf42","ref":"refs/heads/master","pushedAt":"2024-05-24T16:33:34.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"allisonport-db","name":"Allison Portis","path":"/allisonport-db","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/89107911?s=80&v=4"},"commit":{"message":"[Spark] Validate the expression in AlterTableAddConstraintDeltaCommand (#3143)\n\n#### Which Delta project/connector is this regarding?\r\n\r\n- [x] Spark\r\n- [ ] Standalone\r\n- [ ] Flink\r\n- [ ] Kernel\r\n- [ ] Other (fill in here)\r\n\r\n## Description\r\n\r\nThis PR fixes an internal error thrown from\r\n`AlterTableAddConstraintDeltaCommand`. This error is thrown when adding\r\na CHECK constraint with a non-existent column in the expression. The\r\nerror is thrown when we check if the expressions returns a boolean. This\r\nworks correctly for most expressions, but will result in an exception if\r\nthe data type of the unresolved column is checked.\r\n\r\nThis PR fixes this issue by making the analyzer responsible for checking\r\nwhether the expression returns a boolean by wrapping the expression with\r\na `Filter` node.\r\n\r\n## How was this patch tested?\r\n\r\nAdded a test\r\n\r\n## Does this PR introduce _any_ user-facing changes?\r\n\r\nYes, `ALTER TABLE ... ADD CONSTRAINT ... CHECK` will now throw a\r\n`UNRESOLVED_COLUMN` error instead of a `INTERNAL_ERROR` in the case\r\ndescribed above.","shortMessageHtmlLink":"[Spark] Validate the expression in AlterTableAddConstraintDeltaCommand ("}},{"before":"1609c38782f21c06fe3b24b32d0f4c97c3c2c755","after":"039a29abb4abc72ac5912651679233dc983398d6","ref":"refs/heads/master","pushedAt":"2024-05-23T22:54:45.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"tdas","name":"Tathagata Das","path":"/tdas","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/663212?s=80&v=4"},"commit":{"message":"[Spark] Test type widening compatibility with other Delta features (#3053)\n\n## Description\r\nAdditional tests covering type widening and:\r\n- Reading CDF\r\n- Column mapping\r\n- Time travel\r\n- RESTORE\r\n- CLONE\r\n\r\n## How was this patch tested?\r\nTest only","shortMessageHtmlLink":"[Spark] Test type widening compatibility with other Delta features (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2280778075\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3053\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3053/hovercard\" href=\"https://github.com/delta-io/delta/pull/3053\">#…</a>"}},{"before":"b043f5d7c2655c27866d4c33e2255e076f7598a2","after":"1609c38782f21c06fe3b24b32d0f4c97c3c2c755","ref":"refs/heads/master","pushedAt":"2024-05-23T16:06:08.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[Spark] Update OptimizeGeneratedSuite to apply constant folding (#3141)\n\n## Description\r\nThe following change in Spark master broke tests in\r\n`OptimizeGeneratedColumnSuite`:\r\nhttps://github.com/apache/spark/commit/7974811218c9fb52ac9d07f8983475a885ada81b\r\n\r\nIt added an execution of the `ConstantFolding` rule after\r\n`PrepareDeltaScan`, causing constant expressions in filters on generated\r\ncolumns to be simplified, which `OptimizeGeneratedColumnSuite` heavily\r\nused.\r\n\r\nThis change:\r\n- updates the expected results in `OptimizeGeneratedColumnSuite` to\r\nsimplify constant expressions\r\n- adds a pass of `ConstantFolding` after `PrepareDeltaScan` so that\r\nDelta on spark 3.5 behaves the same as Delta on spark master.\r\n\r\n## How was this patch tested?\r\nUpdated tests","shortMessageHtmlLink":"[Spark] Update OptimizeGeneratedSuite to apply constant folding (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2312091783\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3141\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3141/hovercard\" href=\"https://github.com/delta-io/delta/pull/3141\">#3141</a>)"}},{"before":"0deef042b18689fd4b73f4b252700dd2f1ab94f8","after":"b043f5d7c2655c27866d4c33e2255e076f7598a2","ref":"refs/heads/master","pushedAt":"2024-05-23T15:57:04.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[Spark] Make ManagedCommit a preview feature (#3137)\n\n## Description\r\nRenames the ManagedCommit feature and config names by replacing -dev\r\nwith -preview to indicate that it is in preview phase.\r\n\r\n## How was this patch tested?\r\nNo new tests.","shortMessageHtmlLink":"[Spark] Make ManagedCommit a preview feature (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2311201803\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3137\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3137/hovercard\" href=\"https://github.com/delta-io/delta/pull/3137\">#3137</a>)"}},{"before":"35c7536a70c2d0ba57e140704d3e213e4e75a516","after":"0deef042b18689fd4b73f4b252700dd2f1ab94f8","ref":"refs/heads/master","pushedAt":"2024-05-23T15:45:39.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[Kernel][Expressions] Add support for LIKE expression (#3103)\n\n## Description\r\nAdd SQL `LIKE` expression support in Kernel list of supported expressions and a default implementation.\r\n\r\nAddresses part of https://github.com/delta-io/delta/issues/2539 (where `STARTS_WITH` as `LIKE 'str%'`)\r\n\r\n## How was this patch tested?\r\nadded unit tests\r\n\r\nSigned-off-by: Krishnan Paranji Ravi <krishna.pr@gmail.com>","shortMessageHtmlLink":"[Kernel][Expressions] Add support for LIKE expression (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2298974520\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3103\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3103/hovercard\" href=\"https://github.com/delta-io/delta/pull/3103\">#3103</a>)"}},{"before":"420d9e059db18845a49b85cb1571752667d39dc6","after":"35c7536a70c2d0ba57e140704d3e213e4e75a516","ref":"refs/heads/master","pushedAt":"2024-05-22T21:58:33.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[INFRA] Improve the java style checks log the errors to sbt console (#3115)\n\n## Description\r\nResolves #3067.\r\n\r\n## How was this patch tested?\r\nOn local machine, intentionally create checkstyle errors in module\r\n`kernelDefaults` (for experimental), then run the `build/sbt compile`\r\nand `build/sbt kernelDefaults/test`.\r\n\r\nSigned-off-by: Tai Le Manh <manhtai.lmt@gmail.com>","shortMessageHtmlLink":"[INFRA] Improve the java style checks log the errors to sbt console (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2304154169\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3115\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3115/hovercard\" href=\"https://github.com/delta-io/delta/pull/3115\">#…</a>"}},{"before":"a5263cc02b450b16d80b65f358dbf5ff092355bf","after":"420d9e059db18845a49b85cb1571752667d39dc6","ref":"refs/heads/master","pushedAt":"2024-05-22T16:43:30.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[Standalone] Introduce FileAction.tagsOrEmpty (#3132)\n\n## Description\r\nThis PR introduces Introduce `FileAction.tagsOrEmpty` to factor out the\r\ncommon pattern `Option(tags).getOrElse(Map.empty)`.\r\n\r\n## How was this patch tested?\r\nExisting unit tests.\r\n\r\n## Does this PR introduce _any_ user-facing changes?\r\nNo\r\n\r\nSigned-off-by: Sergiu Pocol <sergiu.pocol@databricks.com>","shortMessageHtmlLink":"[Standalone] Introduce FileAction.tagsOrEmpty (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2309617721\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3132\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3132/hovercard\" href=\"https://github.com/delta-io/delta/pull/3132\">#3132</a>)"}},{"before":"d5e9a26195742728ea4693a6abca493c5e6e2241","after":"a5263cc02b450b16d80b65f358dbf5ff092355bf","ref":"refs/heads/master","pushedAt":"2024-05-22T16:42:20.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[Standalone] AddFile Long Tags Accessor + Memory Optimization (#3131)\n\n## Description\r\nThis PR introduces AddFile.longTag which factors out the pattern\r\n`tag(...).map(_.toLong)` and also converts the insertion time tag lazy\r\nval to a method in order to save memory.\r\n\r\n## How was this patch tested?\r\nExisting unit tests.\r\n\r\n## Does this PR introduce _any_ user-facing changes?\r\nNo\r\n\r\nSigned-off-by: Sergiu Pocol <sergiu.pocol@databricks.com>","shortMessageHtmlLink":"[Standalone] AddFile Long Tags Accessor + Memory Optimization (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2309535571\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3131\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3131/hovercard\" href=\"https://github.com/delta-io/delta/pull/3131\">#3131</a>)"}},{"before":"0c35eea4100d83040a11417f66016b48c246c466","after":"d5e9a26195742728ea4693a6abca493c5e6e2241","ref":"refs/heads/master","pushedAt":"2024-05-22T16:40:32.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"vkorukanti","name":"Venki Korukanti","path":"/vkorukanti","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/1719945?s=80&v=4"},"commit":{"message":"[Spark] DynamoDBCommitOwner: add logging, get dynamic confs from sparkSession (#3130)\n\n## Description\r\nUpdates DynamoDBCommitOwner:\r\n\r\n - Added logging around table creation flow\r\n - Get wcu, rcu, and awsCredentialsProvider from SparkSession\r\n- Return -1 as the table version if registerTable has already been\r\ncalled but no actual commits have gone through the owner. This is done\r\nby tracking an extra flag in DynamoDB.\r\n\r\n## How was this patch tested?\r\nExisting tests\r\n\r\n## Does this PR introduce _any_ user-facing changes?\r\nYes, introduces new configs (see DeltaSQLConf changes) which can be used\r\nto configure the DynamoDBCommitOwner.","shortMessageHtmlLink":"[Spark] DynamoDBCommitOwner: add logging, get dynamic confs from spar…"}},{"before":"529717bb6f171ff5252e3a913dd3667d53a2095c","after":"0c35eea4100d83040a11417f66016b48c246c466","ref":"refs/heads/master","pushedAt":"2024-05-22T13:07:11.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"tdas","name":"Tathagata Das","path":"/tdas","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/663212?s=80&v=4"},"commit":{"message":"[Spark] Column Mapping DROP FEATURE (#3124)\n\n<!--\r\nThanks for sending a pull request!  Here are some tips for you:\r\n1. If this is your first time, please read our contributor guidelines:\r\nhttps://github.com/delta-io/delta/blob/master/CONTRIBUTING.md\r\n2. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]\r\nYour PR title ...'.\r\n  3. Be sure to keep the PR description updated to reflect all changes.\r\n  4. Please write your PR title to summarize what this PR proposes.\r\n5. If possible, provide a concise example to reproduce the issue for a\r\nfaster review.\r\n6. If applicable, include the corresponding issue number in the PR title\r\nand link it in the body.\r\n-->\r\n\r\n#### Which Delta project/connector is this regarding?\r\n<!--\r\nPlease add the component selected below to the beginning of the pull\r\nrequest title\r\nFor example: [Spark] Title of my pull request\r\n-->\r\n\r\n- [x] Spark\r\n- [ ] Standalone\r\n- [ ] Flink\r\n- [ ] Kernel\r\n- [ ] Other (fill in here)\r\n\r\n## Description\r\nAllow column mapping feature to be dropped.\r\n\r\n```\r\nALTER TABLE <table_name> DROP FEATURE columnMapping\r\n```\r\nFeature is hidden behind a flag.\r\n<!--\r\n- Describe what this PR changes.\r\n- Describe why we need the change.\r\n \r\nIf this PR resolves an issue be sure to include \"Resolves #XXX\" to\r\ncorrectly link and close the issue upon merge.\r\n-->\r\n\r\n## How was this patch tested?\r\nnew unit tests\r\n<!--\r\nIf tests were added, say they were added here. Please make sure to test\r\nthe changes thoroughly including negative and positive cases if\r\npossible.\r\nIf the changes were tested in any way other than unit tests, please\r\nclarify how you tested step by step (ideally copy and paste-able, so\r\nthat other reviewers can test and check, and descendants can verify in\r\nthe future).\r\nIf the changes were not tested, please explain why.\r\n-->\r\n\r\n## Does this PR introduce _any_ user-facing changes?\r\n\r\n<!--\r\nIf yes, please clarify the previous behavior and the change this PR\r\nproposes - provide the console output, description and/or an example to\r\nshow the behavior difference if possible.\r\nIf possible, please also clarify if this is a user-facing change\r\ncompared to the released Delta Lake versions or within the unreleased\r\nbranches such as master.\r\nIf no, write 'No'.\r\n-->\r\nNo","shortMessageHtmlLink":"[Spark] Column Mapping DROP FEATURE (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2308000207\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3124\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3124/hovercard\" href=\"https://github.com/delta-io/delta/pull/3124\">#3124</a>)"}},{"before":"f2d6c8b4e1ccdd1fcdffc44f87536f1f56408d31","after":"529717bb6f171ff5252e3a913dd3667d53a2095c","ref":"refs/heads/master","pushedAt":"2024-05-22T05:46:56.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"allisonport-db","name":"Allison Portis","path":"/allisonport-db","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/89107911?s=80&v=4"},"commit":{"message":"[Spark] Metadata Cleanup for Unbackfilled Delta Files (#3094)\n\n#### Which Delta project/connector is this regarding?\r\n\r\n- [x] Spark\r\n- [ ] Standalone\r\n- [ ] Flink\r\n- [ ] Kernel\r\n- [ ] Other (fill in here)\r\n\r\n## Description\r\n\r\nUnbackfilled deltas eligible for deletion:\r\n- Version <= max(backfilled-delta-deleted-versions)\r\n\r\n## How was this patch tested?\r\n\r\nUnit tests\r\n\r\n## Does this PR introduce _any_ user-facing changes?\r\n\r\nNo","shortMessageHtmlLink":"[Spark] Metadata Cleanup for Unbackfilled Delta Files (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2296051299\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3094\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3094/hovercard\" href=\"https://github.com/delta-io/delta/pull/3094\">#3094</a>)"}},{"before":"0ee9fd0996e2d34630cd094123f54562211570af","after":"f2d6c8b4e1ccdd1fcdffc44f87536f1f56408d31","ref":"refs/heads/master","pushedAt":"2024-05-22T05:46:24.000Z","pushType":"pr_merge","commitsCount":1,"pusher":{"login":"allisonport-db","name":"Allison Portis","path":"/allisonport-db","primaryAvatarUrl":"https://avatars.githubusercontent.com/u/89107911?s=80&v=4"},"commit":{"message":"[Spark] Apply filters pushed down into DeltaCDFRelation (#3127)\n\n#### Which Delta project/connector is this regarding?\r\n\r\n- [x] Spark\r\n- [ ] Standalone\r\n- [ ] Flink\r\n- [ ] Kernel\r\n- [ ] Other (fill in here)\r\n\r\n## Description\r\n\r\nThis PR modifies `DeltaCDFRelation` to apply the filters that are pushed\r\ndown into this. This enables both partition pruning and row group\r\nskipping to happen when reading the Change Data Feed.\r\n\r\n## How was this patch tested?\r\n\r\nUnit tests\r\n\r\n## Does this PR introduce _any_ user-facing changes?\r\n\r\nNo","shortMessageHtmlLink":"[Spark] Apply filters pushed down into DeltaCDFRelation (<a class=\"issue-link js-issue-link\" data-error-text=\"Failed to load title\" data-id=\"2308910403\" data-permission-text=\"Title is private\" data-url=\"https://github.com/delta-io/delta/issues/3127\" data-hovercard-type=\"pull_request\" data-hovercard-url=\"/delta-io/delta/pull/3127/hovercard\" href=\"https://github.com/delta-io/delta/pull/3127\">#3127</a>)"}}],"hasNextPage":true,"hasPreviousPage":false,"activityType":"all","actor":null,"timePeriod":"all","sort":"DESC","perPage":30,"cursor":"djE6ks8AAAAEWXOQRQA","startCursor":null,"endCursor":null}},"title":"Activity · delta-io/delta"}