Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improve][CDC] Bump the version of debezium to 1.9.8.Final #6740

Merged
merged 7 commits into from
May 23, 2024
Merged

[Improve][CDC] Bump the version of debezium to 1.9.8.Final #6740

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
a339b2e
[Improve][CDC] Bump the version of debezium to 1.9.8.Final
dailai Apr 16, 2024
ab70475
[Improve][CDC] Optimize logminer latency
dailai Apr 23, 2024
52394ac
[Improve][CDC] Update the link with debezium in cdc docs
dailai Apr 23, 2024
35ceb30
[Imporve][CDC] Fix the classloader to load the EmbeddedDatabaseHistory
dailai Apr 24, 2024
92b6106
[Imporve][CDC] Handle the default value of debezium column of mysql
dailai Apr 25, 2024
f59c4d6
[Improve][CDC] Set temporalAdjuster at defaultValue process
dailai Apr 25, 2024
b61f2a7
[Imporve][CDC] Spotless
dailai May 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/en/connector-v2/source/MySQL-CDC.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,7 @@ When an initial consistent snapshot is made for large databases, your establishe
- `interactive_timeout`: The number of seconds the server waits for activity on an interactive connection before closing it. See [MySQL’s documentation](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_interactive_timeout) for more details.
- `wait_timeout`: The number of seconds the server waits for activity on a non-interactive connection before closing it. See [MySQL’s documentation](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_wait_timeout) for more details.

*For more database settings see [Debezium MySQL Connector](https://github.com/debezium/debezium/blob/1.6/documentation/modules/ROOT/pages/connectors/mysql.adoc#set-up)*
*For more database settings see [Debezium MySQL Connector](https://debezium.io/documentation/reference/1.9/connectors/mysql.html#setting-up-mysql)*

## Data Type Mapping

Expand Down Expand Up @@ -196,7 +196,7 @@ When an initial consistent snapshot is made for large databases, your establishe
| inverse-sampling.rate | Integer | No | 1000 | The inverse of the sampling rate used in the sample sharding strategy. For example, if this value is set to 1000, it means a 1/1000 sampling rate is applied during the sampling process. This option provides flexibility in controlling the granularity of the sampling, thus affecting the final number of shards. It's especially useful when dealing with very large datasets where a lower sampling rate is preferred. The default value is 1000. |
| exactly_once | Boolean | No | false | Enable exactly once semantic. |
| format | Enum | No | DEFAULT | Optional output format for MySQL CDC, valid enumerations are `DEFAULT`、`COMPATIBLE_DEBEZIUM_JSON`. |
| debezium | Config | No | - | Pass-through [Debezium's properties](https://github.com/debezium/debezium/blob/1.6/documentation/modules/ROOT/pages/connectors/mysql.adoc#connector-properties) to Debezium Embedded Engine which is used to capture data changes from MySQL server. |
| debezium | Config | No | - | Pass-through [Debezium's properties](https://debezium.io/documentation/reference/1.9/connectors/mysql.html#mysql-connector-properties) to Debezium Embedded Engine which is used to capture data changes from MySQL server. |
| common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details |

## Task Example
Expand Down
7 changes: 6 additions & 1 deletion docs/en/connector-v2/source/Oracle-CDC.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,11 @@
The Oracle CDC connector allows for reading snapshot data and incremental data from Oracle database. This document
describes how to set up the Oracle CDC connector to run SQL queries against Oracle databases.

## Notice

The Debezium Oracle connector does not rely on the continuous mining option. The connector is responsible for detecting log switches and adjusting the logs that are mined automatically, which the continuous mining option did for you automatically.
So, you can not set this property named `log.mining.continuous.mine` in the debezium.

## Supported DataSource Info

| Datasource | Supported versions | Driver | Url | Maven |
Expand Down Expand Up @@ -240,7 +245,7 @@ exit;
| inverse-sampling.rate | Integer | No | 1000 | The inverse of the sampling rate used in the sample sharding strategy. For example, if this value is set to 1000, it means a 1/1000 sampling rate is applied during the sampling process. This option provides flexibility in controlling the granularity of the sampling, thus affecting the final number of shards. It's especially useful when dealing with very large datasets where a lower sampling rate is preferred. The default value is 1000. |
| exactly_once | Boolean | No | false | Enable exactly once semantic. |
| format | Enum | No | DEFAULT | Optional output format for Oracle CDC, valid enumerations are `DEFAULT`、`COMPATIBLE_DEBEZIUM_JSON`. |
| debezium | Config | No | - | Pass-through [Debezium's properties](https://github.com/debezium/debezium/blob/1.6/documentation/modules/ROOT/pages/connectors/oracle.adoc#connector-properties) to Debezium Embedded Engine which is used to capture data changes from Oracle server. |
| debezium | Config | No | - | Pass-through [Debezium's properties](https://debezium.io/documentation/reference/1.9/connectors/oracle.html#oracle-connector-properties) to Debezium Embedded Engine which is used to capture data changes from Oracle server. |
| common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details |

## Task Example
Expand Down
2 changes: 1 addition & 1 deletion docs/en/connector-v2/source/Postgre-CDC.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ ALTER TABLE your_table_name REPLICA IDENTITY FULL;
| inverse-sampling.rate | Integer | No | 1000 | The inverse of the sampling rate used in the sample sharding strategy. For example, if this value is set to 1000, it means a 1/1000 sampling rate is applied during the sampling process. This option provides flexibility in controlling the granularity of the sampling, thus affecting the final number of shards. It's especially useful when dealing with very large datasets where a lower sampling rate is preferred. The default value is 1000. |
| exactly_once | Boolean | No | false | Enable exactly once semantic. |
| format | Enum | No | DEFAULT | Optional output format for Postgre CDC, valid enumerations are `DEFAULT`、`COMPATIBLE_DEBEZIUM_JSON`. |
| debezium | Config | No | - | Pass-through [Debezium's properties](https://github.com/debezium/debezium/blob/1.6/documentation/modules/ROOT/pages/connectors/postgresql.adoc#connector-configuration-properties) to Debezium Embedded Engine which is used to capture data changes from Postgre server. |
| debezium | Config | No | - | Pass-through [Debezium's properties](https://debezium.io/documentation/reference/1.9/connectors/postgresql.html#postgresql-connector-properties) to Debezium Embedded Engine which is used to capture data changes from Postgre server. |
| common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details |

## Task Example
Expand Down
2 changes: 1 addition & 1 deletion docs/en/connector-v2/source/SqlServer-CDC.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ describes how to setup the Sql Server CDC connector to run SQL queries against S
| sample-sharding.threshold | int | No | 1000 | This configuration specifies the threshold of estimated shard count to trigger the sample sharding strategy. When the distribution factor is outside the bounds specified by `chunk-key.even-distribution.factor.upper-bound` and `chunk-key.even-distribution.factor.lower-bound`, and the estimated shard count (calculated as approximate row count / chunk size) exceeds this threshold, the sample sharding strategy will be used. This can help to handle large datasets more efficiently. The default value is 1000 shards. |
| inverse-sampling.rate | int | No | 1000 | The inverse of the sampling rate used in the sample sharding strategy. For example, if this value is set to 1000, it means a 1/1000 sampling rate is applied during the sampling process. This option provides flexibility in controlling the granularity of the sampling, thus affecting the final number of shards. It's especially useful when dealing with very large datasets where a lower sampling rate is preferred. The default value is 1000. |
| exactly_once | Boolean | No | false | Enable exactly once semantic. |
| debezium.* | config | No | - | Pass-through Debezium's properties to Debezium Embedded Engine which is used to capture data changes from SqlServer server.<br/>See more about<br/>the [Debezium's SqlServer Connector properties](https://github.com/debezium/debezium/blob/1.6/documentation/modules/ROOT/pages/connectors/sqlserver.adoc#connector-properties) |
| debezium.* | config | No | - | Pass-through Debezium's properties to Debezium Embedded Engine which is used to capture data changes from SqlServer server.<br/>See more about<br/>the [Debezium's SqlServer Connector properties](https://debezium.io/documentation/reference/1.9/connectors/sqlserver.html#sqlserver-connector-properties) |
| format | Enum | No | DEFAULT | Optional output format for SqlServer CDC, valid enumerations are "DEFAULT"、"COMPATIBLE_DEBEZIUM_JSON". |
| common-options | | no | - | Source plugin common parameters, please refer to [Source Common Options](common-options.md) for details. |

Expand Down
5 changes: 5 additions & 0 deletions seatunnel-connectors-v2/connector-cdc/connector-cdc-base/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,11 @@
<groupId>org.apache.seatunnel</groupId>
<artifactId>seatunnel-format-compatible-debezium-json</artifactId>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>${commons-lang3.version}</version>
</dependency>

<dependency>
<groupId>junit</groupId>
Expand Down