[Feature][Connector-v2][PulsarSink]Add Pulsar Sink Connector. #4382

lightzhao · 2023-03-21T10:52:11Z

Purpose of this pull request

Add Pulsar Sink Connector. #3018

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

lightzhao · 2023-03-24T08:48:10Z

Hello everyone, ci check cannot automatically, ci check require approval?

lightzhao · 2023-03-27T02:41:44Z

Hello everyone, it has been running stably in the test environment for two weeks, who has time to help review.

hailin0

please update release-note.md
please add e2e testcase

reference
https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-kafka-e2e

hailin0 · 2023-03-27T08:28:09Z

docs/en/connector-v2/sink/pulsar.md

+- [x] [batch](../../concept/connector-v2-features.md)
+- [x] [stream](../../concept/connector-v2-features.md)


Suggested change

- [x] [batch](../../concept/connector-v2-features.md)

- [x] [stream](../../concept/connector-v2-features.md)

https://github.com/apache/incubator-seatunnel/blob/dev/docs/en/concept/connector-v2-features.md#sink-connector-features

hailin0 · 2023-03-27T08:28:26Z

docs/en/connector-v2/sink/pulsar.md

+- [ ] [column projection](../../concept/connector-v2-features.md)
+- [x] [parallelism](../../concept/connector-v2-features.md)
+- [ ] [support user-defined split](../../concept/connector-v2-features.md)


Suggested change

- [ ] [column projection](../../concept/connector-v2-features.md)

- [x] [parallelism](../../concept/connector-v2-features.md)

- [ ] [support user-defined split](../../concept/connector-v2-features.md)

lightzhao · 2023-03-27T08:42:24Z

please update release-note.md please add e2e testcase

reference https://github.com/apache/incubator-seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-kafka-e2e

waiting #3990 ,In this pr I added the pulsar-e2e module. doc has been updated.

lightzhao · 2023-03-27T08:50:19Z

Hello everyone, why can't the automatic ci check be performed after updating the code now, and it needs to be approved?

TyrantLucifer · 2023-04-03T09:48:21Z

Hello everyone, why can't the automatic ci check be performed after updating the code now, and it needs to be approved?

Yes, infra changed the rule of this. It needs to be approved.

lightzhao · 2023-04-04T03:41:20Z

Hello everyone, why can't the automatic ci check be performed after updating the code now, and it needs to be approved?

Yes, infra changed the rule of this. It needs to be approved.

please review #3990，and then add pulsar sink e2e test. thanks.

# Conflicts: # seatunnel-connectors-v2/connector-pulsar/pom.xml

…eatunnel into create-pulsar-sink

lightzhao · 2023-05-26T10:38:10Z

please approve ci.

lightzhao · 2023-05-30T08:19:13Z

@Hisoka-X @hailin0 @TyrantLucifer please approve ci.

lightzhao · 2023-06-16T08:20:12Z

@Hisoka-X @hailin0 @EricJoy2048 please approve ci.

lightzhao · 2023-06-26T01:32:46Z

@TyrantLucifer @hailin0 @EricJoy2048 @Hisoka-X PTAL.

lightzhao · 2023-06-28T01:21:50Z

@ALL Can anyone help to review this pr?

lightzhao · 2023-07-19T10:44:20Z

@TyrantLucifer @hailin0 @EricJoy2048 @Hisoka-X PTAL.

lightzhao · 2023-07-24T10:57:29Z

It's been a long time, @TyrantLucifer @hailin0 @EricJoy2048 @Hisoka-X PTAL.

Carl-Zhou-CN · 2023-12-01T08:50:21Z

docs/en/connector-v2/sink/pulsar.md

@@ -0,0 +1,174 @@
+# Apache Pulsar


docs/en/connector-v2/sink/puslar.md -> Please refer to the "kafka.md" file for some upgrade instructions,
https://github.com/apache/seatunnel/blob/dev/docs/en/connector-v2/sink/Kafka.md

Carl-Zhou-CN · 2023-12-01T08:55:34Z

...r/src/main/java/org/apache/seatunnel/connectors/seatunnel/pulsar/sink/PulsarSinkFactory.java

+import static org.apache.seatunnel.connectors.seatunnel.pulsar.config.SourceProperties.CLIENT_SERVICE_URL;
+
+@AutoService(Factory.class)
+public class PulsarSinkFactory implements TableSinkFactory {


Support TableSourceFactory/TableSinkFactory on connector，please refer to the following link for the upgrade instructions: #5651.

Carl-Zhou-CN · 2023-12-01T08:59:49Z

...r-pulsar/src/main/java/org/apache/seatunnel/connectors/seatunnel/pulsar/sink/PulsarSink.java

+ * {@link PulsarSinkWriter} and {@link PulsarSinkCommitter}.
+ */
+@AutoService(SeaTunnelSink.class)
+public class PulsarSink


Support multi-table sink feature: please refer to the following link for the upgrade instructions: #5652

Do kafka and pulsar need the support of multiple table sinks? Multiple topics are supported.

@Hisoka-X cc

Supporting multiple tables is mainly to accept multiple upstream tables, which can be implemented separately in the future.

Carl-Zhou-CN · 2023-12-01T09:45:35Z

...ar/src/main/java/org/apache/seatunnel/connectors/seatunnel/pulsar/state/PulsarSinkState.java

+public class PulsarSinkState implements Serializable {
+
+    /** The transaction id. */
+    private final TxnID txnID;


Wouldn't it be better to store the 'topic' as part of the state?

Only TxnID is required when commit, and the topic parameter is not required.

Carl-Zhou-CN · 2023-12-19T01:33:03Z

...ar/src/main/java/org/apache/seatunnel/connectors/seatunnel/pulsar/sink/PulsarSinkWriter.java

+    /**
+     * get message SerializationSchema
+     *
+     * @param rowType


The comment of these parameters is useless, can we delete it？

Carl-Zhou-CN · 2023-12-19T01:33:34Z

...ar/src/main/java/org/apache/seatunnel/connectors/seatunnel/pulsar/sink/PulsarSinkWriter.java

+    /**
+     * get key SerializationSchema
+     *
+     * @param keyFieldNames


Same as above

Carl-Zhou-CN · 2023-12-19T01:33:47Z

...ar/src/main/java/org/apache/seatunnel/connectors/seatunnel/pulsar/sink/PulsarSinkWriter.java

+    /**
+     * get partition key field list
+     *
+     * @param pluginConfig


Same as above

lightzhao · 2023-12-22T03:03:29Z

@hailin0 @Carl-Zhou-CN @Hisoka-X PTAL.

Carl-Zhou-CN

LGTM

hailin0

LGTM

...l-e2e/seatunnel-connector-v2-e2e/connector-pulsar-e2e/src/test/resources/fake_to_pulsar.conf

…src/test/resources/fake_to_pulsar.conf Co-authored-by: hailin0 <hailin088@gmail.com>

lightzhao · 2024-01-11T02:10:27Z

@hailin0 @Hisoka-X @Carl-Zhou-CN PTAL.

lightzhao added 4 commits March 21, 2023 18:50

Add Pulsar sink connector.

3491092

Add license header.

3046c38

Merge branch 'dev' into create-pulsar-sink

43aad76

Modify asynchronous write mode to improve write throughput performance

ab1e71c

hailin0 reviewed Mar 27, 2023

View reviewed changes

lightzhao added 2 commits March 27, 2023 16:37

update doc.

0b8d97f

Merge branch 'dev' into create-pulsar-sink

5584b72

lightzhao mentioned this pull request Apr 13, 2023

[Improve][Connector][PulsarSource]Improve pulsar deserialization #3990

Closed

5 tasks

lightzhao mentioned this pull request May 12, 2023

[Improve][Connector-V2][Pulsar-Sink] Support pulsar sink #4737

Closed

3 tasks

lightzhao and others added 5 commits May 24, 2023 09:36

Merge branch 'dev' into create-pulsar-sink

b7cde11

# Conflicts: # seatunnel-connectors-v2/connector-pulsar/pom.xml

Merge branch 'dev' into create-pulsar-sink

0a94a8d

Merge branch 'dev' into create-pulsar-sink

50d3ad7

add e2e test.

ce34460

Merge branch 'create-pulsar-sink' of github.com:lightzhao/incubator-s…

86f1c0d

…eatunnel into create-pulsar-sink

lightzhao requested a review from hailin0 May 26, 2023 10:37

ic4y closed this Aug 5, 2023

Carl-Zhou-CN reviewed Dec 1, 2023

View reviewed changes

lightzhao added 5 commits December 1, 2023 18:59

Merge branch 'dev' into create-pulsar-sink

918729d

Merge branch 'dev' into create-pulsar-sink

a80e32c

Support multi-table sink feature and update doc.

c9ca686

Support multi-table sink feature and update doc.

612d000

Merge branch 'dev' into create-pulsar-sink

994242b

Carl-Zhou-CN reviewed Dec 19, 2023

View reviewed changes

lightzhao added 2 commits December 19, 2023 18:55

remove unless code.

ba5a187

Merge branch 'dev' into create-pulsar-sink

0122d77

lightzhao requested review from Hisoka-X and Carl-Zhou-CN December 21, 2023 10:57

Carl-Zhou-CN previously approved these changes Dec 24, 2023

View reviewed changes

github-actions bot added approved reviewed labels Dec 24, 2023

hailin0 reviewed Dec 26, 2023

View reviewed changes

...l-e2e/seatunnel-connector-v2-e2e/connector-pulsar-e2e/src/test/resources/fake_to_pulsar.conf Outdated Show resolved Hide resolved

Update seatunnel-e2e/seatunnel-connector-v2-e2e/connector-pulsar-e2e/…

126874d

…src/test/resources/fake_to_pulsar.conf Co-authored-by: hailin0 <hailin088@gmail.com>

EricJoy2048 dismissed Carl-Zhou-CN’s stale review via 126874d January 5, 2024 07:34

github-actions bot removed approved reviewed labels Jan 5, 2024

EricJoy2048 approved these changes Jan 5, 2024

View reviewed changes

github-actions bot added approved reviewed labels Jan 5, 2024

lightzhao requested review from hailin0 and Carl-Zhou-CN January 10, 2024 08:54

hailin0 approved these changes Jan 11, 2024

View reviewed changes

hailin0 merged commit 543d2c5 into apache:dev Jan 11, 2024
7 checks passed

		- [x] [batch](../../concept/connector-v2-features.md)
		- [x] [stream](../../concept/connector-v2-features.md)

	- [ ] [column projection](../../concept/connector-v2-features.md)
	- [x] [parallelism](../../concept/connector-v2-features.md)
	- [ ] [support user-defined split](../../concept/connector-v2-features.md)

[Feature][Connector-v2][PulsarSink]Add Pulsar Sink Connector. #4382

[Feature][Connector-v2][PulsarSink]Add Pulsar Sink Connector. #4382

Conversation

lightzhao commented Mar 21, 2023 • edited

Purpose of this pull request

Check list

lightzhao commented Mar 24, 2023 • edited

lightzhao commented Mar 27, 2023

hailin0 left a comment • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lightzhao commented Mar 27, 2023

lightzhao commented Mar 27, 2023

TyrantLucifer commented Apr 3, 2023

lightzhao commented Apr 4, 2023

lightzhao commented May 26, 2023

lightzhao commented May 30, 2023

lightzhao commented Jun 16, 2023

lightzhao commented Jun 26, 2023

lightzhao commented Jun 28, 2023

lightzhao commented Jul 19, 2023

lightzhao commented Jul 24, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lightzhao commented Dec 22, 2023

Carl-Zhou-CN left a comment

Choose a reason for hiding this comment

hailin0 left a comment

Choose a reason for hiding this comment

lightzhao commented Jan 11, 2024

lightzhao commented Mar 21, 2023 •

edited

lightzhao commented Mar 24, 2023 •

edited

hailin0 left a comment •

edited