[CORE-2736] Adds output batch compression for Data Transforms #18514

oleiman · 2024-05-15T21:22:12Z

This PR adds the ability to configure output batch compression on WASM transforms, either at deploy time or post facto, via metadata patch request. Includes rpk experience for same

TODO:

rpk bits and tests for deploying w/ compression turned on

Backports Required

Release Notes

Improvements

Add output batch compression for Data Transforms (configurable per deployed transform)

oleiman · 2024-05-16T05:45:59Z

/ci-repeat 1

oleiman · 2024-05-16T19:50:09Z

/ci-repeat 1
release
skip-units
skip-redpanda-build

vbotbuildovich · 2024-05-16T21:33:14Z

new failures in https://buildkite.com/redpanda/redpanda/builds/49248#018f82f5-d2fc-4bd5-b204-a1774afd79bf:

"rptest.tests.connection_virtualizing_test.TestVirtualConnections.test_handling_invalid_ids"

new failures in https://buildkite.com/redpanda/redpanda/builds/49248#018f82f5-d2ff-46d2-8840-d9e063a709e2:

"rptest.tests.connection_virtualizing_test.TestVirtualConnections.test_no_head_of_line_blocking.different_clusters=False.different_connections=False"

src/v/redpanda/admin/api-doc/transform.json

rockwotj · 2024-05-19T23:41:02Z

src/go/rpk/pkg/cli/transform/deploy.go

@@ -149,6 +153,7 @@ The --var flag can be repeated to specify multiple variables like so:
 	cmd.Flags().StringSliceVarP(&fc.outputTopics, "output-topic", "o", []string{}, "The output topic to write the transform results to (repeatable)")
 	cmd.Flags().StringVar(&fc.functionName, "name", "", "The name of the transform")
 	cmd.Flags().Var(&fc.env, "var", "Specify an environment variable in the form of KEY=VALUE")
+	cmd.Flags().StringVar(&fc.compression, "compression", "", "Output batch compression type")


Shouldn't this be an explicit "none"?

yeah fair point

With the current flag-overrides-file model, wouldn't this preclude the ability to control the setting via YAML?

Yeah I think so, if you ask me!

rockwotj · 2024-05-19T23:42:09Z

src/go/rpk/pkg/cli/transform/list_test.go

-NAME      INPUT TOPIC  OUTPUT TOPIC     RUNNING  LAG
-foo2bar   foo          bar              2 / 3    7
-scrubber  pii          cleaned, munged  0 / 1    99
+NAME      INPUT TOPIC  OUTPUT TOPIC     RUNNING  LAG   COMPRESSION


Is this useful? We don't print env vars and this feels like that configuration. Maybe in the detail view? Doesn't feel like there is a user story where someone wants this

good point, yeah I suppose not.

rockwotj · 2024-05-19T23:50:40Z

src/go/rpk/pkg/cli/transform/meta.go

@@ -81,3 +81,40 @@ Subsequent 'rpk transform list' operations will show transform processors as
 	}
 	return cmd
 }
+
+func newSetCompressionCommand(fs afero.Fs, p *config.Params) *cobra.Command {


Probably confusing for users that this is not sticky across deploys

You mean like on the backend (i.e. sticks to (name, transform))? Or writing back to the config (i.e. sticks to the transform itself)?

I am not sure! I just wanted to point out that the following sequence of commands would end up with a non expected outcome.

rpk transform deploy rpk transform set compression=gzip rpk transform deploy # no more gzip!

I personally would just not have this command, but otherwise I think writing back to the file is a reasonable choice (or adding a warning if there is no config file)

No, it's a great point. The set command doesn't seem particularly useful. Thanks!

Feediver1 · 2024-05-20T17:43:07Z

src/go/rpk/pkg/cli/transform/meta.go

+at a cost. So, while it does occur asynchronously with respect to transform
+execution, compression may introduce latency on the output topic.


Suggested change

at a cost. So, while it does occur asynchronously with respect to transform

execution, compression may introduce latency on the output topic.

at a cost. So, while it occurs asynchronously with respect to transform

execution, compression may introduce latency on the output topic.

boost::lexical_cast<compression>("bogus) should raise bad_lexical cast. Previous behavior: runtime_error("Fell off the end of a string-switch") Failure to match the source string to one of the compression lexical cases should set the fail bit on the istream instead of allowing the runtime_error to escape. Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Also adds bool transform_metadata_patch::empty() Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman · 2024-05-31T20:08:15Z

force push to sync w/ dev after a couple weeks of inactivity

Deflaimun · 2024-06-03T14:07:38Z

src/v/redpanda/admin/api-doc/transform.json

+            "lz4",
+            "zstd"
+          ],
+          "description": "Output batch compression mode (TODO(oren): might want to limit which?)"


todo removal reminder

Deflaimun · 2024-06-03T14:07:41Z

src/v/redpanda/admin/api-doc/transform.json

+            "zstd"
+          ],
+          "required": false,
+          "description": "Output batch compression mode (TODO(oren): might want to limit which?)"


todo removal reminder

Deflaimun · 2024-06-03T14:07:46Z

src/v/redpanda/admin/api-doc/transform.json

+            "zstd"
+          ],
+          "required": false,
+          "description": "Output batch compression mode (TODO(oren): might want to limit which?)"


todo removal reminder

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Respecting transform_metadata::compression_mode Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

- tranform_from_json - deploy Signed-off-by: Oren Leiman <oren.leiman@redpanda.com> dt/rpk

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman · 2024-06-07T04:30:25Z

force push contents:

remove set_compression command (compression set only at deploy time to avoid confusion)
fix up some tests to account for above
remove compression column from list transforms output table (still present in JSON representation)

oleiman self-assigned this May 15, 2024

github-actions bot added area/rpk area/redpanda area/wasm WASM Data Transforms labels May 15, 2024

oleiman force-pushed the xform/core-2736/compressed-batches branch 2 times, most recently from 4adf296 to 58df36c Compare May 16, 2024 05:45

oleiman commented May 17, 2024

View reviewed changes

src/v/redpanda/admin/api-doc/transform.json Outdated Show resolved Hide resolved

oleiman force-pushed the xform/core-2736/compressed-batches branch 2 times, most recently from d9f6691 to f13207f Compare May 17, 2024 18:31

oleiman marked this pull request as ready for review May 17, 2024 18:32

oleiman requested review from a team, twmb, r-vasquez, gene-redpanda and Deflaimun as code owners May 17, 2024 18:32

oleiman added the doc-needed label May 17, 2024

oleiman requested review from a team and michael-redpanda and removed request for a team May 17, 2024 22:55

rockwotj reviewed May 19, 2024

View reviewed changes

Feediver1 reviewed May 20, 2024

View reviewed changes

oleiman marked this pull request as draft May 31, 2024 19:56

oleiman added 2 commits May 31, 2024 12:56

model/xform: Add compression mode to metadata & patch

866a3e6

Also adds bool transform_metadata_patch::empty() Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

oleiman force-pushed the xform/core-2736/compressed-batches branch from f13207f to 1e48f71 Compare May 31, 2024 20:07

Deflaimun reviewed Jun 3, 2024

View reviewed changes

oleiman added 8 commits June 3, 2024 09:09

admin/swagger: Add compression mode to transform endpoints

8e155dd

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

admin/xform: Wire up compression mode to deploy/list/patch

4a91cb1

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

xform/api: Respect compression_mode for patch upsert

fc93436

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

xform/proc: Compress record batches before producing

ad543d9

Respecting transform_metadata::compression_mode Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

rpk/xform/api: Add compression mode to metadata wrappers

a3d7bb1

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

rpk/xform: Add compression to project config

52f432c

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

rpk/xform: Add compression mode to deploy command

527273b

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

rpk/xform: Add compression mode to list command output

60c6df0

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

Feediver1 previously approved these changes Jun 4, 2024

View reviewed changes

oleiman added 3 commits June 6, 2024 21:14

dt/rpk: Wire up compression for transforms

7297591

- tranform_from_json - deploy Signed-off-by: Oren Leiman <oren.leiman@redpanda.com> dt/rpk

dt/xform: Test for setting compression mode

e768141

Signed-off-by: Oren Leiman <oren.leiman@redpanda.com>

mucking with deploy

47536f4

oleiman dismissed Feediver1’s stale review via 47536f4 June 7, 2024 04:29

oleiman force-pushed the xform/core-2736/compressed-batches branch from 1e48f71 to 47536f4 Compare June 7, 2024 04:29

oleiman marked this pull request as ready for review June 7, 2024 04:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CORE-2736] Adds output batch compression for Data Transforms #18514

[CORE-2736] Adds output batch compression for Data Transforms #18514

oleiman commented May 15, 2024 •

edited

oleiman commented May 16, 2024

oleiman commented May 16, 2024

vbotbuildovich commented May 16, 2024 •

edited

rockwotj May 19, 2024

oleiman May 31, 2024

oleiman Jun 3, 2024

rockwotj Jun 3, 2024

rockwotj May 19, 2024

oleiman May 31, 2024

rockwotj May 19, 2024

oleiman Jun 3, 2024

rockwotj Jun 4, 2024

oleiman Jun 4, 2024

Feediver1 May 20, 2024

oleiman commented May 31, 2024

Deflaimun Jun 3, 2024

Deflaimun Jun 3, 2024

Deflaimun Jun 3, 2024

oleiman commented Jun 7, 2024

		at a cost. So, while it does occur asynchronously with respect to transform
		execution, compression may introduce latency on the output topic.

[CORE-2736] Adds output batch compression for Data Transforms #18514

Are you sure you want to change the base?

[CORE-2736] Adds output batch compression for Data Transforms #18514

Conversation

oleiman commented May 15, 2024 • edited

Backports Required

Release Notes

Improvements

oleiman commented May 16, 2024

oleiman commented May 16, 2024

vbotbuildovich commented May 16, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleiman commented May 31, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oleiman commented Jun 7, 2024

oleiman commented May 15, 2024 •

edited

vbotbuildovich commented May 16, 2024 •

edited