[CORE-2069]: Changes to support using S3 FIPS Endpoints #18571
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
s3_client::self_configure() is not compatible with fips mode right now, because fips mode requires the use of virtual_host, and this is needed to set early on the server url
s3_configuration::make_configuration. read fip mode from node config. this bool is used to set virtual_host mode and skip s3_client::self_configure the change moves a if else block earlier in the function for the next commit
this is critical for fips mode: server_addr needs to point to the full virtual_host uri (if using virtual_host style, a requirement for fips) to do so, wire in bucket_name in make_configuration. note: s3_client will build its full uri for the HTTP Host field, to support self discovery (not in fips mode). so it still get only the base s3 endpoint as input
When using virtual addressing, the S3 client must connect to <bucket>.s3.<region>.amazonaws.com. This is especially important when targeting an S3 FIPS endpoint: <bucket>.s3-fips.<region>.amazonaws.com Signed-off-by: Michael Boquard <michael@redpanda.com>
Made changes to the S3 client and redpanda infrastructure to allow for testing FIPS compliant S3 endpoints. This also selects path style addressing whenever minio is in use. Signed-off-by: Michael Boquard <michael@redpanda.com>
7 tasks
|
/dt |
andijcr
changed the title
this helps catching refressions that would othrwise cause tests to be killed due to timeout
needed for virtual_host s3 addressing, where s3_client forms [bucket].[server_addr] Hosts
|
/dt |
this is for fixture tests
|
/dt |
|
/rp-unit-test |
|
/ci-repeat 1 skip-units skip-redpanda-build |
|
new failures in https://buildkite.com/redpanda/redpanda/builds/49412#018f9f7d-9aff-4963-ba42-d89601b08d78: new failures in https://buildkite.com/redpanda/redpanda/builds/49412#018f9f7d-9afc-4b66-b365-873b71c91a21: new failures in https://buildkite.com/redpanda/redpanda/builds/49412#018f9f7d-9afa-4cb3-a6b0-db2863d31d35: new failures in https://buildkite.com/redpanda/redpanda/builds/49412#018f9f7d-9b02-4589-ba61-d65137c2c52f: new failures in https://buildkite.com/redpanda/redpanda/builds/49412#018f9f7d-9b07-46c1-a043-e39c457d7325: new failures in https://buildkite.com/redpanda/redpanda/builds/49412#018f9f7d-9b09-46d7-a9f7-493687c965d2: new failures in https://buildkite.com/redpanda/redpanda/builds/49412#018f9f7d-9b0c-4bf0-831e-8b20591ec290: new failures in https://buildkite.com/redpanda/redpanda/builds/49412#018f9f7d-9b0e-4d49-9f9e-75dd0c8df9f1: |
std::visit can accetps multiple variants and this makes the function cleared, in preparation for the next commit
needed for the next commit, where we rebuild s3_configuration from a future
pass a value that will take precedence over cluster property "url_style". default to nullopt. this is to support auto discovery what style of url the s3 backend will use: if "url_style" is set for autodiscovery, a s3 client will be consstructed, discover the value for url_style, and then a new configuration will be built passing an actual value
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes to testing and cloud storage infrastructure in order to be able to point Redpanda at FIPS compliant S3 endpoints in AWS (go to "Amazon Simple Storage Service (S3)" row on table in linked page).
waiting for this #18106 to merge
Backports Required
Release Notes