[AWS GovCloud S3] Support for MFA-enabled User and Search S3 Connector

[architecthutch]

Problem Description

In using an MFA-enabled AWS GovCloud user's Access + Secret keys when configuring the S3 search connector in ECE 3.6.2, Stack Pack 8.13.3, I receive the following error when attempting to sync my data:

Connector error ClientError: An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records.

• Chat thread from the #search-enterprise channel in Slack: https://elasticstack.slack.com/archives/C018C7B9T5F/p1715621502150859
• Documentation: https://www.elastic.co/guide/en/enterprise-search/8.13/connectors-s3.html

I leverage an MFA-enabled named profile within AWS CLI to perform automation with the Boto3 libraries as well as maintain Zero Trust compliance when authenticating. This also ensures my credentials are secure, as the Configuration section of the Connector setup states:
Encryption for data source credentials is unavailable in this version. Your data source credentials will be stored, unencrypted, in Elasticsearch.

Federal guidelines (NIST 800-53 Rev5, NIST 800-207, EO 14028, etc) require securing data in transit and data at rest, such as that of credentials and S3 Bucket contents. In this scenario I have an AWS CMK-encrypted, non-Public S3 bucket the Connector is attempting to access. These security measures need to remain in place for compliance reasons.

Proposed Solution

• Enable the capability for the connector to handle MFA.
• If not feasible currently, request contact with AWS to determine viability
• Validate functionality in an AWS GovCloud environment with

Additional Context

This recommendation was output from a discussion with Elastic over their slack channel mentioned above. The intent is to leverage data from this connector as an indexing point with ELSER and semantic search.