Enable and update packages using agent service deployer - DO NOT MERGE

[mrodm]

Proposed commit message

Test to enable independent Elastic Agents for all packages using the latest changeset of elastic-package main branch.
Migrate auditd_manager and oracle packages that were using custom agents to use this new mode to run Elastic Agents.

• Configuration/Settings for Elastic Agents is now defined in the configuration files for each system test.
• _dev/deploy/agent folders removed, no longer needed.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• Requires a new elastic-package version

How to test this PR locally

# Build/Install elastic-package from main
cd /path/elastic-package
make install

cd /path/integrations

# Start Elastic stack
elastic-package stack up -v -d

# Run system tests in any of the packages from integrations
cd /path/integrations/packages/<package>
ELASTIC_PACKAGE_TEST_ENABLE_INDEPENDENT_AGENT=true elastic-package test system -v

# Tear down Elastic stack
elastic-package stack down -v

Related issues

• Relates Add ability to deploy custom elastic-agents elastic-package#787

[mrodm]

@flash1293 this PR that tests with the latest changes of elastic-package reports a package (zeronetworks) failing due to ignored fields cc @jsoriano

Here it is the link to the error: https://buildkite.com/elastic/integrations/builds/11433#018f778c-9382-496a-a782-af5796a25b70/679-1176

EDIT: there is at least one more package aws_bedrock (not finished yet the build):

• https://buildkite.com/elastic/integrations/builds/11433#018f778c-9103-4ece-8a71-edfeb0725e5e

[flash1293]

@mrodm Ideally, the mapping issue should be investigated and fixed - e.g. if it is a keyword field that's running into ignore_above, maybe we don't need to index it at all or should index it as text? However, you should be able to also mute the error like this: #9823

[mrodm]

@mrodm Ideally, the mapping issue should be investigated and fixed - e.g. if it is a keyword field that's running into ignore_above, maybe we don't need to index it at all or should index it as text? However, you should be able to also mute the error like this: #9823

Ok! This PR is not intended to be merged (at least for now). However, this is something that needs to be taken into account when a new elastic-package version is released and updated in the integrations repository, because it would require changes in packages. cc @elastic/ecosystem

[flash1293]

Yeah, the linked PR was meant to take care of this problem, but maybe some additional integrations had changes that made the check fail in new places. Lets wait for the build to finish, then I can create a PR to mute these errors as well.

Shows that we should release this additional check quickly so integration authors see the problem right away.

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[flash1293]

Opened PR here: #9871

[mrodm]

/test

[elasticmachine]

⏳ Build in-progress, with failures

• Buildkite Build
• Commit: 3083a71

Failed CI Steps

• Check integrations aws_bedrock
• Check integrations symantec_endpoint_security
• Check integrations zeronetworks

History

• 💔 Build #11736 failed 6a7f300
• 💔 Build #11729 failed e5936e4
• 💔 Build #11716 failed e5936e4
• 💔 Build #11713 failed 71015b5
• 💔 Build #11580 failed e5fa260

cc @mrodm

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube