[CI] Separated extended testing pipeline

[pazone]

What does this PR do?

Depends on #4738
Extracting long running integration tests to a separate pipeline.
Added changeset filters to the main the new pipeline
Added notification context for integration tests

Important: Introduces a concurrency group for each type of exteneded tests. The concurrency groups are limited to 8. That means we can run only 8 jobs of a type simultaneously

For example:

steps:
  - label: "Serverless integration test"
    key: "serverless-integration-tests"
    concurrency_group: elastic-agent-extended-testing/serverless-integration # <- concurrency group
    concurrency: 8 # <- concurrency limit
    env:
      BEAT_NAME: "elastic-agent"
    command: ...

Why is it important?

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

Related issues

• Relates Skip integration tests for changes that do not modify Go source files #3674
• Relates Separate expensive test in elastic-agent CI to a separate pipeline #4710

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[mergify]

This pull request does not have a backport label. Could you fix it @pazone? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 8./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[pchila]

@pazone I don't see anything where we add the required check for PRs to have a green build of the new pipeline... Did I miss it or you still need to add it ?

Looking at the checks for this PR I would say we definitely didn't add the required check in GH 😅

and the only listed buildkite build does not contain any integration testing...

If we need to merge the pipeline definition before the GH PR check then we should not remove the existing steps for integration testing until we are fully set up with the new pipeline and check

[rdner]

@pchila if I understood correctly this PR only extracts the steps into a separate pipeline and another PR #4738 is supposed to actually create resources to run it.

Once both of them are merged, we should be able to add a required check in the repository settings.

[pchila]

We need to hold off merging this until we have the check for the extended testing pipeline on GH PRs

[pchila]

I'll remove the request changes review, but I am still worried that some bug may be merged while there are no integration tests running since this PR is removing those steps and there will be no replacement until the new pipeline is created and the check added in the GH repo settings as pointed out by @rdner

[cmacknz]

Why are this and #4738 not a single PR?

[pazone]

Why are this and #4738 not a single PR?

First PR creates the new pipeline on main. Then this PR configures it.

[pazone]

Found a problem with packaging after extracting the pipeline. Solving it

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube