[UII] Surface option to delete diagnostics files

[jen-huang]

Summary

Resolves #167366.

This PR introduces the ability to manually delete diagnostics files before their ILM policy kicks in. This PR:

• Adds a new DELETE /api/fleet/agents/files/{fileId} route which returns {id: string, deleted: boolean}
• Deletes the file from .fleet-fileds-fromhost-data-agent data stream when a request is received, and updates the corresponding meta information in .fleet-fileds-fromhost-meta-agent to set it to DELETED status
  • This is similar to how the meta information reconciliation via task manager is done when the ILM policy deletes diagnostics file
• Updates the Agent details > Diagnostics tab:
  • Surface Delete action in files table for requests that resulted in files
  • Refactors existing UI around the copy text and generate button
  • Add Show expired requests toggle
    • Off by default, which means the following will be shown:
      • Files that are on disk
      • Files being generated
      • File requests which errored out but are still not expired (i.e. users can see errors with recent requests)
    • When toggled on, will additionally show:
      • File requests which errored out AND are expired
      • File requests that are just expired (i.e. edge case where a file was deleted by ILM but the meta info had not yet reconciled)

FYI, the expiration threshold is currently only 3 minutes. This is a bug, see: #183692

The main reason for adding this toggle is to keep the initial list view clean. The items in this list are built from all REQUEST_DIAGNOSTICS agent actions that the user submits, which can be on a single agent or bulk agents.

When a file is deleted manually with this new work, or by the existing ILM policy, we can correctly flag the associated action as having DELETED files and hide it from view. But when a request errors out or otherwise results in no files being generated, we still want to keep the history of the request (we have no precedent of deleting agent activity). Over time, this history is no longer useful for the user and just pollutes the table, so it is better to hide these items from the initial view.

Testing

Use the single and bulk request diagnostics feature and test the delete functionality. Go nuts :)

Checklist

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[apmmachine]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• /oblt-deploy : Deploy a Kibana instance using the Observability test environments.
• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[kilfoyle]

LGTM for the docs. 👍
Here's a docs update PR for the new delete option.

[kpollich]

Code + tests LGTM, I did not pull down to test locally 🚀

I expect the next thing we'll be asked for on this page would be bulk delete. Do you think it makes sense to put an issue for that on the backlog for now?

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: cf0ceab

Failed CI Steps

• FTR Configs #39

Test Failures

• [job] [logs] FTR Configs #39 / discover/group4 discover esql view inspector shows Discover and Lens requests in Inspector

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
fleet	1184	1187	+3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
fleet	1.3MB	1.3MB	+1.7KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
fleet	164.1KB	164.4KB	+364.0B

Unknown metric groups

API count

id	before	after	diff
fleet	1305	1308	+3

History

• 💚 Build #210864 succeeded 9e6eea4
• 💚 Build #210625 succeeded dc19575
• 💛 Build #210621 was flaky 6bf8e27
• 💔 Build #210619 failed 7330a12

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jen-huang

[jen-huang]

@kpollich I opened #183857