[search source] ES Query rule loads fewer fields on query execution

[mattkime]

Summary

tldr; ES Query alert execution creates less field_caps traffic, date fields being accessed in alert message via fields.* might not render aside from the timestamp field.

--

This PR reduces the number of fields loaded via field caps to the minimum required to run a query, rather than the full field list. It adds a createLazy method to the Search Source Service which internally loads fields via a DataViewLazy object and then adds them to a DataView object. This is to minimize changes and ship code quickly - SearchSource objects expose the DataView object they use and kibana apps may use this. It will take time to migrate away from this since the DataView object is used both internally and referenced externally. A key element of this code is the ability to extract a field list from a query so a limited (rather than complete) set of fields can be loaded.*

One side effect of loading fewer fields is that date fields available via fields.* in the alert message may no longer work. Previously, all fields were loaded including all date fields. Now, date fields are only loaded if they're part of the query. This has been determined to be a small corner case and an acceptable tradeoff.

Only the ES Query rule is using this new method of loading fields. While further work is needed before wider adoption, this should prevent significant data transfer savings via a reduction in field_caps usage.

Depends upon #183573

---

* We don't need to load all fields to create a query, rather we need to load all the fields where some attribute will change the output of a query. Sometimes the translation from KQL to DSL is the same no matter the field type (or any other attribute) and sometimes the translation is dependent field type and other attributes. Generally speaking, we need the latter.

There are additional complexities - we need to know which fields are dates (and date nanos) when their values are displayed so their values can be made uniform. In some circumstances we need to load a set of fields due to source field exclusion - its not supported in ES so Kibana submits a list of individual field names.

Finally, there are times where we solve a simpler problem rather than the problem definition. Its easier to get a list of all fields referenced in a KQL statement instead of only getting the subset we need. A couple of extra fields is unlikely to result in performance degradation.

---

Places where the field list is inspected -

packages/kbn-es-query/src/es_query/filter_matches_index.ts
packages/kbn-es-query/src/es_query/from_nested_filter.ts
packages/kbn-es-query/src/es_query/migrate_filter.ts
packages/kbn-es-query/src/kuery/functions/exists.ts
packages/kbn-es-query/src/kuery/functions/is.ts
packages/kbn-es-query/src/kuery/functions/utils/get_fields.ts

This looks like its worth closer examination since it looks at the length of the field list - https://github.com/elastic/kibana/blob/main/packages/kbn-es-query/src/kuery/functions/is.ts#L110

Next steps -

• Discuss above usage and make sure all cases are covered in this PR
• Add statement to PR on lack of date formatting
• Add test to verify reduction of fields requested

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[dej611]

Left some comments on the code review, mostly about rewriting in a concise way some lengthy code.
I've tested this PR locally and tested some dashboards: while I think this is a great approach for search source, from my measures the performance impact on a dashboard of this optimization is mostly negligible as any Lens chart using the same dataView will load the entire fields list no matter what.
From that I can take 2 actions:

• create a new Lens follow up issue to optimize the flow within the embeddable context as well
• noticed that @elastic/kibana-presentation controls will load additionally the default dataView with all its fields no matter what as well.

[dej611]

I wonder if it's possible to rewrite this in a concise and clearer way.
Some idea:

const isIndexName = typeof searchSourceFields.index === 'string';
...
  if(!isDataViewLazy){
    fields.index = isIndexName ? await indexPatterns.get(searchSourceFields.index) : await indexPatterns.create(searchSourceFields.index);
  } else {
    dataViewLazy = isIndexName ? await indexPatterns.getDataViewLazy(searchSourceFields.index): await indexPatterns.createDataViewLazy(searchSourceFields.index);
    // prefetch async data all at once
    const [spec, shortDotsEnable, metaFields] = await Promise.all([
      dataViewLazy.toSpec(),
      searchSourceDependencies.dataViews.getShortDotsEnable(),
      searchSourceDependencies.dataViews.getMetaFields()
    ]);
    const dataView = new DataView({
      spec,
       // field format functionality is not used within search source
       fieldFormats: {} as FieldFormatsStartCommon,
       shortDotsEnable,
       metaFields,
     });

     fields.index = dataView;
  }

[mattkime]

For some reason using isIndexName in the conditional causes typescript to lose the more specific searchSourceFields.index type. Otherwise I've adopted these improvements.

[ymao1]

response ops changes LGTM

[mattkime]

while I think this is a great approach for search source, from my measures the performance impact on a dashboard of this optimization is mostly negligible as any Lens chart using the same dataView will load the entire fields list no matter what.

@dej611 You're correct, the rollout is going to be a long process and the initial focus is on alerts since they run frequently, increasing our COGS, but we'll eventually make our way to lens and dashboards.

[dej611]

Actually, as after thought it might not be convenient to optimize per chart, as that may have the side effect of multiple partial fields requests. Perhaps that should be lifted up at dashboard level who's the only one who has the global context in view.

[kertal]

Actually, as after thought it might not be convenient to optimize per chart, as that may have the side effect of multiple partial fields requests. Perhaps that should be lifted up at dashboard level who's the only one who has the global context in view.

@mattkime @dej611 I don't see much benefit how DataViewLazy would be useful on a Dashboard given we already have Browser caching in place. Because we should omit many different field_caps requests for several visualizations. So I think we already applied the biggest improvement in this area

[lukasolson]

Did a bunch of different tests and seems to be working correctly. Added a couple of comments but after adding the unit test this can be merged!

[mattkime]

/ci

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: e79c326

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
data	2576	2585	+9

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
data	420.3KB	421.5KB	+1.2KB

Unknown metric groups

API count

id	before	after	diff
data	3185	3194	+9

History

• 💚 Build #213396 succeeded 885d3dc
• 💚 Build #213305 succeeded 48f6695
• 💚 Build #213209 succeeded 5499ae0
• 💛 Build #213141 was flaky f77b0f2
• 💚 Build #212994 succeeded 72d709c
• 💚 Build #212969 succeeded e6f70cf

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @mattkime