enhancement(reduce transform): New setting for reduce transform: end_every_period_ms

[charlesconnell]

I propose a new setting available for use on reduce transforms: end_every_period_ms.

In my organization's use of vector, we are tailing Java log files to look for garbage collection events. We want to know the duration of the longest garbage collection that happened lately, for a rolling "lately," and report that value to Prometheus. I don't believe we can accomplish this with vector at present, but this PR will enable it. We can collect values and reduce them with a max merge strategy, then flush them every 60 seconds.

I am sure that this will be useful to many users in the long run. Periodically flushing a reducer surely has many applications.

I have compiled vector with this change and deployed it within my organization, and confirmed that it works for us. For example, here is a snippet of our config:

      "gc-log/parse0/g1Pauses0/pauseTimeMax0": {
         "end_every_period_ms": 60000,
         "group_by": [
            "reason",
            "subtype",
            "type"
         ],
         "inputs": [
            "gc-log/parse0/g1Pauses0"
         ],
         "merge_strategies": {
            "gc_millis": "max"
         },
         "type": "reduce"
      },

This PR does not include new test coverage. I am new to Rust and not familiar with best testing practices. My new code has behavior dependent on the system clock, and that seems hard to test.

[bits-bot]

All committers have signed the CLA.

[jszwedko]

Thanks for this @charlesconnell ! We actually had a previous attempt at this that we had to revert: #16146. I think your PR resolves the issue that led to the revert by adding a new option rather than changing the behavior of an existing one.

I think this would close #16695.

[lukesteensen]

Hi @charlesconnell, sorry for the delay in reviewing this! What you have here makes good sense to me and I'm glad to hear it's working well for you internally. I think successful use in production is a good substitute for tests, at least initially 😄. I agree they're a bit tricky with the time aspect, and that's something we can add after the initial PR if we deem it worthwhile.

It looks like the PR is slightly out of date, so would you mind merging or rebasing with master? Otherwise I think this is largely good to go.

[charlesconnell]

thank you @lukesteensen , I've rebased now