feat(observability)(WIP): allow custom latency distribution buckets

[jfreeland]

Description

Work In Progress.

This is intended to address #7641 and add support for users to provide an alternative list of distributions thresholds.

No tests. Not entirely sure this is the best approach. Not validated at all yet.

Issue reference

Please reference the issue this PR will close: #[issue number]

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• Code compiles correctly
• Created/updated tests
• Unit tests passing
• End-to-end tests passing
• Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• Specification has been updated / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• Provided sample for the feature / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

[yaron2]

@jfreeland are you still working on this?

[jfreeland]

@jfreeland are you still working on this?

my wedding, honeymoon, and day job have gotten in the way. i won't have time to revisit and test for the next couple weeks, unfortunately. if someone else has time and wants to make it happen elsewhere that'd be awesome. if not i'll revisit as soon as i can.

[jfreeland]

I think there's still work to do here but I think this is 'closer'. I didn't get a chance to dig around the Makefile and reverse engineer the quickest path to building my own containers and pushing to my own registry so that I could test this out in my development environment.

I seem to recall a couple months ago that I regenerated the generated files myself but I didn't do that tonight. I don't recall seeing that I needed to in CONTRIBUTING.md, but maybe I wasn't looking in the right place.

It might be cool if there was some simple app that could be used for local testing with https://tilt.dev like https://github.com/hashicorp-demoapp/hashicups-setups.

I also spotted a little typo https://github.com/dapr/dapr/blob/master/pkg/diagnostics/grpc_monitoring.go#L57 healthProbeRoundripLatency should be Roundtrip along the way that bothered me enough to make a note. Probably worthy of a separate PR just to ensure consistency and searchability.

[jfreeland]

I seem to recall a couple months ago that I regenerated the generated files myself but I didn't do that tonight. I don't recall seeing that I needed to in CONTRIBUTING.md, but maybe I wasn't looking in the right place.

I found @ItalyPaleAle 's reference to make code-generate and ran it again but there was no diff. I updated configuration.yaml by hand in this PR. That was probably not correct.

[jfreeland]

Update...

Some of the unit test errors were because I was missing the new-ish statusKey. Maybe that's also related to integration test failures but I didn't quickly determine that.

I haven't yet tracked down what's causing the runtime/service_invocation/res_recv_latency_ms: cannot register view "runtime/service_invocation/res_recv_latency_ms"; a different view with the same name is already registered errors. I removed this from handler_test.go in d5c590a but that was actually probably the wrong thing to do. I believe there's something about these tests running in parallel that was working fine before but no longer is the case after the distribution bucket changes, which is not what I expected. Now that I've addressed the statusKey change I've got more cannot register view errors in e.g. https://github.com/jfreeland/dapr/blob/8519d226924000dab0793bdecbc8191cea10a0a5/pkg/diagnostics/resiliency_monitoring_test.go#L188.