Component errors metric gone

[jpds]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Somewhere between the 0.34.0 release and 0.38.0, the vector_component_errors_total metrics appears to have disappeared(?).

I see the following in my logs:

May 17 08:07:27 host vector[1136]: 2024-05-17T08:07:27.872408Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] has been suppressed 3 times.
May 17 08:07:27 host vector[1136]: 2024-05-17T08:07:27.872542Z  WARN vector::internal_events::dnstap: Recoverable error occurred while parsing dnstap data. error=Encountered error from TrustDns: unexpected end of input reached stage="processing" error_type="parser_failed" internal_log_rate_limit=true
May 17 08:07:27 host vector[1136]: 2024-05-17T08:07:27.873715Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] is being suppressed to avoid flooding.
May 17 08:13:20 host vector[1136]: 2024-05-17T08:13:20.391033Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] has been suppressed 3 times.
May 17 08:13:20 host vector[1136]: 2024-05-17T08:13:20.391127Z  WARN vector::internal_events::dnstap: Recoverable error occurred while parsing dnstap data. error=Encountered error from TrustDns: more than one edns record present stage="processing" error_type="parser_failed" internal_log_rate_limit=true
May 17 08:13:20 host vector[1136]: 2024-05-17T08:13:20.391838Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] is being suppressed to avoid flooding.
May 17 09:15:11 host vector[1136]: 2024-05-17T09:15:11.058375Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] has been suppressed 3 times.
May 17 09:15:11 host vector[1136]: 2024-05-17T09:15:11.058462Z  WARN vector::internal_events::dnstap: Recoverable error occurred while parsing dnstap data. error=Encountered error from TrustDns: unknown OpCode: 1 stage="processing" error_type="parser_failed" internal_log_rate_limit=true
May 17 09:15:11 host vector[1136]: 2024-05-17T09:15:11.059278Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] is being suppressed to avoid flooding.
May 17 09:26:12 host vector[1136]: 2024-05-17T09:26:12.854541Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] has been suppressed 3 times.
May 17 09:26:12 host vector[1136]: 2024-05-17T09:26:12.854659Z  WARN vector::internal_events::dnstap: Recoverable error occurred while parsing dnstap data. error=Encountered error from TrustDns: unexpected end of input reached stage="processing" error_type="parser_failed" internal_log_rate_limit=true
May 17 09:26:12 host vector[1136]: 2024-05-17T09:26:12.855431Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] is being suppressed to avoid flooding.
May 17 09:41:14 host vector[1136]: 2024-05-17T09:41:14.241636Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] has been suppressed 3 times.
May 17 09:41:14 host vector[1136]: 2024-05-17T09:41:14.241733Z  WARN vector::internal_events::dnstap: Recoverable error occurred while parsing dnstap data. error=Encountered error from TrustDns: unknown OpCode: 1 stage="processing" error_type="parser_failed" internal_log_rate_limit=true
May 17 09:41:14 host vector[1136]: 2024-05-17T09:41:14.241979Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] is being suppressed to avoid flooding.
May 17 09:45:30 host vector[1136]: 2024-05-17T09:45:30.053433Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] has been suppressed 3 times.
May 17 09:45:30 host vector[1136]: 2024-05-17T09:45:30.053588Z  WARN vector::internal_events::dnstap: Recoverable error occurred while parsing dnstap data. error=Encountered error from TrustDns: unexpected end of input reached stage="processing" error_type="parser_failed" internal_log_rate_limit=true
May 17 09:45:30 host vector[1136]: 2024-05-17T09:45:30.053970Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] is being suppressed to avoid flooding.
May 17 09:45:40 host vector[1136]: 2024-05-17T09:45:40.511702Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] has been suppressed 3 times.
May 17 09:45:40 host vector[1136]: 2024-05-17T09:45:40.511801Z  WARN vector::internal_events::dnstap: Recoverable error occurred while parsing dnstap data. error=Encountered error from TrustDns: unknown OpCode: 1 stage="processing" error_type="parser_failed" internal_log_rate_limit=true
May 17 09:45:40 host vector[1136]: 2024-05-17T09:45:40.512078Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] is being suppressed to avoid flooding.
May 17 09:52:40 host vector[1136]: 2024-05-17T09:52:40.936106Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] has been suppressed 3 times.
May 17 09:52:40 host vector[1136]: 2024-05-17T09:52:40.936213Z  WARN vector::internal_events::dnstap: Recoverable error occurred while parsing dnstap data. error=Encountered error from TrustDns: unknown OpCode: 1 stage="processing" error_type="parser_failed" internal_log_rate_limit=true
May 17 10:12:43 host vector[1136]: 2024-05-17T10:12:43.021913Z  WARN vector::internal_events::dnstap: Recoverable error occurred while parsing dnstap data. error=Encountered error from TrustDns: unknown OpCode: 1 stage="processing" error_type="parser_failed" internal_log_rate_limit=true
May 17 10:12:43 host vector[1136]: 2024-05-17T10:12:43.022860Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] is being suppressed to avoid flooding.
May 17 10:15:50 host vector[1136]: 2024-05-17T10:15:50.614936Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] has been suppressed 3 times.
May 17 10:15:50 host vector[1136]: 2024-05-17T10:15:50.617455Z  WARN vector::internal_events::dnstap: Recoverable error occurred while parsing dnstap data. error=Encountered error from TrustDns: unknown OpCode: 1 stage="processing" error_type="parser_failed" internal_log_rate_limit=true
May 17 10:15:50 host vector[1136]: 2024-05-17T10:15:50.617852Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] is being suppressed to avoid flooding.
May 17 10:41:54 host vector[1136]: 2024-05-17T10:41:54.338652Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] has been suppressed 3 times.
May 17 10:41:54 host vector[1136]: 2024-05-17T10:41:54.338730Z  WARN vector::internal_events::dnstap: Recoverable error occurred while parsing dnstap data. error=Encountered error from TrustDns: unknown OpCode: 1 stage="processing" error_type="parser_failed" internal_log_rate_limit=true
May 17 10:41:55 host vector[1136]: 2024-05-17T10:41:55.339354Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] is being suppressed to avoid flooding.
May 17 11:01:42 host vector[1136]: 2024-05-17T11:01:42.415285Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] has been suppressed 3 times.
May 17 11:01:42 host vector[1136]: 2024-05-17T11:01:42.415383Z  WARN vector::internal_events::dnstap: Recoverable error occurred while parsing dnstap data. error=Encountered error from TrustDns: more than one edns record present stage="processing" error_type="parser_failed" internal_log_rate_limit=true
May 17 11:01:42 host vector[1136]: 2024-05-17T11:01:42.415558Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] is being suppressed to avoid flooding.
May 17 11:47:49 host vector[1136]: 2024-05-17T11:47:49.058416Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] has been suppressed 3 times.
May 17 11:47:49 host vector[1136]: 2024-05-17T11:47:49.058546Z  WARN vector::internal_events::dnstap: Recoverable error occurred while parsing dnstap data. error=Encountered error from TrustDns: unrecognized label code: 1101001 stage="processing" error_type="parser_failed" internal_log_rate_limit=true
May 17 11:48:26 host vector[1136]: 2024-05-17T11:48:26.789594Z  WARN vector::internal_events::dnstap: Recoverable error occurred while parsing dnstap data. error=Encountered error from TrustDns: unrecognized label code: 1110010 stage="processing" error_type="parser_failed" internal_log_rate_limit=true
May 17 11:48:26 host vector[1136]: 2024-05-17T11:48:26.797605Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] is being suppressed to avoid flooding.
May 17 12:14:47 host vector[1136]: 2024-05-17T12:14:47.779896Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] has been suppressed 1 times.
May 17 12:14:47 host vector[1136]: 2024-05-17T12:14:47.780053Z  WARN vector::internal_events::dnstap: Recoverable error occurred while parsing dnstap data. error=Encountered error from TrustDns: unknown OpCode: 1 stage="processing" error_type="parser_failed" internal_log_rate_limit=true
May 17 12:14:47 host vector[1136]: 2024-05-17T12:14:47.789105Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] is being suppressed to avoid flooding.
May 17 12:28:28 host vector[1136]: 2024-05-17T12:28:28.604621Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] has been suppressed 3 times.
May 17 12:28:28 host vector[1136]: 2024-05-17T12:28:28.604796Z  WARN vector::internal_events::dnstap: Recoverable error occurred while parsing dnstap data. error=Encountered error from TrustDns: unrecognized label code: 1100100 stage="processing" error_type="parser_failed" internal_log_rate_limit=true
May 17 12:28:28 host vector[1136]: 2024-05-17T12:28:28.638766Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] is being suppressed to avoid flooding.
May 17 14:06:45 host vector[1136]: 2024-05-17T14:06:45.992122Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] has been suppressed 35 times.
May 17 14:06:45 host vector[1136]: 2024-05-17T14:06:45.992290Z  WARN vector::internal_events::dnstap: Recoverable error occurred while parsing dnstap data. error=Encountered error from TrustDns: unrecognized label code: 1101111 stage="processing" error_type="parser_failed" internal_log_rate_limit=true
May 17 14:06:46 host vector[1136]: 2024-05-17T14:06:46.135579Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] is being suppressed to avoid flooding.
May 17 14:26:09 host vector[1136]: 2024-05-17T14:26:09.865810Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] has been suppressed 11 times.
May 17 14:26:09 host vector[1136]: 2024-05-17T14:26:09.865914Z  WARN vector::internal_events::dnstap: Recoverable error occurred while parsing dnstap data. error=Encountered error from TrustDns: more than one edns record present stage="processing" error_type="parser_failed" internal_log_rate_limit=true
May 17 14:26:09 host vector[1136]: 2024-05-17T14:26:09.866355Z  WARN vector::internal_events::dnstap: Internal log [Recoverable error occurred while parsing dnstap data.] is being suppressed to avoid flooding.

( These errors might be of interest to @esensar ).

However, if I go to Grafana and explore under vector_ - I do not see any metric names that have error. I did have a dashboard with vector_component_errors_total that did have HTTP error metrics at version 0.34.0, but is now empty.

No output for error on the local Vector Prometheus endpoint with the above log:

$ curl -sSf http://localhost:9598/metrics | rg error
$ curl -sSf http://localhost:9598/metrics | rg util
# HELP vector_utilization utilization
# TYPE vector_utilization gauge
vector_utilization{component_id="loki-dnstap",component_kind="sink",component_type="loki",host="host"} 0.00007385239480221804 1715956751632
...

Configuration

No response

Version

0.38.0

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

Hi @jpds !

Those logs you shared are actually just warnings rather than errors which is why they don't increment component_errors_total. Note that you will only see component_errors_total if there an error occurred (that is, it isn't published if there were no errors).

I think that resolves this question, but feel free to reopen if not.

[jpds]

Should we have a metric for warnings then?

[jszwedko]

We could, that would be an enhancement if you want to open a request for it.