You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This might be related to the same problem as in #3663 but on a different API call.
I am trying to simply scrape my Kafka cluster with the admin and consumer client. However, very often, do I get
The source code of the application I am trying to build can be found here
The problem is, you can't quite catch this exception, and the process just completely gets killed.
How to reproduce
At first, this seemed to be very random, it'd happen after a while, or very quickly.
As it turns out, tweaking the code to create an AdminClient each time before calling on the functions, instead of re-using and existing admin client, one leads to this problem every single time, right away.
Config
bootstrap.servers: : kafka.my-domain.tld:9092
security.protocol: sasl_ssl
sasl.mechanisms: OAUTHBEARER
sasl.oauthbearer.method: oidc
sasl.oauthbearer.client.id: <clientID>
sasl.oauthbearer.client.secret: <clientSecret>
sasl.oauthbearer.token.endpoint.url: <token_url>
sasl.oauthbearer.scope: <scope>
client.dns.lookup: use_all_dns_ips
session.timeout.ms: 10000 # played with many different values, doesn't seem to have an impact at all.
acks: all
socket.keepalive.enable: true
debug: admin,consumer
Checklist
IMPORTANT: We will close issues where the checklist has not been completed.
Please provide the following information:
librdkafka version (release number or git tag): <REPLACE with e.g., v0.10.5 or a git sha. NOT "latest" or "current">
Apache Kafka version: Confluent Cloud, others
librdkafka client configuration: <REPLACE with e.g., message.timeout.ms=123, auto.reset.offset=earliest, ..>
So far the only "workaround" I have found is to forcibly close and create a new set of clients each time, which would be preferable not to have to do.
EDIT: tinkering on this, given that at some point it seems the admin client was set to close, what happens to the existing requests that might be in the queue? Is that what the Local: Timeout might be referring to?
Same happens in confluent-kafka-go when using AdminClient's ListConsumerGroupOffsets method.
If cluster configuration changes (e.g. node restart) app crashes with:
Assertion failed: (eonce->refcnt > 0), function rd_kafka_enq_once_del_source_return, file rdkafka_queue.h, line 1052.
bad thing about go that CGO errors cannot be handled and whole app just crashes completely
Description
This might be related to the same problem as in #3663 but on a different API call.
I am trying to simply scrape my Kafka cluster with the admin and consumer client. However, very often, do I get
python: rdkafka_queue.h:1052: rd_kafka_enq_once_del_source_return: Assertion
eonce->refcnt > 0' failed.`The source code of the application I am trying to build can be found here
The problem is, you can't quite catch this exception, and the process just completely gets killed.
How to reproduce
At first, this seemed to be very random, it'd happen after a while, or very quickly.
As it turns out, tweaking the code to create an AdminClient each time before calling on the functions, instead of re-using and existing admin client, one leads to this problem every single time, right away.
Config
Checklist
IMPORTANT: We will close issues where the checklist has not been completed.
Please provide the following information:
<REPLACE with e.g., v0.10.5 or a git sha. NOT "latest" or "current">
<REPLACE with e.g., message.timeout.ms=123, auto.reset.offset=earliest, ..>
Logs
EDIT 1: This is repeatable as soon as the DescribeConsumerGroups times outs, everything goes wrong
The text was updated successfully, but these errors were encountered: