Onprem SQL to AWS Confluent Cloud Kafka "Not selecting any broker for cluster connection"

[pwbriggs23]

Description

Trying to connect an on-prem HVR hub server to an AWS Confluent Cloud Kafka instance. With proper API username, password, and endpoint, error displayed is "Local: Broker transport failure' during fetching metadata for Kafka broker"
Digging into the log files indicates that the HVR hub server uses the librdkafka library to connect.

From log file:
2024-01-05T15:00:34-05:00: hvrhubserver: $HVR_KAFKA_LIB_TRACE: 7-CONNECT: rdkafka#producer-1: [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request
...
y259p.us-east-1.aws.confluent.cloud:9092/bootstrap: Broker changed state CONNECT -> SSL_HANDSHAKE
2024-01-05T15:00:34-05:00: hvrhubserver: $HVR_KAFKA_LIB_TRACE: 7-BROADCAST: rdkafka#producer-1: [thrd:sasl_ssl://bootstr]: Broadcasting state change
2024-01-05T15:00:34-05:00: hvrhubserver: $HVR_KAFKA_LIB_TRACE: 7-CONNECT: rdkafka#producer-1: [thrd:app]: Cluster connection already in progress: application metadata request
2024-01-05T15:00:34-05:00: hvrhubserver: $HVR_KAFKA_LIB_TRACE: 7-CONNECT: rdkafka#producer-1: [thrd:main]: Cluster connection already in progress: no cluster connection
2024-01-05T15:00:34-05:00: hvrhubserver: $HVR_KAFKA_LIB_TRACE: 7-CONNECT: rdkafka#producer-1: [thrd:main]: Cluster connection already in progress: no cluster connection
2024-01-05T15:00:34-05:00: hvrhubserver: $HVR_KAFKA_LIB_TRACE: 7-CONNECT: rdkafka#producer-1: [thrd:main]: Cluster connection already in progress: no cluster connection
2024-01-05T15:00:34-05:00: hvrhubserver: $HVR_KAFKA_LIB_TRACE: 7-CONNECT: rdkafka#producer-1: [thrd:main]: Cluster connection already in progress: no cluster connection
2024-01-05T15:00:34-05:00: hvrhubserver: $HVR_KAFKA_LIB_TRACE: 7-CONNECT: rdkafka#producer-1: [thrd:main]: Cluster connection already in progress: no cluster connection
2024-01-05T15:00:34-05:00: hvrhubserver: $HVR_KAFKA_LIB_TRACE: 7-CONNECT: rdkafka#producer-1: [thrd:app]: Not selecting any broker for cluster connection: still suppressed for 49ms: application metadata request

How to reproduce

RedHat 8.8 and v1.9.2 of library

Checklist

IMPORTANT: We will close issues where the checklist has not been completed.

Please provide the following information:

• librdkafka version (release number or git tag): 1.9.2
• Apache Kafka version: AWS managed service
• librdkafka client configuration: not aware of any config file
• Operating system: RedHat 8.8
• Provide logs (with debug=.. as necessary) from librdkafka: 'not sure how'
• Provide broker log excerpts: 'don't have any'
• Critical issue 'HVR and Confluent stuck too'

[emasab]

There seems to be a connectivity issue in your environment, the connection doesn't go beyond the SSL_HANDSHAKE state.
Could you enable the "debug" property? Don't know how to pass it in the HVR hub server