hi, thingsboard can work with kafka+ssl.
in configuration this use confluent sub name, but don't let this confuse you.
test these parameters with clonfluent=true feature and configure ThingsBoard to work with your Kafka service\cluster.
example:

@Value("${queue.kafka.confluent.ssl.algorithm}")
private String sslAlgorithm;

@Value("${queue.kafka.confluent.sasl.mechanism}")
private String saslMechanism;

@Value("${queue.kafka.confluent.sasl.config}")
private String saslConfig;

@Value("${queue.kafka.confluent.security.protocol}")
private String securityProtocol;

    if (useConfluent) {
        props.put("ssl.endpoint.identification.algorithm", sslAlgorithm);
        props.put("sasl.mechanism", saslMechanism);
        props.put("sasl.jaas.config", saslConfig);
        props.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, securityProtocol);
    }

`

Hi @ban2derlog, I tested the environment variable TB_QUEUE_KAFKA_USE_CONFLUENT_CLOUD to true, but I still get the SSL handshake error with the message

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

It seems the root cause is no option for JKS file path

@FawenYo, ThingsBoard has queue.kafka.other section where you can put any Kafka parameter you need.
It can be a quick win for you without a custom code.
Kafka has numerous parameters, and other section allows you to adjust any!

queue:
...
  kafka:
...
    other: # In this section you can specify custom parameters for Kafka consumer/producer and expose the env variables to configure outside
      - key: "request.timeout.ms" # refer to https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html#producerconfigs_request.timeout.ms
        value: "${TB_QUEUE_KAFKA_REQUEST_TIMEOUT_MS:30000}" # (30 seconds)
      - key: "session.timeout.ms" # refer to https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#consumerconfigs_session.timeout.ms
        value: "${TB_QUEUE_KAFKA_SESSION_TIMEOUT_MS:10000}" # (10 seconds)
        
### PUT YOUR KAFKA SSL PARAMETERS HERE ###

Hi @smatvienko-tb , thanks for your advice.

But in TbKafkaSettings it didn't import the package org.apache.kafka.common.config.SslConfigs, I think it is not possible to use some SSL settings such as

props.put(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, sslTruststoreLocation);

or do I miss something?

Hi @smatvienko-tb , thanks for your advice.

But in TbKafkaSettings it didn't import org.apache.kafka.common.config.SslConfigs, I think it is not possible to use some SSL settings such as

props.put(SslConfigs.SSL_TRUSTSTORE_LOCATION_CONFIG, sslTruststoreLocation);

?

Hi @FawenYo !
Props for Kafka effectively use String as key and value.
See the constant value here:
public static final String SSL_TRUSTSTORE_LOCATION_CONFIG = "ssl.truststore.location";

in the yaml file you need just put a text values:

queue:
  kafka:
    other: 
      - key: "ssl.truststore.location"
        value: "${TB_QUEUE_KAFKA_SSL_TRUSTSTORE_LOCATION:truststore.jks}"

Additional Kafka client parameters:
https://kafka.apache.org/documentation/#security_configclients

Hi, thanks for your help!

I didn't find your mentioned code in the thingsboard project
public static final String SSL_TRUSTSTORE_LOCATION_CONFIG = "ssl.truststore.location";
Can you please give a hint of the line number of the code?

Besides, I checked the yaml file you mentioned, so in that way, I still need to modify the yaml and build the docker image myself? Or I can just set some values to Docker environment variable?

Hi, thanks for your help!

I didn't find your mentioned code in the thingsboard project public static final String SSL_TRUSTSTORE_LOCATION_CONFIG = "ssl.truststore.location"; Can you please give a hint of the line number of the code?

Hi!
Just drill down the constant (Ctrl+B for my IntelliJ Idea)

package org.apache.kafka.common.config;
public class SslConfigs {
public static final String SSL_TRUSTSTORE_LOCATION_CONFIG = "ssl.truststore.location";

Besides, I checked the yaml file you mentioned, so in that way I still need to modify the yaml and build the docker image myself? Or I can just set some environment variable to Docker environment variable?

You need to map a custom yaml file with a standard docker image. After that, you will be able to apply new environment variables.
Or inject your custom lines to the YAML with init container in Kubernetes. Or use a custom init script.

Yes, it's not super easy for future updates. But the scope of work is minimal compared to custom build.

Hi @smatvienko-tb, so I tried to use volumeMount to mount the YAML file to /usr/share/thingsboard/conf/thingsboard.yml, and here is part of my modified YAML

other: # In this section you can specify custom parameters for Kafka consumer/producer and expose the env variables to configure outside
  - key: "request.timeout.ms" # refer to https://docs.confluent.io/platform/current/installation/configuration/producer-configs.html#producerconfigs_request.timeout.ms
    value: "${TB_QUEUE_KAFKA_REQUEST_TIMEOUT_MS:30000}" # (30 seconds)
  - key: "session.timeout.ms" # refer to https://docs.confluent.io/platform/current/installation/configuration/consumer-configs.html#consumerconfigs_session.timeout.ms
    value: "${TB_QUEUE_KAFKA_SESSION_TIMEOUT_MS:10000}" # (10 seconds)
  #TODO: Wait Thingsboard to add SSL option in future build. Below part is self-added code
  - key: "security.protocol"
    value: "${TB_KAFKA_PROTOCOL:none}"
  - key: "ssl.truststore.location"
    value: "${TB_KAFKA_SSL_TRUSTSTORE_LOCATION:none}"
  - key: "ssl.truststore.password"
    value: "${TB_KAFKA_SSL_TRUSTSTORE_PASSWORD:none}"
  - key: "ssl.keystore.location"
    value: "${TB_KAFKA_SSL_KEYSTORE_LOCATION:none}"
  - key: "ssl.truststore.password"
    value: "${TB_KAFKA_SSL_KEYSTORE_PASSWORD:none}"
  - key: "ssl.key.password"
    value: "${TB_KAFKA_SSL_KEY_PASSWORD:none}"

And I set the value with

- name: TB_KAFKA_PROTOCOL
  value: "SSL"
- name: TB_KAFKA_SSL_TRUSTSTORE_LOCATION
  value: /etc/thingsboard/conf/jks/{{ .Values.jks.truststore_name }}
- name: TB_KAFKA_SSL_TRUSTSTORE_PASSWORD
  valueFrom:
    secretKeyRef:
      name: {{ .Values.jks.secret_name }}
      key: {{ .Values.jks.truststore_password_name }}
- name: TB_KAFKA_SSL_KEYSTORE_LOCATION
  value: /etc/thingsboard/conf/jks/{{ .Values.jks.keystore_name }}
- name: TB_KAFKA_SSL_KEYSTORE_PASSWORD
  valueFrom:
    secretKeyRef:
      name: {{ .Values.jks.secret_name }}
      key: {{ .Values.jks.keystore_password_name }}
- name: TB_KAFKA_SSL_KEY_PASSWORD
  valueFrom:
    secretKeyRef:
      name: {{ .Values.jks.secret_name }}
      key: {{ .Values.jks.keystore_password_name }}

But after modified the pod still keep restarting, below is the error log. Do I missed some settings?

@FawenYo the issue you send has a clear cause (the keystore password is not specified) :

Please, set the value and check the result

Thanks! I miss type the key

- key: "ssl.truststore.password"
    value: "${TB_KAFKA_SSL_KEYSTORE_PASSWORD:none}"

Thanks for your help! It seems to be working now. So maybe this can be implemented in a future build?

@FawenYo Thank you for your feature request and very first PR as well.
This feature is in the backlog and waiting to be implemented, tested, and documented as well. @ban2derlog is tracking it.

@FawenYo could you try to attach the second yml file and leave the original yml as is?
It might be much simpler to deploy and upgrade next time.

/usr/share/thingsboard/conf/thingsboard-override.yml

with only your lines (no need to copy-paste the whole yaml)

And set environment variables for your deployment like

SPRING.CONFIG.ADDITIONALLOCATION=classpath:thingsboard-override.yml
or put the exact file location as described https://docs.spring.io/spring-boot/docs/current/reference/html/features.html#features.external-config.files

It works for me fine

Hi @smatvienko-tb , I tested it with my environment, and it seems to work! Maybe this can be added to README/docs as a temporary Kafka SSL solution since it's a more elegant way to do?

the hack with TB_QUEUE_KAFKA_USE_CONFLUENT_CLOUD=true seems to work