Vector fails to connect to elasticsearch sink with a SSL error when using fully qualified DNS

[s-at-ik]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

When using a (valid) fully qualified DNS record (i.e. with a trailing dot) for the elasticsearch sink, vector fails to connect with the following error:

WARN sink{component_kind="sink" component_id=es1 component_type=elasticsearch}:http: vector::internal_events::http_client: HTTP error. error=error trying to connect: error:0A000417:SSL routines:(unknown function):ssl/tls  │
│ alert illegal parameter:ssl/record/rec_layer_s3.c:865:SSL alert number 47 error_type="request_failed" stage="processing" internal_log_rate_limit=true

Configuration

sinks:
  es1:
    type: elasticsearch
    inputs:
      - normalise
    auth:
      strategy: basic
      user: "${ELASTIC_USERNAME:?missing elastic credentials}"
      password: "${ELASTIC_PASSWORD:?missing elastic credentials}"
    endpoints:
      - https://ingest.es1.svc.cluster.local.:9200
    tls:
      verify_certificate: false
    mode: data_stream
    data_stream:
      dataset: "application"
      namespace: "prod"
      type: "logs"

Version

vector 0.38.0 (x86_64-unknown-linux-gnu ea0ec6f 2024-05-07 14:34:39.794027186)

Debug Output

No response

Example Data

No response

Additional Context

Both vector and elastic are running in the same kubernetes cluster.

Using curl works as expected:

~ # curl -sSfk https://ingest.es1.svc.cluster.local:9200
curl: (22) The requested URL returned error: 401
~ # curl -sSfk https://ingest.es1.svc.cluster.local.:9200
curl: (22) The requested URL returned error: 401

References

No response

[jszwedko]

Hi @s-at-ik ,

We just defer to OpenSSL for the connection so I'm not sure if there is anything to be done in Vector itself. Are you able to successfully connect using openssl s_client and the FQDN including the trailing .?

[s-at-ik]

Hi @jszwedko ,

openssl s_client does work with or without the trailing ..

without trailing dot

root@vector-67487b4c66-tr5bl:/# openssl s_client -connect ingest.es1.svc.cluster.local:9200
CONNECTED(00000003)
depth=1 OU = es1, CN = es1-http
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 OU = es1, CN = es1-http
verify return:1
depth=0 OU = es1, CN = es1-es-http.es1.cluster.local
verify return:1
---
Certificate chain
 0 s:OU = es1, CN = es1-es-http.es1.cluster.local
   i:OU = es1, CN = es1-http
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: May  7 11:26:18 2024 GMT; NotAfter: May  7 11:36:18 2025 GMT
 1 s:OU = es1, CN = es1-http
   i:OU = es1, CN = es1-http
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Apr 23 13:54:39 2024 GMT; NotAfter: Apr 23 14:04:39 2025 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
subject=OU = es1, CN = es1-es-http.es1.cluster.local
issuer=OU = es1, CN = es1-http
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2452 bytes and written 433 bytes
Verification error: self-signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self-signed certificate in certificate chain)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: F6F2D7B3F6ED0BEA45515F8977F1F14B9D0736C5297D26C7046451AF80624F87
    Session-ID-ctx:
    Resumption PSK: E13F36004F9D62F2E24E6D0B8447638B739B86BD8EC70228F100300CB9F59C81089C4B02E18F5A69860581AE1BC3DD3E
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    [...]

    Start Time: 1716449919
    Timeout   : 7200 (sec)
    Verify return code: 19 (self-signed certificate in certificate chain)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
^C

with trailing dot

root@vector-67487b4c66-tr5bl:/# openssl s_client -connect ingest.es1.svc.cluster.local.:9200
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 OU = es1, CN = es1-http
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 OU = es1, CN = es1-http
verify return:1
depth=0 OU = es1, CN = es1-es-http.es1.cluster.local
verify return:1
---
Certificate chain
 0 s:OU = es1, CN = es1-es-http.es1.cluster.local
   i:OU = es1, CN = es1-http
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: May  7 11:26:18 2024 GMT; NotAfter: May  7 11:36:18 2025 GMT
 1 s:OU = es1, CN = es1-http
   i:OU = es1, CN = es1-http
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Apr 23 13:54:39 2024 GMT; NotAfter: Apr 23 14:04:39 2025 GMT
---
Server certificate
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
subject=OU = es1, CN = es1-es-http.es1.cluster.local
issuer=OU = es1, CN = es1-http
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2452 bytes and written 377 bytes
Verification error: self-signed certificate in certificate chain
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 19 (self-signed certificate in certificate chain)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 38C6D1F1CA62461CAC283493CED04DECB9D6700452D6940C96B20D03801C81F3
    Session-ID-ctx:
    Resumption PSK: 28CBE74173714C6B82B024AB6322BD04A1533E2D2F27309EED3C3E25998CBFFFBFF96ADE81F9B4496253BFB60D87B0A3
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    [...]

    Start Time: 1716449914
    Timeout   : 7200 (sec)
    Verify return code: 19 (self-signed certificate in certificate chain)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
^C

[jszwedko]

Interesting! Thanks for trying that. I'm not sure what could be going on here then 🤔

[s-at-ik]

More findings:

• I encountered the same issue outside of kubernetes with a simple docker run (official 0.38.0 image).
• The problem does not seem to occur if the endpoint uses port 443 or does not specify a port (e.g. something like https://ingest.es1.svc.cluster.local.:443 would work).