Allow insecure HTTPS

[dodie]

Problem Description

When I try to connect to an Elasticsearch cluster with HTTPS, it is only possible to do so with a certificate with valid CA that is set up via the ca_certs configuration option. In my case, the target Elasticsearch is a test instance used for trials only, but it only accepts HTTPS traffic. For ad-hoc tests it would be convenient to access it without verifying the certificates. (Similarly to how curl can be used with the --insecure flag.)

Proposed Solution

Allow setting the verify_certs property via the connector config.yml, e.g.:

connectors:
-
  ...

elasticsearch:
  host: "https://<ES_HOST>:9200"
  verify_certs: false    # Defaults to true
  api_key: "..."

Additional Context

As workaround to allow insecure access via HTTPS, one might locally change the following flag to False:

connectors/connectors/es/client.py

Line 93 in bed2e63

options["verify_certs"] = True

[Goldini500]

We are currently facing the exact same issue.
Unfortunately changing options["verify_certs"] to "False" and then rebuilding the container does not change anything.

We are getting the same issue:

[FMWK][15:44:34][INFO] Waiting for Elasticsearch at https://elasticsearch-test-cluster-es-http.mgt.svc:9200 (so far: 1 secs)
[FMWK][15:44:34][ERROR] Could not connect to the Elasticsearch server
[FMWK][15:44:34][ERROR] Cannot connect to host elasticsearch-test-cluster-es-http.mgt.svc:9200 ssl:True
[SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1007)')]

@dodie is this working for you?