confluent-cloud-hybrid-demo

Detailed video of every step here:
---->

https://drive.google.com/file/d/1MQC5X1bzX7G6sivb77Nao2t7I-O15nA8/view?usp=sharing

First clone the git repo, and enter in the folder:

git clone https://github.com/angietd94/confluent-cloud-hybrid-demo.git
cd confluent-cloud-hybrid-demo

Modify first the config.sh file with your values! If done, skip this part.

###Guide: to find values in Confluent Cloud:

CLOUD_KEY=""
CLOUD_SECRET=""

ORG_ID="----****"

ENVID="env-****"

export CLUSTERID="lkc-***"
insidde the cluster information . Cluster > Cluster Settings.

#Either us, eu
export GEO="eu"
export REGION="eu-central-1"

CLUSTERAPIKEY=""
CLUSTERAPISECRET=""



#for example, aws or gcp
CLOUDPROVIDER="aws"

SR_API_KEY=""
SR_API_SECRET=""
SCHEMA_REGISTRY_URL="***+"
The URL in the down-right side!

BOOTSTRAP_SERVERS="*****:****"

CLICKHOUSE DATA

export TOPIC=""
export DATABASE_NAME="default"
export DATABASE_PASSWORD=""
export DATABASE_HOSTNAME="**"
export CLICKHOUSE_HOSTNAME=""

############

Second part: Launching the script!

then:

chmod +x executable.sh 
chmod +x config_files.sh
chmod +x config.sh
chmod +x ccloud-generate-cp-configs.sh
./executable.sh

More information on the part of Auto generating:
Example: Autogenerate Self-Managed Component Configs for Confluent Cloud
https://docs.confluent.io/cloud/current/cp-component/auto-generate-configs.html
Can you see the Control center here?
http://localhost:9021/clusters

and the plugins here?

http://localhost:8083/connector-plugins
What I see:

if YES!

CREATE A TOPIC , I called it simple_example_avro for example with a simple schema:

{
  "doc": "Sample schema to help you get started.",
  "fields": [
    {
      "name": "age",
      "type": "int"
    },
    {
      "doc": "The string is a unicode character sequence.",
      "name": "name",
      "type": "string"
    }
  ],
  "name": "sampleRecord",
  "namespace": "com.mycorp.mynamespace",
  "type": "record"
}

and then after this launch config_files.sh making sure all values are good (they should) This will add a connector for clickhouse

./config_file.sh

You can verify it here, or in the control center!
http://localhost:8083/connectors/

CREATE A TABLE IN CLICKHOUSE SIMILAR TO THE STRUCTURE OF YOUR TOPIC
for example:

CREATE OR REPLACE TABLE simple_example_avro
(
    age Int64,
    name String
)
ENGINE = MergeTree
ORDER BY name;

Try send messages in CC something lik:

{
	"name": "angelica",
	"orderid": 29
}

And the outcome would look like this!

Also some videos here that go through all the process: https://drive.google.com/file/d/1xUpOMX-cFQDH5BVCfKHOc4IMn8uYmNxV/view https://drive.google.com/file/d/1Zr-Nkpmpd2leGuErGQRAPJWxmiheWZTB/view?usp=sharing