<center><h1>Setting up cdc->kafka cluster with debezium on windows </h1></center> <h2> Download kafka with debezium plugin and zookeeper</h2> * https://drive.google.com/drive/folders/17c8eNwqJI-jvQwGepY0XR7e8cHayz_UL?usp=sharing * Note that you have to set zookeeper, kafka , connetor and worker properties <h2> Step 3- Setup kafka cluster </h2> * Set env variable ZOOKEEPER_HOME to ~Zookeeper/bin relevant path * open cmd and execute zkserver to check if its working * Add ~/Zookeeper/bin. to Path env variable ``` zkserver ``` * Setup zookeeper config in ~:/Zookeeper/conf/zoo.cfg ``` dataDir=E:/MiscDownloads/KafkaMisc/Zookeeper/apache-zookeeper-3.6.0-bin/data ``` * start zookeeper by below ``` zkserver ``` * start kafka cluster ``` E:\kafka_2.11-2.4.0\bin\windows\kafka-server-start.bat ``` <h2> Step 4- Setup debezium </h2> * create connector.properties in notepad and save in kafka directory ``` name=192.168.3.20 connector.class=io.debezium.connector.sqlserver.SqlServerConnector database.hostname=192.168.3.20 database.port=1433 database.user=username database.password=pw database.applicationIntent=ReadOnly database.dbname=myCMTest_Internal_4.0 database.server.name=192.168.3.20 table.whitelist=dbo.CRS_Partner database.history.kafka.bootstrap.servers=localhost:9092 database.history.kafka.topic=CRSPartner #If kafka is TLS authenticated, uncomment below lines. #database.history.producer.security.protocol=SSL #database.history.producer.ssl.truststore.location=/tmp/kafka.client.truststore.jks ``` * please note that debezium will only publish changes to host/servername.dbo.tablename * create worker.properties in notepad and save in kafka directory ``` offset.storage.file.filename=D:/kafka/kafka_2.11-2.4.0/tmp/connect.offsets bootstrap.servers=localhost:9092 offset.flush.interval.ms=10000 internal.key.converter=org.apache.kafka.connect.json.JsonConverter internal.value.converter=org.apache.kafka.connect.json.JsonConverter internal.key.converter.schemas.enable=false internal.value.converter.schemas.enable=false key.converter=org.apache.kafka.connect.json.JsonConverter value.converter=org.apache.kafka.connect.json.JsonConverter plugin.path=D:/Kafka/kafka_2.11-2.4.0/plugins #If kafka is TLS authenticated, uncomment below lines. #security.protocol=SSL #ssl.truststore.location=/tmp/kafka.client.truststore.jks #producer.security.protocol=SSL #producer.ssl.truststore.location=/tmp/kafka.client.truststore.jks ``` * Please note despite declaration of topic name in connector debezium will publish cdc changes onto hostname.dbo.tablename topic * Start connector ``` E:\kafka_2.11-2.4.0\bin\windows\connect-standalone.bat ``` * Connector is your producer .. start consuming the topic ``` bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic topicName --from-beginning ``` * make changes in your table... you will see the results here ..