# Kafka MirrorMaker - Kafka Mirror is used for replication of messages from one kafka cluster to another. ## Working - In simple terms kafka mirror is nothing but a producer and consumer combined together in one. - When a message is produced to a topic in source cluster, it is consumed by kafka mirror and it produces the same message to the target cluster. ## Setup - First we need to two config files to start kafka mirror maker. - One provides information for consuming data from source cluster and the other to produce data to target cluster - You need to point the consumer to the source cluster. - For a new cluster we can use offset as earliest. But for an existing cluster use it as latest to avoid producer errors due to bad metadata. **File - consumer.config** ``` bootstrap.servers=<source-kafka-cluster-ip:port> exclude.internal.topics = true # To skip internal topics such as _consumer.offsets auto.offset.reset=earliest # Change based on required offset group.id=test-kafkamirror-consumer-group # Change the consumer group id as required ``` - And the producer to the targer cluster. **File - producer.config** ``` bootstrap.servers=<target-kafka-cluster-ip:port> acks=1 batch.size=100 max.request.size=524288000 ``` ### Start the Kafka Mirror - To consume all the topics from kafka cluster use ``` /usr/bin/kafka-mirror-maker --new.consumer --consumer.config /path-to/consumer.config --producer.config /path-to/producer.config --whitelist ".*" ``` - To consume only to specific topic ``` /usr/bin/kafka-mirror-maker --new.consumer --consumer.config /path-to/consumer.config --producer.config /path-to/producer.config --whitelist "kafka-mirror-maker-topic" ``` - There are also options for black list of certain topics. For which please ref to official documentation.