19_12 ipvo - HackMD

# 19_12 ipvo ## messaging system - pozeljno je da je eksperimentali rad gotov do 9.1 jer ce onda biti konzultacije di ce komentirat radove svakom posebno ### ZOOKEEPER zeroMQ - messaging library ```docker pull zookeeper``` ``` docker run --name some-zookeeper --restart always -d zookeeper e8cbbfff9ff140191b65bdfc9e538c4bda0fb5ddce32e127726fff5664181539 ``` ``` docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES e8cbbfff9ff1 zookeeper "/docker-entrypoint.…" 25 seconds ago Up 23 seconds 2181/tcp, 2888/tcp, 3888/tcp, 8080/tcp some-zookeeper ``` - koristi se localhost adresa: localhost:8080 ``` nano docker-compose.yml korisnik@ODJ-O365-109:~$ docker-compose up Creating network "korisnik_default" with the default driver Creating korisnik_zoo2_1 ... done Creating korisnik_zoo1_1 ... done Creating korisnik_zoo3_1 ... done Attaching to korisnik_zoo2_1, korisnik_zoo3_1, korisnik_zoo1_1 ``` - dobili smo repliciran sustav, tj nekoliko cvorova unutar distibuiranog sustava ### ZAD1 modificirajte docker-compose.yml da radni na 5 cvorova umjesto kao do sad na 3 cvora ``` version: '3.1' services: zoo1: image: zookeeper restart: always hostname: zoo1 ports: - 2181:2181 environment: ZOO_MY_ID: 1 ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181 server.4=zoo4:2888:3888;2181 server.5=zoo5:2888:3888;2181 zoo2: image: zookeeper restart: always hostname: zoo2 ports: - 2182:2181 environment: ZOO_MY_ID: 2 ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181 server.4=zoo4:2888:3888;2181 server.5=zoo5:2888:3888;2181 zoo3: image: zookeeper restart: always hostname: zoo3 ports: - 2183:2181 environment: ZOO_MY_ID: 3 ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181 server.4=zoo4:2888:3888;2181 server.5=zoo5:2888:3888;2181 zoo4: image: zookeeper restart: always hostname: zoo4 ports: - 2184:2181 environment: ZOO_MY_ID: 4 ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181 server.4=zoo4:2888:3888;2181 server.5=zoo5:2888:3888;2181 zoo5: image: zookeeper restart: always hostname: zoo5 ports: - 2185:2181 environment: ZOO_MY_ID: 5 ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181 server.4=zoo4:2888:3888;2181 server.5=zoo5:2888:3888;2181 ``` - servers su isti jer su navedeni serveri koje po id-ju povezuje - portovi se koriste unutar docker mreze ``` korisnik@ODJ-O365-109:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f64239290f45 zookeeper "/docker-entrypoint.…" 8 minutes ago Up 8 minutes 2888/tcp, 3888/tcp, 8080/tcp, 0.0.0.0:2182->2181/tcp, :::2182->2181/tcp korisnik_zoo2_1 ed54d7045fe6 zookeeper "/docker-entrypoint.…" 8 minutes ago Up 8 minutes 2888/tcp, 3888/tcp, 8080/tcp, 0.0.0.0:2183->2181/tcp, :::2183->2181/tcp korisnik_zoo3_1 f1b1a1668da6 zookeeper "/docker-entrypoint.…" 8 minutes ago Up 8 minutes 2888/tcp, 3888/tcp, 8080/tcp, 0.0.0.0:2185->2181/tcp, :::2185->2181/tcp korisnik_zoo5_1 de84f32347f0 zookeeper "/docker-entrypoint.…" 8 minutes ago Up 8 minutes 2888/tcp, 3888/tcp, 8080/tcp, 0.0.0.0:2184->2181/tcp, :::2184->2181/tcp korisnik_zoo4_1 e3cc5fcbada8 zookeeper "/docker-entrypoint.…" 8 minutes ago Up 8 minutes 2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 8080/tcp korisnik_zoo1_1 e8cbbfff9ff1 zookeeper "/docker-entrypoint.…" 22 minutes ago Up 22 minutes 2181/tcp, 2888/tcp, 3888/tcp, 8080/tcp ``` - tu nije dostupan 8080 port jer nmea -> - nakon dodavanja porta 8080 je dostupan ``` version: '3.1' services: zoo1: image: zookeeper restart: always hostname: zoo1 ports: - 2181:2181 - 8080:8080 environment: ZOO_MY_ID: 1 ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181 server.4=zoo4:2888:3888;2181 server.5=zoo5:2888:3888;2181 zoo2: image: zookeeper restart: always hostname: zoo2 ports: - 2182:2181 - 8081:8081 environment: ZOO_MY_ID: 2 ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181 server.4=zoo4:2888:3888;2181 server.5=zoo5:2888:3888;2181 zoo3: image: zookeeper restart: always hostname: zoo3 ports: - 2183:2181 - 8082:8082 environment: ZOO_MY_ID: 3 ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181 server.4=zoo4:2888:3888;2181 server.5=zoo5:2888:3888;2181 zoo4: image: zookeeper restart: always hostname: zoo4 ports: - 2184:2181 - 8083:8083 environment: ZOO_MY_ID: 4 ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181 server.4=zoo4:2888:3888;2181 server.5=zoo5:2888:3888;2181 zoo5: image: zookeeper restart: always hostname: zoo5 ports: - 2185:2181 - 8084:8084 environment: ZOO_MY_ID: 5 ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181 server.4=zoo4:2888:3888;2181 server.5=zoo5:2888:3888;2181 ``` ``` docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES d813ee0120ce zookeeper "/docker-entrypoint.…" 14 seconds ago Up 10 seconds 2888/tcp, 3888/tcp, 8080/tcp, 0.0.0.0:8082->8082/tcp, :::8082->8082/tcp, 0.0.0.0:2183->2181/tcp, :::2183->2181/tcp korisnik_zoo3_1 2526929e4451 zookeeper "/docker-entrypoint.…" 14 seconds ago Up 10 seconds 2888/tcp, 3888/tcp, 8080/tcp, 0.0.0.0:8083->8083/tcp, :::8083->8083/tcp, 0.0.0.0:2184->2181/tcp, :::2184->2181/tcp korisnik_zoo4_1 1cb3d3ad33dd zookeeper "/docker-entrypoint.…" 14 seconds ago Up 10 seconds 2888/tcp, 3888/tcp, 8080/tcp, 0.0.0.0:8081->8081/tcp, :::8081->8081/tcp, 0.0.0.0:2182->2181/tcp, :::2182->2181/tcp korisnik_zoo2_1 0369decc7278 zookeeper "/docker-entrypoint.…" 14 seconds ago Up 11 seconds 2888/tcp, 3888/tcp, 8080/tcp, 0.0.0.0:8084->8084/tcp, :::8084->8084/tcp, 0.0.0.0:2185->2181/tcp, :::2185->2181/tcp korisnik_zoo5_1 cdee64608f5c zookeeper "/docker-entrypoint.…" 14 seconds ago Up 10 seconds 2888/tcp, 0.0.0.0:2181->2181/tcp, :::2181->2181/tcp, 0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 3888/tcp korisnik_zoo1_1 e8cbbfff9ff1 zookeeper "/docker-entrypoint.…" 25 minutes ago Up 24 minutes 2181/tcp, 2888/tcp, 3888/tcp, 8080/tcp some-zookeeper ``` http://localhost:8080/commands ### KAFKA ``` docker network ls NETWORK ID NAME DRIVER SCOPE bb8be45b75cf bridge bridge local cc24bdfcc132 host host local 9eeea86e69dc korisnik_default bridge local 24a42c9ca28b none null local 765509a0e608 some-network bridge local korisnik@ODJ-O365-109:~$ docker network prune WARNING! This will remove all custom networks not used by at least one container. Are you sure you want to continue? [y/N] y Deleted Networks: some-network korisnik@ODJ-O365-109:~$ docker network ls NETWORK ID NAME DRIVER SCOPE bb8be45b75cf bridge bridge local cc24bdfcc132 host host local 9eeea86e69dc korisnik_default bridge local 24a42c9ca28b none null local korisnik@ODJ-O365-109:~$ ``` - kosa crta na kraj reda je prekid retka - u novi treminal pokreni - zookeeper mora bit pokrenut ``` docker run --name kafka-server \ > --network korisnik_default \ > -e ALLOW_PLAINTEXT_LISTENER=yes \ > -e KAFKA_CFG_ZOOKEEPER_CONNECT=zoo1:2181 \ > bitnami/kafka:latest Unable to find image 'bitnami/kafka:latest' locally latest: Pulling from bitnami/kafka f8c1c832ce65: Pull complete 1164ae0729d7: Pull complete Digest: sha256:b0c23c3faa95bb12aaec10ba300b83049f31424ee0df1cdc14b4fde016309072 Status: Downloaded newer image for bitnami/kafka:latest kafka 16:04:54.63 kafka 16:04:54.63 Welcome to the Bitnami kafka container kafka 16:04:54.63 Subscribe to project updates by watching https://github.com/bitnami/containers kafka 16:04:54.63 Submit issues and feature requests at https://github.com/bitnami/containers/issues kafka 16:04:54.63 kafka 16:04:54.63 INFO ==> ** Starting Kafka setup ** kafka 16:04:54.67 WARN ==> You set the environment variable ALLOW_PLAINTEXT_LISTENER=yes. For safety reasons, do not use this flag in a production environment. kafka 16:04:54.68 INFO ==> Initializing Kafka... kafka 16:04:54.68 INFO ==> No injected configuration files found, creating default config files kafka 16:04:54.83 INFO ==> ** Kafka setup finished! ** kafka 16:04:54.84 INFO ==> ** Starting Kafka ** [2022-12-19 16:04:55,653] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$) [2022-12-19 16:04:56,042] INFO Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation (org.apache.zookeeper.common.X509Util) [2022-12-19 16:04:56,135] INFO Registered signal handlers for TERM, INT, HUP (org.apache.kafka.common.utils.LoggingSignalHandler) [2022-12-19 16:04:56,137] INFO starting (kafka.server.KafkaServer) [2022-12-19 16:04:56,137] INFO Connecting to zookeeper on zoo1:2181 (kafka.server.KafkaServer) [2022-12-19 16:04:56,153] INFO [ZooKeeperClient Kafka server] Initializing a new session to zoo1:2181. (kafka.zookeeper.ZooKeeperClient) [2022-12-19 16:04:56,157] INFO Client environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, built on 04/08/2021 16:35 GMT (org.apache.zookeeper.ZooKeeper) [2022-12-19 16:04:56,157] INFO Client environment:host.name=8f745380c973 (org.apache.zookeeper.ZooKeeper) [2022-12-19 16:04:56,157] INFO Client envir . . . ``` - zookepeer se otvra samo prema jednom portu zato je 8081 prazan kod connections a 8080 ima popunjene podatke - 8084 je leader, a ostali su followeri jer ima najveci ID ili zadnji pokrenut (to ni on nije siguran) ### ZAD2 digni jos jednu kafku i spoji ju na isti zookeeper, a treću kafku spoji na drugi zookeeper ``` docker run --name kafka-server1 \ --network korisnik_default \ -e ALLOW_PLAINTEXT_LISTENER=yes \ -e KAFKA_CFG_ZOOKEEPER_CONNECT=zoo1:2181 \ bitnami/kafka:latest ``` ``` docker run --name kafka-server2 \ --network korisnik_default \ -e ALLOW_PLAINTEXT_LISTENER=yes \ -e KAFKA_CFG_ZOOKEEPER_CONNECT=zoo2:2181 \ bitnami/kafka:latest ```