--- title: Microservices tags: TB DevOps --- # Microservices [toc] ## TODO * ``-b release-3.2`` --- 預設.env + tb version = 3.2.2，可以成功部署成功並登入 (http://192.168.1.140 sysadmin@thingsboard.org) * ``-b release-3.4`` 503 Service Unavailable ## Architecture diagram (MSA) ![](https://i.imgur.com/eiDUw1Q.png) TB 微服務架構將服務拆分為五類，分別是： * (1) Microservice: [Transport](https://thingsboard.io/docs/reference/msa/#transport-microservices) * HTTP * MQTT * CoAP * LwM2M * (2) Microservice: [Core](https://thingsboard.io/docs/reference/msa/#thingsboard-node) * (3) Microservice: [WEB UI](https://thingsboard.io/docs/reference/msa/#web-ui-microservices) * (4) Microservice: Rule Engine * (5) Microservice: [JavaScript Executor](https://thingsboard.io/docs/reference/msa/#javascript-executor-microservices) ### Clustering mode :::info * **Service Discovery** ThingsBoard uses Zookeeper for service discovery. All ThingsBoard nodes are identical and registered as ephemeral in Zookeeper. (common coordination services: Consul, etcd) ::: ## docker-compose deployment ``` apt install sudo vim git net-tools ``` ### Docker engine & Docker compose * ... ```! sudo apt -y install apt-transport-https ca-certificates curl gnupg2 software-properties-common ``` * ... ``` curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null ``` * ... ``` sudo apt update sudo apt -y install docker-ce docker-ce-cli containerd.io ``` * ... ``` # Docker version 20.10.18, build b40c2f6 docker version sudo systemctl status docker sudo usermod -aG docker swae logout sudo curl -L "https://github.com/docker/compose/releases/download/v2.17.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose sudo chmod +x /usr/local/bin/docker-compose # Docker Compose version v2.17.2 docker compose version ``` > View last logs in runtime ``` docker compose logs -f docker compose logs -f tb-core1 tb-core2 tb-rule-engine1 tb-rule-engine2 ``` * RabbitMQ Installation [here](https://www.rabbitmq.com/install-debian.html#apt-quick-start-cloudsmith) ``` sudo apt-get install erlang sudo apt-get install rabbitmq-server rabbitmqadmin --version rabbitmqadmin 3.8.9 sudo systemctl start rabbitmq-server.service sudo systemctl enable rabbitmq-server.service swae@jun:/sbin$ sudo ./rabbitmq-diagnostics status -n rabbit@localhost swae@jun:/sbin$ sudo ./rabbitmqctl environment swae@jun:/sbin$ sudo ./rabbitmqctl list_users Listing users ... user tags guest [administrator] swae@jun:/sbin$ sudo ./rabbitmqctl list_connections Listing connections ... swae@jun:/sbin$ sudo ./rabbitmq-diagnostics listeners Asking node rabbit@jun to report its protocol listeners ... Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0 swae@jun:/sbin$ sudo ./rabbitmqctl add_user tbqs 5giotlead [sudo] password for swae: Adding user "tbqs" ... Done. Don't forget to grant the user permissions to some virtual hosts! See 'rabbitmqctl help set_permissions' to learn more. swae@jun:/sbin$ sudo ./rabbitmqctl set_user_tags tbqs administrator Setting tags for user "tbqs" to [administrator] ... swae@jun:/sbin$ sudo ./rabbitmqctl set_permissions -p / tbqs ".*" ".*" ".*" Setting permissions for user "tbqs" in vhost "/" ... //RabbitMQ ships with multiple command line tools, each with a set of related commands: //rabbitmqctl for service management and general operator tasks //rabbitmq-diagnostics for diagnostics and health checking //rabbitmq-plugins for plugin management //rabbitmq-queues for maintenance tasks on queues, in particular quorum queues //rabbitmq-streams for maintenance tasks on streams //rabbitmq-upgrade for maintenance tasks related to upgrades they can be found under the sbin directory in installation root. ``` * Docker ``` docker login -u starlinking docker pull thingsboard/tb-node:3.5.0 docker pull thingsboard/tb-web-ui:3.5.0 docker pull thingsboard/tb-js-executor:3.5.0 docker pull thingsboard/tb-http-transport:3.5.0 docker pull thingsboard/tb-mqtt-transport:3.5.0 docker pull thingsboard/tb-coap-transport:3.5.0 docker pull thingsboard/tb-lwm2m-transport:3.5.0 docker pull thingsboard/tb-snmp-transport:3.5.0 cd workspace git clone -b release-3.5 https://github.com/thingsboard/thingsboard.git cd thingsboard/docker/ vi .env +line 1 //二選一 TB_QUEUE_TYPE=kafka | rabbitmq +line 23 //二選一 DATABASE=postgres | hybrid +line 28 //Grafana and Prometheus services MONITORING_ENABLED=false | true vi queue-rabbitmq.env TB_QUEUE_TYPE=rabbitmq TB_QUEUE_RABBIT_MQ_HOST=localhost TB_QUEUE_RABBIT_MQ_PORT=5672 TB_QUEUE_RABBIT_MQ_USERNAME=YOUR_USERNAME TB_QUEUE_RABBIT_MQ_PASSWORD=YOUR_PASSWORD ./docker-create-log-folders.sh Create and chown: user 799 group 799 dir tb-transports/coap/log Create and chown: user 799 group 799 dir tb-transports/lwm2m/log Create and chown: user 799 group 799 dir tb-transports/http/log Create and chown: user 799 group 799 dir tb-transports/mqtt/log Create and chown: user 799 group 799 dir tb-transports/snmp/log Create and chown: user 799 group 799 dir tb-vc-executor/log Create and chown: user 999 group 999 dir tb-node/postgres Create and chown: user 799 group 799 dir tb-node/data Create and chown: user 799 group 799 dir tb-integration-executor/log Create and chown: user 999 group 999 dir tb-node/cassandra Create and chown: user 1001 group 1001 dir tb-node/redis-data swae@tb-app1:~/workspace/tb-ce-docker-compose$ ./docker-install-tb.sh --loadDemo WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "WEB_REPORT_DOCKER_NAME" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. [+] Running 3/0 ✔ Container basic-redis-1 Running 0.0s ✔ Container basic-postgres-1 Running 0.0s ✔ Container basic-cassandra-1 Running 0.0s WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "WEB_REPORT_DOCKER_NAME" variable is not set. Defaulting to a blank string. Starting ThingsBoard installation ... 03:26:07,697 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml] 03:26:07,697 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/config/logback.xml] 03:26:07,699 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs multiple times on the classpath. 03:26:07,699 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/usr/share/thingsboard/conf/logback.xml] 03:26:07,699 |-WARN in ch.qos.logback.classic.LoggerContext[default] - Resource [logback.xml] occurs at [file:/config/logback.xml] 03:26:07,883 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set 03:26:07,904 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Will scan for changes in [file:/config/logback.xml] 03:26:07,904 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - Setting ReconfigureOnChangeTask scanning period to 10 seconds 03:26:07,908 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender] 03:26:07,918 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [fileLogAppender] 03:26:07,938 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@1917513796 - setting totalSizeCap to 3 GB 03:26:07,944 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@1917513796 - Archive files will be limited to [100 MB] each. 03:26:07,980 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@1917513796 - No compression will be used 03:26:07,983 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@1917513796 - Will use the pattern /var/log/thingsboard/tb-monolith/thingsboard.%d{yyyy-MM-dd}.%i.log for the active file 03:26:07,986 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@4678c730 - The date pattern is 'yyyy-MM-dd' from file name pattern '/var/log/thingsboard/tb-monolith/thingsboard.%d{yyyy-MM-dd}.%i.log'. 03:26:07,986 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@4678c730 - Roll-over at midnight. 03:26:07,996 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@4678c730 - Setting initial period to Tue May 23 03:26:07 UTC 2023 03:26:07,998 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property 03:26:08,033 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[fileLogAppender] - Active log file name: /var/log/thingsboard/tb-monolith/thingsboard.log 03:26:08,033 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[fileLogAppender] - File property is set to [/var/log/thingsboard/tb-monolith/thingsboard.log] 03:26:08,034 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender] 03:26:08,036 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT] 03:26:08,037 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property 03:26:08,038 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.thingsboard.server] to INFO 03:26:08,038 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [com.google.common.util.concurrent.AggregateFuture] to OFF 03:26:08,038 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.kafka.common.utils.AppInfoParser] to WARN 03:26:08,038 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.kafka.clients] to WARN 03:26:08,039 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [com.microsoft.azure.servicebus.primitives.CoreMessageReceiver] to OFF 03:26:08,039 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO 03:26:08,039 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [fileLogAppender] to Logger[ROOT] 03:26:08,039 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT] 03:26:08,039 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration. 03:26:08,040 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@6767c1fc - Registering current configuration as safe fallback point ______ __ _ ____ __ /_ __/ / /_ (_) ____ ____ _ _____ / __ ) ____ ____ _ _____ ____/ / / / / __ \ / / / __ \ / __ `/ / ___/ / __ | / __ \ / __ `/ / ___/ / __ / / / / / / / / / / / / / / /_/ / (__ ) / /_/ / / /_/ // /_/ / / / / /_/ / /_/ /_/ /_/ /_/ /_/ /_/ \__, / /____/ /_____/ \____/ \__,_/ /_/ \__,_/ /____/ =================================================== :: ThingsBoard :: (v3.5.0) =================================================== Starting ThingsBoard Installation... Installing DataBase schema for entities... Installing SQL DataBase schema part: schema-entities.sql Installing SQL DataBase schema indexes part: schema-entities-idx.sql Installing SQL DataBase schema PostgreSQL specific indexes part: schema-entities-idx-psql-addon.sql Installing SQL DataBase schema views and functions: schema-views-and-functions.sql Successfully executed query: DROP VIEW IF EXISTS device_info_view CASCADE; Successfully executed query: CREATE OR REPLACE VIEW device_info_view AS SELECT * FROM device_info_active_attribute_view; Installing DataBase schema for timeseries... Installing Cassandra DataBase schema part: schema-keyspace.cql Installing Cassandra DataBase schema part: schema-ts.cql Loading system data... Creating default notification configs for system admin Creating default notification configs for all tenants Loading demo data... WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.nustaq.serialization.FSTClazzInfo (jar:file:/usr/share/thingsboard/bin/thingsboard.jar!/BOOT-INF/lib/fst-2.57.jar!/) to field java.lang.String.value WARNING: Please consider reporting this to the maintainers of org.nustaq.serialization.FSTClazzInfo WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Installation finished successfully! ``` ``` swae@jun:~/workspace/thingsboard/docker$ ./docker-start-services.sh WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. [+] Running 66/34 ⠿ zookeeper Pulled 45.6s ⠿ grafana Pulled 32.8s ⠿ haproxy Pulled 13.0s ⠿ prometheus Pulled 18.4s ⠿ tb-coap-transport Pulled 2.7s ⠿ tb-http-transport1 Pulled 2.8s ⠿ tb-http-transport2 Pulled 2.7s ⠿ tb-js-executor Pulled 2.6s ⠿ tb-lwm2m-transport Pulled 2.8s ⠿ tb-mqtt-transport1 Pulled 2.8s ⠿ tb-mqtt-transport2 Pulled 2.7s ⠿ tb-snmp-transport Pulled 2.8s ⠿ tb-vc-executor1 Pulled 60.4s ⠿ tb-vc-executor2 Pulled 60.4s ⠿ tb-web-ui1 Pulled 2.7s ⠿ tb-web-ui2 Pulled 2.7s [+] Running 33/33 ⠿ Volume "docker_grafana_data" Created 0.0s ⠿ Volume "docker_prometheus_data" Created 0.0s ⠿ Container docker-tb-web-ui2-1 Started 2.5s ⠿ Container docker-prometheus-1 Started 2.4s ⠿ Container docker-redis-1 Running 0.0s ⠿ Container docker-zookeeper-1 Started 2.4s ⠿ Container docker-postgres-1 Running 0.0s ⠿ Container docker-tb-js-executor-10 Started 8.3s ⠿ Container docker-tb-web-ui1-1 Started 2.1s ⠿ Container docker-tb-js-executor-1 Started 13.6s ⠿ Container docker-tb-js-executor-2 Started 2.6s ⠿ Container docker-tb-js-executor-3 Started 3.6s ⠿ Container docker-tb-js-executor-4 Started 4.4s ⠿ Container docker-tb-js-executor-5 Started 5.4s ⠿ Container docker-tb-js-executor-6 Started 7.1s ⠿ Container docker-tb-js-executor-7 Started 9.5s ⠿ Container docker-tb-js-executor-8 Started 11.2s ⠿ Container docker-tb-js-executor-9 Started 1.9s ⠿ Container docker-grafana-1 Started 2.5s ⠿ Container docker-tb-rule-engine2-1 Started 16.3s ⠿ Container docker-tb-rule-engine1-1 Started 16.4s ⠿ Container docker-tb-core2-1 Started 22.4s ⠿ Container docker-tb-core1-1 Started 22.9s ⠿ Container docker-tb-vc-executor1-1 Started 39.9s ⠿ Container docker-tb-coap-transport-1 Start... 27.5s ⠿ Container docker-tb-http-transport1-1 Star... 36.3s ⠿ Container docker-tb-http-transport2-1 Star... 33.9s ⠿ Container docker-tb-mqtt-transport1-1 Star... 39.7s ⠿ Container docker-tb-snmp-transport-1 Start... 28.2s ⠿ Container docker-tb-vc-executor2-1 Started 28.8s ⠿ Container docker-tb-lwm2m-transport-1 Star... 40.5s ⠿ Container docker-tb-mqtt-transport2-1 Star... 37.0s ⠿ Container haproxy-certbot Started 48.0s ``` * wisight ``` swae@tb-app1:~/tb-ce-docker-compose$ ./docker-install-tb.sh --loadDemo WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. [+] Running 4/4 ✔ Network wisight_default Created 0.0s ✔ Container wisight-redis-1 Started 0.6s ✔ Container wisight-cassandra-1 Started 0.5s ✔ Container wisight-postgres-1 Started 0.6s WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. WARN[0000] The "JAVA_OPTS" variable is not set. Defaulting to a blank string. Starting ThingsBoard installation ... ______ __ _ ____ __ /_ __/ / /_ (_) ____ ____ _ _____ / __ ) ____ ____ _ _____ ____/ / / / / __ \ / / / __ \ / __ `/ / ___/ / __ | / __ \ / __ `/ / ___/ / __ / / / / / / / / / / / / / / /_/ / (__ ) / /_/ / / /_/ // /_/ / / / / /_/ / /_/ /_/ /_/ /_/ /_/ /_/ \__, / /____/ /_____/ \____/ \__,_/ /_/ \__,_/ /____/ =================================================== :: ThingsBoard :: (v3.5.0) =================================================== Starting ThingsBoard Installation... Installing DataBase schema for entities... Installing SQL DataBase schema part: schema-entities.sql Installing SQL DataBase schema indexes part: schema-entities-idx.sql Installing SQL DataBase schema PostgreSQL specific indexes part: schema-entities-idx-psql-addon.sql Installing SQL DataBase schema views and functions: schema-views-and-functions.sql Successfully executed query: DROP VIEW IF EXISTS device_info_view CASCADE; Successfully executed query: CREATE OR REPLACE VIEW device_info_view AS SELECT * FROM device_info_active_attribute_view; Installing DataBase schema for timeseries... Installing Cassandra DataBase schema part: schema-keyspace.cql Installing Cassandra DataBase schema part: schema-ts.cql Loading system data... Creating default notification configs for system admin Creating default notification configs for all tenants Loading demo data... WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.nustaq.serialization.FSTClazzInfo (jar:file:/usr/share/thingsboard/bin/thingsboard.jar!/BOOT-INF/lib/fst-2.57.jar!/) to field java.lang.String.value WARNING: Please consider reporting this to the maintainers of org.nustaq.serialization.FSTClazzInfo WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release Installation finished successfully! ``` ## References * https://github.com/rabbitmq/rabbitmq-server/blob/main/deps/rabbit/docs/rabbitmq.conf.example