TB with Shelly EM Integration

--- title: TB with Shelly EM Integration tags: ThingsBoard image: https://i.imgur.com/jM0kQLT.png --- # TB w/ Shelly EM Integration :::info Shelly Gen 1 設備，感測端回傳數值分別有各自相對應 topic，連同感測數值，不符合 TB telemetry JSON 格式 (key: value)。 When configured for MQTT Shelly EM reports data on: * ``shellies/shellyem-<deviceid>/emeter//pf power factor`` * ``shellies/shellyem-<deviceid>/emeter//power instantaneous active power in Watts`` * ``shellies/shellyem-<deviceid>/emeter//reactive_power instantaneous reactive power in Watts`` * ``shellies/shellyem-<deviceid>/emeter//total total energy in Wh (accumulated in device's non-volatile memory)`` * ``shellies/shellyem-<deviceid>/emeter//total_returned total energy returned to the grid in Wh (accumulated in device's non-volatile memory)`` * ``shellies/shellyem-<deviceid>/emeter//voltage grid voltage in Volts`` 要讓 Shelly EM，透過 MQTT，回傳感測數值至 TB，其一解法詳見本文。 TB <> TB IoT Gateway (ShellyEMMqttUplinkConverter) <> Mosquitto Broker <> Device: Shelly EM ::: [toc] ## ThingsBoard * Dashbaords > Gateways > Add device * Devices > Shelly-GW > Copy access token ## ThingsBoard IoT Gateway w/ Mosquitto Mqtt Broker ### Mosquitto mqtt broker ##### Package Installation ``` sudo apt install mosquitto ``` ``` $ sudo systemctl status mosquitto.service ● mosquitto.service - Mosquitto MQTT v3.1/v3.1.1 Broker Loaded: loaded (/lib/systemd/system/mosquitto.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2022-08-08 02:15:31 UTC; 12s ago Docs: man:mosquitto.conf(5) man:mosquitto(8) Main PID: 1512271 (mosquitto) Tasks: 3 (limit: 19098) Memory: 1.4M CGroup: /system.slice/mosquitto.service └─1512271 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf Aug 08 02:15:31 sw systemd[1]: Starting Mosquitto MQTT v3.1/v3.1.1 Broker... Aug 08 02:15:31 sw mosquitto[1512271]: [3616570.595474]~DLT~1512271~INFO ~FIFO /tmp/dlt cannot be opened. Retrying late> Aug 08 02:15:31 sw systemd[1]: Started Mosquitto MQTT v3.1/v3.1.1 Broker. ``` #### Config and Debug * Config files: /etc/mosquitto/mosquitto.conf (/etc/mosquitto/cond.d/local.conf) :::spoiler more ```! sudo wget https://github.com/eclipse/mosquitto/blob/master/mosquitto.conf -O /etc/mosquitto/conf.d/local.conf sudo vi /etc/mosquitto/conf.d/local.conf +line 233 listener 1883 +line 531 allow_anonymous true ``` ::: * Log files: /var/log/mosquitto/mosquitto.log ### ThingsBoard IoT Gateway #### Package Installation 1. Download the deb file ``wget https://github.com/thingsboard/thingsboard-gateway/releases/latest/download/python3-thingsboard-gateway.deb`` 2. Install the gateway using apt ``` sudo apt install ./python3-thingsboard-gateway.deb -y $ pip3 show thingsboard-gateway Name: thingsboard-gateway Version: 3.1 Summary: Thingsboard Gateway for IoT devices. Home-page: https://github.com/thingsboard/thingsboard-gateway Author: ThingsBoard Author-email: info@thingsboard.io License: Apache Software License (Apache Software License 2.0) Location: /usr/lib/python3/dist-packages Requires: PyInquirer Required-by: ``` 3. Check gateway status * Failed to start ThingsBoard Gateway ```! star@sw:~/workspace$ sudo systemctl status thingsboard-gateway.service ● thingsboard-gateway.service - ThingsBoard Gateway Loaded: loaded (/etc/systemd/system/thingsboard-gateway.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Mon 2022-08-08 08:34:28 UTC; 14min ago Process: 1527100 ExecStart=/usr/bin/python3 -c from thingsboard_gateway.tb_gateway import daemon; daemon() (code=exited, status=1/FAILURE) Main PID: 1527100 (code=exited, status=1/FAILURE) Aug 08 08:34:28 sw systemd[1]: thingsboard-gateway.service: Scheduled restart job, restart counter is at 5. Aug 08 08:34:28 sw systemd[1]: Stopped ThingsBoard Gateway. Aug 08 08:34:28 sw systemd[1]: thingsboard-gateway.service: Start request repeated too quickly. Aug 08 08:34:28 sw systemd[1]: thingsboard-gateway.service: Failed with result 'exit-code'. Aug 08 08:34:28 sw systemd[1]: Failed to start ThingsBoard Gateway. ``` * Check and ==upgrade protobuf version== Active: active (running) :::spoiler terminal details ``` star@sw:~/workspace$ sudo pip3 show protobuf Name: protobuf Version: 3.6.1 Summary: Protocol Buffers Home-page: https://developers.google.com/protocol-buffers/ Author: None Author-email: None License: 3-Clause BSD License Location: /usr/lib/python3/dist-packages Requires: Required-by: star@sw:~/workspace$ sudo pip3 install protobuf==3.20.1 Collecting protobuf==3.20.1 Downloading protobuf-3.20.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB) |████████████████████████████████| 1.0 MB 901 kB/s Installing collected packages: protobuf Attempting uninstall: protobuf Found existing installation: protobuf 3.6.1 Not uninstalling protobuf at /usr/lib/python3/dist-packages, outside environment /usr Can't uninstall 'protobuf'. No files were found to uninstall. Successfully installed protobuf-3.20.1 star@sw:~/workspace$ pip3 show protobuf Name: protobuf Version: 3.20.1 Summary: Protocol Buffers Home-page: https://developers.google.com/protocol-buffers/ Author: None Author-email: None License: BSD-3-Clause Location: /usr/local/lib/python3.8/dist-packages Requires: Required-by: star@sw:~/workspace$ sudo systemctl status thingsboard-gateway.service ● thingsboard-gateway.service - ThingsBoard Gateway Loaded: loaded (/etc/systemd/system/thingsboard-gateway.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Mon 2022-08-08 08:34:28 UTC; 25min ago Process: 1527100 ExecStart=/usr/bin/python3 -c from thingsboard_gateway.tb_gateway import daemon; daemon() (code=exited, status=1/FAILURE) Main PID: 1527100 (code=exited, status=1/FAILURE) Aug 08 08:34:28 sw systemd[1]: thingsboard-gateway.service: Scheduled restart job, restart counter is at 5. Aug 08 08:34:28 sw systemd[1]: Stopped ThingsBoard Gateway. Aug 08 08:34:28 sw systemd[1]: thingsboard-gateway.service: Start request repeated too quickly. Aug 08 08:34:28 sw systemd[1]: thingsboard-gateway.service: Failed with result 'exit-code'. Aug 08 08:34:28 sw systemd[1]: Failed to start ThingsBoard Gateway. star@sw:~/workspace$ sudo systemctl start thingsboard-gateway.service star@sw:~/workspace$ sudo systemctl status thingsboard-gateway.service ● thingsboard-gateway.service - ThingsBoard Gateway Loaded: loaded (/etc/systemd/system/thingsboard-gateway.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2022-08-08 09:00:03 UTC; 4s ago Main PID: 1527767 (python3) Tasks: 11 (limit: 19098) Memory: 28.9M CGroup: /system.slice/thingsboard-gateway.service └─1527767 /usr/bin/python3 -c from thingsboard_gateway.tb_gateway import daemon; daemon() Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()" Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription suc> Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()" Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription suc> Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()" Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription suc> Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()" Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription suc> Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [tb_device_mqtt.py] - tb_device_mqtt - _on_connect - 139 - connection SUCCESS" Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [tb_gateway_mqtt.py] - tb_gateway_mqtt - gw_subscribe_to_attribute - 175 - Subscribed to *|* with id ``` ::: 4. Configure the gateway * /etc/thingsboard-gateway/config/tb_gateway.yaml :::spoiler more ``` thingsboard: host: 192.168.1.202 port: 1884 remoteShell: false remoteConfiguration: false statsSendPeriodInSeconds: 3600 minPackSendDelayMS: 0 checkConnectorsConfigurationInSeconds: 60 security: accessToken: 1NR2FQ8R3u6vIlbQc23E qos: 1 storage: type: memory read_records_count: 100 max_records_count: 100000 # type: file # data_folder_path: ./data/ # max_file_count: 10 # max_read_records_count: 10 # max_records_per_file: 10000 # type: sqlite # data_file_path: ./data/data.db # messages_ttl_check_in_hours: 1 # messages_ttl_in_days: 7 grpc: enabled: false serverPort: 9595 keepaliveTimeMs: 10000 keepaliveTimeoutMs: 5000 keepalivePermitWithoutCalls: true maxPingsWithoutData: 0 minTimeBetweenPingsMs: 10000 minPingIntervalWithoutDataMs: 5000 connectors: - name: MQTT Broker Connector type: mqtt configuration: mqtt.json ``` ::: * /etc/thingsboard-gateway/config/mqtt.json :::spoiler more ``` { "broker": { "name": "Default Local Broker", "host": "127.0.0.1", "port": 1883, "clientId": "ThingsBoard_gateway", "maxMessageNumberPerWorker": 20, "maxNumberOfWorkers": 100, "security": { "type": "anonymous" } }, "mapping": [ { "topicFilter": "shellies/+/emeter/0/power", "converter": { "type": "custom", "extension": "ShellyEMMqttUplinkConverter" } }, { "topicFilter": "shellies/+/emeter/0/reactive_power", "converter": { "type": "custom", "extension": "ShellyEMMqttUplinkConverter" } }, { "topicFilter": "shellies/+/emeter/0/pf", "converter": { "type": "custom", "extension": "ShellyEMMqttUplinkConverter" } }, { "topicFilter": "shellies/+/emeter/0/voltage", "converter": { "type": "custom", "extension": "ShellyEMMqttUplinkConverter" } }, { "topicFilter": "shellies/+/emeter/0/total", "converter": { "type": "custom", "extension": "ShellyEMMqttUplinkConverter" } }, { "topicFilter": "shellies/+/emeter/0/total_returned", "converter": { "type": "custom", "extension": "ShellyEMMqttUplinkConverter" } } ], "connectRequests": [ { "topicFilter": "sensor/connect", "deviceNameJsonExpression": "${SerialNumber}" }, { "topicFilter": "sensor/+/connect", "deviceNameTopicExpression": "(?<=sensor\/)(.*?)(?=\/connect)" } ], "disconnectRequests": [ { "topicFilter": "sensor/disconnect", "deviceNameJsonExpression": "${SerialNumber}" }, { "topicFilter": "sensor/+/disconnect", "deviceNameTopicExpression": "(?<=sensor\/)(.*?)(?=\/disconnect)" } ], "attributeRequests": [ { "retain": false, "topicFilter": "v1/devices/me/attributes/request", "deviceNameTopicExpression": "${SerialNumber}", "attributeNameJsonExpression": "${sensorModel}" } ], "attributeUpdates": [ { "retain": true, "deviceNameFilter": "SmartMeter.*", "attributeFilter": "uploadFrequency", "topicExpression": "sensor/${deviceName}/${attributeKey}", "valueExpression": "{\"${attributeKey}\":\"${attributeValue}\"}" } ], "serverSideRpc": [ { "deviceNameFilter": ".*", "methodFilter": "echo", "requestTopicExpression": "sensor/${deviceName}/request/${methodName}/${requestId}", "responseTopicExpression": "sensor/${deviceName}/response/${methodName}/${requestId}", "responseTimeout": 10000, "valueExpression": "${params}" }, { "deviceNameFilter": ".*", "methodFilter": "no-reply", "requestTopicExpression": "sensor/${deviceName}/request/${methodName}/${requestId}", "valueExpression": "${params}" } ] } ``` ::: * /var/lib/thingsboard_gateway/extensions/mqtt/shellyem_mqtt_uplink_converter.py :::spoiler more ``` ... ``` ::: #### Config and Debug * Config files: /etc/thingsboard-gateway/config/* * connected_devices.json * grpc_connector_1.json * logs.conf * ==mqtt.json== * request.json * tb-cloud-chain.pem * rest.json * ==tb_gateway.yaml== * Log files: /var/log/thingsboard-gateway/* * ==connector.log== * ==converter.log== * extension.log * service.log * storage.log * ==tb_connection.log== ## Shellies ### Shelly EM * shellies/+/emter/0/+ (or shellies/+/emter/1/+) or if we configured a custom prefix in the device settings: shellies/(custom_prefix)/(device_name)/# * Try using software such as ==mqtt explorer== to listen in on your mqtt broker. It helps you determine which messages are being sent, and the topics on which they are broadcasted. * Connect Shelly EM to Mosquitto Broker (allow_anonymous true) ## Validations ![](https://i.imgur.com/jM0kQLT.png)