---
# System prepended metadata

title: TB with Shelly EM Integration
tags: [ThingsBoard]

---

---
title: TB with Shelly EM Integration
tags: ThingsBoard
image: https://i.imgur.com/jM0kQLT.png
---

# TB w/ Shelly EM Integration

:::info
Shelly Gen 1 設備，感測端回傳數值分別有各自相對應 topic，連同感測數值，不符合 TB telemetry JSON 格式 (key: value)。

When configured for MQTT Shelly EM reports data on:

* ``shellies/shellyem-<deviceid>/emeter/<i>/pf power factor``
* ``shellies/shellyem-<deviceid>/emeter/<i>/power instantaneous active power in Watts``
* ``shellies/shellyem-<deviceid>/emeter/<i>/reactive_power instantaneous reactive power in Watts``
* ``shellies/shellyem-<deviceid>/emeter/<i>/total total energy in Wh (accumulated in device's non-volatile memory)``
* ``shellies/shellyem-<deviceid>/emeter/<i>/total_returned total energy returned to the grid in Wh (accumulated in device's non-volatile memory)``
* ``shellies/shellyem-<deviceid>/emeter/<i>/voltage grid voltage in Volts``

要讓 Shelly EM，透過 MQTT，回傳感測數值至 TB，其一解法詳見本文。
TB <> TB IoT Gateway (ShellyEMMqttUplinkConverter) <> Mosquitto Broker <> Device: Shelly EM
:::

[toc]

## ThingsBoard

* Dashbaords > Gateways > Add device
* Devices > Shelly-GW > Copy access token

## ThingsBoard IoT Gateway w/ Mosquitto Mqtt Broker 

### Mosquitto mqtt broker

##### Package Installation

```
sudo apt install mosquitto
```
```
$ sudo systemctl status mosquitto.service
● mosquitto.service - Mosquitto MQTT v3.1/v3.1.1 Broker
     Loaded: loaded (/lib/systemd/system/mosquitto.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2022-08-08 02:15:31 UTC; 12s ago
       Docs: man:mosquitto.conf(5)
             man:mosquitto(8)
   Main PID: 1512271 (mosquitto)
      Tasks: 3 (limit: 19098)
     Memory: 1.4M
     CGroup: /system.slice/mosquitto.service
             └─1512271 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

Aug 08 02:15:31 sw systemd[1]: Starting Mosquitto MQTT v3.1/v3.1.1 Broker...
Aug 08 02:15:31 sw mosquitto[1512271]: [3616570.595474]~DLT~1512271~INFO     ~FIFO /tmp/dlt cannot be opened. Retrying late>
Aug 08 02:15:31 sw systemd[1]: Started Mosquitto MQTT v3.1/v3.1.1 Broker.
```

#### Config and Debug

* Config files: /etc/mosquitto/mosquitto.conf (/etc/mosquitto/cond.d/local.conf)
    :::spoiler more
    ```!
    sudo wget https://github.com/eclipse/mosquitto/blob/master/mosquitto.conf -O /etc/mosquitto/conf.d/local.conf
    
    sudo vi /etc/mosquitto/conf.d/local.conf
    +line 233
    listener 1883
    +line 531
    allow_anonymous true
    ```
    :::
* Log files: /var/log/mosquitto/mosquitto.log

### ThingsBoard IoT Gateway

#### Package Installation

1. Download the deb file
``wget https://github.com/thingsboard/thingsboard-gateway/releases/latest/download/python3-thingsboard-gateway.deb``
2. Install the gateway using apt
    ```
    sudo apt install ./python3-thingsboard-gateway.deb -y

    $ pip3 show thingsboard-gateway
    Name: thingsboard-gateway
    Version: 3.1
    Summary: Thingsboard Gateway for IoT devices.
    Home-page: https://github.com/thingsboard/thingsboard-gateway
    Author: ThingsBoard
    Author-email: info@thingsboard.io
    License: Apache Software License (Apache Software License 2.0)
    Location: /usr/lib/python3/dist-packages
    Requires: PyInquirer
    Required-by:
    ```
3. Check gateway status
* <span style="color:red">Failed to start ThingsBoard Gateway</span>
    ```!
    star@sw:~/workspace$ sudo systemctl status thingsboard-gateway.service
    ● thingsboard-gateway.service - ThingsBoard Gateway
         Loaded: loaded (/etc/systemd/system/thingsboard-gateway.service; enabled; vendor preset: enabled)
         Active: failed (Result: exit-code) since Mon 2022-08-08 08:34:28 UTC; 14min ago
        Process: 1527100 ExecStart=/usr/bin/python3 -c from thingsboard_gateway.tb_gateway import daemon; daemon() (code=exited, status=1/FAILURE)
       Main PID: 1527100 (code=exited, status=1/FAILURE)

    Aug 08 08:34:28 sw systemd[1]: thingsboard-gateway.service: Scheduled restart job, restart counter is at 5.
    Aug 08 08:34:28 sw systemd[1]: Stopped ThingsBoard Gateway.
    Aug 08 08:34:28 sw systemd[1]: thingsboard-gateway.service: Start request repeated too quickly.
    Aug 08 08:34:28 sw systemd[1]: thingsboard-gateway.service: Failed with result 'exit-code'.
    Aug 08 08:34:28 sw systemd[1]: Failed to start ThingsBoard Gateway.
    ```
* Check and ==upgrade protobuf version== <span style="color:green">Active: active (running)</span>
  :::spoiler terminal details
    ```
    star@sw:~/workspace$ sudo pip3 show protobuf
    Name: protobuf
    Version: 3.6.1
    Summary: Protocol Buffers
    Home-page: https://developers.google.com/protocol-buffers/
    Author: None
    Author-email: None
    License: 3-Clause BSD License
    Location: /usr/lib/python3/dist-packages
    Requires:
    Required-by:

    star@sw:~/workspace$ sudo pip3 install protobuf==3.20.1
    Collecting protobuf==3.20.1
      Downloading protobuf-3.20.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
         |████████████████████████████████| 1.0 MB 901 kB/s
    Installing collected packages: protobuf
      Attempting uninstall: protobuf
        Found existing installation: protobuf 3.6.1
        Not uninstalling protobuf at /usr/lib/python3/dist-packages, outside environment /usr
        Can't uninstall 'protobuf'. No files were found to uninstall.
    Successfully installed protobuf-3.20.1
    star@sw:~/workspace$ pip3 show protobuf
    Name: protobuf
    Version: 3.20.1
    Summary: Protocol Buffers
    Home-page: https://developers.google.com/protocol-buffers/
    Author: None
    Author-email: None
    License: BSD-3-Clause
    Location: /usr/local/lib/python3.8/dist-packages
    Requires:
    Required-by:
    star@sw:~/workspace$ sudo systemctl status thingsboard-gateway.service
    ● thingsboard-gateway.service - ThingsBoard Gateway
         Loaded: loaded (/etc/systemd/system/thingsboard-gateway.service; enabled; vendor preset: enabled)
         Active: failed (Result: exit-code) since Mon 2022-08-08 08:34:28 UTC; 25min ago
        Process: 1527100 ExecStart=/usr/bin/python3 -c from thingsboard_gateway.tb_gateway import daemon; daemon() (code=exited, status=1/FAILURE)
       Main PID: 1527100 (code=exited, status=1/FAILURE)

    Aug 08 08:34:28 sw systemd[1]: thingsboard-gateway.service: Scheduled restart job, restart counter is at 5.
    Aug 08 08:34:28 sw systemd[1]: Stopped ThingsBoard Gateway.
    Aug 08 08:34:28 sw systemd[1]: thingsboard-gateway.service: Start request repeated too quickly.
    Aug 08 08:34:28 sw systemd[1]: thingsboard-gateway.service: Failed with result 'exit-code'.
    Aug 08 08:34:28 sw systemd[1]: Failed to start ThingsBoard Gateway.
    star@sw:~/workspace$ sudo systemctl start thingsboard-gateway.service
    star@sw:~/workspace$ sudo systemctl status thingsboard-gateway.service
    ● thingsboard-gateway.service - ThingsBoard Gateway
         Loaded: loaded (/etc/systemd/system/thingsboard-gateway.service; enabled; vendor preset: enabled)
         Active: active (running) since Mon 2022-08-08 09:00:03 UTC; 4s ago
       Main PID: 1527767 (python3)
          Tasks: 11 (limit: 19098)
         Memory: 28.9M
         CGroup: /system.slice/thingsboard-gateway.service
                 └─1527767 /usr/bin/python3 -c from thingsboard_gateway.tb_gateway import daemon; daemon()

    Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
    Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription suc>
    Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
    Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription suc>
    Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
    Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription suc>
    Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 326 - ()"
    Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [mqtt_connector.py] - mqtt_connector - _on_subscribe - 333 - "MQTT Broker Connector" subscription suc>
    Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [tb_device_mqtt.py] - tb_device_mqtt - _on_connect - 139 - connection SUCCESS"
    Aug 08 09:00:03 sw python3[1527767]: ""2022-08-08 09:00:03" - |INFO| - [tb_gateway_mqtt.py] - tb_gateway_mqtt - gw_subscribe_to_attribute - 175 - Subscribed to *|* with id 
    ```
  :::
4. Configure the gateway
  * /etc/thingsboard-gateway/config/tb_gateway.yaml
    :::spoiler more
    ```
    thingsboard:
      host: 192.168.1.202
      port: 1884
      remoteShell: false
      remoteConfiguration: false
      statsSendPeriodInSeconds: 3600
      minPackSendDelayMS: 0
      checkConnectorsConfigurationInSeconds: 60
      security:
        accessToken: 1NR2FQ8R3u6vIlbQc23E
      qos: 1
    storage:
      type: memory
      read_records_count: 100
      max_records_count: 100000
    #  type: file
    #  data_folder_path: ./data/
    #  max_file_count: 10
    #  max_read_records_count: 10
    #  max_records_per_file: 10000
    #  type: sqlite
    #  data_file_path: ./data/data.db
    #  messages_ttl_check_in_hours: 1
    #  messages_ttl_in_days: 7
    grpc:
      enabled: false
      serverPort: 9595
      keepaliveTimeMs: 10000
      keepaliveTimeoutMs: 5000
      keepalivePermitWithoutCalls: true
      maxPingsWithoutData: 0
      minTimeBetweenPingsMs: 10000
      minPingIntervalWithoutDataMs: 5000
    connectors:
      -
        name: MQTT Broker Connector
        type: mqtt
        configuration: mqtt.json
    ```
    :::
  * /etc/thingsboard-gateway/config/mqtt.json
    :::spoiler more
    ```
      {
      "broker": {
        "name": "Default Local Broker",
        "host": "127.0.0.1",
        "port": 1883,
        "clientId": "ThingsBoard_gateway",
        "maxMessageNumberPerWorker": 20,
        "maxNumberOfWorkers": 100,
        "security": {
         "type": "anonymous"
        }
      },
      "mapping": [
        {
          "topicFilter": "shellies/+/emeter/0/power",
          "converter": {
            "type": "custom",
            "extension": "ShellyEMMqttUplinkConverter"
            }
        },
        {
          "topicFilter": "shellies/+/emeter/0/reactive_power",
          "converter": {
            "type": "custom",
            "extension": "ShellyEMMqttUplinkConverter"
            }
        },
        {
          "topicFilter": "shellies/+/emeter/0/pf",
          "converter": {
            "type": "custom",
            "extension": "ShellyEMMqttUplinkConverter"
            }
        },
        {
          "topicFilter": "shellies/+/emeter/0/voltage",
          "converter": {
            "type": "custom",
            "extension": "ShellyEMMqttUplinkConverter"
            }
        },
        {
          "topicFilter": "shellies/+/emeter/0/total",
          "converter": {
            "type": "custom",
            "extension": "ShellyEMMqttUplinkConverter"
            }
        },
        {
          "topicFilter": "shellies/+/emeter/0/total_returned",
          "converter": {
            "type": "custom",
            "extension": "ShellyEMMqttUplinkConverter"
            }
        }
      ],
      "connectRequests": [
        {
          "topicFilter": "sensor/connect",
          "deviceNameJsonExpression": "${SerialNumber}"
        },
        {
          "topicFilter": "sensor/+/connect",
          "deviceNameTopicExpression": "(?<=sensor\/)(.*?)(?=\/connect)"
        }
      ],
      "disconnectRequests": [
        {
          "topicFilter": "sensor/disconnect",
          "deviceNameJsonExpression": "${SerialNumber}"
        },
        {
          "topicFilter": "sensor/+/disconnect",
          "deviceNameTopicExpression": "(?<=sensor\/)(.*?)(?=\/disconnect)"
        }
      ],
      "attributeRequests": [
        {
          "retain": false,
          "topicFilter": "v1/devices/me/attributes/request",
          "deviceNameTopicExpression": "${SerialNumber}",
          "attributeNameJsonExpression": "${sensorModel}"
        }
      ],
      "attributeUpdates": [
        {
          "retain": true,
          "deviceNameFilter": "SmartMeter.*",
          "attributeFilter": "uploadFrequency",
          "topicExpression": "sensor/${deviceName}/${attributeKey}",
          "valueExpression": "{\"${attributeKey}\":\"${attributeValue}\"}"
        }
      ],
      "serverSideRpc": [
        {
          "deviceNameFilter": ".*",
          "methodFilter": "echo",
          "requestTopicExpression": "sensor/${deviceName}/request/${methodName}/${requestId}",
          "responseTopicExpression": "sensor/${deviceName}/response/${methodName}/${requestId}",
          "responseTimeout": 10000,
          "valueExpression": "${params}"
        },
        {
          "deviceNameFilter": ".*",
          "methodFilter": "no-reply",
          "requestTopicExpression": "sensor/${deviceName}/request/${methodName}/${requestId}",
          "valueExpression": "${params}"
        }
      ]
    }
    ```
    :::
  * /var/lib/thingsboard_gateway/extensions/mqtt/shellyem_mqtt_uplink_converter.py
    :::spoiler more
    ```
    ...
    ```
    :::


#### Config and Debug

* Config files: /etc/thingsboard-gateway/config/*
  * connected_devices.json
  * grpc_connector_1.json
  * logs.conf
  * ==mqtt.json==
  * request.json
  * tb-cloud-chain.pem
  * rest.json
  * ==tb_gateway.yaml==
* Log files: /var/log/thingsboard-gateway/*
  * ==connector.log==
  * ==converter.log==
  * extension.log
  * service.log
  * storage.log
  * ==tb_connection.log==


## Shellies

### Shelly EM

* shellies/+/emter/0/+ (or shellies/+/emter/1/+)
or if we configured a custom prefix in the device settings: shellies/(custom_prefix)/(device_name)/#
* Try using software such as ==mqtt explorer== to listen in on your mqtt broker.
It helps you determine which messages are being sent, and the topics on which they are broadcasted.
* Connect Shelly EM to Mosquitto Broker (allow_anonymous true)


## Validations

![](https://i.imgur.com/jM0kQLT.png)

