---
# System prepended metadata

title: Store RPi Maintenance / Troubleshoot
tags: [' maintenance', ' troubleshoot', mPOS]

---

---
tags: mPOS, maintenance, troubleshoot
---

# Store RPi Maintenance / Troubleshoot

[TOC]

# Quick diagnose (no SSH)
  
  ## 確認裝置有連上網: ping 裝置
  - **ping RPi** 
    - `10.*.*.248` 
      - 10.2.219.248, 10.2.113.248...
    - `10.*.*.245` 
      - 10.7.1.245, 10.6.5.245...
  - **ping 發票印表機** `10.*.*.241`
  ## 確認 API Server 正常啟動
- 確認 API 正常啟動 並 連的到 SC
    - `cURL` or **Postman** `GET <RPi IP>/mPOS/status`; 或直接browser打開網址: `result == 1`

      ```json
      /* Status is OK if result == 1 */
      {
        "result": 1,
        "message": "",
        "returnList": {
          "storeShortName": "內湖",
          "creditPercent": 100,
          "server": "0.2.2 210426-1047_f72051c",
          "storeType": "2",
          "storeTel": "0277203999",
          "receiptMSG": null,
          "storePrincipal": null,
          "IP": null,
          "DBtime": "MON 17 MAY 2021 09:47:33 AM CST",
          "TKGUINo": "69568009",
          "storeNo": "00705",
          "passport": 1,
          "storeName": "內湖訓練教室",
          "addr": "內湖燦坤TEST"
        }
      }
      ```
      ![](https://i.imgur.com/o6uewVR.png)

 
- 確認 RPi 與 發票印表機 有連上: `PRN2.state == 2`
    > Note: 非即時, 每分鐘從印表程式接收狀態
    - `cURL` or **Postman** `GET <RPi IP>/mPOS/status/printer`

        ```json
        /* Status is OK : 
         *   result == 1 AND 
         *   returnList.PRN<id>.state == 2 AND 
         *   returnList.PRN<id>.status_text has "ONLINE"
         */
        {
          "result": 1,
          "message": "",
          "returnList": {
            "PRN0": {
              "id": "0", "vendor": null,  "model": "RP700",
              "state": 2,
              "status_value": 0,
              "status_text": "ONLINE",
              "timestamp": 1621321578805
            },
            "PRN2": {
              "id": "2", "vendor": "Partner", "model": "RP700",
              "state": 2,
              "status_value": 2,
              "status_text": "ONLINE|DRAWER",
              "timestamp": 1621321578805
            }
          }
        }
        ```
## Logs
`https://<RPi IP>/logs`

:::info
可從 dashboard 
![](https://i.imgur.com/8WCtaCO.png)
:::

![](https://i.imgur.com/2PHquCk.png)


---

# Troubleshoot (Remote SSH)
## SSH RPi access

:::spoiler login info
- id: `pi`
- pwd: `tk3cmp0s`
:::

## Tomcat / API Server 檢查

- API status (SC DB query store info)
    ```bash
    curl localhost:8080/mPOS/status | jq
    ```
  - 
    ```json
    {
      "result": 1,
      "message": "",
      "returnList": {
        "storeShortName": "內湖",
        "creditPercent": 100,
        "server": "0.2.2 210426-1047_f72051c",
        "storeType": "2",
        "storeTel": "0277203999",
        "receiptMSG": null,
        "storePrincipal": null,
        "IP": null,
        "DBtime": "MON 17 MAY 2021 09:47:33 AM CST",
        "TKGUINo": "69568009",
        "storeNo": "00705",
        "passport": 1,
        "storeName": "內湖訓練教室",
        "addr": "內湖燦坤TEST"
      }
    }
    ```
- API server ping (純測試 API Server, 不 query SC)
  ```bash
  curl localhost:8080/mPOS/status/ping | jq
  ```
  -   
    ```json
    {
      "result": 1,
      "message": "OK"
    }
    ```

## 發票印表機 檢查

- ping 發票印表機 
  - `192.168.1.251` 
  - `10.*.*.241`

- 確認 印表程式服務是否正常運行 ([參: Services](#Services))
  ```bash
  systemctl status pos-print
  ```
  
  ![](https://i.imgur.com/fVDsSKA.png)
  
  若出現 沒有正常運行: `failed`, `inactive` 則重啟服務:
  
  ```bash
  sudo systemctl restart pos-print
  ```

- 確認 RPi 與 發票印表機 有連上: `PRN2.state == 2`
  
  > Note: 非即時, 被動式 每分鐘從印表程式接收狀態

  ```bash
  curl localhost:8080/mPOS/status/printer | jq
  ```
  
  ```json
  {
    "result": 1,
    "message": "",
    "returnList": {
      "PRN0": {
        "id": "0", "vendor": null,  "model": "RP700",
        "state": 2,
        "status_value": 0,
        "status_text": "ONLINE",
        "timestamp": 1621321578805
      },
      "PRN2": {
        "id": "2", "vendor": "Partner", "model": "RP700",
        "state": 2,
        "status_value": 2,
        "status_text": "ONLINE|DRAWER",
        "timestamp": 1621321578805
      }
    }
  }
  ```

## Logs

- tomcat (API Server)
  ```
  /opt/tomcat/latest/logs/catalina.*
  ```
- TK3CPOSPrint (發票印表程式)
  ```
  /opt/tk3c/logs/TK3CPOSPrint.log*
  /opt/tk3c/logs/TK3CPOSPrint.error.log
  ```
- poskeeper ()
  ```
  /opt/tk3c/logs/poskeeper.log*
  /opt/tk3c/logs/poskeeper.error.log*
  ```
- nginx
  ```
  /var/log/nginx/access.log*
  ```
- mosquitto (MQTT broker)
  ```
  /var/log/mosquitto/mosquitto.log*
  ```


## Services
### Service list

- **nginx**
    - `nginx.service`
- **tomcat (API Server)**
    - `tomcat.service`
- **mosquitto (MQTT Broker)**
    - `mosquitto.service`
- **pos-print (發票印表程式)**
    - `pos-print.service`
- **pos-keeper**
    - `pos-keeper.service`
- **pos-cleanup (Daily cleanup script)**
    - `pos-cleanup.timer`
    - `pos-cleanup.service`
- ~~**pos-pickup_activities (即時發佈 activity log)**~~ (20210930 Removed)
    ~~- `pos-pickup_activities.service`~~
    
### Check service status

- with `grep`
  ```bash
  sudo systemctl -all | grep -E "(nginx|mosquitto|tomcat|pos-print|pos-pickup_activities|pos-keeper|pos-clean)"
  ```
- with `awk`
  ```bash
  sudo systemctl -all | awk '/(nginx|mosquitto|tomcat|pos-print|pos-pickup_activities|pos-keeper|pos-clean)/{ printf "%-30s%-9s%-9s%-9s%s\n",$1,$2,$3,$4,(substr($0, index($0, $5))) }'
  ```
    Output:
    ```
    mosquitto.service             loaded   active   running  Mosquitto MQTT v3.1/v3.1.1 Broker
    nginx.service                 loaded   active   running  A high performance web server and a reverse proxy server
    pos-cleanup.service           loaded   inactive dead     mPOS server cleanup    
    pos-keeper.service            loaded   active   running  mPOS Keeper            
    pos-print.service             loaded   active   running  mPOS Printing          
    tomcat.service                loaded   active   running  TomEE Plus             
    pos-cleanup.timer             loaded   active   waiting  mPOS server cleanup 
    ```

- detailed
  ```bash
  systemctl status tomcat.service nginx.service pos-print.service mosquitto.service pos-pickup_activities.service pos-cleanup.timer --lines=0
  ```
<div style="margin: 0 -50px">

![](https://i.imgur.com/t2qwHMm.png)
</div>
  



### 手動重啟 failed service
```bash
sudo systemctl restart nginx.service
```

```bash
sudo systemctl restart mosquitto.service
```

```bash
sudo systemctl restart pos-print.service
```

```bash
sudo systemctl restart tomcat.service
```

```bash
sudo systemctl restart pos-pickup_activities.service
```

```bash
sudo systemctl restart pos-keeper.service
```

---

# Troubleshoot (Using mpos_tool.py)

script file location: `10.1.62.97:/home/tk3c/`

files: 
- mpos_tool.py
- storelist
- deployment_version.ini

usage:

```shell
./mpos_tool.py --help
```

```shell
./mpos_tool.py <command> --help
```

```shell
./mpos_tool.py <command> [--only STORE_NO/IP] [--include STORE_NO/IP] [--exclude STORE_NO/IP]
```

commands:
- `apistatus` - check api status, remote execute `curl localhost:8080/mPOS/status`
- `rprinter` - check receipt printer status,  `curl localhost:8080/mPOS/status/printer`
- `apilog` - check tomcat log
- `service`- performs {start, stop, restart} on systemd service
- `service-summary`- shows status of systemd services
- `jmaptomcat` - shows current tomcat memory usage 
- `clearcache` - clear tomcat jcache
- `apiver` - shows current api version
- `deploy` (CAUTIOUS) - remote deploy tomcat war

optional arguments:
- `-h`, `--help` - show command help
- `--only`, `--include`, `--exclude` - filter storeno or ip
  - example
    - `./mpos_tool.py apistatus --only 219`
    - `./mpos_tool.py apistatus --only 219,228`
    - `./mpos_tool.py apistatus --exclude 705`
    - `./mpos_tool.py apistatus --include 10.1.62.97`



## Tomcat / API Server 檢查

commands: 
- `./mpos_tool.py apistatus`
- `./mpos_tool.py apilog <day_offset=1>`
- `./mpos_tool.py service status tomcat`


### apistatus
```bash
./mpos_tool.py apistatus --only 219
```
![](https://i.imgur.com/QSq9uuz.png)

### apilog
```bash
./mpos_tool.py apilog 2 --only 219
```
![](https://i.imgur.com/nunwM3V.png)


### service status
```bash
./mpos_tool.py service status tomcat --only 219
```
![](https://i.imgur.com/nskdKXQ.png)

## 發票印表機 檢查
commands: 
- `./mpos_tool.py rprinter`
- `./mpos_tool.py service status pos-print`

### rprinter
```bash
./mpos_tool.py rprinter --only 219,598
```
![](https://i.imgur.com/tf1aV5m.png)


### service status

```bash
./mpos_tool.py service status pos-print --only 219
```
![](https://i.imgur.com/ZlN6VHH.png)


## Service
commands: 
- `./mpos_tool.py service status <service(s)>`
- `./mpos_tool.py service start <service(s)>`
- `./mpos_tool.py service stop <service(s)>`
- `./mpos_tool.py service restart <service(s)>`
- `./mpos_tool.py service-summary`

service list:
```javascript
[
  tomcat,
  pos-print, 
  nginx, 
  mosquitto, 
  pos-cleanup, 
  pos-keeper, 
  pos-pickup_activites
]
```

### service status
```bash
./mpos_tool.py service status tomcat pos-print --only 219
```
![](https://i.imgur.com/dY7kzcy.png)

### service restart
```bash
./mpos_tool.py service restart tomcat --only 10.1.62.132
```
![](https://i.imgur.com/Ow3sjG8.png)


### service-summary
```bash
./mpos_tool.py service-summary --exclude 705
```
![](https://i.imgur.com/mJmE2fg.png)



# MISC

## speculation of RP-700(receipt printer) is unabled to reconnect without rebooting RP-700
<div style="margin: 0 -50px">

![](https://i.imgur.com/xMop4oU.png)
</div>


