Let's Discuss with Bagas: pram.hao@gapp.nthu.edu.tw
In this notespace, I will share my original journey of thinkering with IoT. I hope through this notespace, beginner (like me) can replicate and maintain any entry-level IoT project.
Cheat sheet of internet related tech from this Giga Chad: https://mpolinowski.github.io/docs/intro
Well, this problem is relatively easy if you use off-the-shelf part like the ESP32. Why ESP32? Because it has good performance. Why does it perform well?
There is also another method to connect with the internet such as Cellular Network (SIM). Mostly you will see SIM7000 or SIM7600 as the communication chips. So, what is the different here? I don't know too, So I googled it for you: https://github.com/vshymanskyy/TinyGSM/issues/595
TLDR: SIM7600 support GSM, better internet
Good product ref: https://www.aliexpress.com/item/4000542688096.html
You can also experiment with better chipset (Raspberry Pico, AVR, AI Chip NICLA, STM32, nRF etc) and other methods of communication such as LoRa, NB-IoT, Mesh Network, WLAN.
You can also DIY your system, but it will be so complex and take a lot of your time, especially if you want to have rapid system iteration.
If you are interested in this topic, you can study more about embedded system.
To thinker with IoT or MCU, basically you need an IDE. For most mortals, we will use Arduino IDE, but if you want try the next level you can use SDK provided by the chip manufacturer such as nRF SDK, STM SDK, and so on.
Installing Arduino IDE
For windows, just follow the general tutorial you can find in the google.
Better to use code example provided in each board library, check whether you can program the MCU (For example: WiFi Scanner, ChipID)
For nerd or ubuntu user :). Environment can be a little harsh to you.
For me, I cannot found the serial library. So, I try to install pyserial first for my env. But I haven't installed python yet. So, I need to install python, but should I install python or python3? I dunno! So I googled it for you.
sudo apt install python-is-python3
sudo apt-get install python3-serial -y
Second problem: the port didn't exist
whoami
sudo adduser <username> dialout
sudo chmod a+rw /dev/ttyUSB0
it worked! the port is opened now. Press the boot button while the flashed is connecting. Voila!
For the example above, I use T18_3.0 From Lilygo, set the board as ESP32-Wrover Module, as written in the chip surface.
Now, checking the WiFi scan property:
So, congrats! Now you already succeed at confirming that you can flash the MCU and it can detect WiFi signal
If you use NB-IoT SIM : https://github.com/FelixLinSY/am7020_arduino
If you want to add external antenna: https://community.home-assistant.io/t/how-to-add-an-external-antenna-to-an-esp-board/131601/31 ufl is also a good option
Advance Topic: Filesystem Uploader to work with SPIFFS Ref
SPIFFS (Serial Peripheral Interface Flash File System) is memory management for your ESP32. More specifically, it’s memory storage that doesn’t erase after powering off your ESP32. You can use it for data logging, saving access codes, storing files, and remembering user-entered settings.
Follow this great implementation of WiFi Event: https://deepbluembedded.com/esp32-connect-to-wifi-network-arduino/
Somehow I suggest you to use 115200 baud when working with WiFi
const char* ssid = "xxx";
const char* password = "xxx";
int WiFiStatus;
String Get_WiFiStatus(int Status){
switch(Status){
case WL_IDLE_STATUS:
return "WL_IDLE_STATUS";
case WL_SCAN_COMPLETED:
return "WL_SCAN_COMPLETED";
case WL_NO_SSID_AVAIL:
return "WL_NO_SSID_AVAIL";
case WL_CONNECT_FAILED:
return "WL_CONNECT_FAILED";
case WL_CONNECTION_LOST:
return "WL_CONNECTION_LOST";
case WL_CONNECTED:
return "WL_CONNECTED";
case WL_DISCONNECTED:
return "WL_DISCONNECTED";
}
}
void setup() {
Serial.begin(115200);
Serial.println("Connecting..");
WiFi.begin(ssid, password);
WiFiStatus = WiFi.status();
while(WiFiStatus != WL_CONNECTED){
delay(250);
WiFiStatus = WiFi.status();
Serial.println(Get_WiFiStatus(WiFiStatus));
}
Serial.println("\nConnected To The WiFi Network");
Serial.print("Local ESP32 IP: ");
Serial.println(WiFi.localIP());
}
WiFi.status() return values:
Cannot connect to WiFi Network Troubleshoot: https://techoverflow.net/2021/01/21/how-to-fix-esp32-not-connecting-to-the-wifi-network/
You can pin restart the ESP if it won't connected to a network
ESP.start()
FALLBACK MECHANISM: MULTIRUN
Use this to give the MCU severals recognized/registered AP, then it will connect to the strongest RSSI
WiFiManager: config wifi wihtout hard-coding
Ref: https://dronebotworkshop.com/wifimanager/
WiFi Mode: STA(Station), AP(Access Point), SoftAP (Bridge STA+AP)
SoftAP implementation: https://techtutorialsx.com/2021/01/04/esp32-soft-ap-and-station-modes/
TBD.
Interesting Prepaid SIM Card: https://www.icshop.com.tw/products/368070100227
TBD. Back-up if I2C is not reliable
Related implementation: https://randomnerdtutorials.com/esp8266-esp-now-wi-fi-web-server/
After you are able to make sure that you have access to the MCU and the communication chip, then we can choose the suitable communication protocol
WiFi Client Implementation:
// Creates a client that can connect to to a specified internet IP address and port
// as defined in client.connect().
WiFiClient client;
// 80 is the default port for HTTP
if (client.connect(server, 80)) {
Serial.println("connected to server");
// Make a HTTP request:
client.println("GET /search?q=arduino HTTP/1.1");
client.println("Host: www.google.com");
client.println("Connection: close");
client.println();
}
while (client.available()) {
char c = client.read();
Serial.write(c);
}
// if the server's disconnected, stop the client:
if (!client.connected()) {
Serial.println();
Serial.println("disconnecting from server.");
client.stop();
// do nothing forevermore:
while (true);
}
client.print(): This method is used to send textual data. It takes various data types as arguments (like strings, integers, floats, etc.) and converts them to their textual representation before sending. It adds line breaks (\r\n) after each call unless specified otherwise.
Use client.write() to send raw binary data.
HTTP Client Implementation
Getting time free API: https://www.timeapi.io/swagger/index.html
HTTPClient http;
String url = "https://www.timeapi.io/api/Time/current/zone?timeZone=Asia/Singapore";
http.begin(url.c_str());
int httpCode = http.GET();
if (httpCode > 0) {
String payload = http.getString();
// Parse JSON
DynamicJsonDocument doc(1024);
DeserializationError error = deserializeJson(doc, payload);
if (error) {
Serial.print(F("$> [!] deserializeJson() failed: "));
Serial.println(error.f_str());
startTime = random(0, 1000);
} else {
String year = doc["year"];
String mon = doc["month"];
String day = doc["day"];
String hour = doc["hour"];
String mins = doc["minute"];
String value = doc["dateTime"];
startTime = year + mon + day + hour + mins;
Serial.print("$> [v] Time: ");
Serial.println(value);
}
} else {
Serial.println(F("$> [!] Error on HTTP request"));
startTime = random(0, 1000);
}
Follow this master:
We need to implement this publish and subscribe library to connect with the MQTT broker(server) using WebSocket.
(Read More: It must run over a transport protocol that provides ordered, lossless, bi-directional connections—typically, TCP/IP.)
Library Prereq: MQTT, ArduinoJSON
PUBLISH:
StaticJsonDocument<200> message;
message["timestamp"] = millis();
message["data"] = analogRead(0); // Or you can read data from other sensors
char messageBuffer[512];
serializeJson(message, messageBuffer);
mqtt.publish(PUBLISH_TOPIC, messageBuffer);
The length of the actual topic string is at most 65536 bytes. This is a limit imposed by the mqtt spec, you can't change it. It is also worth noting that the topic is encoded with utf-8, so you may have less than 65536 characters available. The payload of the message is limited to 268,435,456 bytes.
SUBSCRIBE AND CONNECT:
void connectToMQTT() {
// Connect to the MQTT broker
mqtt.begin(MQTT_BROKER_ADRRESS, MQTT_PORT, network);
// Create a handler for incoming messages
mqtt.onMessage(messageHandler);
Serial.print("ESP32 - Connecting to MQTT broker");
while (!mqtt.connect(MQTT_CLIENT_ID, MQTT_USERNAME, MQTT_PASSWORD)) {
Serial.print(".");
delay(100);
}
Serial.println();
if (!mqtt.connected()) {
Serial.println("ESP32 - MQTT broker Timeout!");
return;
}
// Subscribe to a topic, the incoming messages are processed by messageHandler() function
if (mqtt.subscribe(SUBSCRIBE_TOPIC))
Serial.print("ESP32 - Subscribed to the topic: ");
else
Serial.print("ESP32 - Failed to subscribe to the topic: ");
Serial.println(SUBSCRIBE_TOPIC);
Serial.println("ESP32 - MQTT broker Connected!");
}
void messageHandler(String &topic, String &payload) {
Serial.println("ESP32 - received from MQTT:");
Serial.println("- topic: " + topic);
Serial.println("- payload:");
Serial.println(payload);
}
Implement mqtt.loop() in void loop before publish data
Here's what mqtt.loop() does:
By calling mqtt.loop() in the main loop of your Arduino sketch, you ensure that MQTT communication is continuously maintained, allowing your device to send and receive messages reliably. Failure to call mqtt.loop() regularly may result in lost messages or disconnection from the MQTT broker.
TBD. Advance method for real time multi client handling.
Ref: https://randomnerdtutorials.com/esp32-web-server-websocket-sliders/
Designed for intrapcb communication protocol. Usually used by modern complex sensors.
Terminology
Lab: https://embedded-lab.com/blog/lab-14-inter-integrated-circuit-i2c-communication/
Typical pull-up resistor values are 4.7k Ohm for 5V devices and 2.4k Ohm for 3.3V devices.
Cable Length
I2C was originally designed for talking between devices separated by fairly short distances. For example, between all the chips inside an iPhone. So a total length of inches (centimeters), not feet (meters).
Dallas Temp Ref: https://randomnerdtutorials.com/esp32-ds18b20-temperature-arduino-ide/
Cable length and config: https://www.analog.com/en/resources/technical-articles/guidelines-for-reliable-long-line-1wire-networks.html
Core Problem: 'esp32_gpioMux' was not declared in this scope
Use this library instead: OneWireNg
Common peripheral with this protocol is SD Card.
Terminology:
Bit resolution is 12 => 0 -> 4095; Voltage logic is 3.3V
We need to include this library: TridentTD_LineNotify
Ref: https://ithelp.ithome.com.tw/articles/10271219
#include <WiFi.h>
#include <WiFiClient.h>
#include <TridentTD_LineNotify.h>
// Get the token for LINE Notify Webpages: Login->Create New Token->Assign Group
#define LINE_TOKEN "xxxx"
// Credential For AP
const char* ssid = "xxx";
const char* password = "xxx";
// Global Program Variables
int counter = 0;
unsigned long previousMillis = 0; // will store last temp was read
const long interval = 2000; // interval at which to read sensor
void setup(void)
{
Serial.begin(115200);
WiFi.mode(WIFI_STA);
WiFi.begin(ssid, password);
Serial.print("\n\r \n\rWorking to connect");
while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}
// DEV: Time out Fall Back Reconnect to Other AP
Serial.println("");
Serial.print("Connected to ");
Serial.println(ssid);
Serial.print("IP address: ");
Serial.println(WiFi.localIP());
}
void loop(void)
{
unsigned long currentMillis = millis();
if(currentMillis - previousMillis >= interval) {
// DEV: Time out Fall Back Reconnect to Other AP
previousMillis = currentMillis;
counter++;
}
Serial.println(LINE.getVersion());
LINE.setToken(LINE_TOKEN);
LINE.notify("\n Counting:" + String(counter) + "!");
delay(10000);
}
Google app script work kinda like a serverless (lambda)
but google drive API work as end point, so you need another server to handle the upload
Ref for Google Service Account (bit complicated): https://randomnerdtutorials.com/esp32-datalogging-google-sheets/
For easier method, host your web app to provide the API (GET Uplink, POST Downlink):
https://iotdesignpro.com/articles/esp32-data-logging-to-google-sheets-with-google-scripts
Web App Script Quota & Limit = 90 mins/day
Google Sheets API Limit: 300 min/day of write
Back-up Plan: REPL auto-ping
Need to find program execution time, to determine API usage
1. Create Google Sheets, get the ID
2. Create New Script:
var sheet_id = "xxxx";
var sheet_name = "iot_data";
function doGet(e){
var now = new Date();
var ss = SpreadsheetApp.openById(sheet_id);
var sheet = ss.getSheetByName(sheet_name);
var sensor = e.parameter.sensor ? Number(e.parameter.sensor) : 0;
var date = now.toUTCString();
Logger.log("time: " + date);
sheet.appendRow([date, sensor]);
}
for image base64
base64 increase the file size by 33%
function doPost(e) {
try {
var parentFolderId = "13_O4T_L0WbmgLWk5RrUNbojJU2IsyiGm";
var folderName = e.parameter.folder;
var fileName = e.parameter.filename;
var mimeType = e.parameter.mimetype;
var imageData = Utilities.base64Decode(e.parameter.image);
var blob = Utilities.newBlob(imageData, mimeType, fileName);
var parentFolder = DriveApp.getFolderById(parentFolderId);
var subFolder = getOrCreateSubfolder(parentFolder, folderName);
var file = subFolder.createFile(blob);
return ContentService.createTextOutput("File Uploaded: " + file.getUrl());
} catch (error) {
return ContentService.createTextOutput("Error: " + error.message);
}
}
// Function to check if subfolder exists, otherwise create it
function getOrCreateSubfolder(parentFolder, folderName) {
var folders = parentFolder.getFoldersByName(folderName);
return folders.hasNext() ? folders.next() : parentFolder.createFolder(folderName);
}
3. Deploy your web app
4. Test your URL API: Go to web.postman.co; insert the params
5. Check on the sheet
References:
PUSH: Server-Sent Event
SSE allows the browser to receive automatic updates from a server via HTTP connection
Client only send get request in the first time to establish EventSource interface, and then the web server will send JSON to client if there is any new readings. The client will receive updates from the server, but it can’t send any data to the server after the initial handshake.
ESP32 Async Web Server (Ref)
The ESP32 Async Web Server is a powerful web server implementation designed specifically for the ESP32 microcontroller. It allows you to create web applications and services that can handle multiple client requests simultaneously in an asynchronous manner, meaning it can perform tasks concurrently without blocking the main execution thread.
Key features of the ESP32 Async Web Server include:
To construct the web server, you require two distinct files: the Arduino sketch and the HTML file. The HTML file must be stored within a directory named "data" within the Arduino sketch folder.
Both SPIFFS (Serial Peripheral Interface Flash File System) and SD card storage can be used to store files for an Async Web Server on an ESP (Espressif) microcontroller, such as the popular ESP8266 or ESP32. Each has its own advantages and considerations:
SPIFFS:
SD Card:
Conclusion:
The reason GET is used here is likely because this operation is idempotent, meaning it doesn't change the state of the server. The server is expected to respond to the GET request by updating some internal state or performing some operation without altering the server's state itself. Also, GET requests are simpler and can be easily cached by browsers, which can improve performance, especially for repeated requests.
However, it's worth noting that using GET for operations that alter the server's state is generally considered a bad practice because GET requests can be cached by browsers and may be inadvertently repeated if users navigate back or refresh the page. In those cases, the more appropriate method would be POST or PUT, depending on the semantics of the operation.
GET:
POST:
In summary, GET requests are generally easier to handle for simple data retrieval operations, while POST requests offer more security and flexibility for sending complex or sensitive data. The choice between them depends on the specific requirements of your application.
Image above is dummy image, i will create my own if I have the time. Courtesy: unbrick.id
To be able to save the HTML template to the SPI File System, please put the HTML file inside a data folder. Then use the ESP32 Sketch Data Upload in the tools section.
Resources
Ref: https://randomnerdtutorials.com/esp32-microsd-card-arduino/
Method:
SD Card Module use 5V instead of 3.3V
SCK and CLK is the same
TBD. Back-up plan
TBD. Back-up plan
Installing server: https://mpolinowski.github.io/docs/Development/Javascript/2021-06-02–mqtt-cheat-sheet/2021-06-02/
Ref with Sleep Implementation: https://microcontrollerslab.com/esp32-esp8266-publish-sensor-readings-google-sheets-via-ifttt/
Stable solution.
Built-peripherals:
All ESP32 GPIO pins (0-15) can operate in Input or Output modes. Except for the pins (34 to 39) those are input-only pins. You just can’t drive them HIGH or LOW. They’re used only in input mode. On our board, we’ve 4 of them namely (GPIO34 – GPIO35 – GPIO36 – GPIO39). They don’t support PWM functionality as well.
increasing the resolution will give you way more levels of control over the PWM’s duty cycle. Which will result in very smooth and precise control.
If we set the resolution for PWM to 12 bit then you don't need to map the value
TBD.
MCU needs to check whether it receives any command/ack from a server or broker
Ref: https://microcontrollerslab.com/esp32-web-server-dc-motor-speed-l298n-driver/
TBD. Back-up plan.
TBD. Back-up plan.
1 Cell Li-Ion Shield: https://www.electroschematics.com/battery-shield/
This shield contains voltage regulator, voltage boost, and also li-ion protector. The DW01-P battery protection IC is designed to protect lithium-ion/polymer battery from damage or degrading the lifetime due to overcharge, overdischarge, and/or overcurrent for one-cell lithium-ion/polymer battery powered systems, such as cellular phones.
T18 Measuring Power: https://electronics.stackexchange.com/questions/628931/how-to-measure-the-battery-voltage-in-a-ttgo-t18
TBD. Topics: Voltage Booster, Voltage Regulator
https://www.taiwansensor.com.tw/product/kis3r33s-同步整流-dc-dc-降壓電源模組-7v-24v轉5v-3a/
Many devices that accept a 5V PWM signal might still interpret a 3.3V signal as a valid logic high, especially if they are designed to be compatible with both 3.3V and 5V logic systems. However, there's no guarantee for this unless explicitly mentioned in the device's datasheet or specifications.
If the driver is tolerant to a 3.3V input signal, you can directly connect your 3.3V PWM source to it. But if it requires a 5V PWM signal, you might need a level shifter or voltage translator to convert the 3.3V signal to 5V before connecting it to the driver. These devices are designed to convert signals between different voltage levels while maintaining signal integrity and compatibility.
https://www.electronicshub.org/bidirectional-logic-level-converter/
Gill Radiation & Sensor shelter
M12 Connector
https://learn.sparkfun.com/tutorials/button-and-switch-basics/all
https://arduino.stackexchange.com/questions/12587/how-can-i-handle-the-millis-rollover
Queue: https://github.com/SMFSW/Queue/tree/master
Electronic sketch: https://www.circuitlab.com/editor/#?id=7pq5wm&from=homepage
https://randomnerdtutorials.com/visualize-esp32-esp8266-sensor-readings-from-anywhere/
Node Red: https://randomnerdtutorials.com/getting-started-node-red-raspberry-pi/
https://randomnerdtutorials.com/esp8266-nodemcu-webserial-library/
EEPROM: https://deepbluembedded.com/esp32-eeprom-library-tutorial-arduino/
https://microdigisoft.com/esp32-with-modbus-rtu-rs485-protocol-using-arduino-ide/
https://github.com/TronixLab/ModBus-RTU-Temperature-Humidity-SHT20-Sensor/tree/main
https://embeddedthere.com/how-to-interface-esp32-with-rs485-modbus-sensors-with-example-code/
Here are some common Modbus return codes defined in the library: