# IoT Data Logger Framework for Indoor Ag

## TL;DR
The objective of this project is to provide means for students or personal in the biomass lab to collect environmental parameters of their experiments in the most ergonomic way. We propose an Internet of Things (IoT) sensor framework that will communicate via wifi to the localized Raspberry Pis (Rpi). The sensors will be coded via an application on their phone by the user. All data will be sent online to a Master Data base of the lab and can be uploaded via a hardrive (USB) or file synchronization.

## Shortcuts
* [Overleaf documennt](#Paper)
* [Desired Outcomes](#Desired-Outcomes)
* [Weekly Updates 2023](#Weekly-updates-2023)
* [Week of Jan 9th 2023](#Week-of-Jan-9th-2023)
* [Week of Jan 20th 2023](#Week-of-Jan-20th-2023)
* [Weekly Updates 2022](#Weekly-updates)
* [Week of August 10th 2022](#Week-of-Aug-10)
* [Week of August 3rd 2022](#Week-of-Aug-3)
* [Week of July 11th 2022](#Week-of-July-11th)
* [Week of July 4th 2022](#Week-of-July-4th)
* [Week of June 27th 2022](#Week-of-June-27th)
* [Lab Survey](#Lab-Survey)
* [Proposed Framework - IoT sensor netwok](#Proposed-Framework-IoT-sensor-Setup)
* [To-Do List](#To-Do-IoT-Network)
* [Toy Data Logger](#Toy-datalogger-on-RPi)
## Paper
https://www.overleaf.com/5223282549dnyqhpyybqbp
## Weekly updates 2023
### Week of August 28th 2023

#### Installation stuff
- [ ] Mo email Jasmine to understand where she is at in experiment
- [ ] Reflash NodeMcu with authetification key
- [ ] check frequency, 1/per minute
- [ ] Relfash rpi wifi bridge
- [ ] Rewiring on bench 3. Shielding isn't going into the pins (jasmine's bench)
#### pig barn
- [ ] Make excel doc for premade kit
- [ ] include pcb board
- [ ] make a unit price
- [ ] buy a spool fo plastic
* 5$ cad for case
* 30$ for 15 pcb boards / 2$ per board
*
- [ ]

### Week of Feb 10th 2023
I suggest we meet this week in person, Thursday or Friday?
Things that need to be addressed:
Purchase more pcb boards (Mo I am assigning this to you) - Jerome confirm that you can plug everything prior to purchase
What is the status of the junction box, how can I help on this front, do you need me to purchase anything?
I will show you the cables I have when we meet.
Soldering of temp sensors needs to be done (I can do)
We need basic testing scripts for the RH/TEMP and VOC/CO2 sensors
Test all current sensors before installing in greenhouse
What is the status of Influx?
### Week of Jan 20th 2023

#### list of items that need to be adressed:
- [x] Plug in sensors and send data from home to public InlfuxDB and verify we can live plot (Jerome & Flo)
- the update on this is need for influx db account and json reformatting
- [ ] Make new admin influxdb account (Mo)
- [ ] Make instructions on webpage for making an influxdb accoutn or link to good url (30days)
- [ ] Better instructions of whole setup process on website for outside ppl
- [ ] setup wifi
- [ ] pick hardware (nodemcu/pi pico w, rpi)
- [ ] setup influx
- [ ] flash nodes
- [x] Setup a rpi or computer permanently in the greenhouse and connect to mcgill wi-fi (Mo)
- Computer will be picked up on Monday
- [ ] Make a local network
- [ ] Use a flashed node and test if we can send data from greenhouse to influxDB (look at live plots)
- [x] Contact Eugene and understand where he is at in his current cycle of growing lettuce (Jerome)
- [x] Finalize protocol and place in overleaf document to share (Yasmeen)
- [x] Complete the SEEF
- [ ] look into integrating data and being able to push directly to https://ceaod.github.io/
- [ ] Mo to share experimental setup at home
- [x] Look into rpico with platformio (Flo)
- [ ] make new manifest
- [ ] add website dropdown selector to switch between node/pico
- [ ] push to github and document
- meet Thursday Mo, Jerome, Yasmeen at greenhouse (meet for lunch at 12)
- 16hrs of light 220umol/m^2s
- measure the cable length for 16 sensors (2 per bed)
- look into rpi from eugene
- [x] Flo: update web flasher JSON

### Week of Jan 9th 2023
* First meeting of 2023, we discussed the next step to collect data in the greenhouse
* Make code flashable for rpico (this will be downstream)
* Making it is easy to for our users to deploy on the cornell open-source database (like a button)
#### list of items that need to be adressed:
- [ ] Plug in sensors and send data from home to public InlfuxDB and verify we can live plot (Yasmeen & Flo)
- [ ]
- [ ] Setup a rpi or computer permanently in the greenhouse and connect to mcgill wi-fi (Mo or Jerome or together)
- [ ] Make a local network
- [ ] Use a flashed node and test if we can send data from greenhouse to influxDB (look at live plots)
- [ ] Contact Eugene and understand where he is at in his current cycle of growing lettuce (Jerome)
- [ ] Finalize protocol and place in overleaf document to share (Yasmeen)
- [ ] look into integrating data and being able to push directly to https://ceaod.github.io/

## Desired Outcomes
* Provide an easy method to collect more data for researchers who are not hardware savy
* Provide dashboards to encourage researchers to track their experiments through time
* Provide a master data base of all ongoing experiments for data sharing purposes
* Provide a toy data logger package for those who are interested in coding as well

## Lab Survey
After talking to Dr. Lefsrud and surveying the lab we have a few take aways:
1. Not everybody has a rpi (there is a shortage)
2. large majority of the lab are not comfortable with python
3. People are not comfortable with command line
4. The preferred data format is CSV
5. The sampling rates range from every 10 seconds to one hour.
6. List of desired sensors:

| Sensor| Model | Range| Accuracy| Price|
| -------- | -------- | -------- | -------|-----|
| RH/Temp | DHT11 <br> DHT 22 <br> BME 280 |0 to 50ºC <br> -40 to 80ºC <br> -40 to 85ºC |+/- 2ºC (at 0 to 50ºC) <br> +/- 0.5ºC (at -40 to 80ºC) <br> +/-0.5ºC (at 25ºC) | 5-6 $ <br> 6-7 $ <br> 8-16$ |
| Camera | [ArduCAM](https://www.instructables.com/ArduCAM-Mini-ESP8266-Web-Camera/) (for ESP 8266) <br> Rpi Camera | 2592 x 1944 pixels <br> 2592 × 1944 pixel | 2 megapixels <br> 5 megapixels | 26$ <br> 18$|
| CO2 & VOC|CCS811| CO2:400ppm to 8192ppm <br> TVOC: 0ppb to 1187ppb| Did not find | 25$|
| Light intensity | VEML7700 <br> Photo diode -- watt output/total energy. (not LUX) |
| Soil Conductivity | stemma soil sensonr (greenwall) |

### Things to check
* Need to find a light sensor that is not just ON/OFF
* Soil conductivity, not sure will ask someone who has used one.
* In theory ArduCAM does work with NodeMcu and owuld be preferred if we wanted to have a standardized base board for every sensor. A link about debugging can be found [here](https://forum.arducam.com/t/some-questions-about-arducam-mini-2mp-plus-with-nodemcu-v3/1011/4)
* If this does not work we could use a RPi Pico
* send live photo every 2 or 3 seconds (on NodeMcu)
* We should pick one of each sensor. I.e. for temp humidity either DHT11, DHT 22 and BME280
* What do people envision to do with Cameras, is anybody actually doing computer vision in the lab?

## Proposed Framework IoT sensor Setup

### Things to note
* Ethernet cable to greenhouse location
* Security issues: left to the user, for privacy the interface is shared with preferred users.
* adding random password on boot
* local

### STEP 1 - Setting up WiFi connection to locations WiFi
1. User takes the C++ script that we will have written and will upload onto board.
* In fact we could also preflash
* Otherwise this means they need to install arduino onto their computers
* Download C++ and flash the node once
2. Through a hotspot connection to the NodeMcu the user gets to a webserver.
* [Here is a tutorial to set this up](https://randomnerdtutorials.com/esp8266-nodemcu-access-point-ap-web-server/)
* In fact there is a library that does just this, [SoftAP](https://arduino-esp8266.readthedocs.io/en/latest/esp8266wifi/soft-access-point-class.html)
3. Once on web server, the user fills in:
* WiFi Network name
* Password to this network
* Name of the user
* Name of the NodeMcu
4. User is asked to reboot the NodeMcu
* (plug and unplug to enable the arguments taken from webserver are now in the C++ code)

### STEP 2 - setting up sensor on NodeMcu through web server
1. The user will go to the website they have configured by adding their name in previous step: `http://USERNAME+NodeMCUNAME.local`
The user selects the different buttons with the sensor options they would like to pick:
* They will drag the switch for the desired sensor from OFF to ON and this will lead to a wiring diagram.
* What this means for us is that the original C++ file will have many IF statements
* [Here is a tutorial to design the web server](https://randomnerdtutorials.com/esp8266-web-server/)
* [Here is how to add photos on web server](https://randomnerdtutorials.com/display-images-esp32-esp8266-web-server/)

2. We will provide a rpi IP address, maybe we make this drop down or have the person add them manually.(ex: 192.168.0.127)
* This enables data to be sent to a RPi in the greenhouse or whatever location of the experiment.
* the Rpi will not be unique to one user, they will be used as brokers to receive the data from the NodeMcus and will upload elsewhere after
3. We ask the frequency at which they would like to sample their data
* Once unplugged and plugged again, the data will start collecting
### STEP 3 - Viewing data and fetching it from RPi
**Note this section is the backend side of things**
1. The data will be sent to RPi via MQTT protocol
2. The data will be stored locally on the rpi but can be viewed and updated every X hours or unit we choose.
* [Here is how to make a simple html to provide dashboards to people in the lab](https://www.phidgets.com/education/learn/projects/pi-website-2/)
3. At the end of their experiments people will be able to:
* rsync their data to their computer or the lab computer (again this can be through a webpage) We ask where they want to upload (ip adress, user name and pass)
* Plug a USB into the RPI that will enable them to dump their csv files into
4. We need to keep track of the amount of GB of data stored on the RPi at all times, to alert people to download their data and we can free up some space.

#### Things to consider in STEP 3
* Do we have the USB dump button on the dashboard html website
* At what frequency do we create one csv file. One a day? This is to ensure that we don't just have one big file that can be lost at some point
* Where do we host the entire LAB data, do we want cloud storage? Do we have this on a website? It would be cool for people to start comparing.

## To-Do IoT Network
### NodeMcu list
- [x] Check availabilities sensor. (amazon & alibaba & digikey & dfrobot)( Jerome)
- [x] Light intensity
- [x] Soil conductivity
- [x] For RH and Temp do we want BME280 or DHT 22?
- [x] Figure out WiFi connection for greenhouse/location of experiment and setup router ( ask for ethernet, ask sarah) (Yasmeen)
- [x] Check wifi strength in GH, take a photo (Mohamed)
- [x] Start a github for project and share with others (PLNT-NODE) name I was thinking
- [x] Figure out how NodeMcu connects to user's phone.
* Use ~~SoftAP~~ [WifiManager lib](https://github.com/tzapu/WiFiManager)
- [x] Start from custom config example ["SPIFFS/AutoConnectWithFSParameters"](https://github.com/tzapu/WiFiManager/blob/master/examples/Parameters/SPIFFS/AutoConnectWithFSParameters/AutoConnectWithFSParameters.ino)
- [x] add parameters (default params included are wifi SSID/pass): node-name, rpi ip address (maybe login for mqtt but prolly later)
- [x] Make Wiring schematics for:
- [x] Single Temp RH to NodeMcu
- [x] Single Light Intensity to NodeMcu
- [x] Single soil conductivity to NodeMcu
- [x] Single Camera to NodeMcu
- [ ] Single VOCs/CO2 to NodeMcu
* For combinations of multiple sensor on one NodeMcu we can do after. i suggest we start basic
- [x] write basic capture code for the above:
- [ ] Put Mqtt in C++ script
- [x] Make a webpage with available sensors to select:
- [x] Webpage look into making on/off swith for sensor on c++ script
- [x] Make C++ backend with if statements for all possible sensors
- [x] Webpage, look into linking image of schematics for on/off switch.
- [x] Ask for frequency
* [Here is a tutorial to design the web server](https://randomnerdtutorials.com/esp8266-web-server/)
* [Here is how to add photos on web server](https://randomnerdtutorials.com/display-images-esp32-esp8266-web-server/)
- [x] Capture RPI IP address to send over to correct broker

### RPi stuff
- [ ] Get a lab computer to host data base
- [x] Have RPi that are flashed with raspbian and ready to go for project
- [x] count how many are available in l ab
- [ ] Enable MQTT protocol
- [ ] Enable SSH protocol for ourselves
- [ ] Make MQTT file to capture the incoming data:
- [ ] Make small npz files for incoming data and send to sensor specific tmp folders:
* [example from my Plant electrical sensor activity](https://github.com/YasmeenVH/Plant_eV/blob/main/software/Plant_eV/mqtt_test.py)
- [ ] Make a python file for npz aggregation into CSV that will be made daily?
- [ ] Empty sensor specific tmp folder with npz files to not overload the memory of rpi
- [ ] Track data load (DISK free GB)
- [x] Make a data visualization board that are NodeMCu specific or project specific
- [x] How to capture, store and visualize data (FLO to flo diagram)
- [x] Node red ( look into Mohamed)
- [ ] Get NGROK or find a way to tunnel data efficiently for back up on the lab computer
- [ ] Build database framework.... Not sure if these should be called by project or not
- [ ] Provide methods for downloading data via USB from dashboard
- [ ] addign abutton on dashboard when USB is connected
- [ ] Providing rsync instructions

* Overall we need to make an instruction page
* Host an overall dashboard for all ongoing experiments
## Toy datalogger on RPi
The idea here is to provide an educational python package with wiring diagrams to introduce RPI for environmental monitoring to those who want to learn how to code or understand how it works.

### STEP 1 - Making a tutorial page to explain basic RPi setup stuff
1. Explain how to get Raspbian on SD card
2. Explain how to use HDMI to code on RPI or enable SSH to do it wireless
3. Explain how to use command line to install python

### STEP 2 - Package
All of this will be hosted on github with instructions
1. Provide wiring diagram for RH TEMP, Camera, CO2&VOC and any other sensor.
2. Provide instructions to install all python requirements via command line.
* This will be a requirements.txt file
3. Have a config file so users can put their values in:
* Data format: csv, hdf5, json
* sample rate
* Picture rate
* Name of user
* Location for data storage
* File creation frequency
4. Have a main.py script that runs everything

### STEP 3 - Running on boot and data dump

1. Tutorial on systemd, This is systemctl
2. Tutorial on rsync
3. Maybe provide some base analysis in this package. The goal is that maybe people will want to add on

TO BE CONTINUED

## Weekly updates

### Week of Aug 10
* Code for Yasmeen and Jerome are on Github.
* This means individual sensor code is up
* Wifi setting and Html to preselect sensors is up
* We need to decide if Adafruit stemma soil sensor is trully needed since it is not suppose to be that reliable
* Need to write CSS811 c++ code
* ESPCAM get one
* Down the line a pipeline for sending pictures from NodeMcu to RPI needs to be put into place, it will not be using INFLUXDB
* make a python script capturing photo names and saving names to influxdb
* Yasmeen finalize html table
* Florian refactor Jerome's code
* Jerome look into css811 code if possible
### Week of Aug 3
* InfluxDB for nodemcu - Jerome has done
* Yasmeen add influxdb creditials - token, organization and bucket (url annd token)
* Yasmeen + Flo finalize second html for all possible sensors. Not blank
* Flo share public IP for grafana and influxdb purposes to make nice dashboards
* Talk to sarah (Mo)

### Week of July 11th
* Switched from arduino to platformio and use Clion IDE
* If you have pycharm educational license jetbrain
* Mohamed:
* Talk to Sarah to ask for extra port in GH
* Install and setup InfluxDB and Grafana
* Make random number generator and store in InfluxDB and plot with Grafana :https://www.youtube.com/watch?v=7M8MHa6W9w0
* https://www.influxdata.com/blog/getting-started-python-influxdb/
* Jerome:
* Finalize list of sensors
* Purchase sensors from ABRA (x2)
* Pico w, 15$ max per board, up to 50$ for MANY
* Florian:
* Overall Guru that is overviewing
* Yasmeen:
* Finalize the setup of nodemcu to desired rpi, capture for sending
* Sensor code with models number from jerome
* jerome will go over with pphysical sensor
* Switch to turn on sensor code
### Week of July 4th
* Jerome:
* validate sensor purchase is easy
* pick a lux light sensor (Jerome to confirm)
* Ask for Soil conductivity Robinson (Jerome)
* Mohamed:
* check wifi scenario in Mac geenhouses and take screenhot
* Look into Nodered for live plotter for rpi. Make a toy example maybe
* Florian:
* Make flowchart for data capture and visualization (how to access data for live plotter)
* Yasmeen:
* Ask for Mohamed to be added to #deepplants channel on slack
* Start C++ file for nodemcu with known sensors: DHT22, CCS811 & etc...

### Week of June 27th
* applied to NVIDIA grant
* walked through sensor framework

## Other considerations
* TASMOTA which already exists and we would not need to make custom webpages. However after discussion with Florian and checking it myself I do think its maybe too much for what we need and we can make an easy interface. Still open however and we can discuss.
* Example of paper published in 202 or [IoT-enabled Controlled Environment Agriculture](https://ieeexplore-ieee-org.proxy3.library.mcgill.ca/stamp/stamp.jsp?tp=&arnumber=9546017)
* ^ might wanna reach out to the MITACS project peoples: https://www.mitacs.ca/en/projects/development-iot-enabled-controlled-environment-indoor-plant-growth-prototype