# Dry run for distributed syncing across devices
pinned:

## Sprint: Raspberry pi platform testing
Sprint status: current
Sprint deadline: suggest deadline
Apps for testing: Syncthing, yggdrasil, papad-api, papad monitor, papad frontend, Cow radio

Description: We need to install all Apps, setup configuration, dockerize and have a single docker-compose file to bring up all the services. For yggdrasil and Cow Radio sanket might already have some scripts.

Below are the tasks to run on raspberry pi 4 as of now, all of the below will need some scripts, for reference we can have a look at the source code of
[Community Server](https://github.com/MoinhoDigital/community-server).

- [x] **Install:** docker and docker-compose, create docker network and configuration
- [x] **Config and setup:** Apart from Yggdrasil and cowradio, other apps are dockerised, and the `docker-compose` `Dockerfile` should be available in respective code repo or pantoto server
- [ ] **Testing:** Prelim test case here for papad and papad-monitor testing [Test run of import / export scripts with syncthing](#Test-run-of-export--import-script-for-syncing) More test cases are needed.
- [ ] **Integration Testing:** After above tasks, we would have a couple of different pi devices with all apps at different locations
- [ ] **with internet:** connected to internet, we can demo test our logical mesh, with services running. files and annotations syncing, adding pi devices to syncthing, +more
- [ ] **with interrupted internet:** TBA
- [ ] **without Internet:** TBA
- [ ] **Documentation:** Test results

---
## Context

The below is the representation of a typical process on the hyper-local platform.


[TOC]

## Limitations
The platform assumes certain underlying limitations as initial requirements for software developers, Devops, Admin teams to consider.

- Devices hosting papad preferably Pi 4
- Syncthing for syncing resources
- yggdrasil for network*
- Audio Id for audio resource identification
- All devices syncing should be in the same network
- Toxbot for phone recordings and ..

### Papad app
Papad monitor script - https://github.com/janastu/papad-monitor
papad Api Server source - https://gitlab.com/servelots/papad-api
Papad browser client source -

- doesn't assume the source (or location) of the files (audio, images), meaning the files could be anywhere accessible by URI
- more to be added
-

## Use cases or user stories

From the initial surveys, we have identified the personas, and documenting the user stories below.
This will give a simplified idea to the developers and other stakeholders on what features we are looking at and how that will benefit the users.

- As a dropout i want to record cooking recipes, so i can share with my neighbors and friends
- As a teacher i want to teach hindi online, so i dont have to worry about gathering students due to covid
- As a poet i want to do reading of my poems, so my friends can hear and comment back
- As a student i want to find a teacher who can teach me dancing, so i can improve my dance skills
- As a beautician i want to share beauty tips so i can improve my reach in the locality
- As a cooking enthusiast i want to bookmark a particular duration in the recipe so i can specifically point at my questions / clarifications
- As a music teacher i want to be able to mark a particular note in a song to explain cord progression to my students.
- more to be added

## Test cases

### Test cases for syncthing
Pantoto syncserver shared 3 folders with sagehome device
sagehome device sharing the folders with arunpi, geeta, madhu and mani laptops and phones.

1. `papad-send` send only type
2. `papad-receive` receive only type
3. `papad-send-receive` send and receive type

#### Test case 1:
created a file on sagehome with name `sagehome-sending.txt` and added this to `/papad-receive` and `/papad-send-receive` folders.

Expected outcome: all shared devices should be able to see `sagehome-sending.txt` in the `/papad-receive` and `/papad-send-receive`

comments:
> [name=bhanu] passed OK
>

#### Test case 2:
Create a send only folder, checkout here on how to do it, https://docs.syncthing.net/users/foldertypes.html
and share the folder with your peers.
Add a new file to the send only type folder,

expected outcome: all shared devices should be able to see the new file

comments:
> [name=bhanu] passed OK

#### Test case 3:
add a new file in papad-send folder

expected outcome: other devices should not be able to see the file.

comments:
> [name=bhanu] passed OK
>

#### questions

Add if anybody got questions

-----

### Test cases for Papad
This space to document the test cases for implementation and integration.

Test cases 1, 2 and 3 was added before the detailed discussions with Arun

---
#### 1

* Test Case Id: papad-export-annos-to-folder
* Assumptions: Choice of db (sqlite) with rest api
* Test Data: papad-api.test.pantoto.org
* Test Steps:
1. [x] deploy papad-api on server
2. [x] A script to export papad-api data to Syncthing outgoing folder (dir name: papad-annos-sync)
* Expected Results:
>The dir papad-annos-sync will have updated data from the DB
* Result: Pass / Fail
* Comments: [x]

---
#### 2

* Test Case Id: papad-sync-annos-to-device
* Assumptions: syncthing installed and configured in n devices
* Test Data: papad-api.test.pantoto.org + some laptop
* Test Steps:
1. deploy multiple papad instances
2. install and setup syncthing in all this devices
3. Start sync papad-annos-sync dir
4. Run script to import data from papad-annos-sync into DB
* Expected Results:
The dir papad-annos-sync will sync to local papad instances, and the browser should show latest audio and tags thats synced

---
#### 3

* Test Case Id: papad-sync-media-to-device
* Assumptions: syncthing installed and configured in n devices
* Test Data: papad-api.test.pantoto.org + some laptop
* Test Steps:
1. deploy multiple papad instances
2. install and setup syncthing in all this devices
3. Start sync of media folders
4. ?
* Expected Results:
TBA

---

#### 4

* Test Case Id: recorderPi-Papad-local
* Assumptions:
* Test Data:
* Test Steps:
1. Record a audio using recorder pi
2. Add recording (audio) to papad db
3. tag audio
* Expected Resuts: Everything should work just fine
* comments if test failed:

---

#### 5

* Test Case Id: recorderPi-Papad-syncthing-to-remote
* Assumptions: That test 4 (recorderPi-Papad-local) is passed
* Test Data:
* Test Steps:
1. Check if the remote server has synced the recording (audio) files
2. Check if remote papad has the entry to the audio file
3. tag audio
* Expected Results: Everything should work just fine
* comments if test failed:
*

#### 6

* Test Case Id: papad-remote-to-local
* Assumptions: Syncthing installed and configured to folder served by web server
* Test Data: from files.janastu.org
* Test Steps:
1. Check sync path for new files
2. Add recording (audio) to papad db, choose a channel
3. tag audio
* Expected Results: Everything should work just fine
* comments if test failed:
*
---

## Discussions

Dinesh, Bhanu with Arun:
suggestion for media syncing with dynamic document root

a. http://da.pantoto.org/audio-girish-radio/FOLDER01/150409-153031.MP3
`<papad-service-name>/<pi-mac-id>/<path/file.ogg>`

papad-service-name = da.pantoto.org
pi-mac-id = audio-girish-radio (unique id across the network)
path = FOLDER01/
file = file.ogg

Will be effectively uuid
`audio-girish-radio/FOLDER01/150409-153031.MP3`
later will be unique by `$_$,$_$,$_$`
`audio-girish-radio$_$FOLDER01$_$150409-153031.MP3`

Arun: having a source URI and

every pi will have a folder named by mac id
and a curated folder in wich file name seperators will be underscore (`_`)

pi 1 -> export csv into -> macid folder -> sync to pi 2 mac id folder -> ---
more TBA

## Questions
questions re. the distributed syncthing papad app

* What will be the target of the annotations?
* How will papad discover the newly synced data?
* One way is to have a discovery service which will add database entries into local papad

* How will devices discover eachother's syncthing?
* how to setup incoming folders on pantoto?
*

## Tasks
[x] deploy papad-api on pantoto server (need help on this one) arun helped with this deployment
[x] Facing issues with this, the deps are not clean, debugging manually, also asking vignesh to help out
[x] solved: Pandas is not building
[x] deploy papad-api in laptops (add laptops here) and devices.
[x]Visual diagram of system process
[x] Mani working on the digital version
[] Install recording pi and papad together on a raspberry pi - Shalini is working on this
[x] Arun is working on a script to manage export import from / to db so that syncthing can sync exported json files
[] Athithya is working on the beta upgrades to annotate fragment of audio - in progress
[] Papad client bug fixes - Athithya
[x] Shalini deployed syncthing on server
[] Syncthing server configuration - Unassigned

## Illustrations

Documenting all versions of illustrations and diagrams

Process diagrams

The below has been asked for some more edits

System architecture diagram

## Resources and readings
- Similar question using sqlite; suggestions in answers (https://github.com/rqlite/rqlite) - (question) https://stackoverflow.com/questions/16032825/method-to-replicate-sqlite-database-across-multiple-servers
- Some limits of rqlite or raft protocol to keep in mind
- For a cluster of N nodes in size to remain operational, at least (N/2)+1 nodes must be up and running, and be in contact with each other.
- Clusters of 3, 5, 7, or 9, nodes are most practical. Clusters of those sizes can tolerate failures of 1, 2, 3, and 4 nodes respectively.
- Clusters with a greater number of nodes start to become unweildy, due to the number of nodes that must be contacted before a database change can take place.

## Test run of export / import script for syncing
29/06/2020

Arun, Athithya, Bhanu

* Install sync scripts
* configure syncthing path in env variable
* Run papa-api locally
* Run sync scripts main.py
* ~~Set env var SYNCTHING_PATH and run dump.py in a another terminal, this will export local db into json file~~ deprecated
* sync the exported json file
* After adding device, the files sync to deafult folder path
* Syncthing might have a diff to see if the file changed
* >Test case: Possible bug: i notify not reading incoming sync json file - bug fixed
* There is something called syncthing inotify - not needed

Test cases passed
1. run dump script, will dump the db contents into a json file, filename is macid.json
2. run main.py will look for changes in macid.json file, and import the changes

## Links
Sanket's pulse audio testing [ServeLots Webinar Pi](/CBbF7cMXTyylW9hutayWqA)