Dry run for distributed syncing across devices

# Dry run for distributed syncing across devices pinned: ## Sprint: Raspberry pi platform testing Sprint status: current Sprint deadline: suggest deadline Apps for testing: Syncthing, yggdrasil, papad-api, papad monitor, papad frontend, Cow radio Description: We need to install all Apps, setup configuration, dockerize and have a single docker-compose file to bring up all the services. For yggdrasil and Cow Radio sanket might already have some scripts. Below are the tasks to run on raspberry pi 4 as of now, all of the below will need some scripts, for reference we can have a look at the source code of [Community Server](https://github.com/MoinhoDigital/community-server). - [x] **Install:** docker and docker-compose, create docker network and configuration - [x] **Config and setup:** Apart from Yggdrasil and cowradio, other apps are dockerised, and the `docker-compose` `Dockerfile` should be available in respective code repo or pantoto server - [ ] **Testing:** Prelim test case here for papad and papad-monitor testing [Test run of import / export scripts with syncthing](#Test-run-of-export--import-script-for-syncing) More test cases are needed. - [ ] **Integration Testing:** After above tasks, we would have a couple of different pi devices with all apps at different locations - [ ] **with internet:** connected to internet, we can demo test our logical mesh, with services running. files and annotations syncing, adding pi devices to syncthing, +more - [ ] **with interrupted internet:** TBA - [x] **without Internet:** TBA - [x] **Documentation:** Test results --- ## Context The below is the representation of a typical process on the hyper-local platform. ![](https://i.imgur.com/N3W7Osw.jpg) [TOC] ## Limitations The platform assumes certain underlying limitations as initial requirements for software developers, Devops, Admin teams to consider. - Devices hosting papad preferably Pi 4 - Syncthing for syncing resources - yggdrasil for network* - Audio Id for audio resource identification - All devices syncing should be in the same network - Toxbot for phone recordings and .. ### Papad app Papad monitor script - https://github.com/janastu/papad-monitor papad Api Server source - https://gitlab.com/servelots/papad-api Papad browser client source - - doesn't assume the source (or location) of the files (audio, images), meaning the files could be anywhere accessible by URI - more to be added - ## Use cases or user stories From the initial surveys, we have identified the personas, and documenting the user stories below. This will give a simplified idea to the developers and other stakeholders on what features we are looking at and how that will benefit the users. - As a dropout i want to record cooking recipes, so i can share with my neighbors and friends - As a teacher i want to teach hindi online, so i dont have to worry about gathering students due to covid - As a poet i want to do reading of my poems, so my friends can hear and comment back - As a student i want to find a teacher who can teach me dancing, so i can improve my dance skills - As a beautician i want to share beauty tips so i can improve my reach in the locality - As a cooking enthusiast i want to bookmark a particular duration in the recipe so i can specifically point at my questions / clarifications - As a music teacher i want to be able to mark a particular note in a song to explain cord progression to my students. - more to be added ## Test cases ### Test cases for syncthing Pantoto syncserver shared 3 folders with sagehome device sagehome device sharing the folders with arunpi, geeta, madhu and mani laptops and phones. 1. `papad-send` send only type 2. `papad-receive` receive only type 3. `papad-send-receive` send and receive type #### Test case 1: created a file on sagehome with name `sagehome-sending.txt` and added this to `/papad-receive` and `/papad-send-receive` folders. Expected outcome: all shared devices should be able to see `sagehome-sending.txt` in the `/papad-receive` and `/papad-send-receive` comments: > [name=bhanu] passed OK > #### Test case 2: Create a send only folder, checkout here on how to do it, https://docs.syncthing.net/users/foldertypes.html and share the folder with your peers. Add a new file to the send only type folder, expected outcome: all shared devices should be able to see the new file comments: > [name=bhanu] passed OK #### Test case 3: add a new file in papad-send folder expected outcome: other devices should not be able to see the file. comments: > [name=bhanu] passed OK > #### questions Add if anybody got questions ----- ### Test cases for Papad This space to document the test cases for implementation and integration. Test cases 1, 2 and 3 was added before the detailed discussions with Arun --- #### 1 * Test Case Id: papad-export-annos-to-folder * Assumptions: Choice of db (sqlite) with rest api * Test Data: papad-api.test.pantoto.org * Test Steps: 1. [x] deploy papad-api on server 2. [x] A script to export papad-api data to Syncthing outgoing folder (dir name: papad-annos-sync) * Expected Results: >The dir papad-annos-sync will have updated data from the DB * Result: Pass / Fail * Comments: [x] --- #### 2 * Test Case Id: papad-sync-annos-to-device * Assumptions: syncthing installed and configured in n devices * Test Data: papad-api.test.pantoto.org + some laptop * Test Steps: 1. deploy multiple papad instances 2. install and setup syncthing in all this devices 3. Start sync papad-annos-sync dir 4. Run script to import data from papad-annos-sync into DB * Expected Results: The dir papad-annos-sync will sync to local papad instances, and the browser should show latest audio and tags thats synced --- #### 3 * Test Case Id: papad-sync-media-to-device * Assumptions: syncthing installed and configured in n devices * Test Data: papad-api.test.pantoto.org + some laptop * Test Steps: 1. deploy multiple papad instances 2. install and setup syncthing in all this devices 3. Start sync of media folders 4. ? * Expected Results: TBA --- #### 4 * Test Case Id: recorderPi-Papad-local * Assumptions: * Test Data: * Test Steps: 1. Record a audio using recorder pi 2. Add recording (audio) to papad db 3. tag audio * Expected Resuts: Everything should work just fine * comments if test failed: --- #### 5 * Test Case Id: recorderPi-Papad-syncthing-to-remote * Assumptions: That test 4 (recorderPi-Papad-local) is passed * Test Data: * Test Steps: 1. Check if the remote server has synced the recording (audio) files 2. Check if remote papad has the entry to the audio file 3. tag audio * Expected Results: Everything should work just fine * comments if test failed: * #### 6 * Test Case Id: papad-remote-to-local * Assumptions: Syncthing installed and configured to folder served by web server * Test Data: from files.janastu.org * Test Steps: 1. Check sync path for new files 2. Add recording (audio) to papad db, choose a channel 3. tag audio * Expected Results: Everything should work just fine * comments if test failed: * --- ## Discussions Dinesh, Bhanu with Arun: suggestion for media syncing with dynamic document root a. http://da.pantoto.org/audio-girish-radio/FOLDER01/150409-153031.MP3 `<papad-service-name>/<pi-mac-id>/<path/file.ogg>` papad-service-name = da.pantoto.org pi-mac-id = audio-girish-radio (unique id across the network) path = FOLDER01/ file = file.ogg Will be effectively uuid `audio-girish-radio/FOLDER01/150409-153031.MP3` later will be unique by `$_$,$_$,$_$` `audio-girish-radio$_$FOLDER01$_$150409-153031.MP3` Arun: having a source URI and every pi will have a folder named by mac id and a curated folder in wich file name seperators will be underscore (`_`) pi 1 -> export csv into -> macid folder -> sync to pi 2 mac id folder -> --- more TBA ## Questions questions re. the distributed syncthing papad app * What will be the target of the annotations? * How will papad discover the newly synced data? * One way is to have a discovery service which will add database entries into local papad * How will devices discover eachother's syncthing? * how to setup incoming folders on pantoto? * ## Tasks [x] deploy papad-api on pantoto server (need help on this one) arun helped with this deployment [x] Facing issues with this, the deps are not clean, debugging manually, also asking vignesh to help out [x] solved: Pandas is not building [x] deploy papad-api in laptops (add laptops here) and devices. [x]Visual diagram of system process [x] Mani working on the digital version [] Install recording pi and papad together on a raspberry pi - Shalini is working on this [x] Arun is working on a script to manage export import from / to db so that syncthing can sync exported json files [] Athithya is working on the beta upgrades to annotate fragment of audio - in progress [] Papad client bug fixes - Athithya [x] Shalini deployed syncthing on server [] Syncthing server configuration - Unassigned ## Illustrations Documenting all versions of illustrations and diagrams Process diagrams ![](https://i.imgur.com/MYMSe96.png) The below has been asked for some more edits ![](https://i.imgur.com/JnLV7QY.jpg) ![](https://i.imgur.com/WLqS2ww.jpg) System architecture diagram ## Resources and readings - Similar question using sqlite; suggestions in answers (https://github.com/rqlite/rqlite) - (question) https://stackoverflow.com/questions/16032825/method-to-replicate-sqlite-database-across-multiple-servers - Some limits of rqlite or raft protocol to keep in mind - For a cluster of N nodes in size to remain operational, at least (N/2)+1 nodes must be up and running, and be in contact with each other. - Clusters of 3, 5, 7, or 9, nodes are most practical. Clusters of those sizes can tolerate failures of 1, 2, 3, and 4 nodes respectively. - Clusters with a greater number of nodes start to become unweildy, due to the number of nodes that must be contacted before a database change can take place. ## Test run of export / import script for syncing 29/06/2020 Arun, Athithya, Bhanu * Install sync scripts * configure syncthing path in env variable * Run papa-api locally * Run sync scripts main.py * ~~Set env var SYNCTHING_PATH and run dump.py in a another terminal, this will export local db into json file~~ deprecated * sync the exported json file * After adding device, the files sync to deafult folder path * Syncthing might have a diff to see if the file changed * >Test case: Possible bug: i notify not reading incoming sync json file - bug fixed * There is something called syncthing inotify - not needed Test cases passed 1. run dump script, will dump the db contents into a json file, filename is macid.json 2. run main.py will look for changes in macid.json file, and import the changes ## Links Sanket's pulse audio testing [ServeLots Webinar Pi](/CBbF7cMXTyylW9hutayWqA)