Dry run for distributed syncing across devices

pinned:

Sprint: Raspberry pi platform testing

Sprint status: current
Sprint deadline: suggest deadline
Apps for testing: Syncthing, yggdrasil, papad-api, papad monitor, papad frontend, Cow radio

Description: We need to install all Apps, setup configuration, dockerize and have a single docker-compose file to bring up all the services. For yggdrasil and Cow Radio sanket might already have some scripts.

Below are the tasks to run on raspberry pi 4 as of now, all of the below will need some scripts, for reference we can have a look at the source code of
Community Server.

  • Install: docker and docker-compose, create docker network and configuration
  • Config and setup: Apart from Yggdrasil and cowradio, other apps are dockerised, and the docker-compose Dockerfile should be available in respective code repo or pantoto server
  • Testing: Prelim test case here for papad and papad-monitor testing Test run of import / export scripts with syncthing More test cases are needed.
  • Integration Testing: After above tasks, we would have a couple of different pi devices with all apps at different locations
    • with internet: connected to internet, we can demo test our logical mesh, with services running. files and annotations syncing, adding pi devices to syncthing, +more
    • with interrupted internet: TBA
    • without Internet: TBA
  • Documentation: Test results

Context

The below is the representation of a typical process on the hyper-local platform.

Limitations

The platform assumes certain underlying limitations as initial requirements for software developers, Devops, Admin teams to consider.

  • Devices hosting papad preferably Pi 4
  • Syncthing for syncing resources
  • yggdrasil for network*
  • Audio Id for audio resource identification
  • All devices syncing should be in the same network
  • Toxbot for phone recordings and ..

Papad app

Papad monitor script - https://github.com/janastu/papad-monitor
papad Api Server source - https://gitlab.com/servelots/papad-api
Papad browser client source -

  • doesn't assume the source (or location) of the files (audio, images), meaning the files could be anywhere accessible by URI
  • more to be added

Use cases or user stories

From the initial surveys, we have identified the personas, and documenting the user stories below.
This will give a simplified idea to the developers and other stakeholders on what features we are looking at and how that will benefit the users.

  • As a dropout i want to record cooking recipes, so i can share with my neighbors and friends
  • As a teacher i want to teach hindi online, so i dont have to worry about gathering students due to covid
  • As a poet i want to do reading of my poems, so my friends can hear and comment back
  • As a student i want to find a teacher who can teach me dancing, so i can improve my dance skills
  • As a beautician i want to share beauty tips so i can improve my reach in the locality
  • As a cooking enthusiast i want to bookmark a particular duration in the recipe so i can specifically point at my questions / clarifications
  • As a music teacher i want to be able to mark a particular note in a song to explain cord progression to my students.
  • more to be added

Test cases

Test cases for syncthing

Pantoto syncserver shared 3 folders with sagehome device
sagehome device sharing the folders with arunpi, geeta, madhu and mani laptops and phones.

  1. papad-send send only type
  2. papad-receive receive only type
  3. papad-send-receive send and receive type

Test case 1:

created a file on sagehome with name sagehome-sending.txt and added this to /papad-receive and /papad-send-receive folders.

Expected outcome: all shared devices should be able to see sagehome-sending.txt in the /papad-receive and /papad-send-receive

comments:

bhanu passed OK

Test case 2:

Create a send only folder, checkout here on how to do it, https://docs.syncthing.net/users/foldertypes.html
and share the folder with your peers.
Add a new file to the send only type folder,

expected outcome: all shared devices should be able to see the new file

comments:

bhanu passed OK

Test case 3:

add a new file in papad-send folder

expected outcome: other devices should not be able to see the file.

comments:

bhanu passed OK

questions

Add if anybody got questions

Test cases for Papad

This space to document the test cases for implementation and integration.

Test cases 1, 2 and 3 was added before the detailed discussions with Arun

1

  • Test Case Id: papad-export-annos-to-folder
  • Assumptions: Choice of db (sqlite) with rest api
  • Test Data: papad-api.test.pantoto.org
  • Test Steps:
    1. deploy papad-api on server
    2. A script to export papad-api data to Syncthing outgoing folder (dir name: papad-annos-sync)
  • Expected Results:

    The dir papad-annos-sync will have updated data from the DB

  • Result: Pass / Fail
  • Comments: [x]

2

  • Test Case Id: papad-sync-annos-to-device
  • Assumptions: syncthing installed and configured in n devices
  • Test Data: papad-api.test.pantoto.org + some laptop
  • Test Steps:
    1. deploy multiple papad instances
    2. install and setup syncthing in all this devices
    3. Start sync papad-annos-sync dir
    4. Run script to import data from papad-annos-sync into DB
  • Expected Results:
    The dir papad-annos-sync will sync to local papad instances, and the browser should show latest audio and tags thats synced

3

  • Test Case Id: papad-sync-media-to-device
  • Assumptions: syncthing installed and configured in n devices
  • Test Data: papad-api.test.pantoto.org + some laptop
  • Test Steps:
    1. deploy multiple papad instances
    2. install and setup syncthing in all this devices
    3. Start sync of media folders
    4. ?
  • Expected Results:
    TBA

4

  • Test Case Id: recorderPi-Papad-local
  • Assumptions:
  • Test Data:
  • Test Steps:
    1. Record a audio using recorder pi
    2. Add recording (audio) to papad db
    3. tag audio
  • Expected Resuts: Everything should work just fine
  • comments if test failed:

5

  • Test Case Id: recorderPi-Papad-syncthing-to-remote
  • Assumptions: That test 4 (recorderPi-Papad-local) is passed
  • Test Data:
  • Test Steps:
    1. Check if the remote server has synced the recording (audio) files
    2. Check if remote papad has the entry to the audio file
    3. tag audio
  • Expected Results: Everything should work just fine
  • comments if test failed:

6

  • Test Case Id: papad-remote-to-local
  • Assumptions: Syncthing installed and configured to folder served by web server
  • Test Data: from files.janastu.org
  • Test Steps:
    1. Check sync path for new files
    2. Add recording (audio) to papad db, choose a channel
    3. tag audio
  • Expected Results: Everything should work just fine
  • comments if test failed:

Discussions

Dinesh, Bhanu with Arun:
suggestion for media syncing with dynamic document root

a. http://da.pantoto.org/audio-girish-radio/FOLDER01/150409-153031.MP3
<papad-service-name>/<pi-mac-id>/<path/file.ogg>

papad-service-name = da.pantoto.org
pi-mac-id = audio-girish-radio (unique id across the network)
path = FOLDER01/
file = file.ogg

Will be effectively uuid
audio-girish-radio/FOLDER01/150409-153031.MP3
later will be unique by $_$,$_$,$_$
audio-girish-radio$_$FOLDER01$_$150409-153031.MP3

Arun: having a source URI and

every pi will have a folder named by mac id
and a curated folder in wich file name seperators will be underscore (_)

pi 1 -> export csv into -> macid folder -> sync to pi 2 mac id folder -> –-
more TBA

Questions

questions re. the distributed syncthing papad app

  • What will be the target of the annotations?

  • How will papad discover the newly synced data?

    • One way is to have a discovery service which will add database entries into local papad

  • How will devices discover eachother's syncthing?

  • how to setup incoming folders on pantoto?

Tasks

[x] deploy papad-api on pantoto server (need help on this one) arun helped with this deployment
[x] Facing issues with this, the deps are not clean, debugging manually, also asking vignesh to help out
[x] solved: Pandas is not building
[x] deploy papad-api in laptops (add laptops here) and devices.
[x]Visual diagram of system process
[x] Mani working on the digital version
[] Install recording pi and papad together on a raspberry pi - Shalini is working on this
[x] Arun is working on a script to manage export import from / to db so that syncthing can sync exported json files
[] Athithya is working on the beta upgrades to annotate fragment of audio - in progress
[] Papad client bug fixes - Athithya
[x] Shalini deployed syncthing on server
[] Syncthing server configuration - Unassigned

Illustrations

Documenting all versions of illustrations and diagrams

Process diagrams

The below has been asked for some more edits

System architecture diagram

Resources and readings

  • Similar question using sqlite; suggestions in answers (https://github.com/rqlite/rqlite) - (question) https://stackoverflow.com/questions/16032825/method-to-replicate-sqlite-database-across-multiple-servers
  • Some limits of rqlite or raft protocol to keep in mind
    • For a cluster of N nodes in size to remain operational, at least (N/2)+1 nodes must be up and running, and be in contact with each other.
    • Clusters of 3, 5, 7, or 9, nodes are most practical. Clusters of those sizes can tolerate failures of 1, 2, 3, and 4 nodes respectively.
    • Clusters with a greater number of nodes start to become unweildy, due to the number of nodes that must be contacted before a database change can take place.

Test run of export / import script for syncing

29/06/2020

Arun, Athithya, Bhanu

  • Install sync scripts
  • configure syncthing path in env variable
  • Run papa-api locally
  • Run sync scripts main.py
  • Set env var SYNCTHING_PATH and run dump.py in a another terminal, this will export local db into json file deprecated
  • sync the exported json file
  • After adding device, the files sync to deafult folder path
  • Syncthing might have a diff to see if the file changed

  • Test case: Possible bug: i notify not reading incoming sync json file - bug fixed

  • There is something called syncthing inotify - not needed

Test cases passed

  1. run dump script, will dump the db contents into a json file, filename is macid.json
  2. run main.py will look for changes in macid.json file, and import the changes

Sanket's pulse audio testing ServeLots Webinar Pi

Select a repo