---
# System prepended metadata

title: 512 P3 - ZooKeeper

---

# 512 P3 - ZooKeeper
## Tom's Idea
1. Additional DistProcess after having a master will be workers.
    - ZKServer after failing to create master Node, will create a sequential worker znode `/dist42/workers/worker-00000X`. 

2. Master should keep track of workers on the platform.
    - Master has a watch on `/dist42/workers`, on new node, will setup a new watch and record new worker available.
    - Try to assign a job to new worker. (put a znode under `/dist42/workers/worker-0000X/task`?)

3. Master should not do computation but delegate to (idle) workers
    - On job request (update in `/dist42/tasks`), put watch again, pick idle worker to do the job:
        - If idle worker: remove the znode in `/dist42/tasks`, put new znode in `/dist42/workers/worker-0000X/task`, result should be put in `/dist42/workers/worker-0000X/task/result` on termination by the worker (so master should put a watch there).
        - If no idle worker: wait for some callback (return).
    - Master should cleanup after sending back result to client.
        - Remove task under the worker.

4. Worker can only execute one task at a time, then idle again. Keep track of who is idle / busy for Master to assign tasks.
    - Can only have one task znode under each `/dist42/workers/worker-0000X`.
    - In order to keep track of idle workers:
        - Can store internal data in the Master (with a `PriorityQueue` like Alvin suggested) and `setData` on the worker. 

5. If no idle workers, wait for worker to become free or new worker joins
    - Try to assign a task when new worker joins or when worker finishes task.

</br>
</br>
</br>
</br>
</br>


# Alvin's Idea
1. Additional DistProcess after having a master will be workers.
    - Sync. `create() -e -s` `/dist42/workers/worker-XX` with default "IDLE" message
    - Stores the worker path
    - Async. `getData()` `/dist42/workers/worker-XX`
        - `callback`
            - If data is not "IDLE", do work
        - `watch`: `NodeDataChanged `
            - Sync. `getData()` `/dist42/workers/worker-XX` with no watcher and same callback as above

2. Master should keep track of workers on the platform.
    - Master (initially) calls `getChildren()` on `/dist42/workers`
    - `callback`
        - Add each unique child to `HashSet<Workers>`
        - If child is new node
            - Add each child to `PriorityQueue<IdleWorkers>`
            - For each tasks in `PriorityQueue<Tasks>`
                - Assign tasks *(see next section)*
    - `watch`: `NodeChildrenChanged`
        - Repeat `getChildren()`
        
3. Master should not do computation but delegate to (idle) workers
    - Master (intially) calls `getChildren()` on `/dist42/tasks` 
    - `callback`: 
        - Check if the tasks is new
        - If `List<idleWorkers>` is not empty:
            - Pop `PriorityQueue<IdleWorkers>`
            - Asynchronously `setData()` `/dist42/workers/worker-XX` to task path
            - Asynchronously `getData()` `/dist42/workers/worker-XX` 
                -  `callback`
                    - If data is "IDLE", then repeat callback above
                - `watch`: `NodeDataChanged`
                    - Push `PriorityQueue<IdleWorkers>`
                    - Repeat callback above
    - `watch`: `NodeChildrenChanged`
        - Repeat `getChildren()`
   
4. Worker can only execute one task at a time, then idle again. Keep track of who is idle / busy for Master to assign tasks.
    - On computation completion:
        - Async. `create() -e` `/dist42/tasks/task-XX/result`
        - Async `removeWatch()` `/dist42/workers/worker-XX` to prevent unconsumed watch
        - Async. `setData()` `/dist42/workers/worker-XX` to "IDLE"
        - Async. `getData()` `/dist42/workers/worker-XX`
            - Repeat callback and watch 

5. If no idle workers, wait for worker to become free or new worker joins
    - If `PriorityQueue<idleWorkers>` is empty 
        - Adds tasks to `PriorityQueue<Tasks>` only it is new