---
title: 'Project documentation: CatCaster - Team 01'
disqus: hackmd
---

Project documentation: CatCaster - Team 01
===

## Table of Contents

[TOC]

## Terminologie
- **Server**: this is the server where our node server process runs. We serve our website from here. This is also where we keep track of registered "clients". Our server keeps websocket connections with the clients. There is only one server.
- **Client**: is either a "screen" or a "controller".
- **Screen**: this is a web browser which makes available its view port to be used for showing the game. Note that one computer could have multiple browser windows and hence multiple "screens", which might be on the same physical display or on different displays. Multiple computers can register multiple "screens" at the same time. **We assume that different "screens" do not overlap.**
- **Controller**: this is a smartphone which is used to:
- Start a game (and the needed screen detection for multi screens)
- control a cat in the game. We make use of the gyroscope to move in a certain direction and we make use of the accelerometer to jump through a wormhole.

Algemene Taakverdeling
---
> Waar overlap wordt er gecommuniceerd en samengewerkt, bv: Client-Server & WebRTC/Controller horen sterk bij elkaar

Voor paasvakantie:
* Server (Express, Websockets,REST)
✖ Corentin, Elias
* WebRTC
✖ Quinten
* Client (Controller, Screen) ; Integratie andere onderdelen met server/client (bv. sensorinput verbinden met client/server)
✖ Quinten, Corentin, Elias
* Controller sensor/gamepad input
✖ Andreh
* Detecting Multi-screen (QR)
✖ Kevin, Arne
* Game logic
✖ Benoit, Axel

Na paasvakantie:
* Game logic
✖ Benoit, Axel
* Planet hopping tussen screens
✖ Corentin, Elias
* UI
✖ Arne, Kevin, Andreh
* Microsoft Azure onderhoud; HackMD onderhoud; debugging
✖ Quinten

## Algemene Planning/Progress
> elke week start met een teamvergadering
### WEEK 1
:::spoiler week 1
- Opening
- Team discussie & taakverdeling
- Vertrouwd geraken met de opdracht
- Git opzet
- Hackmd opzet
- First steps
:::
### WEEK 2
:::spoiler week 2
- Deelgroepen werken verder op hun taken
- discussie over architectuur/protocol (Team Server & Team WebRTC/webpages (2 uur)
- Prof. Nuyens heeft ons opgelegd om niet met socket.io te werken
:::
### WEEK 3
:::spoiler week 3
- WebRTC (Quinten) hoort eigenlijk bij client-server in de taakverdeling, moet aangepast worden
- Afspraak om tegen week 5? aparte delen proberen af te krijgen
- Afspraak om tegen week 7 een semi-werkend iets te verkrijgen
- Deelgroepen werken verder aan hun taken
- Add "Event" rubriek in hackmd (Quinten)
:::

### WEEK 4
- Deelgroepen werken verder op hun taken

### WEEK 5
:::spoiler week 5
- Doel: Aparte delen afmaken zodat integratie vanaf volgende week kan
- Mergen/Integratie Sensors & DataChannels
- QR & gamelogica werken verder aan hun ding
:::
### WEEK 6
:::spoiler week 6
- Doel: in week 6 starten met gamelogica en QR samenvoegen met de rest (lijkt haalbaar tot nu toe)
- QR team werkt verder op QR → QR algoritme werkt nagenoeg perfect, zowel localisatie (met eigen algoritme) als decoderen van verschillende QR codes door ze te isoleren (uitsnijden) en jsQR te gebruiken
- Duidelijke graphical interface voor QR codes te genereren (voor testing purposes)
- Gamelogica branch & Communicatie(=ClientConnection) zijn gemerged in
'gamelogic-comms-merge'.
- Gamelogica team werkt verder aan enkele features gamelogica
- 'Connection/Communicatie' Team integreert gamelogica met datachannels/sockets met behulp van gamelogica team
:::
### WEEK 7
:::spoiler week 7
* Doel: Iets quasi af krijgen zodat verslagen geschreven kunnen worden met een vooruitblik
* => Spel werkt op 1 scherm:
- 1. main branch
- 2. npm run build-and-start
- 3. open ip:8000/screen on desktop
- 4. scan QR (open link) with phone
- 5. 'play single screen'
* Beginselen Voronoi diagram
* Beginselen Multi screen setup
:::
### Paasvakantie
cooldown

### WEEK 8
* Work on support for multi screen
* UI for start screens
* Debugging windows
* HackMD refresh

### WEEK 9
* Doel: Multi screen support zou af moeten zijn
* Vergadering over het game objective (ipv gewoon rondwandelen)
* Vergadering om checklist te overlopen
* Debuggen
* UI

### WEEK 10
* Bugfixes
* UI
* Sound
* Scoreboard
* ...

### Extra Doelen
- [ ] een "how to set up after download from github" file
- [ ] Verklaar "HR Bullshit" in verslag

## Functional requirements
You are allowed to make changes to the requirements. The functionality should be equivalent. E.g. you could opt for screens to be identified by a combination of two or three words instead of the short id if you have a good reason for that. Or you could decide to establish the WebRTC connection between the controller and the screen only at the time that your cat actually appears on that screen. Etc...
### Broad requirements:
- [x] [M1] No GDPR statements. That means: no cookies, no user accounts, ...
- [ ] [M2] Limit the number of external libraries and keep track of their licenses. There should be an "about" item which lists all used external libraries in a correct way. Also include your own copyright license..
- [ ] Own license file
- [ ] readme.md file!
- [ ] "about" item
- [x] [M3] The game should feel smooth when played. (There are examples of using smartphones as controllers for games in a browser...)
- [ ] [M4] The whole setup should work on all modern browsers and smartphones.

### Client-server requirements (Corentin & Elias):
- [x] [CS1] Clients load the game in a web browser.
- [x] [CS2] The URL for a screen ends in `/screen/`.
- [x] [CS3] (The URL for a controller ends in `/controller/`. (You probably do not need this.))
- [x] [CS4] Each client gets an `id` generated by the server.
- [x] [CS5] The `id` is a hexadecimal string (like the hashes from git) and the server guarantees that the first 8 characters (alternatively 4) can be used as a short and unique version of the `id`.
- [x] [CS6] As soon as a client knows its `id` it will reload the page and add the `id` as a GET parameter. (The technical team thought this would help if anything goes astray, then the user can just reload the page and at least we have retained our `id`. They think that things can get implemented in such a way that connections are automatically restored and there is minimal disruption in the game.)
- [x] [CS7] If the GET parameter is already present then the server does not generate a new `id`.
- [x] [CS8] Each client establishes a websocket connection to the server.
- [x] [I4]
- [x] [CS9] The client sends its `id` over the websocket connection by a message `{ id: "..." }` as a kind of "helo" (note that this is not a typo, this spelling is actually used in the SMTP protocol).
- [x] [I2]
- [x] [CS10] The server closes connections for unknown `id`'s.
- [x] [CS11] We are currently not protecting against hijacking of `id`'s.
- [x] [CS12] If the websocket connection disconnects then the client tries to open it again. After a number of tries it gives up and displays an error message "Cannot connect to server.". This error message should be clearly displayed.
- ~~https://socket.io/docs/v4/client-options/#reconnectionattempts~~
- ~~zal niet meer id message verwachten [CS9] bij reconnection. Mss id checken bij 'connect' event?~~
- manual reconnect als we [I4] willen doen?
- [x] Define in config file
- [x] [CS13] The server purges a record of a client when the websocket has been closed/lost for longer than 10 minutes (of course this number is a **configuration variable** somewhere).
- [x] Modularize socket and id functions in client?
- [x] File names socket.ts vs client.ts class for browser?
- [x] To keep socket object private, expose socket.emit like functions?

### Screen-controller-server
- [x] [S1] A screen can be in two states: **free** or **game**.
- [x] [S2] When a screen is in free mode it displays a QR code which contains the URL of the server with `/screen/{short-id}` at the end, where the `{short-id}` is the 8 (or 4) character short id for this screen. (Note that you could type in this URL manually and it should also work...)
- [x] [S3] When a screen is in free mode and a controller scanning this QR code goes to that URL (by using the builtin functionality of the smartphone), then a menu will be presented on the smartphone:
- Play on single screen
- Detect multi-screen
- ... (other possible options)
- [x] [S4] The option "Play on single screen" will let the controller play the CatCaster game on the single screen.
- [x] [S5] The option "Detect multi-screen" is explained further and lets the controller take a picture of the multiple screens and identify its physical setup (from a specific point of view; note that the physical setup is different from different points of view!).
- [x] [S6] When the screen is in game mode then scanning the QR code will let another controller join the game.
- [x] [S7] In free mode the QR code is placed in such a way on the "screen" that it is possible to identify their physical position from a picture of multiple "screens", possibly on different physical displays. This analysis can either be done on the controller device (smartphone) or on the server.
- [x] [S8] In game mode the QR code is placed somewhere out of the way such that the game can take up most of the "screen" size.
- [x] [S9] Note that a screen will be instructed by the server to change mode (through the websocket connection): `{ mode: "free" | "CatCaster" }`. (Note that we say "CatCaster" instead of "game"...)

### Detecting the multi-screen setup (Kevin & Arne):
- [x] [D1] The controller (aka smartphone) takes a picture of all of the screens from a specific point of view. (The point of view is important, other players cannot be too far off this point of view as then the straight line connections between the screens will not be correct anymore.)
- [X] [D2] In the "Detect multi-screen" mode the controller can upload this picture to the server. (Alternatively the analysis can be done on the smartphone.)
- [x] [D3] Analysis of the picture consists of scanning for all screen QR codes in the picture.
- [x] [D4] The position of the QR code on each screen should reveal the size and position of the screen in the picture. (E.g. put the QR code horizontally and vertically centered and take note of the relative height and width w.r.t. the screen size.)
- [x] [D5] We are assuming all screens to be placed under a normal viewing angle. (So no funny perspective is going on.)
- [x] [D6] Relevant information is passed by the screen to the server on each resize of the browser window (and on each connect or reconnect of the websocket).
- [x] Resize of browser
- [x] Socket connection.
- [x] [D7] When joining screens into a multi-screen only screens in the free state will be considered.
- [x] [D8] The screens then receive a list of their neighbors and establish WebRTC data connections to all of their neighbors. (Quinten: OK +/-)
- [x] [D9] The connected screens draw lines from the centers of the screens to their neighboring screens. (You can do this by a Voronoi diagram.)
- [x] [D10] These connection lines stay visible in the background also when the CatCaster game is on.
- [x] [D11] The lines disappear when the multi-screen is deleted.
- [ ] [D12] The lines connecting the different screens should look equal in thickness on the different physical displays.
- [x] [D13] Scanning a QR code on one of the connected screens adds a menu item "Play on multi-screen" and also "Delete multi-screen".

### Technical and debug information on a screen (always visible) (Hoofdverantwoordelijken: Benoit & Axel):
- [x] [I1] A screen will have a designated area where technical information is shown.
- [x] [I2] It shows the short id of the screen. (implemented in client-server branch)
- [ ] [I3] It shows the state the screen is in ("free" or "CatCaster" (game)). **Vinden we duidelijk genoeg door UI**
- [x] [I4] It shows the connection status with the server (websocket). (Also show when the screen is trying to reconnect...)
- [x] [I5] It shows the connection status with all other screens in a multi-screen (WebRTC data). The other screens are identified by their short id.
- [x] [I6] It shows the connection status with all connected controllers (WebRTC data) to this screen. The controllers are identified by their short id.
- [ ] [I7] If the screen is part of a multi-screen then the picture which was used to detect the screens is shown in small. (This will allow a user to go stand at the correct point of view.) -> **HR Bullshit**

### CatCaster (game logic) (Benoit & Axel):
- [x] [CC1] When a controller selects "Play on single screen" or "Play on multi-screen" the CatCaster game starts on all the screens involved.
- [x] [CC2] When a controller scans a QR code of a screen in game mode it will join the running game (this could be a multi-screen game). **Zie S8**
- [x] [CC3] When a controller joins a game it establishes **WebRTC data channels to the screens** involved. We use the websocket to the server as our **signaling channel** to set up the WebRTC connection. (Quinten)
- [x] [CC4] The cat is spawned at a random planet (of a random screen in multi-screen mode), similar to when a cat died and is re-spawned (see further).
**Team Design choice:** We do a random first spawn but respawn is only random on same screen.

- [x] [CC5] The `id` of the controller should be visible somewhere on or above the cat.
- [x] [CC6] Cats should be distinguishable, e.g., by different colors.
- [x] [CC7] When a screen starts the CatCaster game it generates between one and three planets. (To make it interesting a single screen game should probably generate three planets.)
- [x] [CC8] A planet is a disk of which the center is fixed but the distribution of weight/cats will make the disk tilt in that direction.
- [x] [CC9] Cats which do not try to move in a certain direction will slide.
- [x] [CC10] Cats which slide off a disk die and re-spawn on a random planet. (The player should be able to recognize its cat. To give the player more time this could be animated, and during this re-spawning period the cat cannot slide off its new planet.)
- [x] [CC11] The planets/disks can be very simply represented by an oval.
- [x] [CC12] The portals are situated where the straight lines from the center of the different disks connect to their neighbors (Voronoi).
- [x] [CC13] Portals can be simply marked by marking an arc on the disk.
- [x] [CC14] If a cat is close enough to a portal (on the edge of the disk) then shaking the controller in the direction of the other planet will cast the cat from one planet to the other. (The shaking does not necessarily need to be in the right direction.)
- [x] [CC15] There is a lot of choice in tuning parameters, **these should all be in a separate configuration file (or at least in a designated place)**.
- [x] [CC16] A controller sends the **orientation data directly to the screen** where its cat is on using the WebRTC connection to the particular screen.
- [x] [CC17] A screen determines the positions of all the cats on that screen.
- [x] [CC18] If a cat switches screens then the old screen passes the cat onto the new screen and **informs the controller to send data to the new screen.**
- [x] [CC19] The WebRTC connection between the controller and the screens can be set into a **UDP kind of mode** by specifying `dataChannelOptions = { ordered: false, maxRetransmit: 0 }`. (Quinten)
- [x] [CC20] If necessary a timestamp/counter can be added to the packages such that the screen can ignore old packages must they arrive out of order. (The screen needs to store the previous value of the timestamp/counter to decide this. Do not compare the timestamp of the controller with the time of the screen.)
- [x] [CC21] We are assuming the number of screens and the number of controllers allows us to **connect everything to everything.**
- [x] [CC22] For **debugging mode** the CatCaster game should be playable on a single screen using the keyboard attached to the computer. This mode could e.g. by activated by pressing `x` after which a cat is spawned which can be controlled by the arrow keys and space bar.
- [x] [CC23] Construct a formula to take into account the weight of the cats and how your planet tilts. (Think of easy unit tests.)
- [ ] [CC24] Construct a formula on how the sensor data from a controller is used to move a cat on a tilted planet. (Think of easy unit tests.)
- [x] [CC25] Construct a formula to detect if a cat is close enough to a portal.

### Controller (Andreh & Quinten):
- [x] [C1] A controller shows its `id` somewhere on the screen.
- [x] [C2] Optionally the `controller` can have buttons to move the cat.
- [x] [C3] The controller shows the player the `color` of its cat (or whatever feature you use to distinguish different cats).
- [x] [C4] The controller shows the short `id` of the screen where its cat is supposed to be.

### Mitigation strategies:
In case of things not completely working as planned:
- [ ] [B1] If the interaction between controller and screen is not fast enough then the cats can be replaced by turtles...
- [ ] [B2] If detecting the physical setup of multiple screens turns out to be too hard then a (partial of full) manual procedure can be implemented. There are different options:
- [ ] [B2.a] A predetermined setup of screens (i.e. a straight line).
- [ ] [B2.b] Let the user mark the screens on the picture. (Either on the smartphone or on one of the computers which controls a screen.)
- [ ] [B3] If detecting the size of the screens is too hard then a user can manually override.
- [ ] [B4] If the Voronoi lines connecting the screens are too hard then a user can manually interact to draw and position connections.

## Planning per pair
### Server (Elias & Corentin)
:::spoiler Week 1
:::
:::spoiler Week 2
- [x] Protocol for reconnection
- [x] Routing problems
- [x] Documentation for functions, usability
- [ ] Run Linter
- [x] socket.data at authentication
- [x] Status Messages at right moment
- [x] Ten minutes/Seconds? Config file
- [x] File names
- [x] Class for client websocket
- [x] provide functions for socket use
:::
### Controller/Screen & WebRTC (Quinten & Andreh)
Quinten: focus webRTC data channel
Andreh: focus controller data
:::spoiler Week 1
* Vertrouwd geraken met opdracht
* Branchen
* RTC Data channels opzoekwerk + meerdere simultaan? (Quinten)
* Eerste code voor WebRTC communicatie (Quinten)
* Fundamenten accelerometer & orientatie (Andreh)
:::
:::spoiler Week 2
* bugfixing & move-herkenning controller/sensoren (Andreh)
* Discussie over structuur/organisatie van Socket & WebRTC communicatie (2 uur, Quinten met Elias & Corentin)
* HackMD aanvullen met resultaten van discussie (Quinten)
* Beginselen communicatieprotocol om webRTC communicatie op te zetten (Quinten)
:::
:::spoiler Week 3
* DataChannel class afwerken (Q)
* Screen & Controller class for client-side to set up datachannel between all controllers and screens & between screens (Q)
:::
:::spoiler Week 4
* Verdere implementatie Screen webpage/klasse met Elias & Corentin (Q)
:::

:::spoiler Week 5
* Merge/Integratie Controller sensor data (Andreh) with Screen & Controller classes (datachannels)(Q,Elias,Corentin):
* Sending the fitting controller data/events to screen (over data channel) = done
* message handlers for data channel messages will be implemented next week to add
:::

:::spoiler Week 6
* na merge met gamelogica werken met Corentin,Elias,Andreh en input van gamelogica team op het samenbrengen van de functionaliteiten
* Beginnen aan deel van verslag (Quinten)
:::
### QR (Kevin & Arne)
:::spoiler Week 1
* Vertrouwd geraken met de opdracht
* Uitzoeken hoe de positie van de schermen te vinden
* Branches / Pull requests:
* Werkende QR code detection voor één QR-code (geeft relatieve positie terug van de QR code)
* QR code genereren in html (client-side)
* Opstellen gedetailleerde planning
:::
:::spoiler Week 2
* Verder onderzoeken van mogelijkheden om meerdere QR codes te scannen in één foto. Onderzochte pistes:
* Er bestaat geen Javascript / Typescript library die de locatie teruggeeft van de gevonden QR codes. Een mogelijkheid is om de finderpattern locater uit een library te gebruiken om zelf al de QR codes te vinden - *wordt onderzocht*
* Verderwerken op code die reeds werd geschreven in het eerste semester. Libraries kunnen echter accurater en met minder fouten QR codes detecteren - *gesloten*
* Gebruik van OpenCV (Python) en dit te importeren in Typescript - *gesloten wegens niet toegelaten*
* Zelf een algoritme schrijven om finder patterns te detecteren en localiseren - *wordt onderzocht*
:::
:::spoiler Week 3
* Uitwerken volgens de uiteindelijk gekozen piste: zelf een algoritme schrijven om de finder patterns te detecteren o.b.v. online bronmateriaal
:::
:::spoiler Week 4
* Verdere uitwerking QR algoritme
:::
:::spoiler Week 5
* Finale uitwerking QR algoritme
* Start integratie met jsQR
:::
:::spoiler Week 6
* Integratie met jsQR voor decoderen
* Optimalisaties voor jsQR voor decoderen
* Documentatie schrijven
:::
### Game Logic (Benoit & Axel)
:::spoiler Week 1
* Vertrouwd geraken met de opdracht
* Een 3D library vinden (uiteindelijke keuze: ThreeJS)
* Experimenteren met ThreeJS
* Physics berekeningen op paier uitwerken
* Redeneren over welke klasses we nodig hebben en hun relaties
:::
:::spoiler Week 2
* Branch aanmaken voor game logic
* Planeten toevoegen
* Kat toevoegen (nog geen echte kat, gewoon een cilinder)
* Physics berekeningen implementeren in de klasses
:::
:::spoiler Week 3
* Portalen toevoegen
* Camera settings tunen
* Schaduwen toevoegen
* Portalen katten laten teleporteren
:::
:::spoiler Week 4
* Katten moeten kunnen doodgaan wanneer ze van een planeet vallen
* 3D model van kat maken (verschillende soorten/kleuren)
* Planeten verschillende kleuren geven en de portalen kleuren in de kleur van hun bestemming
* Katten moeten draaien in de richting waarin de speler wilt bewegen
:::
:::spoiler Week 5
* Physics aanpassen
* Kat modellen importeren
* Loading screen toevoegen
* Modellen voor planeten en portalen maken + toevoegen
:::
:::spoiler Week 6
* Mergen met clientConnection branch
* Problemen met resizen oplossen
* Movecats functie opdaten om beter te werken met WebRTC
:::
:::spoiler Week 7
* Bugs ten gevolge van de merge met clientConnection oplossen
* Kat laten draaien als de player beweegt
:::
Events
---
> Message protocols moeten vooraf/voorzichtig/nauwkeurig uitgedacht worden gezien deze de core van het hele systeem vormen (- Quinten)

#### Sockets
('EventName',data)
* **'sendScreens'** : data bevat array scherm IDs zodat WebRTC connections established kunnen worden (server -> Schermen,Controllers) This event tells a screen that it should switch from free mode to game mode and start a game. **Currently not sending 'only the neighbouring screens'**
* **'sendController'** : data bevat een controller ID zodat een WebRTC connections established kan worden (server -> Schermen) (controllers moeten niet met elkaar p2p verbonden worden)

('EventName',fromID,toID,data)
* **'SDP'** : data bevat SDP van fromID
* **'ICE'** : data = ice candidate van fromID

('EventName')
* '**endGame**' : This event is sent to screens and controllers (server -> Schermen,Controllers) to indicate the end of a CatCaster game. Screens should switch back to free mode.

#### DataChannel

// Van Controller naar Scherm
export type movementEvent =
{
    type: 'movement',
    data: [number,number], // x z 
    timestamp: number
};

export type jumpEvent = 
{
    type:'jump',
    data: 1,
    timestamp: number;
};

// x: horizontaal
// y: 0 als ge ni wilt springen
// z: naar u toe



// Van Scherm naar Controller

//succesfully jumped through portal
export type jumpedEvent = 
{
    type: 'jumped',
    data: string // new ScreenID to which succesfully jumped through portal
};

Overzicht werking
---
Algemeen overzicht:

set up WebRTC via sockets:

WebRTC communicatie van game-moves:

Uitgewerkt
---
* Schermen gaan naar de link `.../screen`
* Server genereert een unieke `id`
* Pagina herlaadt met de link `.../screen?id=...`.
* Na het verkrijgen van een id wordt een websocket connectie opgezet tussen client en server om de id door te geven
* De sockets krijgen een gelijkaardige functie als de emit en on functies van socket.io sockets: https://socket.io/docs/v4/emitting-events/
* `JSON.stringify()` oproepen om objecten door te sturen is niet nodig (wordt geregeld door `.emit()` en `.on()`)

    //Client
    const ws = new Socket(id, socketStatusElement);
    ws.on('greeting', (arg1) => {
        console.log(arg1) //Generel Kenobi
    });
    ws.emit('greeting', 'hello there')


    //Server: each Client object (or inheriting objects) has access to .emit and .on
    this.on('greeting', (arg1) => {
        console.log(arg1); //hello there
        this.emit('greeting', 'General Kenobi');
    });

* Tussen de schermen wordt een peer-to-peer connectie opgezet
* ...
* Een controller gaat naar `/controller/` en trekt een foto van de schermen (automatisch geüpload)
* Wordt geconnecteerd met alle schermen in free-mode en zij vormen samen één spel --> schermen gaan over naar game-mode

Message Diagram(s) (?)
---
> Read more about sequence-diagrams here: http://bramp.github.io/js-sequence-diagrams/
### WebSocket Setup

Browser->Server: HTTP-request: upgrade to websocket
Server->Browser: handshake response
Note right of Browser: WebSocket protocol
Browser-->Server: id
Note over Server: this.isConnected == true
Server-->Browser: 1
Note over Browser: socket.isConnected == true