---
# System prepended metadata

title: 20th May

---



# 20th May
### General:
- Meeting on teams:
  - Discussed what is left to be done:
    - Complete API gateway integration
    - Complete adding logging to all services
    - Improve icao24 check against database injection
    - Complete web app (to work correctly on all services)
    - Improve look and usability of web app
    - Implement unit testing for flink
    - Perform load testing 
    - Perform some stress testing
    - Do costs analysis
    - Do scalability analysis
    - Do reliability analysis
    - Create presentation
  - Al + Grik: complete logging and fixes for go services.
  - Nico + enri + Grik: work on web app.
  - Nico + enri: implement unit testing for flink.
  - Gio-R: finalize deploy pipeline
  - Al+Gio-R: test deployment on different user and perform load testing, consider stress testing (costs limitations) etc.
### Al & Grik:
  - Implemented better icao24 check in aircraft info and positions.
  - Implemented logs in aircraft list
  - Implemented logs in 30d history
  - Implemented logs in realtime history
  - Implemented logs in history calculator
  - Changed 30d history to also return the values used for the interval and the resolution.
### nico:
### Grik:
### enri:

# 21th May
### Al & Grik:
  - Solved conflicts between log changes and develop
  - Tested code after conflict solve.
  - Removed ADC env var check and instead always only use ADC as authorization
  - Merged logs and icao check changes into develop.
  - Removed all log related env vars to simplify deployment configuration.

# 23th May
### Al (with help from Gio-R):
  - Set up terraform and everything needed to deploy on personal gcloud.
  - Followed instructions to test deployment on a different cloud fro scratch.
  - Diagnosed various bugs and issues with Gio-R that were encountered along the way.
  - Managed to get the infrastructure up but there were still a couple of issues to fix.
    - Airspace positions icao check was incorrect and was refusing valid icaos.

# 24th May
### Al (with help from the others):
- Refactored aircraft positions code to use different files for each structure (client, pubsublistener, hub, timer)
- Fixed incorrect Icao24 check in aircraft positions that was rejecting valid icaos.
- Fixed a terraform issue that wasn't pulling the correct image for flink
- Fixed a terraform API gateway configuration issue that was blocking access to static resources of web app.
- Tested Web app interface with the others, found some issues and improvements to handle:
  - Show selected icao in the aircraft list dropbox
  - Don't use alert when the websocket is disconnected.
  - Show sign of life when a request fails for some reason.
  - Show some indicator that the websocket is listening for positions.
  - Fix realtime history viewer to use timestamp instead of starttime.
  - Fix 30d history viewer to access "history": field of the json response.

# 26th May
### Al (with help from Gio-R and enri):
- Tested changes to the web app.
- Tested changes in terraform api enabling
- Started learning how to use gatling to write load tests

# 27th May
### Al & Grik:
- Wrote and tried some load testing scenarios using gatling:
  - Aircraft list
  - Aircraft list + Aircraft info
  - Airspace realtime history
- Had to fight a lot with gatling syntax, also having to recompile each time to check whether it actually worked (no intellisense).
- Performed stress testing for aircraft list.
- Performed stress testing for aircraft list locally after changing:
  - Storing the list in bytes to avoid conversion on request time
  - Avoid the conversion step inside the state.Write() call (shouldn't matter)
- Found some issues with container scaling and fixed it.
- Moved max container concurrency back to 100 by looking at stress testing results.
- Gatling seems to incur in some issues with high volume of requests (400 concurrent).
- Using hey (https://github.com/rakyll/hey) for stress testing seemed to work fine on the other had.
- Further testing from Al found that aircraft list instances on cloud would still not scale under heavy load.

# 28th May
### Al:
- Local testing of gatling shows that it apparently runs out of virtual tcp ports causing errors and incorrect results.
- Using Hey instead did not incur in the problem
  - Hey local testing showed aircraft list able to sustain 500rps with 99% response time under 0.031 and average of 0.0023
- Changed aircraft info to avoid converting string to []byte in request handler. Store the list directly as byte array.
- Redeployed and tested on the deployed aircraft info using hey.
- Discussed with Gio-R about ability to clean up some left over things from deployment.
- Discussed with everyone about doing and keeping up a production deployment on one of our accounts and use that account only for the deployment hoping it would last.
- Tried registering for the 300 dollars free trial but it requires credit card (prepaid not ok) and paypal option failed with unsepcified error. Gio-R also had the same issue.
- Looked into perfKit to understand whether it was something useful to us, but it seems not at this stage since we already locked on the cloud provider.
- Identified an issue when trying to update a single service in the deployment (Gio-R found solution and fixed it).
- Continued with aircraft list and info stress tests to see whether they would start scaling:
  - Info service does seem to start scaling under heavy load (14 instances of 15 max at certain points), but not reliably.
  - List went to a max of 2 instances. It might be a problem of cpu utilization, very unreliably.
  - The CPU utilization is low even under heavy load so something else is the bottleneck?
  - Further local testing can see if bottleneck is app or something else.
  - Alternative is that the second service is failing on startup but there aren't logs suggesting it.
  - Testing without rate limiting requests to 500rpm got better response time results which makes no sense.
    - 100 threads, 40 0000 requests:
      - 10 seconds, 3670rps, 100% of requests < 0.779s response time, average of 0.026s.
  - For some reason rate limiting requests to 1 per second per thread is taking more time?
  - Perhaps because all the 500 requests happen at the exact same time so some are delayed, but still it doesn't make sense.
  - Maybe the 500 concurrent were triggering the scaling which slowed overall performance due to instance creation in the background.
- Talked with enri about cost estimation to model number of requests and accurately quantify resource usage for cloud run apps.
  - Most important factors that influence the cost are total number of requests and the CPU time for single request.
  - A service under constant 200rps 24/7 at 20ms per request would cost ~190$ a month. Those are very pessimistic numbers.
    - Both 200rps and 20ms are higher than the model and cpu time actual values.
  - Kubernetes on the other hand managed to cost more money today doing nothing than the could run services being stress tested.
- Tried stress testing realtime history service as well but found that the backend cluster had actually failed at some point:
  - No data was generated on the db for today
  - Looking at the log showed errors about IP_SPACE_EXHAUSTED. 
    - Gio-R had already encountered this before but measures should have been in place to avoid it so we don't know what went wrong yet.
    - Spent time trying to destroy the deployment but it failed and had to be cleaned up manually.

# 29th May
### Al:
- Looked into high kubernetes costs from the day before 
- Found potential ways to reduce the costs a bit but ended up dicussing about hosting on premise.
- Gio-R created a docker-compose for flink and history-calculator to run locally
- Discussed with Gio-R about using single image registry and implementing automatic image publishing on github push.
- Al tested running back end on spare machine.
- Encountered and fixed various issues to start it.
- Finally started it crashed with socket timeout.
- Diagnosed the issue in various ways until we found that OpenSky API was responding after 60 seconds or giving bad gateway.
- Nico changed flink code to survive exceptions when calling OpenSky.
- Tested and iterated on the changes with Nico until the backend was able to survive without opensky.
- Liaisoned with Enri and Nico for their cost analysis regarding Firestore data access patterns and other values.
- Deployed front facing services on cloud for stress and load testing.
- Tried helping Grik to get Gatling tests to work.
- Grik managed to set up websocket load testing but there were still issues with the ws not closing, so the test never ending.


# 30th May
### General:
- Meeting on teams:
  - Discussed what is left to do:
    - Complete loadtesting
    - Complete costs analysis 
    - complete stress testing
    - Deploy app on main
    - Start producing the presentations.
  - Discussed what to bring to presentation and how to split things to talk about
  - Focus on things that are important for the course, reliability, scalability, security, costs etc
    - Nico: 
      - Presentation of project objectives and architecture (requirements, expected load, architecture)
      - Explain limitations of time and budget
      - give overview of what every person did 
      - Start explaining the backend
      - Why flink
      - Choices during flink implementation
      - scalability of backend
      - reliability of backend (fault resistent)
      - Costs of backedn (flink is chunky)
    - Al:
      - Front facing services (using microservices rather than using monolithic application)
      - Objectives of front facing services
      - Language decision (use scala or something else?) (gin)
      - Distroless images
      - Aircraft info service
      - Database embedded and considered alternatives
      - Icao 24 check for database
      - Immutability
      - Positions service
      - Considered alternatives to the architecture and why we choose this.
      - Cloud Pub Sub and why
      - Limitations of PB
      - How wb are handled 
      - listener for each icao
      - icao check 
      - Subscirptions uuid
      - Handle gracefully error on ws read, ws write, pb err, pb timeout.
      - Usage of readWrite locks to allow high concurrency while avoiding inconsistent state.
      - scalability of positions 
      - reliability of positions
      - Logging
      - Load testing
      - stress testing
    - Grik:
      - The remaining services have to do with history and the aircraft list.
      - Big query to compute history. Very good on paper. Low complexity of code, db does all the work.
      - Testing big query bad perf
      
      (Al): Vabbè sono le 3:00 di notte, decidete poi voi se avete voglia o tempo di finire qui.
       
### Al, Grik:
- Completed implementation of loadtesting:
  - 2 new users per second using the app for 3 minutes, using all the available services. 250 concurrent active, 600 users in 5 minutes
  - Performed testing, found 95% optimal results, mean result below 100ms
  - Very few outlier errors (9 ws erros in 20k), some long resp times
### Al
- Continued monitoring local backend deployment for issues using logs.
  - Identified issue with 5m data having timestamps that weren't aligned to 5 minutes (clock drift of flink probably)
    - This created issues with the history calculator that would not be able to correctly pinpoint the documents later for subtraction operations.
    - Notified Nico who immediately managed to find a solution and fix it (subtract with previous timestamp to see if difference if 5m, if not approximate based on whether the difference is higher or lower than 5)
  - Identified potential issue with airplanes distance values (too high based on personal approximations)
  - Waited 02 AM to observe whether daily value and db cleanup worked correctly at the end of the day.
  - Identified an issue with database cleanup, old values were not deleted correctly.
    - Found and fixed the bug that was using time.Time vals instead of string dates in database query.

# 31th May
### Al:
- Consulted with Nico about the incorrect distance calculation.
- Nico found the issue and implemented a fix. 
- Tested the fix and verified that the new values were realistic.
- Deployed services on cloud for stress testing.
### Al & Grik:
- Identified all the endpoints to test (can't test ws due to most stress test tools not supporting them)
- Researched best practices and tools for stress testing
- Started performing stress testing using Hey on cloud deployed services:
  - 100 threads, 60s, from cable connected 8 core 16 threads machine.
- Results for most services where around 4000rps and with 3-12 replicated instances on cloud side. 
- Recorded result data for each endpoint using the same test parameters.
- Monitoring resource usage on test source machine we found that the uplink bandwidth was being a bottleneck (10MBps)
- Increasing the number of threads would only increase the latency due to more requests waiting for bandwidth.
- enpoints/positions/url showed higher latency and around 100 504 response errors. Highest latency was 15s.
  - Looking into the issue it appears that the gateway was not able to find the service instance for those cases for some reason.
  - This was highly repetable for only that endpoint, all other endpoint never showed this problem even with multiple tests.
  - Bandwidth consumption was very low for /url endpoint, and the code serving it is extremely simple. 
  - Tested the endpoint locally with impeccable performance and 0 errors.
  - We looked into potential configuration differences of the endpoint but found nothing.
  - We have no idea what the issue might be.
- Surprisingly monthly history performed exceptionally well considering it requires access to the db for each request.
- We looked into cloud based solutions to workaround the bandwidth limitation (all other group members have laptops and no cabled connection).
- Basically all of them require credit card registration and cost money or have too low loads offering for free tier.
- Gatling has enterprise trial for 2 weeks. Still have to see if it asks for credit card.
- Found that with Hey, if one request was stalling, the thread would stall and overall rps would drop.
### Al:
- Performed stress test with Hey at the same time as Gio-R to see whether the services would be able to hold (approx) double the load.
  - Both bandwidth were saturated and and the rps remained as high as tests from a single machine. 
  - So services seem to be able to scale much more than bandwidth.
  - Later asked another friend with 20MB in up and we managed to get 12000 cumulative requests per second.
- Looked for hey alternatives. Found wrk cited in various places by other developers.
  - wrk needs to be compiled and is linux only.
  - Used windows subsistem for linux to compile it (took 30m)
  - With wrk you can define the number of connection each thread handles, if one connection is stalling the thread continues to work on all the other ones.
  - Ran wrk on aircraft list with slightly better results than hey, but still bandwidth limited.
  - Ran wrk on enpoints/position/url. Same errors but overall better rps (800->2800)
- Consulted with Gio-R about finally pushing everything onto main.
- Did the PR and found that apparently logging also needs to have parameters validated.
- Ignored the problem for now to see if everything else worked correctly and it did.
- 

# 1st June
### Al:
- Tried creating a docker-compose for front facing services to more easily test them locally.
  - Docker compose was getting stuck on attaching to the running containers, but they are distroless s they don't have a shell.
  - After some struggling finally gave up and started all the services manually.
- Used wrk to test all the services locally:
  - info, realtime history and endpoints/url were exceptional being around 100 000rps
  - list was a bit slower at 65 000 rps, likely due to the bigger message to send.
  - web UI was at 6k rps, we hypothesize due to the size of the data and the fact that it is serving files from the fs.
  - monthly history was predictably the slowest at 2k rps.
    - This is expected since it needs to query the db with each request, read the db data and re elaborate it for the user.
- Spend time trying to get gatling to work testing the websockets.
  - When it finally worked it registered that no messages were received
  - Dubiously assumed it to be ok since manual testing shows that it actually receives the messages but only after a few seconds.
  - Gatling is sadly unable to work at highloads (500rps) even on a local environment, the connections are simply refused outright.
  - Tested up to 200rps and it worked fine.
  - We don't have any tool to test this well. 
  - Only solution would be to develop our own tool and hope we don't mess up.
  - Time budget will not allow it.
  - As it is we are unable to stress test the websocket endpoint past 200rps. We have no reason to suspect issues but testing would be much better.
  
# 2nd June 
### Al:
- Started deployment of app from github to start producing history values and watch for any issues.
- Used image of architecture from Gio-R on excalidraw to make high res image of architecture, (small refinements).
- Added image to repo readme.
- Changed number of minimal service instance from 0 to 1 to provide a more responsive experience to new users.
## Al & Grik:
- Started working together on presentation to split what to talk about each.
- Already started working on the slides. 

# 3rd June
### General:
- Meeting on teams:
  - Shared planned topics for each person and discussed how to optimize the presentation to avoid splitting talking too much.
  - Did a small preview for each person for the presentation. 
  - Discussed which topics to leave as additional slides, what to add or remove etc.
- Gio-R identified an issue with 30d-history 1d-bucked values recording hourly vals instead of daily vals.
- Al found and fixed the bug in the code.
## Al & Grik:
- Continued work on slides and presentation.



# xxth June
### General:
### Al:
### Gio-R:
### nico:
### Grik:
### enri: