HackMDHolo: Performance Testing With Network Variables
tags: holo-notes holo-notes-testing
Note: This doc does not describe the actual test cases for the performance test. The Holochain team should be well suited for test the initial baseline, and we(Holo) can base our off those test, since our value is based off their baseline.
This Doc is a suggestion of how we can split our Performance tests to get a better understanding of the data from our performance tests and understand the diffrent variables that need to be controled while performing those tests.
To understand the performance of the holo infrastructure, we need to split it up into multiple sections and gain baselines at all the different levels.
We will have to talk about network performance and system performance
Most of Holo's performance testing is going to end at the network layers.
Mary Camacho Can this be clarified? Are we saying that if we understand the baseline of EACH network layer (ie. Internet performance, Holochain performance and Holo's Zero-Tier network) then everything else is actually testing our software?
end at the network layers
A Hosted user's performance is based on three layers of networking.
If we are able to control these network layers we can get good system performance stats.
Bellow you can see how our stack can be visualized.
Notes to better understand this diagram
- The Red Boxes are Local Network i.e. (Local internet provider)
- The Green Box is
ZeroTier(Should we call this something else - instead of the name of our vendor? We conceivably could change that out.) - The Blue Cloud is
Holochain Networkingi.e. uses the kitsune lib - The White boxes are services/application that we use in the entire infrastructure.
Holochain Baseline (HC-2-HC):
As seen in the diagram this is a holochain node to holochain node interaction. And the only variable is the individual local network.
Suggestion:
This is the test that the holochain team sets up for themselves and gives us a good baseline of what they think the speed of their network should look like given variable of the individual local network
Mary CamachoWill the networking changes Holochain does to switch the quic protocol to web-rtc change the broadstroke performance of Holochain networking - is that something DavidB could tell us?
Holo Baseline:
As you can see in the diagram, Holo baselining comes in two variations
- Holoport to holoport (HP-2-HP)
- holoport to holochain (HP-2-HC)
Holoport to Holoport (HP-2-HP):
As seen in the diagram this is the interaction between holoport runing a holochain node to a different holoport running a holochain node. Unlike the holochain baselines here we have two diffrent local networks (variables) on each side of the p2p.
Because of the above reason it would be useful to break the test into two phases.
Phase 1:
In Phase 1 we get a baseline of holochain interactions between HP-2-HP.
This would be getting a baseline of a self-hosted happ.
At this phase our variable is just one local network on each side. And we can follow the same set of baselines that the holochain team sets in their baseline.
Mary CamachoDo we have a way of installing self-hosted hApps other than the holo hApps?
Suggestions:
We break the test group based on our traunches and get a baseline on each traunches that we create
Possible Learnings:
- Delta of HC2HC vs Phase1
- If this baseline is different than the holochain baseline it would be because of the Zerotier.
Mary Camacho 1) The assumption is that Holochain baseline tests use the same hApp as we do. 2) How are we dealing with physical distance?
- This will tell us the slow down that zerotier causes
AlastairThe baseline may also be different due to hardware differences. In particular the known issue that HPs use HDDs which Sqlite3 does not fully support.
Phase 2:
This is the Hosted user interactions. Here we need to control two local network as a variable on each side.
Variable 1 = Hosted User network speed
Variable 2 = Holoport Local network speed
We can get as granular as we need but the lease would be a simple matrix below
| Variable 1 | Variable 2 | |
|---|---|---|
| High | x | x |
| Low | x | x |
Possible Learnings:
- The difference between this and the Phase 1 tests are going to tell us how well our infrastructure performs.
- After knowing HC-2-HC and Phase 1, this will give a better understanding of how performant our infrastructure is compared to running vanila holochain.
Holoport to holochain (HP-2-HC):
We will need to get a baseline of this interaction if we see a difference between HP-2-HP Phase-1 baseline vs HC-2-HC baseline.
If they are the same it would mean that our holoports have no performance degradation when acting like a holochain node.
If they are not the same then by having this baseline we can get a better understanding of the performance issues of running holochain on a holoport is having.
