# Reserve Commitment Sampler
## development process

1) Sanity checking the prototype

3) Measuring CPU usage while running the prototype on a single CPU typical node – experiment and optimise it from here to reduce the current 40-minute time as much as possible. As other processes must run concurrently alongside the RC calculation
4) Measure and test the finalised sampler with the reduced time for performance guarantees
5) Post-optimisation snapshot testing - how much of the reserve sample is actually common between nodes in the neighbourhood (take a snapshot and see how many elements match)

Ultimately, the shorter it takes the less variance is introduced by live-sync. A longer calculation period means more chunks are in flux introducing more variance in the sample, and therefore the likelihood of being slashed when acting honestly (and increasing the need for a potential temporal component).

## Specifications / Methodology

---

- ### We will only make a RC when we know we are playing in a lottery round:
And ‘when you play’ is determined when the nonces from the last round are known, i.e. the truth selection anchor with the current randomness source, which reveals once per round.

The next neighbourhood selection anchor comes from the previous round’s truth selection anchor.

The sampler needs to iterate through the reserve in a matter of minutes (2 instead of 40 currently), with the initial node storage at 16Gb we are on the limit of what is possible as it already uses 20-40% of the average SSD bandwidth.

12.5 minutes round durations won’t be long enough to iterate and calculate RCs within this window and nodes will be forced to skip rounds if the initial storage size were to increase and RC not calculated in time. There is also the option to adapt when the RC is calculated - only when that node’s neighbourhood is playing and not create an RC every round regardless.

RC generation every round (irregardless of your neighbourhood playing or not) is a high-cost operation, increasing electricity consumption of the node and therefore decreasing the profitability of the node operator (effect is minuscule). If you have a 1CPU setup then this extra work, of reading the disc and doing the hashes and system operations, would make nodes less responsive and cause waves of unreliability. But if only 1 locality is selected and forms an RC then it doesn’t affect 99% of the network.

---

- ### Round length: We will adjust the round length depending on how efficient we can make the reserve commitment sampling on a single CPU cluster node (as the minimum hardware requirements). How much can we reduce the 40-minute sampling time?
The Claim period length can be used for calculating the RC; because by the claim period no more reveals can be done, therefore the new randomness is already known.

The claim period plus the commitment period (in block numbers) must be long enough for the RC calculation to occur.

Then begins the reveal period.

---
- ### Sample size: We iterate through all of the chunks with a valid postage stamp and belong to our storage depth proximity.
This should essentially be the same as the contents of the pull-sync index.

The main aim of this iteration and the formation of the RC is that you can only produce the same RC as your neighbourhood if you have the content ready when preparing to calculate your RC.

Metrics and Parameters: Slashing chances being >2% when behaving honestly is unacceptable. Therefore require 0.1% sample deviation coming from live-sync at most at any moment, with a sample size of 16.

----
### Implementation notes

- #### Filter:

The period of time for the RC should correspond to half the time of a lottery round (storage being 2 to the 22nd number of chunks); this may be restrictive and later need a filter function.

If the RC takes n(seconds) and the lottery round is > K, then take this filter function that reduces K via a smaller random sample. Thus solving the consistent bottleneck of lottery round length if the sampler cannot be dramatically reduced in time.

- #### iterator:
pull-sync provides iterators, subscribe to them as other peers do via historical sync etc etc. This should dictate that all nodes start syncing at the same time.

Pretend you are a new node and join and subscribe to the bins. It feeds a channel, workers pick up on this and feed it to the buffer to keep the (1st? / smallest? / what?) chunks.

- #### 3 step process
1) gather and subscribe - push hashes into the process & pull-sync.
2) retrieval - retrieve data and inputs to the worker channel - retrieves the data from the db/sharkey or from the filter.
Then calculate the modified hash and feed this into the modified channel.
4) selection process - keeps the buffer of the sample size data of 'n' smallest hashes

---

## miscellaneous additional notes & RC Sampler discussion for future versioning

1) #### Dan’s idea (phase 3 inclusion-proofs alternative)
From the RC can we not use chunk density as a schelling point which reduces the need for inclusion-proofs?
The amount of content in the network is 2 on the power of 12 times 8-16Gb. We don’t know the actual number 0f 8-16, but phase 4 gives inclusion proofs and therefore an accurate reading of what is in the sample.
And from the sample density we can estimate the actual number of chunks that is somewhere between 2-4 million etc.

When we commit, we can get a non-inclusion proofed last sample, and if you want to stay on the same schelling point as the others you are motivated to report this correctly. If you don’t have the orderSample doesn’t matter as your RC hash differs anyway, and if you misreport you don’t gain anything anyway.

Reserve size estimate – when you use the reveal function you have to specify the last sample element transformed hash as well as the schelling point. So, if you have the canonical sample of the nhood and you want to stay on the schelling point as others then you’ll correctly name the last sample element transformed hash (motivated to stay on that point). So, inclusion proofs can be less required or substituted.

Maybe need to inclusion-proof not just the last sample element, but a random deterministic element from an arbitrary location in the RC. That’s selected by the truth selection anchor, so reveals have to be done first before you have truth that can point at a random member of the RC.

2) #### Age limit – no time limits / timestamps
Timestamps would cause problems by introducing discrepancies via injecting backdated content and it doesn’t solve the issue but change the behaviour and cause requirements for pull-sync.
Timestamps do not matter currently but if we have minimum age then they become influential. This can create an attack surface whereby you can create discrepancies by injecting backdated timestamps and have it propagated by nodes.

A node would then have to conditionalize if a valid stamp is still accepted every time. Timestamps need to be signed, so to reupload content you would need to resign with a new correct stamp – this could potentially cause content indexing issues?

This minimum age requirement means that your lag in syncing cannot be more than the set time/age requirement and therefore that node would not be able to participate or else include those chunks in it’s reserve. Therefore, a more well-defined definition of the reserve is needed that all nodes can adhere to with minimum discrepancies in the RC.
Vik’s issues:
- It might leave us with a small sample size, or honest node’s will be slashed too often without
- You would be motivated to not sync chunks that would not match the RC
- Creates a strong dependency of RC creation and the lottery round. When in reality there are many more lottery rounds than RCs, you can also enforce nodes to create RCs even when not playing (an extra security feature).
- Creation of the RC that you play with should be independent of the lottery round itself, this way there’s no constraint on the interval of the lottery round to generate the RC
- Timestamps backdating: can only use recently stamped chunks if we have an age requirement, which is not ideal.

3) #### Periodicity how it works when making the reserve sample

Let’s say depth is 10 so 1000 nhoods, 1 nhood is on avg selected every 1000 lottery rounds. This means if we only make the RC when our nhood is playing it 1000x less work than doing the RC sampling when there’s a new round.

Therefore, the larger the network becomes the less energy efficient participating in the storage incentives becomes. If there is depth 20 and a million nhoods, then you make this RC a million times more work by the time you play once.

Energy efficiency is related to node operator profitability, so if costs go up then pay-outs would need to increase – so if we can get away with doing RC when your nhood is actually playing then it’s much better off than if you have to this periodically.

Before you commit you are supposed to have the RC ready and calculated already, the round length cannot be shorter than making a RC, even if prepped before you would still need another RC for any other round. This would prevent you from say preparing a metadata sample of RC weeks before, dropping all data, but playing and providing the old RC sample; the RC needs to change every round to force you to keep the content but produce it whenever.

Yes to all of the above points but it feels like a temporal component is missing (timestamps etc). As the nodes need to sync around which precise chunks would be in the reserve at any given time and there are chunks constantly going in and out. All chunks have stamps (many to one, but right now they’re one to one), can use them to define in time what the consensus of the sample should be.