Background
----------

The Timely Vote Credits (TVC) feature is activated in two stages by two separate features:

`7axKe5BTYBDD87ftzWbk5DfzWMGyRvqmWTduuo22Yaqy`: enables in-place modification of vote accounts to expand their size and make available room for recording vote latencies

`2oXpeh141pPZCTCFHBsvCwG2BtaHZZAtrVhwaxSy6brS`: enables the use of the space made available in the prior change to allow validators to track the latency of votes and then assign vote credits to votes based on this latency information

These features are activated in sequence and in the order presented above. The first feature has already been activated on testnet, and demonstrated that it causes no cluster disruption and has the expected and intended effect of upgrading vote accounts.

The second feature has not been activated on testnet.

A new small testnet has been created to test the enablement of both of these features, as a way to demonstrate the effects of TVC in practice. There are three aspects to this:

1. Demonstrating that enablement of the TVC features does not "break" the cluster
2. Demonstrating that enablement of the TVC features results in the expected credits accounting changes
3. Demonstrating that geographically diverse validators are not intrinsically disadvantaged in credits earnings as a result of TVC

Methodology
-----------

A separate test cluster was spun up to test TVC.

The cluster consisted of 7 nodes:

| Vote Account | Operator | Location | Stake Weight | Notes |
|--------------|----------|----------|--------------|-------|
|`FGd5g9uYs9MiGvJtbJeoL1QMAB3yzECpRe8J5jSQkn1h`|Anza|Frankfurt Germany|19%|stock voting|
|`CLLH3k44keLPYrfzF49C2HhcwLQVWXSXtcxyXh2zd27G`|Anza|Frankfurt Germany|19%|stock voting|
|`459KZryRbx5BburUgrrbBg1r9DxBfe5mQocP2oRHtXUA`|Anza|Frankfurt Germany|19%|stock voting|
|`F176raFT8mtKnBi13ErBqTMMJYXQpugzQbReTXjouNLm`|Anza|Frankfurt Germany|19%|stock voting|taken offline before TVC testing began|
|`meowZQvyYd6wCHhD5QfAPJmn21nuGeg7z9CJdTkWP9f`|Pumpkin's Pool|Tokyo Japan|8%|stock voting|
|`zanveSXNMKjPZtBwnBJRPPNESkwc6hPC7XtcFugp6mA`|Shinobi Systems|Santiago Chile|9%|voting mods at various times|
|`ash2VSKkKrNZmP8wP9pvX3RTejpCJZ93f3pgkMqWmTu`|Ashwin (Anza)|Chicago USA|8%|consensus-only voting|

Of these nodes, there were four Anza operated nodes, three of which just acted like normal nodes with no modifications, and one of which was a normal node but was taken offline before the first TVC feature was activated. One node was run by Pumpkin's Pool out of Tokyo, running unmodified code. One node was run by Shinobi Systems out of Santiago, in various epochs running different modifications to test their effects. And one node was run by Ashwin at Anza, running a modification that caused the node to only vote for already-confirmed blocks (representing a "worst case" validator voting mod behavior).

The cluster was run with very short epochs of 8192 slots in order to allow a faster epoch turnover for faster testing.

In addition, the cluster was put into high skip/fork behavior by the consistent delinquency of 19% of its stake, as well as some of the Anza nodes enabling testing code that attempted to cause forks by delivering block data late. Thus the cluster acted as a highly skippy/forky cluster which magnifies the impact of both stock code "fork avoidance" methods and validator voting mods.

What Was Tested
---------------

1. Enablement of the first feature

a. Test that a vote account with insufficient lamports to support self-upgrade continued to function normally
- Shinobi Systems set up its vote account to be not upgrade-capable by reducing its lamports to an amount that would not support the upgrade, and setting its validator commission to 0% so that it would not receive lamports into its vote account that would alter this fact. Thus this vote account could not auto-upgrade.

b. Test that a vote account with sufficient lamports to support self-upgrade did so and continued to function normally
- All other vote accounts were in this category

c. Test that vote credits continued to function normally

d. Test that a vote account with mods intended to capture more vote credits even if it produces higher vote latency (HIGH-LAG MODS), earns significantly more credits than the rest of the cluster, before the latency-based credits feature is activated

2. Enablement of the second feature

a. Test that a vote account that was not upgraded continued to function normally except that it received the minimum of 1 credit per vote instead of more credits based on latency

b. Test that vote accounts which were upgraded continued to function normally and earned credits commensurate with their vote latencies.

3. Post-enablement testing

a. Test that a vote account which was not upgraded would self-upgrade when sufficient lamports were added to it, and that it would then earn credits corresponding to vote latency

b. Test that unmodded validators at "distant" locations from the stake majority did not earn significantly different vote credits than validators in "close" locations.

c. Test that a vote account that voted only for confirmed blocks received lower credits due to the inherent latency of this voting method.

d. Test that a vote account from (1d) now earns significantly fewer vote credits due to the increased vote latency of its modifications.

e. Test that a vote account that earns additional credits through mods that add minimal vote latency (LOW-LAG MODS) do earn more credits still (the expected behavior) but that the additional earnings are much reduced from the (1d) case.

Results
-------

All tests passed and demonstrated the expected effects. In particular:

- The cluster operated as normal and survived the high skip rate/high delinquency/high forking behavior caused by the stress testing behavior of the Anza nodes.

- The distant nodes in Tokyo and Santiago did not earn significantly fewer credits in any situation (the Santiago validator running without voting mods did earn approximately 0.01% fewer credits after TVC was enabled, but this reduction is exceedingly small; the Tokyo validator had no difference at all)

- The validator which voted only after consensus did earn significantly more credits before TVC and significantly fewer credits after TVC.

- The validator which was modded to optimize vote credits through higher vote latency fork avoidance (HIGH-LAG MODS) did earn significantly more credits before TVC and significantly fewer credits after TVC.

- This validator which was then modded to optimize vote credits through no additional vote latency (LOW-LAG MODS) earned a small amount of additional credits after TVC (which was expected)

Specifics
---------

Here are the credits earnings in epoch 63, before any TVC features were activated:

|Voter|Credits|Notes|
|-----|-------|-----|
|`459KZ`|8192||
|`CLLH3`|8192||
|`F176r`|8192||
|`FGd5g`|8192||
|`ash2V`|2874|partial epoch for this validator|
|`meowZ`|8191||
|`zanve`|8192||

All validators were operating on stock code and earned essentially identical credits, except `ash2V` which had started part-way through the epoch.

Here are the credits earnings in epoch 64, after the first TVC feature was activated:

|Voter|Credits|Notes|
|-----|-------|-----|
|`459KZ`|7211||
|`CLLH3`|7211||
|`F176r`|7211||
|`FGd5g`|7211||
|`ash2V`|7028||
|`meowZ`|7211||
|`zanve`|7532||

`ash2V` was fully online and `zanve` had enabled voting mods so those are the only ones with credits differences.

From epoch 64 - 79, there were no changes in how any vote account or the cluster operated, except that in epoch 77, the `F176r` validator was taken off-line. Here are average credits earned by each validator in epochs 64 - 79:

|Voter|Avg Credits|Notes|
|-----|-------|-----|
|`459KZ`|5505.562||
|`CLLH3`|5505.562||
|`F176r`|4825.000|decreased credits due to delinquency|
|`FGd5g`|5505.562||
|`ash2V`|5882.750|increased credits due to consensus-only vote mods|
|`meowZ`|5505.625||
|`zanve`|6871.250|vote optimizing mods (HIGH-LAG MODS) highly effective due to cluster skip/fork rate|

In epoch 80, the second TVC feature was activated. At this time, the `zanve` validator upgraded part-way through the epoch, only received 1 credit per vote for some of its votes. This is reflected in credit earnings this epoch:

|Voter|Credits|Notes|
|-----|-------|-----|
|`459KZ`|8316||
|`CLLH3`|8316||
|`F176r`|0|offline|
|`FGd5g`|8316||
|`ash2V`|4715|immediately reduced credits due to TVC|
|`meowZ`|8316|same credits as Anza despite being in Tokyo|
|`zanve`|6028|immediately reduced credits (HIGH-LAG MODS) due to TVC + non-upgraded vote account|

In epochs 81 and 82, the `zanve` validator ran with a fully upgraded vote account and at "max lag for credits":

|Voter|Credits|Notes|
|-----|-------|-----|
|`459KZ`|8384||
|`CLLH3`|8384||
|`F176r`|0|offline|
|`FGd5g`|8384||
|`ash2V`|4781|highly reduced credits due to TVC|
|`meowZ`|8384||
|`zanve`|6016|highly reduced credits (HIGH-LAG MODS) due to TVC|

|Voter|Credits|Notes|
|-----|-------|-----|
|`459KZ`|8176||
|`CLLH3`|8176||
|`F176r`|0|offline|
|`FGd5g`|8176||
|`ash2V`|4611|highly reduced credits due to TVC|
|`meowZ`|8168||
|`zanve`|5916|highly reduced credits (HIGH-LAG MODS) due to TVC|

In epoch 83, the `zanve` validator altered its voting mods to reduce their lag. This had the effect of increasing overall cluster credits considerably, because prior to this, the total amount of non-delinquent stake that was voting with heavy lag (`zanve` and `ash2V`) left the cluster at less than 66% of stake voting without induced lag, and this caused an extreme amount of skipping/forking, resulting in highly reduced credits for all validators. But in epoch 83, the reduced lag of `zanve` allowed a supermajority to vote quickly and much reduced skipping/forking.

|Voter|Credits|Notes|
|-----|-------|-----|
|`459KZ`|40136||
|`CLLH3`|40136||
|`F176r`|0|offline|
|`FGd5g`|40136||
|`ash2V`|29217|highly reduced credits due to TVC|
|`meowZ`|40136||
|`zanve`|45514|increased credits (LOW-LAG MODS) due to voting mods|

In epochs 84 - 107 (the final epochs recorded for this report), `ash2V` continued to run with mods, but `zanve` disabled all voting mods. The purpose of this was to see what an unmodded validator vote credits looked like from a distant location like Santiago, Chile after TVC was enabled. The average vote credits for 84 - 107 are:

|Voter|Avg Credits|Notes|
|-----|-------|-----|
|`459KZ`|40764.625||
|`CLLH3`|40764.625||
|`F176r`|0|offline|
|`FGd5g`|40764.625||
|`ash2V`|33545.833|highly reduced credits due to TVC|
|`meowZ`|40763.667|0.002% fewer credits due to distant Tokyo location|
|`zanve`|40764.542|0.0002% fewer credits (NO MODS) due to distant Santiago location|

Conclusions
-----------

- The TVC cluster test demonstrated very clearly that unmodified validators are not disadvantaged by TVC, even if distant from the rest of the network
- Modified validators are disadvantaged heavily by TVC if they induce extra vote latency ("lag")
- The TVC features are safe and do not break any cluster functionality