Solana Timely Vote Credits Feature Test Results

Background

The Timely Vote Credits (TVC) feature is activated in two stages by two separate features:

7axKe5BTYBDD87ftzWbk5DfzWMGyRvqmWTduuo22Yaqy: enables in-place modification of vote accounts to expand their size and make available room for recording vote latencies

2oXpeh141pPZCTCFHBsvCwG2BtaHZZAtrVhwaxSy6brS: enables the use of the space made available in the prior change to allow validators to track the latency of votes and then assign vote credits to votes based on this latency information

These features are activated in sequence and in the order presented above. The first feature has already been activated on testnet, and demonstrated that it causes no cluster disruption and has the expected and intended effect of upgrading vote accounts.

The second feature has not been activated on testnet.

A new small testnet has been created to test the enablement of both of these features, as a way to demonstrate the effects of TVC in practice. There are three aspects to this:

Demonstrating that enablement of the TVC features does not "break" the cluster
Demonstrating that enablement of the TVC features results in the expected credits accounting changes
Demonstrating that geographically diverse validators are not intrinsically disadvantaged in credits earnings as a result of TVC

Methodology

A separate test cluster was spun up to test TVC.

The cluster consisted of 7 nodes:

Vote Account	Operator	Location	Stake Weight	Notes
`FGd5g9uYs9MiGvJtbJeoL1QMAB3yzECpRe8J5jSQkn1h`	Anza	Frankfurt Germany	19%	stock voting
`CLLH3k44keLPYrfzF49C2HhcwLQVWXSXtcxyXh2zd27G`	Anza	Frankfurt Germany	19%	stock voting
`459KZryRbx5BburUgrrbBg1r9DxBfe5mQocP2oRHtXUA`	Anza	Frankfurt Germany	19%	stock voting
`F176raFT8mtKnBi13ErBqTMMJYXQpugzQbReTXjouNLm`	Anza	Frankfurt Germany	19%	stock voting
`meowZQvyYd6wCHhD5QfAPJmn21nuGeg7z9CJdTkWP9f`	Pumpkin's Pool	Tokyo Japan	8%	stock voting
`zanveSXNMKjPZtBwnBJRPPNESkwc6hPC7XtcFugp6mA`	Shinobi Systems	Santiago Chile	9%	voting mods at various times
`ash2VSKkKrNZmP8wP9pvX3RTejpCJZ93f3pgkMqWmTu`	Ashwin (Anza)	Chicago USA	8%	consensus-only voting

Of these nodes, there were four Anza operated nodes, three of which just acted like normal nodes with no modifications, and one of which was a normal node but was taken offline before the first TVC feature was activated. One node was run by Pumpkin's Pool out of Tokyo, running unmodified code. One node was run by Shinobi Systems out of Santiago, in various epochs running different modifications to test their effects. And one node was run by Ashwin at Anza, running a modification that caused the node to only vote for already-confirmed blocks (representing a "worst case" validator voting mod behavior).

The cluster was run with very short epochs of 8192 slots in order to allow a faster epoch turnover for faster testing.

In addition, the cluster was put into high skip/fork behavior by the consistent delinquency of 19% of its stake, as well as some of the Anza nodes enabling testing code that attempted to cause forks by delivering block data late. Thus the cluster acted as a highly skippy/forky cluster which magnifies the impact of both stock code "fork avoidance" methods and validator voting mods.

What Was Tested

Enablement of the first feature

a. Test that a vote account with insufficient lamports to support self-upgrade continued to function normally
- Shinobi Systems set up its vote account to be not upgrade-capable by reducing its lamports to an amount that would not support the upgrade, and setting its validator commission to 0% so that it would not receive lamports into its vote account that would alter this fact. Thus this vote account could not auto-upgrade.
b. Test that a vote account with sufficient lamports to support self-upgrade did so and continued to function normally
- All other vote accounts were in this category
c. Test that vote credits continued to function normally

d. Test that a vote account with mods intended to capture more vote credits even if it produces higher vote latency (HIGH-LAG MODS), earns significantly more credits than the rest of the cluster, before the latency-based credits feature is activated
Enablement of the second feature

a. Test that a vote account that was not upgraded continued to function normally except that it received the minimum of 1 credit per vote instead of more credits based on latency

b. Test that vote accounts which were upgraded continued to function normally and earned credits commensurate with their vote latencies.
Post-enablement testing

a. Test that a vote account which was not upgraded would self-upgrade when sufficient lamports were added to it, and that it would then earn credits corresponding to vote latency

b. Test that unmodded validators at "distant" locations from the stake majority did not earn significantly different vote credits than validators in "close" locations.

c. Test that a vote account that voted only for confirmed blocks received lower credits due to the inherent latency of this voting method.

d. Test that a vote account from (1d) now earns significantly fewer vote credits due to the increased vote latency of its modifications.

e. Test that a vote account that earns additional credits through mods that add minimal vote latency (LOW-LAG MODS) do earn more credits still (the expected behavior) but that the additional earnings are much reduced from the (1d) case.

Results

All tests passed and demonstrated the expected effects. In particular:

The cluster operated as normal and survived the high skip rate/high delinquency/high forking behavior caused by the stress testing behavior of the Anza nodes.
The distant nodes in Tokyo and Santiago did not earn significantly fewer credits in any situation (the Santiago validator running without voting mods did earn approximately 0.01% fewer credits after TVC was enabled, but this reduction is exceedingly small; the Tokyo validator had no difference at all)
The validator which voted only after consensus did earn significantly more credits before TVC and significantly fewer credits after TVC.
The validator which was modded to optimize vote credits through higher vote latency fork avoidance (HIGH-LAG MODS) did earn significantly more credits before TVC and significantly fewer credits after TVC.
This validator which was then modded to optimize vote credits through no additional vote latency (LOW-LAG MODS) earned a small amount of additional credits after TVC (which was expected)

Specifics

Here are the credits earnings in epoch 63, before any TVC features were activated:

Voter	Credits	Notes
`459KZ`	8192
`CLLH3`	8192
`F176r`	8192
`FGd5g`	8192
`ash2V`	2874	partial epoch for this validator
`meowZ`	8191
`zanve`	8192

All validators were operating on stock code and earned essentially identical credits, except ash2V which had started part-way through the epoch.

Here are the credits earnings in epoch 64, after the first TVC feature was activated:

Voter	Credits	Notes
`459KZ`	7211
`CLLH3`	7211
`F176r`	7211
`FGd5g`	7211
`ash2V`	7028
`meowZ`	7211
`zanve`	7532

ash2V was fully online and zanve had enabled voting mods so those are the only ones with credits differences.

From epoch 64 - 79, there were no changes in how any vote account or the cluster operated, except that in epoch 77, the F176r validator was taken off-line. Here are average credits earned by each validator in epochs 64 - 79:

Voter	Avg Credits	Notes
`459KZ`	5505.562
`CLLH3`	5505.562
`F176r`	4825.000	decreased credits due to delinquency
`FGd5g`	5505.562
`ash2V`	5882.750	increased credits due to consensus-only vote mods
`meowZ`	5505.625
`zanve`	6871.250	vote optimizing mods (HIGH-LAG MODS) highly effective due to cluster skip/fork rate

In epoch 80, the second TVC feature was activated. At this time, the zanve validator upgraded part-way through the epoch, only received 1 credit per vote for some of its votes. This is reflected in credit earnings this epoch:

Voter	Credits	Notes
`459KZ`	8316
`CLLH3`	8316
`F176r`	0	offline
`FGd5g`	8316
`ash2V`	4715	immediately reduced credits due to TVC
`meowZ`	8316	same credits as Anza despite being in Tokyo
`zanve`	6028	immediately reduced credits (HIGH-LAG MODS) due to TVC + non-upgraded vote account

In epochs 81 and 82, the zanve validator ran with a fully upgraded vote account and at "max lag for credits":

Voter	Credits	Notes
`459KZ`	8384
`CLLH3`	8384
`F176r`	0	offline
`FGd5g`	8384
`ash2V`	4781	highly reduced credits due to TVC
`meowZ`	8384
`zanve`	6016	highly reduced credits (HIGH-LAG MODS) due to TVC

Voter	Credits	Notes
`459KZ`	8176
`CLLH3`	8176
`F176r`	0	offline
`FGd5g`	8176
`ash2V`	4611	highly reduced credits due to TVC
`meowZ`	8168
`zanve`	5916	highly reduced credits (HIGH-LAG MODS) due to TVC

In epoch 83, the zanve validator altered its voting mods to reduce their lag. This had the effect of increasing overall cluster credits considerably, because prior to this, the total amount of non-delinquent stake that was voting with heavy lag (zanve and ash2V) left the cluster at less than 66% of stake voting without induced lag, and this caused an extreme amount of skipping/forking, resulting in highly reduced credits for all validators. But in epoch 83, the reduced lag of zanve allowed a supermajority to vote quickly and much reduced skipping/forking.

Voter	Credits	Notes
`459KZ`	40136
`CLLH3`	40136
`F176r`	0	offline
`FGd5g`	40136
`ash2V`	29217	highly reduced credits due to TVC
`meowZ`	40136
`zanve`	45514	increased credits (LOW-LAG MODS) due to voting mods

In epochs 84 - 107 (the final epochs recorded for this report), ash2V continued to run with mods, but zanve disabled all voting mods. The purpose of this was to see what an unmodded validator vote credits looked like from a distant location like Santiago, Chile after TVC was enabled. The average vote credits for 84 - 107 are:

Voter	Avg Credits	Notes
`459KZ`	40764.625
`CLLH3`	40764.625
`F176r`	0	offline
`FGd5g`	40764.625
`ash2V`	33545.833	highly reduced credits due to TVC
`meowZ`	40763.667	0.002% fewer credits due to distant Tokyo location
`zanve`	40764.542	0.0002% fewer credits (NO MODS) due to distant Santiago location

Conclusions

The TVC cluster test demonstrated very clearly that unmodified validators are not disadvantaged by TVC, even if distant from the rest of the network
Modified validators are disadvantaged heavily by TVC if they induce extra vote latency ("lag")
The TVC features are safe and do not break any cluster functionality

cfl0ws / chainflow

2024/02/27 18:53:52

on testnet

"on the current testnet"

Zantetsu

2024/02/28 18:30:19

'testnet' is a well known term and I don't want to have to clarify that. I have used "TVC testnet" to describe this test cluster as a differentiating term.

2024/02/27 18:54:15

enabled

Maybe change "enabled" to "activated" throughout, to stay consistent with language.

2024/02/28 18:29:47

I have changed "enabled" to "activated" wherever the TVC features are referenced. I left "enabled" as a description of enabling TVC as a whole as the concept of "activating" TVC as a concept leaves me with cognitive dissonance.

2024/02/27 18:55:45

validator voting mods.

Were these mods similar to or the same as mods that validators have used on mainnet to-date?

2024/02/28 18:30:31

They were exactly the same, at least for me.

2024/02/28 18:31:07

But tuned to a higher-lag mode for the HIGH-LAG MOD testing. Tuned to the no lag I use on mainnet for the LOW-LAG MOD testing.

2024/02/27 18:56:25

insufficient lamports to support self-upgrade continued to function normally

Might be worth explaining how this works, i.e. how many lamports are needed and why.

2024/02/27 18:57:29

earned credits commensurate with their vote latencies

Where can we learn more about this relationship between latency and credits?

2024/02/27 18:58:42

“distant”

How was "distant" defined? I understand in this case it was Tokyo and Chile that were distant from Frankfurt. Might be worth mentioning that here or a parallel setup on mainnet that this was intended to mirror.

2024/02/28 18:32:29

Locations distant from Europe were defined as distant. In particular, east Asia is generally recognized as a "distant" region when we talk about mainnet, and Chile is *definitely* "distant" in all contexts in which validator locations are discussed.

2024/02/27 18:59:20

confirmed blocks

, also know as "consensus-only voting",

2024/02/27 19:13:49

Here are the credits

Why the drop between 63 and 64?

2024/02/28 18:33:58

Because of the skipping/forking that the SL nodes were doing was causing all votes to have additional average latency. This effect scaled to all nodes which means that there was no effect on relative credit earnings as a result of this.

2024/02/27 19:15:49

epoch 64 - 79

Similar questions, why the differences among the epochs, i.e. 8k, 7k, 5k, 8k, are the fluctuations among the averages due to TVC or more attributable to the general cluster performance?

2024/02/28 18:35:23

I described in a later section how the HIGH-LAG MODS being run by zanve were causing the cluster overall to have a much higher skip rate, fork rate, and also to just have slower consensus. This was the cause of the low credits that occurred after TVC was enabled and before the HIGH-LAG MODS validator switched to LOW-LAG MODs. In terms of why sometimes 7k credits and why sometimes 5k, I believe that's just happenstance.

2024/02/27 19:17:32

of non-delinquent stake that was voting with heavy lag (zanve and ash2V) left the cluster at less than 66% of stake voting without induced lag, and this caused an extreme amount of skipping/forking

I see now, maybe want to refer to this point a bit earlier in the results section.

2024/02/27 19:19:59

The credits in 83 are close to these averages from 84-107. The way I'm reading it is that zanve mods were turning off after 83, so I thought the credits in 84-107 would drop back to the 5k-8k range. Did this not happen because non-modded in this case also means zanve wasn't intentionally lagging? The 5x difference feels quite significant.

michael-laine

2024/02/28 18:05:59

with TVC enabled the max credit per vote is 8 instead of 1, therefore the total credits earned in normal conditions with TVC enabled should be approximately 6-8x of "normal"

2024/02/28 18:33:54

Thanks for clarifying, probably useful to mention in this doc and the upcoming proposal.

2024/02/28 18:36:09

It's all in the SIMD though. Reading and understanding the SIMD is an assumed prerequisite for reading and understanding the test results.

Blake Bartenbach

2024/02/28 19:13:47

highly reduced credits due to TVC

HIGH-LAG MODS. Might want to highlight the differences in the table for this epoch between LOW-LAG and HIGH-LAG mods (Edited)