# Staking parachain PoV limits PoV upper bounds and configs: ```rust pub const MILLISECS_PER_BLOCK: u64 = 12000; pub const SLOT_DURATION: u64 = MILLISECS_PER_BLOCK; /// 0.5 of a second of compute with a ~12 second average block time. const MAXIMUM_BLOCK_WEIGHT: Weight = Weight::from_parts( WEIGHT_REF_TIME_PER_SECOND.saturating_div(2), // 500_000_000_000 cumulus_primitives_core::relay_chain::MAX_POV_SIZE as u64, // 5_242_880 (5120kb) ); // async backing disabled: const UNINCLUDED_SEGMENT_CAPACITY: u32 = 1; const BLOCK_PROCESSING_VELOCITY: u32 = 1; // aura.AllowMultipleBlocksPerSlot = ConstBool<false>; ``` To reproduce: - follow https://github.com/paritytech/polkadot-sdk/tree/staking-parachain-exp/templates/parachain *Note*: using unsigned submission only, but PoV should map 1-1 with signed phase (1 paged submission per block) **Notes** - It seems that the snapshot creation or the export phase are the bottleneck wrt the amount of storage r/w required per block - Adding too many stakers in genesis may overflow the wasm allocation: > going to fail due to allocating 41018542 > Error: Service(Other("wasm call error Requested allocation size is too large")) ### Working bounds/configs ``` *genesis* n_validators: 1_500 n_nominators: 30_000 (8 edges) // staking pub const MaxValidatorSet: u32 = 1_500; staking.ValidatorCount = 1_500; // EPM pub Pages: PageIndex = 3; pub MaxWinnersPerPage: u32 = 1_500; pub MaxBackersPerWinner: u32 = 5_000; pub VoterSnapshotPerBlock: VoterIndex = 10_000; pub TargetSnapshotPerBlock: TargetIndex = 1_500; ``` ``` // Targets snapshot [Parachain] [#22]: 🎯🎯🎯 🗳 created target snapshot with 1500 targets. [Parachain] PoV size { header: 0.2177734375kb, extrinsics: 3.5830078125kb, storage_proof: 74.2119140625kb } // Voters snapshot page 2: [Parachain] [#23]: 🎯🎯🎯 🗳 created voter snapshot with 2500 voters. [Parachain] PoV size { header: 0.2177734375kb, extrinsics: 3.5830078125kb, storage_proof: 3319.3525390625kb } page 1: [Parachain] [#24]: 🎯🎯🎯 🗳 voter snapshot progressed: page 1 with 2500 voters [Parachain] PoV size { header: 0.2177734375kb, extrinsics: 3.5830078125kb, storage_proof: 3350.6201171875kb } page 0: [Parachain] [#25]: 🎯🎯🎯 🗳 voter snapshot progressed: page 0 with 2500 voters [Parachain] PoV size { header: 0.2177734375kb, extrinsics: 3.5830078125kb, storage_proof: 3348.458984375kb } // Unsigned submission + verification page 2: [Parachain] [#27]: 🎯🎯🎯 🗳 now 27 - current phase Unsigned(26), next election 30, remaining: 3 [Parachain] [#27]: 🎯🎯🎯 🗳 queued sync solution with score ElectionScore { minimal_stake: 253309505716, sum_stake: 2744596071297945, sum_stake_squared: 5410361055743539901774942387 } (page 2) [Parachain] PoV size { header: 0.2177734375kb, extrinsics: 83.9814453125kb, storage_proof: 777.2119140625kb } page 1: [Parachain] [#28]: 🎯🎯🎯 🗳 now 28 - current phase Unsigned(26), next election 30, remaining: 2 [Parachain] [#28]: 🎯🎯🎯 🗳 queued sync solution with score ElectionScore { minimal_stake: 253309505716, sum_stake: 2744596071297945, sum_stake_squared: 5410361055743539901774942387 } (page 2) [Parachain] PoV size { header: 0.2177734375kb, extrinsics: 83.9814453125kb, storage_proof: 777.33984375kb } page 0 (+ full verification) [Parachain] [#30]: 🎯🎯🎯 🗳 now 30 - current phase Unsigned(26), next election 30, remaining: 0 [Parachain] PoV size { header: 0.2177734375kb, extrinsics: 3.5830078125kb, storage_proof: 5.2998046875kb } // elect() + store electable stashes page 2: [Parachain] [32]: 🎯🎯🎯 💸 elect(): next election in 3 pages, start fetching solution pages. [Parachain] [32]: 🎯🎯🎯 💸 new validator set of size 1500 has been processed for era 1 [Parachain] PoV size { header: 0.2177734375kb, extrinsics: 3.5830078125kb, storage_proof: 1084.013671875kb } page 1: [Parachain] [#32]: 🎯🎯🎯 🗳 now 32 - current phase Export(32), next election 32, remaining: 0 [Parachain] PoV size { header: 0.2177734375kb, extrinsics: 3.5830078125kb, storage_proof: 1084.013671875kb } ❌ [Parachain] panicked at /Users/gpestana/.rustup/toolchains/nightly-aarch64-apple-darwin/lib/rustlib/src/rust/library/alloc/src/vec/mod.rs:2162:13: `at` split index (is 52) should be <= len (is 14) 2024-05-07 01:50:36.086 ERROR tokio-runtime-worker aura::cumulus: [Parachain] err=Error { inner: Proposing .... ``` --- **Test case 1.** single-page, 10K voters/ snapshot page ❌ bricking the collation by overflowing the storage proof. - 10K voters per snapshot overflows the storage proof by ~3600kb ``` *genesis* n_validators: 1_000 n_nominators: 10_000 // staking pub const MaxValidatorSet: u32 = 1_000; staking.ValidatorCount = 1_000; // EPM pub Pages: PageIndex = 1; pub MaxWinnersPerPage: u32 = 1_000; pub MaxBackersPerWinner: u32 = 5_000; pub VoterSnapshotPerBlock: VoterIndex = 10_000; pub TargetSnapshotPerBlock: TargetIndex = 1_000; ``` ``` // Target snapshot [Parachain] [#7]: 🎯🎯🎯 🗳 created target snapshot with 1000 targets. [Parachain] [#7]: 🎯🎯🎯 🗳 target snapshot created with 1000 targets Parachain] PoV size { header: 0.216796875kb, extrinsics: 3.5810546875kb, storage_proof: 50.9462890625kb } [Parachain] Compressed PoV size: 47.9091796875kb // Voter snapshot [#8]: 🎯🎯🎯 🗳 now 8 - current phase Snapshot(1), next election 10, remaining: 2 Parachain] [8]: 🎯🎯🎯 💸 generated 10000 npos voters, 481 from validators and 9519 nominators [Parachain] [#8]: 🎯🎯🎯 🗳 created voter snapshot with 10000 voters. [#8]: 🎯🎯🎯 🗳 voter snapshot progressed: page 0 with 10000 voters [Parachain] PoV size { header: 0.216796875kb, extrinsics: 3.5810546875kb, storage_proof: 8787.5908203125kb } [Parachain] Compressed PoV size: 2300.1845703125kb parachain::collator-protocol: [Relaychain] Collation wasn't advertised to any validator. candidate_hash=0x8ba7a25f4e96b533930373d7fb80797b3dbff3d4049c72535ac516ddc7faf694 pov_hash=0xd74d1799fed57cfa1c6057db5c5fd0a344be46bedc038ce813f9e7b62563b6e5 traceID=185633098290831878312032420129021131131 > bricked, parachain blocks not progressing ``` **Test case 2.** single-page, 5K voters/ snapshot page ✅ - Voter snapshot (5k/page) in the limit of the storage proof size (5115kb vs 5120 kb max) ```rust! // single page election score ElectionScore { minimal_stake: 13_042_714_285_497, sum_stake: 14_261_412_733_134_461, sum_stake_squared: 203451499923879607620859254597 } ``` ``` *genesis* n_validators: 1_000 n_nominators: 10_000 // staking pub const MaxValidatorSet: u32 = 1_000; staking.ValidatorCount = 1_000; // EPM pub Pages: PageIndex = 1; pub MaxWinnersPerPage: u32 = 1_000; pub MaxBackersPerWinner: u32 = 5_000; pub VoterSnapshotPerBlock: VoterIndex = 5_000; pub TargetSnapshotPerBlock: TargetIndex = 1_000; ``` ``` // Target snapshot [Parachain] [#7]: 🎯🎯🎯 🗳 now 7 - current phase Off, next election 10, remaining: 3 [Parachain] [7]: 🎯🎯🎯 💸 generated 1000 npos targets [Parachain] [#7]: 🎯🎯🎯 🗳 created target snapshot with 1000 targets. [Parachain] [#7]: 🎯🎯🎯 🗳 target snapshot created with 1000 targets [Parachain] 🏆 Imported #7 (0x1ebb…493f → 0x77e4…b879) [Parachain] PoV size { header: 0.216796875kb, extrinsics: 3.5810546875kb, storage_proof: 50.9462890625kb } [Parachain] Compressed PoV size: 47.9091796875kb // Voter snapshot [Parachain] [#8]: 🎯🎯🎯 🗳 now 8 - current phase Snapshot(1), next election 10, remaining: 2 [Parachain] [8]: 🎯🎯🎯 💸 generated 5000 npos voters, 0 from validators and 5000 nominators [Parachain] [#8]: 🎯🎯🎯 🗳 created voter snapshot with 5000 voters. [Parachain] [#8]: 🎯🎯🎯 🗳 voter snapshot progressed: page 0 with 5000 voters [Parachain] PoV size { header: 0.216796875kb, extrinsics: 3.5810546875kb, storage_proof: 5115.0859375kb } [Parachain] Compressed PoV size: 1705.0869140625kb // Unsigned submission + verification [Parachain] [#10]: 🎯🎯🎯 🗳 now 10 - current phase Unsigned(9), next election 10, remaining: 0 [Parachain] [#10]: 🎯🎯🎯 🗳 validate_unsigned OK [Parachain] [#10]: 🎯🎯🎯 🗳 queued sync solution with score ElectionScore { minimal_stake: 13042714285497, sum_stake: 14261412733134461, sum_stake_squared: 203451499923879607620859254597 } (page 0) [Parachain] [#10]: 🎯🎯🎯 🗳 finalizing verification of a correct solution, replacing old score None with ElectionScore { minimal_stake: 13042714285497, sum_stake: 14261412733134461, sum_stake_squared: 203451499923879607620859254597 } [Parachain] [#10]: 🎯🎯🎯 🗳 finalize verification outcome: Ok(()) offchain-worker runtime::multiblock-election: [#10]: 🎯🎯🎯 🗳 offchain miner cache cleared. [Parachain] PoV size { header: 0.216796875kb, extrinsics: 311.560546875kb, storage_proof: 2736.2041015625kb } [Parachain] Compressed PoV size: 727.3427734375kb // elect() + store electable stashes [Parachain] [14]: 🎯🎯🎯 💸 elect(): next election in 1 pages, start fetching solution pages. [Parachain] [#14]: 🎯🎯🎯 🗳 elect(): provided the last supports page, rotating round. [Parachain] [14]: 🎯🎯🎯 💸 new validator set of size 1000 has been processed for era 1 [Parachain] [#14]: 🎯🎯🎯 🗳 now 14 - current phase Off, next election 14, remaining: 0 [Parachain] PoV size { header: 0.216796875kb, extrinsics: 3.5810546875kb, storage_proof: 3850.3623046875kb } [Parachain] Compressed PoV size: 916.2607421875kb ``` **Test case 4.** single-page, 10k targets and 5K voters/ snapshot page; 10K max backers per winner ❌ bricking the collation by overflowing the storage proof. - 5K (out of 20Ks) voters per snapshot overflows the storage proof by ~1300kb ``` *genesis* n_validators: 10_000 n_nominators: 20_000 // staking pub const MaxValidatorSet: u32 = 1_000; staking.ValidatorCount = 1_000; // EPM pub Pages: PageIndex = 1; pub MaxWinnersPerPage: u32 = 10_000; pub MaxBackersPerWinner: u32 = 10_000; pub VoterSnapshotPerBlock: VoterIndex = 5_000; pub TargetSnapshotPerBlock: TargetIndex = 10_000; ``` ``` // Target snapshot [Parachain] [7]: 🎯🎯🎯 💸 generated 10000 npos targets [Parachain] [#7]: 🎯🎯🎯 🗳 created target snapshot with 10000 targets. [Parachain] [#7]: 🎯🎯🎯 🗳 target snapshot created with 10000 targets [Parachain] PoV size { header: 0.216796875kb, extrinsics: 3.5810546875kb, storage_proof: 463.5048828125kb } [Parachain] Compressed PoV size: 405.9716796875kb // Voter snapshot [Parachain] [8]: 🎯🎯🎯 💸 generated 5000 npos voters, 0 from validators and 5000 nominators [Parachain] [#8]: 🎯🎯🎯 🗳 created voter snapshot with 5000 voters. [Parachain] [#8]: 🎯🎯🎯 🗳 voter snapshot progressed: page 0 with 5000 voters [Parachain] PoV size { header: 0.216796875kb, extrinsics: 3.5810546875kb, storage_proof: 6427.046875kb } [Parachain] Compressed PoV size: 3703.509765625kb 2024-05-07 00:42:26.074 WARN tokio-runtime-worker parachain::collator-protocol: [Relaychain] Collation wasn't advertised to any validator. candidate_hash=0x21458f63bf60680185843d95f1b9bc8d8a853bc55075c9ca5d722c2b3766a6bf pov_hash=0xa3cc666e0dfe5b18559bdedb385b48db0cc0477c27d7fb6d5c76d20e790a58dc traceID=44225700631530722921976040915954416781 > bricked, parachain blocks not progressing ``` **Test case 5.** multi-page (3), 10k targets and 2.5K voters/ snapshot page; 10K max backers per winner ✅ ``` *genesis* n_validators: 10_000 n_nominators: 20_000 // staking pub const MaxValidatorSet: u32 = 1_000; staking.ValidatorCount = 1_000; // EPM pub Pages: PageIndex = 3; pub MaxWinnersPerPage: u32 = 10_000; pub MaxBackersPerWinner: u32 = 10_000; pub VoterSnapshotPerBlock: VoterIndex = 2_500; pub TargetSnapshotPerBlock: TargetIndex = 10_000; ``` --- (*old*) ### Theoretical limits Dumps from test in https://github.com/paritytech/polkadot-sdk/blob/gpestana/staking_para_weights/substrate/frame/election-provider-multi-phase/test-staking-e2e/src/lib.rs#L392 Note that we assume only the max `BlockLimit` weight in this exercise, so take it with a pinch of salt. Given ```rust= // `MaxActiveValidators` in Polkadot. let targets = 1_200; // `MaxElectionVoters` in Polkadot. let voters = 22_500; // we assume 1/2 of the voters are active. let active_voters = 22_500 / 2; // `MaxWinners` in Polkadot. let desired_targets = 1_200; // parachain block limit (using same as AssetHub in Rococo) let weight_limit = Weight::from_parts( // ref_time 500_000_000_000, // proof_size 5_242_880, ); ``` We have: ``` snapshot weight: Weight { ref_time: 4548650500, proof_size: 0 }. fits? true. Max voters: 2830950 submit_unsigned: Weight { ref_time: 68979654500, proof_size: 12482638 }. fits? false. Max voters: 9410 elect weight: Weight { ref_time: 4447471000, proof_size: 8702723 }. fits? false get_npos_voters weight: Weight { ref_time: 108390856050, proof_size: 44909090 }. fits? false get_npos_targets weight: Weight { ref_time: 5945960664, proof_size: 3027510 }. fits? true ``` ### Practical limits