# ToT Degradation ![image](https://hackmd.io/_uploads/r1Fh6MlpC.png) ### Events ## 11-09-2024 The relay was running with 6 validators: 4 of which were running version 1.15.1. 2 of them version 1.14.1 (updated to 1.15.2 after observing the issue). Relay runtime version 1.28.0 No `NodeFeatures` activated in `HostConfig`. No `AsyncBacking`. New collator is spun up to run AH with the following spec: - https://github.com/user-attachments/files/16954915/asset-hub-paseo-local.raw.json Some blocks after the new collator started running we observed the relay blocktime will go up by 6 seconds every 2 blocks. No other migration/upgrade/change was enacted during that time. Changes applied to `HostConfig`: Namely, we changed the above mentioned ones. ``` { maxCodeSize: 3,145,728 maxHeadDataSize: 32,768 maxUpwardQueueCount: 8 maxUpwardQueueSize: 1,048,576 maxUpwardMessageSize: 51,200 maxUpwardMessageNumPerCandidate: 5 hrmpMaxMessageNumPerCandidate: 5 validationUpgradeCooldown: 2 validationUpgradeDelay: 2 asyncBackingParams: { maxCandidateDepth: 3 allowedAncestryLen: 2 } maxPovSize: 5,242,880 maxDownwardMessageSize: 1,048,576 hrmpMaxParachainOutboundChannels: 4 hrmpSenderDeposit: 0 hrmpRecipientDeposit: 0 hrmpChannelMaxCapacity: 8 hrmpChannelMaxTotalSize: 8,192 hrmpMaxParachainInboundChannels: 4 hrmpChannelMaxMessageSize: 1,048,576 executorParams: [ { MaxMemoryPages: 8,192 } { PvfExecTimeout: [ Backing 2,500 ] } { PvfExecTimeout: [ Approval 15,000 ] } ] codeRetentionPeriod: 1,200 coretimeCores: 0 onDemandRetries: 0 onDemandQueueMaxSize: 10,000 onDemandTargetQueueUtilization: 25.00% onDemandFeeVariability: 3.00% onDemandBaseFee: 10,000,000 onDemandTtl: 5 groupRotationFrequency: 20 parasAvailabilityPeriod: 4 schedulingLookahead: 1 maxValidatorsPerCore: null maxValidators: null disputePeriod: 6 disputePostConclusionAcceptancePeriod: 100 noShowSlots: 2 nDelayTranches: 25 zerothDelayTrancheWidth: 0 neededApprovals: 2 relayVrfModuloSamples: 2 pvfVotingTtl: 2 minimumValidationUpgradeDelay: 5 minimumBackingVotes: 2 nodeFeatures: 0b01000000 approvalVotingParams: { maxApprovalCoalesceCount: 6 } } ``` Validator logs (Sadly we couldn't get the ones from the start of the issue): ``` Sep 11 17:12:42 Paseo-Test-Validator-1 systemd[1]: Stopped "paseo systemd service". Sep 11 17:12:42 Paseo-Test-Validator-1 systemd[1]: paseo.service: Consumed 28min 46.012s CPU time. Sep 11 17:12:42 Paseo-Test-Validator-1 systemd[1]: Started "paseo systemd service". Sep 11 17:12:42 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:42 Parity Polkadot Sep 11 17:12:42 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:42 ✌️ version 1.15.2-16b0fd09d9e Sep 11 17:12:42 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:42 ❤️ by Parity Technologies <admin@parity.io>, 2017-2024 Sep 11 17:12:42 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:42 📋 Chain specification: Paseo Testnet Sep 11 17:12:42 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:42 🏷 Node name: capricious-underwear-9678 Sep 11 17:12:42 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:42 👤 Role: AUTHORITY Sep 11 17:12:42 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:42 💾 Database: RocksDb at /opt/paseo/node1/chains/paseo/db/full Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 🚀 Using prepare-worker binary at: "/usr/local/sbin/polkadot-prepare-worker" Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 🚀 Using execute-worker binary at: "/usr/local/sbin/polkadot-execute-worker" Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 🏷 Local node identity is: 12D3KooWBY1M9NZt3c7hxg1nQCEUW93z6qe1kCS73mSzUoo8VNUJ Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 Running libp2p network backend Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 🔍 Discovered new external address for our node: /ip4/192.96.202.174/tcp/10802/p2p/12D3KooWBY1M9NZt3c7hxg1nQCEUW93z6qe1kCS73mSzUoo8VNUJ Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 💻 Operating system: linux Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 💻 CPU architecture: x86_64 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 💻 Target environment: gnu Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 💻 CPU: Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 💻 CPU cores: 2 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 💻 Memory: 15737MB Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 💻 Kernel: 5.14.0-427.28.1.el9_4.x86_64 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 💻 Linux distribution: Red Hat Enterprise Linux 9.4 (Plow) Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 💻 Virtual machine: yes Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 📦 Highest known block at #1674902 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 〽️ Prometheus exporter started at 127.0.0.1:9615 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 Running JSON-RPC server: addr=0.0.0.0:10000, allowed origins=["*"] Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 👶 Starting BABE Authorship worker Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 🥩 BEEFY gadget waiting for BEEFY pallet to become available... Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 discovered: 12D3KooWRMRQvoEdGcfcZJhekP3AD89GqfnZNF2qACC9PHS2NCuh /ip4/10.50.118.21/tcp/30334 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 discovered: 12D3KooWDiYHgHcVsA1cBFmT5R8zEK7tjLDXYJ2nn66Jtqj7kVb1 /ip4/10.50.118.16/tcp/30334 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 discovered: 12D3KooWP2ybUBq9DKrmzKYvQbd3YJzZ4xPoNsutx7bENAT1Mwov /ip4/10.50.118.14/tcp/30334 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 discovered: 12D3KooWF6bu75BZcmWCVWNZk2A5cqs33o3JbuYxizvzDDpBbnZs /ip4/10.50.118.15/tcp/30334 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 discovered: 12D3KooWDzQrEFvNPDxMvqw5R5aii23re3wqGvPEvXfL1wmqJkmW /ip4/10.50.118.14/tcp/30332 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 discovered: 12D3KooWApG17wTzkm4RJHfnWTa9VtJiaSs8i6aVTmFy89PMwzjk /ip4/10.50.118.16/tcp/30332 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 discovered: 12D3KooWDMv2weJXuVQsnxetpu5o3sUyxcTzKniKfJb7U3nmCLUP /ip4/10.50.118.20/tcp/30334 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 discovered: 12D3KooWAoux46wBtj987pnj6cAJMRMQkPAb62TPVC1Tfj4mDnF5 /ip4/10.50.118.18/tcp/30334 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 discovered: 12D3KooWScv6oyza3QivUYVcrgqEds1su7Rj15ALWskyN7zPmwsm /ip4/10.50.118.17/tcp/30332 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 discovered: 12D3KooWGSr4EVQ5CN8vFfp8uqscgujUp9WYEnHBUZvjULNZEPPc /ip4/10.50.118.15/tcp/30332 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 discovered: 12D3KooWPpeZFYtKkHdb6Q6jdNox6qN5dRPtHmtj3rmDqdzQPkcG /ip4/10.50.118.20/tcp/30332 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 discovered: 12D3KooWBmfx5Pr8UvxdR2jdn81aFCFN2RtUjWoyUNAPodmguw7V /ip4/10.50.118.17/tcp/30334 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 discovered: 12D3KooWGD3wr6wtqKNCBR1xFupEfZF2UBt57twzMewAJzmuXZgr /ip4/10.50.118.22/tcp/30334 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 discovered: 12D3KooWMTZz2t27THPr9wi2JAsEF3uyhCxyNL4obyvw2Tg3PCHC /ip4/10.50.118.23/tcp/30334 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 discovered: 12D3KooWPdJMmoXgd5ANiUB8RWHFo2LJ3wDVkbbMzbdNy2YNzqrk /ip4/10.50.118.13/tcp/30332 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 discovered: 12D3KooWCjZTGDVpRUk3UVJRENJnaBeTKmqHieTWjvLJ6wffYETm /ip4/10.50.118.19/tcp/30334 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 discovered: 12D3KooWPJXB67H5asANBpbWBsfLtNM7abiGLzqfuptr7wn9hp2k /ip4/10.50.118.13/tcp/30334 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 discovered: 12D3KooWCgsfgM61z8TUMRse9xRuHJRKrCj4kPNwYiodwdv4cRAn /ip4/10.50.118.11/tcp/30334 Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 🚨 Some security issues have been detected. Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: Running validation of malicious PVF code has a higher risk of compromising this machine. Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: - Optional: Cannot enable landlock (ABI 1), a Linux 5.13+ kernel security feature: not available: Could not fully enable: NotEnforced Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 👮‍♀️ Running in Secure Validator Mode. It is highly recommended that you operate according to our security guidelines. Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: More information: https://wiki.polkadot.network/docs/maintain-guides-secure-validator#secure-validator-mode Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 🔍 Discovered new external address for our node: /ip4/192.96.202.161/tcp/30334/p2p/12D3KooWBY1M9NZt3c7hxg1nQCEUW93z6qe1kCS73mSzUoo8VNUJ Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 🔍 Discovered new external address for our node: /ip4/192.96.202.174/tcp/30334/p2p/12D3KooWBY1M9NZt3c7hxg1nQCEUW93z6qe1kCS73mSzUoo8VNUJ Sep 11 17:12:48 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:48 Backing off claiming new slot for block authorship: finality is lagging. Sep 11 17:12:48 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:48 Received msg before first active leaves update. This is not expected - message will be dropped. msg=DetermineUndisputedChain { base: (1674697, 0xe9743387049eb87e20fa0ebb35f4af4b239b553a30163ed61e3ec24300c29825), block_descriptions: [], tx: Sender { complete: false } } Sep 11 17:12:48 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:48 Call to `DetermineUndisputedChain` failed error=DetermineUndisputedChainCanceled(Canceled) Sep 11 17:12:50 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:50 Received msg before first active leaves update. This is not expected - message will be dropped. msg=DetermineUndisputedChain { base: (1674697, 0xe9743387049eb87e20fa0ebb35f4af4b239b553a30163ed61e3ec24300c29825), block_descriptions: [], tx: Sender { complete: false } } Sep 11 17:12:50 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:50 Call to `DetermineUndisputedChain` failed error=DetermineUndisputedChainCanceled(Canceled) Sep 11 17:12:52 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:52 💤 Idle (5 peers), best: #1674902 (0x344a…74f1), finalized #1674697 (0xe974…9825), ⬇ 23.3kiB/s ⬆ 28.7kiB/s Sep 11 17:12:53 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:53 Report 12D3KooWMdvy6kPy4DiFnoQB5fvsjZKGiUpC6jvxoNWVQ5K9MgCU: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting. Sep 11 17:12:53 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:53 Received msg before first active leaves update. This is not expected - message will be dropped. msg=DetermineUndisputedChain { base: (1674697, 0xe9743387049eb87e20fa0ebb35f4af4b239b553a30163ed61e3ec24300c29825), block_descriptions: [], tx: Sender { complete: false } } Sep 11 17:12:53 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:53 Call to `DetermineUndisputedChain` failed error=DetermineUndisputedChainCanceled(Canceled) Sep 11 17:12:54 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:54 Backing off claiming new slot for block authorship: finality is lagging. Sep 11 17:12:57 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:57 💤 Idle (5 peers), best: #1674902 (0x344a…74f1), finalized #1674697 (0xe974…9825), ⬇ 7.2kiB/s ⬆ 4.1kiB/s Sep 11 17:12:57 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:57 Received msg before first active leaves update. This is not expected - message will be dropped. msg=DetermineUndisputedChain { base: (1674697, 0xe9743387049eb87e20fa0ebb35f4af4b239b553a30163ed61e3ec24300c29825), block_descriptions: [], tx: Sender { complete: false } } Sep 11 17:12:57 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:57 Call to `DetermineUndisputedChain` failed error=DetermineUndisputedChainCanceled(Canceled) Sep 11 17:12:59 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:59 Report 12D3KooWAerghr5s8u8YeWZ97qX6bhq7M7HVh1iEpfVR7KPvVfKu: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting. Sep 11 17:13:01 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:01 Received msg before first active leaves update. This is not expected - message will be dropped. msg=DetermineUndisputedChain { base: (1674697, 0xe9743387049eb87e20fa0ebb35f4af4b239b553a30163ed61e3ec24300c29825), block_descriptions: [], tx: Sender { complete: false } } Sep 11 17:13:01 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:01 Call to `DetermineUndisputedChain` failed error=DetermineUndisputedChainCanceled(Canceled) Sep 11 17:13:02 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:02 💤 Idle (5 peers), best: #1674902 (0x344a…74f1), finalized #1674697 (0xe974…9825), ⬇ 8.2kiB/s ⬆ 5.9kiB/s Sep 11 17:13:05 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:05 Received msg before first active leaves update. This is not expected - message will be dropped. msg=DetermineUndisputedChain { base: (1674697, 0xe9743387049eb87e20fa0ebb35f4af4b239b553a30163ed61e3ec24300c29825), block_descriptions: [], tx: Sender { complete: false } } Sep 11 17:13:05 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:05 Call to `DetermineUndisputedChain` failed error=DetermineUndisputedChainCanceled(Canceled) Sep 11 17:13:07 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:07 💤 Idle (5 peers), best: #1674902 (0x344a…74f1), finalized #1674697 (0xe974…9825), ⬇ 5.2kiB/s ⬆ 4.8kiB/s Sep 11 17:13:09 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:09 Received msg before first active leaves update. This is not expected - message will be dropped. msg=DetermineUndisputedChain { base: (1674697, 0xe9743387049eb87e20fa0ebb35f4af4b239b553a30163ed61e3ec24300c29825), block_descriptions: [], tx: Sender { complete: false } } Sep 11 17:13:09 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:09 Call to `DetermineUndisputedChain` failed error=DetermineUndisputedChainCanceled(Canceled) Sep 11 17:13:12 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:12 💤 Idle (5 peers), best: #1674902 (0x344a…74f1), finalized #1674697 (0xe974…9825), ⬇ 5.4kiB/s ⬆ 5.9kiB/s Sep 11 17:13:13 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:13 Received msg before first active leaves update. This is not expected - message will be dropped. msg=DetermineUndisputedChain { base: (1674697, 0xe9743387049eb87e20fa0ebb35f4af4b239b553a30163ed61e3ec24300c29825), block_descriptions: [], tx: Sender { complete: false } } Sep 11 17:13:13 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:13 Call to `DetermineUndisputedChain` failed error=DetermineUndisputedChainCanceled(Canceled) Sep 11 17:13:14 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:14 Report 12D3KooWHSFAzDvYfUErk5wh9j4R9MHyTXTnc49K5WpfwnQTsxZJ: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting. Sep 11 17:13:17 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:17 💤 Idle (5 peers), best: #1674902 (0x344a…74f1), finalized #1674697 (0xe974…9825), ⬇ 4.6kiB/s ⬆ 5.0kiB/s Sep 11 17:13:17 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:17 Received msg before first active leaves update. This is not expected - message will be dropped. msg=DetermineUndisputedChain { base: (1674697, 0xe9743387049eb87e20fa0ebb35f4af4b239b553a30163ed61e3ec24300c29825), block_descriptions: [], tx: Sender { complete: false } } Sep 11 17:13:17 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:17 Call to `DetermineUndisputedChain` failed error=DetermineUndisputedChainCanceled(Canceled) Sep 11 17:13:21 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:21 Received msg before first active leaves update. This is not expected - message will be dropped. msg=DetermineUndisputedChain { base: (1674697, 0xe9743387049eb87e20fa0ebb35f4af4b239b553a30163ed61e3ec24300c29825), block_descriptions: [], tx: Sender { complete: false } } Sep 11 17:13:21 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:21 Call to `DetermineUndisputedChain` failed error=DetermineUndisputedChainCanceled(Canceled) Sep 11 17:13:22 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:22 💤 Idle (5 peers), best: #1674902 (0x344a…74f1), finalized #1674697 (0xe974…9825), ⬇ 7.8kiB/s ⬆ 4.8kiB/s Sep 11 17:13:24 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:24 🏆 Imported #1674903 (0x344a…74f1 → 0xd2a7…ab19) Sep 11 17:13:24 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:24 Node uses the following topology indices index_in_gossip_topology=Some(ValidatorIndex(2)) index_in_parachain_authorities=Some(Active(ValidatorIndex(2))) Sep 11 17:13:27 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:27 💤 Idle (5 peers), best: #1674903 (0xd2a7…ab19), finalized #1674697 (0xe974…9825), ⬇ 17.2kiB/s ⬆ 3.4kiB/s Sep 11 17:13:30 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:30 Backing off claiming new slot for block authorship: finality is lagging. Sep 11 17:13:32 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:32 💤 Idle (5 peers), best: #1674903 (0xd2a7…ab19), finalized #1674697 (0xe974…9825), ⬇ 5.3kiB/s ⬆ 4.9kiB/s Sep 11 17:13:33 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:13:33 Report 12D3KooWFEarPetbCWzTp4x6U6hTn1GMXEkSEYxsdMa3QxZqLpYR: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting. ``` Following logs are not relevant, the machine doesn't have landlock support, hence the warning. ``` Sep 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: 2024-09-11 17:12:47 🚨 Some security issues have been detected. Se`p 11 17:12:47 Paseo-Test-Validator-1 paseo-n1[1348942]: Running validation of malicious PVF code has a higher risk of copromising this machine. ``` ## 12-09-2024 https://polkadot.js.org/apps/?rpc=wss://rpc.stake.plus/tot#/explorer ![image](https://hackmd.io/_uploads/rJ1zvWZaA.png) Same as above set up but all validators running @ 1.15.2 Relay runtime version 1.28.0 Registered two parachains - Assigned slots. Sent changes to HostConfig Prior to changes HostConfig was - https://github.com/paseo-network/runtimes/blob/9b3215eb7f65453ea8b2605823653128ecf63ba1/relay/paseo/src/genesis_config_presets.rs#L91 ``` { maxCodeSize: 3,145,728 maxHeadDataSize: 32,768 maxUpwardQueueCount: 174,762 maxUpwardQueueSize: 1,048,576 maxUpwardMessageSize: 65,531 maxUpwardMessageNumPerCandidate: 16 hrmpMaxMessageNumPerCandidate: 10 validationUpgradeCooldown: 60 validationUpgradeDelay: 60 asyncBackingParams: { maxCandidateDepth: 3 allowedAncestryLen: 2 } maxPovSize: 5,242,880 maxDownwardMessageSize: 1,048,576 hrmpMaxParachainOutboundChannels: 128 hrmpSenderDeposit: 0 hrmpRecipientDeposit: 0 hrmpChannelMaxCapacity: 1,000 hrmpChannelMaxTotalSize: 102,400 hrmpMaxParachainInboundChannels: 128 hrmpChannelMaxMessageSize: 102,400 executorParams: [ { MaxMemoryPages: 8,192 } { PvfExecTimeout: [ Backing 2,500 ] } { PvfExecTimeout: [ Approval 15,000 ] } ] codeRetentionPeriod: 1,200 coretimeCores: 0 onDemandRetries: 0 onDemandQueueMaxSize: 10,000 onDemandTargetQueueUtilization: 25.00% onDemandFeeVariability: 3.00% onDemandBaseFee: 10,000,000 onDemandTtl: 5 groupRotationFrequency: 20 parasAvailabilityPeriod: 4 schedulingLookahead: 2 maxValidatorsPerCore: null maxValidators: null disputePeriod: 6 disputePostConclusionAcceptancePeriod: 100 noShowSlots: 2 nDelayTranches: 25 zerothDelayTrancheWidth: 0 neededApprovals: 2 relayVrfModuloSamples: 2 pvfVotingTtl: 2 minimumValidationUpgradeDelay: 5 minimumBackingVotes: 2 nodeFeatures: 0b11000000 approvalVotingParams: { maxApprovalCoalesceCount: 1 } } ``` ## 13-09-2024 At some point we were only running 3 validators, which it seems mixed with `minimumBackingVotes: 2` and `neededApprovals: 2` could have been a problem due to the small size of the network. Finality lagging behind is the cause of block production degradataion. Dispatching `initializer.forceApprove` helps and brings back the blocktime to normal. Which hints the cause of the issue might be in the approval process. Bringing back the 4th validator seemed to help with finality, as expected. Block production seems to continue and finality is catching up. Dispatched a change for HostConfig to bring down `neededApprovals` and `minimumBackingVotes` to `1`.