# AH Downtime Postmortem – 23rd of October 2025

Today, **23rd of October 2025**, Asset Hub **stopped** producing blocks at **11:32:42 AM CET**.
**R0GUE** noticed and started investigating at **11:48:00 AM CET**, following alerts from the collator maintainers.

## Scenario

After attending a request from **Hydra (2034)** to execute a privileged call to set a new head and Wasm for their chain, we executed a [`hrmp.force_clean_hrmp`](https://paseo.subscan.io/extrinsic/8737640-3) call to remove HRMP-related state that was blocking their runtime from producing blocks after having pushed their new Wasm and head into the relay state.

This is a task and series of calls that we perform fairly often, as different teams request to hot-swap their code and head registered in the relay chain. We immediately assisted with re-opening the HRMP channels they had open at the time.

Bidirectional with the following parachains:
- `1,000`
- `1,111`
- `2,030`
- `4,001`

However, this time the action resulted in two chains stopping their block production unexpectedly:
- **Asset Hub (`1000`)**
- **PAsset Hub (`1111`)**

## Findings

We believe the root cause of the issue is as follows:

After closing the above-mentioned HRMP channels and re-opening them, we noticed that the following storage key in the parachains wasn’t properly updated:

- [`LastHrmpMqcHeads`](https://github.com/paritytech/polkadot-sdk/blob/31001cd5379c191aba605ac075419be9979d5742/cumulus/pallets/parachain-system/src/lib.rs#L874)

This caused the runtime of these chains to panic due to a mismatch between the MQC head value stored in their local state and the one they were pulling from the relay chain.

More concretely, this is the assertion that caused the panic:

- https://github.com/paritytech/polkadot-sdk/blob/4866b0827ee6c78192a57162dc2d46eb92c96f3c/cumulus/pallets/parachain-system/src/lib.rs#L1240

We believe the `LastHrmpMqcHeads` state wasn’t updated correctly for the following reason:

- The Asset Hub MQC head for Hydra (2034) at the time was:
`0x387b9a372b5759dd293dd006027224cac0e2c1d4763616516d1b1f7fb597ebc5`.
This matched the state kept for the corresponding channel in the relay:

Can be retrieved by querying key:
- 0x6a0da05ca59913bc38a8630590f2627cb6604cff828a6e3f579ca6c59ace013d5d6339a4014c8ab9f2070000e8030000
- at block hash: 0x16618700c2ed854713c50717ad912c0e3f9661558b7742ed9e519697857ae2a9

{
  maxCapacity: 1,000
  maxTotalSize: 102,400
  maxMessageSize: 102,400
  msgCount: 0
  totalSize: 0
  mqcHead: 0x387b9a372b5759dd293dd006027224cac0e2c1d4763616516d1b1f7fb597ebc5
  senderDeposit: 0
  recipientDeposit: 0
}

When removing the channels, the entries on the relay were cleared, including the `mqcHead` value for this channel.
If these values have no state, the default value for the corresponding type is used instead. In this case:
`0x0000000000000000000000000000000000000000000000000000000000000000`

This resulted in the following logs on Asset Hub collators:

Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:52.133  INFO tokio-runtime-worker txpool: [Parachain] maintain txs=(1, 0) a=1 i=4 views=[(3292003, 1, 0)] event=Finalized { hash: 0xc1d92548540ffbcdac0ce69719d1254717126bb2da80984054f4a0e6d16456be, tree_route: [0x1a08a7b6a4086be99232e2deb707f31df7e72553ade8f14049355a9e7e322a4d, 0x558894501b5a9a773c896bbf556d3cdb720250bf61e36d34cc1680519dd10c6a] } duration=150.551µs
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:52.826  INFO tokio-runtime-worker minimal-polkadot-node: [Parachain] Received imported block via RPC: #8737815 (0x4de6…b9b7 -> 0xb0b9…7d7f)
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:52.827  INFO tokio-runtime-worker cumulus_client_service: [Parachain] Update at relay chain block #8737815 (0xb0b9…7d7f) - included: #985748 (0xc112…f84c), #985749 (0xcda9…8ec9), #985750 (0x4181…3735), #6549775 (0xfca3…6b24), #2435786 (0x6279…a69b), #5787359 (0x1930…75e5), #2518733 (0x5d47…59cd), #12528833 (0xa47e…fbd7), #6949233 (0xa642…8b6c), #4186900 (0xeba3…b15b), #887779 (0xde96…ad0c), #6462406 (0x00c5…ca9a), #2577233 (0x5e08…6ad7), #992317 (0xde3b…219c), #1388866 (0x6a6a…bfb1), #2384988 (0xaaf0…eeb6), #3332213 (0xb55c…c5c6), #6386563 (0x6f1b…b352), #230399 (0xa781…9cdb), #3292001 (0xaf3f…f1ec), #7130515 (0x3a20…a635), #143 (0x1d10…3dce), #3993341 (0x08d2…7f0c), #2671209 (0xd44b…bd9e), #9538506 (0xe9bc…d34b), #1875746 (0x25d2…183e), #2609952 (0x0e13…ecb7), timed out: #1626254 (0xe940…5985)
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:52.844  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [Parachain] 🙌 Starting consensus session on top of parent 0xaf3f9f9a9328bd7ccc9b22a0efbbecd1e8d8856dac80c7e061da7b3187b0f1ec (#3292001)
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:52.846 ERROR tokio-runtime-worker runtime: [Parachain] panicked at /home/builder/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/cumulus-pallet-parachain-system-0.22.1/src/lib.rs:1240:4:
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]: assertion `left == right` failed: HRMP head mismatch
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:   left: 0x387b9a372b5759dd293dd006027224cac0e2c1d4763616516d1b1f7fb597ebc5
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:  right: 0x0000000000000000000000000000000000000000000000000000000000000000
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:52.847  WARN tokio-runtime-worker sp_state_machine::overlayed_changes::changeset: [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:52.847  WARN tokio-runtime-worker sp_state_machine::overlayed_changes::changeset: [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:52.847  WARN tokio-runtime-worker basic-authorship: [Parachain] ❗️ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]: WASM backtrace:
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]: error while executing at wasm backtrace:
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:     0: 0x9f72d8 - asset_hub_paseo_runtime.wasm!__rustc[4794b31dd7191200]::rust_begin_unwind
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:     1:   0x46cf - asset_hub_paseo_runtime.wasm!core::panicking::panic_fmt::hd534225921b41838
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:     2:  0x731b7 - asset_hub_paseo_runtime.wasm!core::panicking::assert_failed_inner::h91413972d8ac72a4
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:     3: 0x534b53 - asset_hub_paseo_runtime.wasm!core::panicking::assert_failed::hd01376200f4fb95a
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:     4: 0x534efb - asset_hub_paseo_runtime.wasm!cumulus_pallet_parachain_system::<impl cumulus_pallet_parachain_system::pallet::Pallet<T>>::check_hrmp_mcq_heads::h8bf9d420bd195111
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:     5: 0x5322c8 - asset_hub_paseo_runtime.wasm!cumulus_pallet_parachain_system::<impl cumulus_pallet_parachain_system::pallet::Pallet<T>>::enqueue_inbound_horizontal_messages::h17e04ec1e6aca284
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:     6: 0x3ae5c2 - asset_hub_paseo_runtime.wasm!cumulus_pallet_parachain_system::pallet::Pallet<T>::set_validation_data::{{closure}}::h7d33bf7b51280f5c
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:     7: 0x3a43a9 - asset_hub_paseo_runtime.wasm!<cumulus_pallet_parachain_system::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::{{closure}}::ha0ef810eb8c03d13
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:     8: 0x27700f - asset_hub_paseo_runtime.wasm!<asset_hub_paseo_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::h2f03ba5c7de16cc5
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:     9: 0x2027c8 - asset_hub_paseo_runtime.wasm!<asset_hub_paseo_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::h217bd9308ff1a606
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:    10: 0x1ca714 - asset_hub_paseo_runtime.wasm!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::do_apply_extrinsic::hcb42677bb66ef307
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:    11: 0x84d4f9 - asset_hub_paseo_runtime.wasm!BlockBuilder_apply_extrinsic. Dropping.
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:52.848 ERROR tokio-runtime-worker runtime: [Parachain] panicked at /home/builder/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/cumulus-pallet-parachain-system-0.22.1/src/lib.rs:329:41:
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]: Missing required set_validation_data inherent. This inherent must be
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:                                 present in every block. This error typically occurs when the set_validation_data
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:                                 execution failed and was rejected by the block builder. Check earlier log entries
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:                                 for the specific cause of the failure.
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:52.848 ERROR tokio-runtime-worker aura::cumulus: [Parachain] err=Error { inner: Proposing
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]: Caused by:
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:     0: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:        WASM backtrace:
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:        error while executing at wasm backtrace:
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:            0: 0x9f72d8 - asset_hub_paseo_runtime.wasm!__rustc[4794b31dd7191200]::rust_begin_unwind
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:            1:   0x46cf - asset_hub_paseo_runtime.wasm!core::panicking::panic_fmt::hd534225921b41838
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:            2:  0xc93fc - asset_hub_paseo_runtime.wasm!core::option::expect_failed::h445a22c5cbc9c22d
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:            3: 0x1cd6f7 - asset_hub_paseo_runtime.wasm!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34,TupleElement35,TupleElement36,TupleElement37,TupleElement38,TupleElement39,TupleElement40,TupleElement41,TupleElement42,TupleElement43,TupleElement44,TupleElement45,TupleElement46,TupleElement47,TupleElement48,TupleElement49,TupleElement50,TupleElement51,TupleElement52,TupleElement53,TupleElement54,TupleElement55) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::h20da6c456dd6fbe5
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:            4: 0x84dab4 - asset_hub_paseo_runtime.wasm!BlockBuilder_finalize_block
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:     1: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:        WASM backtrace:
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:        error while executing at wasm backtrace:
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:            0: 0x9f72d8 - asset_hub_paseo_runtime.wasm!__rustc[4794b31dd7191200]::rust_begin_unwind
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:            1:   0x46cf - asset_hub_paseo_runtime.wasm!core::panicking::panic_fmt::hd534225921b41838
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:            2:  0xc93fc - asset_hub_paseo_runtime.wasm!core::option::expect_failed::h445a22c5cbc9c22d
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:            3: 0x1cd6f7 - asset_hub_paseo_runtime.wasm!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34,TupleElement35,TupleElement36,TupleElement37,TupleElement38,TupleElement39,TupleElement40,TupleElement41,TupleElement42,TupleElement43,TupleElement44,TupleElement45,TupleElement46,TupleElement47,TupleElement48,TupleElement49,TupleElement50,TupleElement51,TupleElement52,TupleElement53,TupleElement54,TupleElement55) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::h20da6c456dd6fbe5
Oct 23 09:32:52 ahpaseo-collator polkadot-parachain[346832]:            4: 0x84dab4 - asset_hub_paseo_runtime.wasm!BlockBuilder_finalize_block }
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:54.452  INFO tokio-runtime-worker minimal-polkadot-node: [Parachain] Received imported block via RPC: #8737816 (0xb0b9…7d7f -> 0xa4d4…201b)
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:54.453  INFO tokio-runtime-worker cumulus_client_service: [Parachain] Update at relay chain block #8737816 (0xa4d4…201b) - backed: #230400 (0x2909…26a8), #2833045 (0x702c…63d0), #2689776 (0x0fe4…fe59), #2671210 (0x6f9e…8ed3), #12528834 (0xa8f8…1dc3), #3791955 (0xdcec…3905), #3478203 (0x5f2a…602d), #4186901 (0x7f80…0ba6), #7130516 (0xdc17…bd0d), #6549776 (0xf84d…ed96), #2384989 (0x8f25…fe25), #9188097 (0xdc3f…adc3), #3993342 (0xa413…dd09), #2609953 (0xd38e…9f8f), #2435787 (0xee29…5df3), #985751 (0xc9ca…d4c7), #5787360 (0x81d7…1001), #3332214 (0x9a14…63e6), #6386564 (0x4890…3053), #2518734 (0x80c5…dc0e), #6949234 (0x10bc…37c9), #3566076 (0x9121…6968), #992318 (0x9744…6b78), #2442474 (0x9743…3cf9), #887780 (0x5c55…91ab), #299507 (0x6480…905b)
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:54.464  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [Parachain] 🙌 Starting consensus session on top of parent 0xaf3f9f9a9328bd7ccc9b22a0efbbecd1e8d8856dac80c7e061da7b3187b0f1ec (#3292001)
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:54.466 ERROR tokio-runtime-worker runtime: [Parachain] panicked at /home/builder/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/cumulus-pallet-parachain-system-0.22.1/src/lib.rs:1240:4:
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]: assertion `left == right` failed: HRMP head mismatch
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:   left: 0x387b9a372b5759dd293dd006027224cac0e2c1d4763616516d1b1f7fb597ebc5
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:  right: 0x0000000000000000000000000000000000000000000000000000000000000000
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:54.467  WARN tokio-runtime-worker sp_state_machine::overlayed_changes::changeset: [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:54.467  WARN tokio-runtime-worker sp_state_machine::overlayed_changes::changeset: [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:54.467  WARN tokio-runtime-worker basic-authorship: [Parachain] ❗️ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]: WASM backtrace:
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]: error while executing at wasm backtrace:
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:     0: 0x9f72d8 - asset_hub_paseo_runtime.wasm!__rustc[4794b31dd7191200]::rust_begin_unwind
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:     1:   0x46cf - asset_hub_paseo_runtime.wasm!core::panicking::panic_fmt::hd534225921b41838
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:     2:  0x731b7 - asset_hub_paseo_runtime.wasm!core::panicking::assert_failed_inner::h91413972d8ac72a4
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:     3: 0x534b53 - asset_hub_paseo_runtime.wasm!core::panicking::assert_failed::hd01376200f4fb95a
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:     4: 0x534efb - asset_hub_paseo_runtime.wasm!cumulus_pallet_parachain_system::<impl cumulus_pallet_parachain_system::pallet::Pallet<T>>::check_hrmp_mcq_heads::h8bf9d420bd195111
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:     5: 0x5322c8 - asset_hub_paseo_runtime.wasm!cumulus_pallet_parachain_system::<impl cumulus_pallet_parachain_system::pallet::Pallet<T>>::enqueue_inbound_horizontal_messages::h17e04ec1e6aca284
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:     6: 0x3ae5c2 - asset_hub_paseo_runtime.wasm!cumulus_pallet_parachain_system::pallet::Pallet<T>::set_validation_data::{{closure}}::h7d33bf7b51280f5c
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:     7: 0x3a43a9 - asset_hub_paseo_runtime.wasm!<cumulus_pallet_parachain_system::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::{{closure}}::ha0ef810eb8c03d13
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:     8: 0x27700f - asset_hub_paseo_runtime.wasm!<asset_hub_paseo_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::h2f03ba5c7de16cc5
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:     9: 0x2027c8 - asset_hub_paseo_runtime.wasm!<asset_hub_paseo_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::h217bd9308ff1a606
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:    10: 0x1ca714 - asset_hub_paseo_runtime.wasm!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::do_apply_extrinsic::hcb42677bb66ef307
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:    11: 0x84d4f9 - asset_hub_paseo_runtime.wasm!BlockBuilder_apply_extrinsic. Dropping.
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:54.467 ERROR tokio-runtime-worker runtime: [Parachain] panicked at /home/builder/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/cumulus-pallet-parachain-system-0.22.1/src/lib.rs:329:41:
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]: Missing required set_validation_data inherent. This inherent must be
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:                                 present in every block. This error typically occurs when the set_validation_data
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:                                 execution failed and was rejected by the block builder. Check earlier log entries
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:                                 for the specific cause of the failure.
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:54.468 ERROR tokio-runtime-worker aura::cumulus: [Parachain] err=Error { inner: Proposing
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]: Caused by:
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:     0: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:        WASM backtrace:
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:        error while executing at wasm backtrace:
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:            0: 0x9f72d8 - asset_hub_paseo_runtime.wasm!__rustc[4794b31dd7191200]::rust_begin_unwind
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:            1:   0x46cf - asset_hub_paseo_runtime.wasm!core::panicking::panic_fmt::hd534225921b41838
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:            2:  0xc93fc - asset_hub_paseo_runtime.wasm!core::option::expect_failed::h445a22c5cbc9c22d
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:            3: 0x1cd6f7 - asset_hub_paseo_runtime.wasm!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34,TupleElement35,TupleElement36,TupleElement37,TupleElement38,TupleElement39,TupleElement40,TupleElement41,TupleElement42,TupleElement43,TupleElement44,TupleElement45,TupleElement46,TupleElement47,TupleElement48,TupleElement49,TupleElement50,TupleElement51,TupleElement52,TupleElement53,TupleElement54,TupleElement55) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::h20da6c456dd6fbe5
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:            4: 0x84dab4 - asset_hub_paseo_runtime.wasm!BlockBuilder_finalize_block
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:     1: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:        WASM backtrace:
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:        error while executing at wasm backtrace:
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:            0: 0x9f72d8 - asset_hub_paseo_runtime.wasm!__rustc[4794b31dd7191200]::rust_begin_unwind
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:            1:   0x46cf - asset_hub_paseo_runtime.wasm!core::panicking::panic_fmt::hd534225921b41838
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:            2:  0xc93fc - asset_hub_paseo_runtime.wasm!core::option::expect_failed::h445a22c5cbc9c22d
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:            3: 0x1cd6f7 - asset_hub_paseo_runtime.wasm!<(TupleElement0,TupleElement1,TupleElement2,TupleElement3,TupleElement4,TupleElement5,TupleElement6,TupleElement7,TupleElement8,TupleElement9,TupleElement10,TupleElement11,TupleElement12,TupleElement13,TupleElement14,TupleElement15,TupleElement16,TupleElement17,TupleElement18,TupleElement19,TupleElement20,TupleElement21,TupleElement22,TupleElement23,TupleElement24,TupleElement25,TupleElement26,TupleElement27,TupleElement28,TupleElement29,TupleElement30,TupleElement31,TupleElement32,TupleElement33,TupleElement34,TupleElement35,TupleElement36,TupleElement37,TupleElement38,TupleElement39,TupleElement40,TupleElement41,TupleElement42,TupleElement43,TupleElement44,TupleElement45,TupleElement46,TupleElement47,TupleElement48,TupleElement49,TupleElement50,TupleElement51,TupleElement52,TupleElement53,TupleElement54,TupleElement55) as frame_support::traits::hooks::OnFinalize<BlockNumber>>::on_finalize::h20da6c456dd6fbe5
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]:            4: 0x84dab4 - asset_hub_paseo_runtime.wasm!BlockBuilder_finalize_block }
Oct 23 09:32:54 ahpaseo-collator polkadot-parachain[346832]: 2025-10-23 09:32:54.949  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (16 peers), best: #3292003 (0x4c38…50e7), finalized #3291999 (0xc1d9…56be), ⬇ 8 B/s ⬆ 1.2kiB/s
O

When re-opening these channels, they were opened empty (no messages had been sent yet) hence the default value (`0x0000…`) remained for the `mqcHead` entry being asserted by the collator.
Meanwhile, the `mqcHead` value on Asset Hub remained unchanged: `0x387b9a372b5759dd293dd006027224cac0e2c1d4763616516d1b1f7fb597ebc5`.

Our initial guess was that we might be looking at a bug, since we expected `LastHrmpMqcHeads` to have been cleared on removal of the channels, or at least not compared while the channel was non-existent.

This potential bug doesn’t seem to be present in runtimes depending on older versions of the Polkadot SDK, as **Bifrost (2030)** had a channel open with Hydra at the time and wasn’t affected like Asset Hub and PAsset Hub.
We also didn’t manage to reproduce this issue with runtimes depending on older SDK versions.

We are not certain from which exact SDK version onward runtimes are affected.

## Fix

With the parachains not producing blocks, updating the state on Asset Hub and PAsset Hub wasn’t really an option.

We considered deploying a patched version of the runtime and having the collators execute a version of the code that would clean the relevant state key via [`ChainSpec::code_substitutes`](https://github.com/paritytech/polkadot-sdk/blob/31001cd5379c191aba605ac075419be9979d5742/substrate/client/chain-spec/src/lib.rs#L426).

The second option was encoding the matching values and pre-writing them in the relay chain state.

After thorough testing and review of the HRMP-related logic, we opted to execute the following calls:

- https://paseo.subscan.io/extrinsic/8741919-2
- https://paseo.subscan.io/extrinsic/8741882-2

These wrote the corresponding values into the relay state, allowing the assertion executed by the collator to pass and bringing block production back to both chains at **2025-10-23 18:19:42 CET**.