# Polkadot Release Analysis v0.9.38

:warning: ***The following report is not an exhaustive list of changes included in the release, but a set of changes that we felt deserved to be highlighted due to their impact on the Polkadot ecosystem builders.***

***Please do not stop reading the [release notes v0.9.38](https://github.com/paritytech/polkadot/releases/tag/v0.9.38). This report is complementary to the changelogs, not a replacement.***

***Highlights are categorized into High Impact, Medium Impact, and Low Impact for ease of navigation.***
***Also there is a section related to all note worthy changes related to tooling integration.***

# Summary

**XCM v3** is included in this release 🎉!

We highly encourage you to look at the actual PR [#4097](https://github.com/paritytech/polkadot/pull/4097) for details on XCM v3.

This is an important release since XCM v3 will land on each respective relay chain with their corresponding runtime upgrades.

Another important change is the inclusion of the `proof_size` field calculation during the pallet benchmarking process. This is basically making effective weights v2 for FRAME pallets and for any pallet migrating to use benchmarking v2 macros.

As usual, new features as well as some bug fixes are being added as part of the release.

## Runtimes built on release v0.9.38

- **Rococo v0.9.38**
- **Westend v0.9.38**
- **Kusama v0.9.38**
- **Polkadot v0.9.38**

# :hammer_and_wrench: Tooling Section

TxWrapper Core - [v5.0.0](https://github.com/paritytech/txwrapper-core/releases/tag/v5.0.0)
Substrate Sidecar API - [v14.4.0](https://github.com/paritytech/substrate-api-sidecar/releases/tag/v14.4.0)
Polkadot JS API - [v9.14.2](https://github.com/polkadot-js/api/releases/tag/v9.14.2)

❗️ As a result of the new XCM version, the `transaction_version` was bumped.

---

# ❗️ High Impact

### XCM v3
PR: https://github.com/paritytech/polkadot/pull/4097

This PR is merging most of the work related to the new version of XCM v3. As this change is massive and there are several improvements and design changes. The highlights for this new version can be simplified around Non-Fungibles (NFT) support, enabling XCM for bridges and multiple enhancements and new features regarding the programability of this messaging format like AssetLock, QueryResponseInfo, an implementable trait in XcmExecutor::Config for the `ExchangeAsset` instruction to use, etc.

In regards XCM v2 <> XCM v3 compatibility and new scenarios, we are working on this living document that we will continue to update:
- https://hackmd.io/@c_14hP4wTliLtuVeh0_OOA/HyjxRZY2s

*❗️ The documentation for this new verison can be found mostly in the PR description. It explains the changes in details and for some of them the corresponding PR is linked. Also, the migration from v2 to v3 is included under the section `Code migration considerations v2 -> v3`.*

**How does this impact the Polkadot builders?**

Very unlikely, but **in case your parachain runtime is still using v0 or v1 versions, you must upgrade at least to v2** since v0 and v1 will no longer be supported.

This PR as well as its companion introduces `transaction_version` breaking changes, meaning that applying these changes is enough for `transaction_version` to be bumped.

**Why is this change interesting for builders?**

This new version of XCM brings a lot of new possibilites for interoperatbility and remote programmability between different consensus systems. This is a major step forward to have a more powerful and performant ecosystem.

---

### XCM: `ExpectTransactStatus` instruction
PR: https://github.com/paritytech/polkadot/pull/6578

**Why is this important?**

With XCM v3 a new register has been added to `xcm-executor`, *Transact Status Register*, it stores the status of the most recent attempt to use `Transact` instruction. The *Transact Status Register* is set according to the result of dispatching the call. The possible values are:

#[derive(Clone, Eq, PartialEq, Encode, Decode, Debug, TypeInfo, MaxEncodedLen)]
pub enum MaybeErrorCode {
	Success,
	Error(BoundedVec<u8, MaxDispatchErrorLen>),
	TruncatedError(BoundedVec<u8, MaxDispatchErrorLen>),
}

This new instruction will compare the given status with the one held in the registry, returning an error, `XcmError::ExpectationFalse`, in case these two are not the same.

**Why is this change interesting for builders?**

Builders now can leverage this instruction to check if transact calls have been dispatched correctly.

**Does this change have a related companion PR?**
Cumulus companion PR: https://github.com/paritytech/cumulus/pull/2112

---

### [Fix] CountedMap::set now takes Counter into account
PR: https://github.com/paritytech/substrate/pull/13214

**Why is this change interesting for builders?**

A [CountedStorageMap](https://paritytech.github.io/substrate/master/frame_support/storage/types/struct.CountedStorageMap.html) is a wrapper around `StorageMap` and has a counter of type `StorageValue<Value=u32>`, which tracks how many items are present in map. When we insert or remove any item from map, this counter is updated.

At the moment, when we set the item in map, this counter is not being updated.

// Current Behaviour
#[pallet::storage]
pub type CountedMap<T> = CountedStorageMap<_, Blake2_128Concat, u8, u16>;

#[test]
fn counted_map_works() {
    new_test_ext().execute_with(|| {
        CountedMap::<Test>::set(3, Some(100));
        assert_eq!(CountedMap::<Test>::count(), 1);
        CountedMap::<Test>::set(3, None);
        // This will fail.
        assert_eq!(CountedMap::<Test>::count(), 0);
    })
}

This PR fixes this issue. Counter will be updated when we set an item.

**How does this impacts the Polkadot builders?**

This PR does not provide any migration. There is a separate [issue](https://github.com/paritytech/substrate/issues/13218) provided for this. If you are using `CountedStorageMap::set`, your stored count and the actual number of elements in the map will differ. It is recommended to call `initialize_counter`.

Related Issues: https://github.com/paritytech/substrate/issues/13212

---

### Add Proof Size to Weight Output
PR: https://github.com/paritytech/substrate/pull/11637

As stated in the PR: This updates the frame-benchmarking-cli to output estimated Proof-of-Validity sizes which are needed for chromatic weights.

Integration guide (this can happen at your own pace):

- If you have a [custom weight template](https://github.com/paritytech/substrate/blob/213e340742153e81cd7cef2fdc07e021a12629f2/.maintain/frame-weight-template.hbs#L61); update it according to the changes in this MR.
- Re-benchmark all your pallets to generate new weights.
- Ensure that the outputted proof sizes will not overflow your block limit for important tasks like democracy.
THIS IS IMPORTANT TO NOT STALL YOUR CHAIN
Runtime upgrades itself should still work since they run in on_initialize which is mandatory and thereby not weight limited.

One of the most remarkable features described in this change is the chance to influence the (benchmark, storage-key) level of the PoV calculation with the new `pov_mode` attribute. This new attribute can be set to `Meassured`, `MaxEncodedLen` or `Ignored`:

#[pov_mode = Measured]
my_benchmark_with_length_witness {

}
    
#[pov_mode = MaxEncodedLen {
	// Use MaxEncodedLen for all except Preimage::PreimageFor which uses Measured instead.
    Preimage::PreimageFor: Measured
    // ...
}]
service_task_fetched {}

---

# ⚠️ Medium Impact

### Scheduler is already at V4
PR: https://github.com/paritytech/substrate/pull/13105

**Why is this change interesting for builders?**

As a part of this [PR](https://github.com/paritytech/substrate/pull/11649), some refactoring and new changes were introduced in [Scheduler](https://paritytech.github.io/substrate/master/pallet_scheduler/pallet/) pallet along with migrations. However scheduler code version was not changed. This PR changes the version from `V3` to `V4`.

**How does this impacts the Polkadot builders?**

If you have configured and deployed scheduler genesis without this patch, you need to set scheduler code version manually from 3 to 4 and not to use `v3::MigrateToV4`.

---

### Fix the storage_size/state_getStorageSize RPC call
PR: https://github.com/paritytech/substrate/pull/13154

**Why is this important?**

This PR introduces a **⚠️breaking change** to `storage_size` RPC call. Allowing unbound execution time **only** when unsafe RPCs is enabled, otherwise a 30 second timeout would prevent this from running forever. If the `storage_size` call doesn't return a result within this time an error is returned.

**Does this change have a related companion PR?**
Polkadot companion PR [#6570](https://github.com/paritytech/polkadot/pull/6570)

---

### Correct arithmetical semantic of PerDispatchClass
PR: https://github.com/paritytech/substrate/pull/13194

**Why is this important?**

This PR introcues **⚠️breaking changes** to [`PerDispatchClass` implementation](https://github.com/paritytech/substrate/blob/a1d42aaa9f52531b27f882832cca9efe370ffcb6/frame/support/src/dispatch.rs#L424). The following public functions have been renamed:
- `add` renamed to `accrue`
- `checked_add` renamed to `checked_accrue`
- `sub` renamed to `reduce`

The following is introduced to `sp_weights::Weight`:

/// Construct [`Weight`] from the same weight for all parts.
pub const fn from_all(value: u64) -> Self {
    Self { ref_time: value, proof_size: value }
}

/// Reduce [`Weight`] by `amount` via saturating subtraction.
pub fn saturating_reduce(&mut self, amount: Self) {
    *self = self.saturating_sub(amount);
}

/// Try to increase `self` by `amount` via checked addition.
pub fn checked_accrue(&mut self, amount: Self) -> Option<()> {
    self.checked_add(&amount).map(|new_self| *self = new_self)
}

/// Try to reduce `self` by `amount` via checked subtraction.
pub fn checked_reduce(&mut self, amount: Self) -> Option<()> {
	self.checked_sub(&amount).map(|new_self| *self = new_self)
}

And implements `From<u64>` and `From<(u64, u64)>` for `Weight` in testing environments, meaning that the implementations are decorated with:
`#[cfg(any(test, feature = "std", feature = "runtime-benchmarks"))]`

**How does this impacts the Polkadot builders?**

In case you or your team are using the renamed functions of `PerDispatchClass` implementation, please apply the relevant changes to avoid errors.

---

### XCM: Add HRMP to SafeCallFilter
PR: https://github.com/paritytech/polkadot/pull/6606
Associated issue: https://github.com/paritytech/polkadot/issues/6602

This change is whitelisting the `HRMP` and `Registrar` pallets on the `SafeCallFilter` set on the relay chains. These new rules let the whitelisted calls from both pallets to be executed on the relay chains through the `Transact` XCM instruction.

In case you need more information about the `SafeCallFilter` you can refer to this PR https://github.com/paritytech/polkadot/pull/6280.

---

# ℹ️ Low Impact

### NIS should retain funds in reserve
PR: https://github.com/paritytech/substrate/pull/12928

**Why is this change interesting for builders?**

NIS pallet (formerly known as Gilt pallet) allows for accounts to auction for being frozen and receive open-ended inflation-protection in return. Learn about the pallet [here](https://paritytech.github.io/substrate/master/pallet_nis/index.html). This PR introduces the following changes:

- Rename `thaw` to `thaw_private`
- Add new dispatchable functions `communify` and `privatize` to create/burn fungible counterparts
- Add new dispatchable function `thaw_communal` to reduce or remove an outstanding receipt
- A new config type `ReserveId`

**How does this impacts the Polkadot builders?**

If you are using NIS pallet, you may have to add this new config type in runtime. For example:

parameter_types! {
    pub const NisReserveId: [u8; 8] = *b"py/nis  ";
}
    
impl pallet_nis::Config for Runtime {
    --- Other Config types ---
    type ReserveId = NisReserveId;
}

**Does this change have a related companion PR?**
Polkadot companion PR [#6569](https://github.com/paritytech/polkadot/pull/6569)

---

### [NFTs] Track item's metadata depositor
PR: https://github.com/paritytech/substrate/pull/13124

**Why is this change interesting for builders?**

In the previous [release-v0.9.37](https://forum.polkadot.network/t/polkadot-release-analysis-v0-9-37/1736#nfts-20-5), a new pallet ([pallet-nfts](https://paritytech.github.io/substrate/master/pallet_nfts/)) has been introduced with some advanced NFT features.

As a part of this PR, changes have been made regarding the reserved item's metadata deposit to track whether the deposit has been made by the owner or a different account.

---

### Enable treasury.spend by Root origin for Polkadot network before Gov2
PR: https://github.com/paritytech/polkadot/pull/6490

**Why is this important?**

There could be scenarios where it is not feasible for anyone to pay the proposal deposit for a treasury expense. This PR allows `Root` to call *treasury.spend* and avoid such a situation by changing the definition of `SpendOrigin` in the Polkadot runtime to:

type SpendOrigin = frame_system::EnsureRootWithSuccess<AccountId, RootSpendOriginMaxAmount>;

---

### Add WeightToFee and LengthToFee impls to transaction-payment Runtime API
PR: https://github.com/paritytech/substrate/pull/13110

**Why is this change interesting for builders?**

[Transaction Payment](https://docs.rs/pallet-transaction-payment/latest/pallet_transaction_payment/) pallet has two methods `length_to_fee` to compute the length portion of a fee and `weight_to_fee` to compute the unadjusted portion of the weight fee. At the moment, these functions are not available in the RuntimeApi so they can not be accessed. As a part of this PR, these two methods have been added to the runtime API.

**How does this impacts the Polkadot builders?**

If you are using `pallet_transaction_payment_rpc_runtime_api`, you will face below compilation error:

 error[E0046]: not all trait items implemented, missing: `query_weight_to_fee`, `query_length_to_fee`

You have to implement both functions in your runtime:

impl pallet_transaction_payment_rpc_runtime_api::TransactionPaymentApi<Block, Balance> for Runtime {
    
    --- Other Code ---
    
    fn query_weight_to_fee(weight: Weight) -> Balance {
        TransactionPayment::weight_to_fee(weight)
    }
    fn query_length_to_fee(length: u32) -> Balance {
        TransactionPayment::length_to_fee(length)
    }
}
    
impl pallet_transaction_payment_rpc_runtime_api::TransactionPaymentCallApi<Block, Balance, RuntimeCall> for Runtime {

    --- Other Code ---

    fn query_weight_to_fee(weight: Weight) -> Balance {
        TransactionPayment::weight_to_fee(weight)
     }
    fn query_length_to_fee(length: u32) -> Balance {
        TransactionPayment::length_to_fee(length)
     }


}
    

For more details, please refer to companion PRs.

**Does this change have a related companion PR?**
Polkadot companion PR [#6536](https://github.com/paritytech/polkadot/pull/6536)
Cumulus companion PR [#2073](https://github.com/paritytech/cumulus/pull/2073)

---

### migrate new benchmarking syntax from frame_support::benchmarking to frame_benchmarking::v2
PR: https://github.com/paritytech/substrate/pull/13235

This migrates all of the new proc-macro-based benchmarking code to live under `frame_benchmarking::v2`.

This is a non-breaking change and is part of a larger umbrella issue [#13132](https://github.com/paritytech/substrate/issues/13132) to bring about benchmarking v2.

This also migrates the old decl-macro-based benchmarking syntax from `frame_benchmarking` to `frame_benchmarking::v1`, but keeps a public export of `v1::*` at the root of frame_benchmarking, allowing existing benchmarking code to still compile without any changes. Sometime in the distant future we would remove this export when the v1 syntax is officially deprecated.

---

### v3::Junction supports small (32-byte max) "vecs"
PR: https://github.com/paritytech/polkadot/pull/6716

This change prevents that the junction `GeneralKey` does not exceed the 32 bytes and keeps the compatibility for chains using this junction with different XCM versions. In case it exceeds, it returns an error.