### TLDR;

While producing block number 2 Gossamer produced and sent the correct parachain inherent with the correct parent block hash but the runtime returned extrinsic was different resulting in a wrong parent block hash causing the `mandatory dispatch error`

### The problem

Gossamer was facing the following problem while trying to create a block in a given slot:

failed to handle slot 281601762: cannot build inherents: error applying inherent: transaction validity error: mandatory dispatch error	pkg=babe

This error comes from the runtime and it is related to apply inherents. Inherents are a piece of information that the client can append to a block and it behaves like extrinsics but different from extrinsics they are not signed.

The point of failure was this source line https://github.com/paritytech/substrate/blob/08f585742c63694f80bd8be53f8f5349d1af6925/frame/executive/src/lib.rs#L593. Now inherents are `mandatory` (since `v0.9.34`) extrinsics which means if the runtime fail to apply them the block build process fails entirely.

Basically, I didn't have much information about what inherently was failing, then I started to debug the runtime interaction with Gossamer through the wasmer logs.

Here is the full log output: https://gist.github.com/EclesioMeloJunior/d7bace9839ad9bd1cc11898a7d1ca020

Here is a short version with some explanation:
https://gist.github.com/EclesioMeloJunior/ebc2895ae9252695dac0e4c17dc890d5

So, gossamer was able to produce block 1 but when it try to produce block 2 in the next available slot it was failing with

transaction validity error: mandatory dispatch error	pkg=babe

### BlockBuilder_inherent_extrinsics

This is a mandatory call that the client should execute, so the client creates the inherents marshal them and calls the runtime function `BlockBuilder_inherent_extrinsics` this function gets the provided inherents execute some logic/validation and return to the client extrinsics to be applied against the runtime again

### BlockBuilder_apply_extrinsic

Once we have the extrinsics the client calls `BlockBuilder_apply_extrinsic` for each extrinsic it gots from the previous step, this time it will calls another layer of busines logic.

For example we have the timestamp inherent, in the first step it check if the provided timestamp is correctly setted and is greater the previous timestamp stored and then it returns a call to the `set` method, this call will be executed in the next step where it checks if the timestamp and slot number matches.

https://github.com/paritytech/substrate/blob/1cc97dd30537997ff4c9827beb2ef0cac9c49063/frame/timestamp/src/lib.rs#L228

### The debug output

In the `paras_inherent` pallet the first step will call the `create_inherent` method.
https://github.com/paritytech/polkadot/blob/ba42b9ce51d25bdaf52d2c61e0763a6e3da50d25/runtime/parachains/src/paras_inherent/mod.rs#L210C2-L210C2

This method executes some business logic (which I am not full awere of) but one check is quite important, ensure the providede parent hash is the same as the runtime known parent hash, and particularly in the case of the inner function `enter_inner` if this check fails it will return a error.

So let's jump to the debug logs and the interaction between gossamer and runtime:

After build block 1 gossamer tries to build block 2 and it starts by calling the `BlockBuilder_inherent_extrinsics` which does:

INFO     built block 1 with hash 0x77d2ca4bd6c9fb35113aabe5b28f3da4cfc07b38233cabddda7f724f90e8e1d5, state root 0xd77aa7ed02b252efdd0768b56c68dd8ff6d4a4ad86adf869a01211793e0bdd9f, epoch 0 and slot 281601761

...

DEBUG    target=runtime::inclusion-inherent message=[create_inherent_inner] bitfields.len(): 0, backed_candidates.len(): 0, disputes.len() 0	ext_logging_log_version_1

DEBUG    key: 0x26aa394eea5630e07c48ae0c9558cef78a42f33323cb5ced3b44dd825fda9fcc	ext_storage_get_version_1

DEBUG    value: 0x77d2ca4bd6c9fb35113aabe5b28f3da4cfc07b38233cabddda7f724f90e8e1d5	ext_storage_get_version_1

The last two logs lines refeer to the runtimes `frame::system::parentHash` key that stores the latest block the runtime knows and should be the parent of the block which is being build. Then we can see that the `enter_inner` method was called and it is using the right parent hash

DEBUG    target=runtime::inclusion-inherent message=[enter_inner] parent_header=0x77d2ca4bd6c9fb35113aabe5b28f3da4cfc07b38233cabddda7f724f90e8e1d5 bitfields.len(): 0, backed_candidates.len(): 0, disputes.len(): 0	ext_logging_log_version_1

Now we go to next step which is: given the extrinsics from the runtime let's execute them:

DEBUG    target=runtime::inclusion-inherent message=[enter_inner] parent_header=0xfb0cf820badbcf6451effc8b6222c7564133d834f65a60e1d664a4d73fc248d4 bitfields.len(): 0, backed_candidates.len(): 0, disputes.len(): 0	ext_logging_log_version_1

But now, look at the parent header field logged, it is not the same as our parent head hash, it is different and this is triggering the validation check which fails to apply the inherent and since all inherents are mandatory it fails with: `transaction validity error: mandatory dispatch error`.