# Interchain Security: IBC Timeouts
This document will summarize how IBC timeouts can affect consumer chains in an Interchain Security environment.
- How to query that a client has expired?
```
interchain-security-pd q ibc client status 07-tendermint-0 --home ~/.isp
status: Expired
```
## Scenario 1: IBC Relayer is turned off longer than the timeout period
Check:
- [X] The timeout period is set by the trusting period in the Hermes config file.
- [X] The trusting period must be shorter than the unbonding period
In order to test this, we only need a single-validator environment with two chains connected to each other. We can test this in a non-Interchain Security setup first to simplify things.
### Test 1: Same trusting period and unbonding time on both chains
- Start both chains (a and b)
```yaml
---
# yamllint disable rule:line-length
all:
vars:
chain_home_clear: true
chain_version: v7.0.0
chain_create_validator: true
ansible_user: root
reboot: false
pruning: nothing
p2p_allow_duplicate_ip: true
chain_gov_testing: true
chain_voting_period: 60s
api_enabled: true
grpc_web_enabled: false
children:
node:
hosts:
ibc-test.stg.earthball.xyz:
fast_sync: false
chain_id: chain-a
chain_moniker: chain-a
node_user: chain-a
chain_home: "{{node_user_home}}/.gaia"
cosmovisor_service_name: cv-chain-a
node_user: chain-a
api_port: 26650
rpc_port: 26651
p2p_port: 26652
grpc_port: 26653
rpc_pprof_port: 26654
config_proxy_app: 'tcp://127.0.0.1:26655'
ibc-test-2.stg.earthball.xyz:
fast_sync: false
chain_id: chain-b
chain_moniker: chain-b
node_user: chain-b
chain_home: "{{node_user_home}}/.gaia"
cosmovisor_service_name: cv-chain-b
node_user: chain-b
api_port: 26660
rpc_port: 26661
p2p_port: 26662
grpc_port: 26663
rpc_pprof_port: 26664
config_proxy_app: 'tcp://127.0.0.1:26665'
```
- Set the unbonding period to 2 minutes in both chains
Proposal to set the unbonding period to 2minutes:
```json
{
"title":"Unbonding period - 2 minutes",
"description":"Unbonding period update",
"changes":[
{
"subspace":"staking",
"key":"UnbondingTime",
"value":"120000000000"
}
],
"deposit":"1uatom"
}
```
To submit and vote on the proposal:
```
gaiad tx gov submit-proposal param-change unbonding-2m.json --home ~/.gaia --from validator -y
gaiad tx gov vote 1 yes --from validator -y
```
- Set the trusting period to Hermes config to 1 minute (if it lets us) for both chains
Inventory file:
```
---
# yamllint disable rule:line-length
all:
vars:
chain_home_clear: true
chain_version: v7.0.2
hermes_version: v1.0.0
ansible_user: root
children:
node:
hosts:
my-chain-1.dev.testnet.com:
chain_id: my-chain-1
chain_create_validator: true
chain_airdrop: true
chain_airdrop_accounts:
- cosmos1r5v5srda7xfth3hn2s26txvrcrntldjumt8mhl # relayer account
my-chain-2.dev.testnet.com:
chain_id: my-chain-2
chain_create_validator: true
chain_airdrop: true
chain_airdrop_accounts:
- cosmos1r5v5srda7xfth3hn2s26txvrcrntldjumt8mhl # relayer account
hermes:
hosts:
ibc-test.stg.earthball.xyz:
hermes_relayer_mnemonics: true
hermes_chains:
chain-a:
hermes_relayer_mnemonic: 'chaina-mnemonic.txt'
hermes_port_name: transfer
hermes_chain_rpc_url_schema: http
hermes_chain_rpc_hostname: ibc-test.stg.earthball.xyz
hermes_chain_rpc_port: 26651
hermes_chain_grpc_url_schema: http
hermes_chain_grpc_hostname: ibc-test.stg.earthball.xyz
hermes_chain_grpc_port: 26653
hermes_chain_websocket_url_schema: ws
hermes_chain_rpc_timeout: '10s'
hermes_chain_account_prefix: 'cosmos'
hermes_chain_key_name: 'testkey'
hermes_chain_store_prefix: 'ibc'
hermes_chain_max_gas: 2000000
hermes_chain_fee_granter: ''
gas_price: "{ price = 0.001, denom = 'uatom' }"
hermes_chain_gas_multiplier: 1.1
hermes_chain_clock_drift: '5s'
hermes_chain_trusting_period: '60seconds'
hermes_chain_trust_threshold: "{ numerator = '1', denominator = '3' }"
chain-b:
hermes_relayer_mnemonic: 'chainb-mnemonic.txt'
hermes_port_name: transfer
hermes_chain_rpc_url_schema: http
hermes_chain_rpc_hostname: ibc-test.stg.earthball.xyz
hermes_chain_rpc_port: 26661
hermes_chain_grpc_url_schema: http
hermes_chain_grpc_hostname: ibc-test.stg.earthball.xyz
hermes_chain_grpc_port: 26663
hermes_chain_websocket_url_schema: ws
hermes_chain_rpc_timeout: '10s'
hermes_chain_account_prefix: 'cosmos'
hermes_chain_key_name: 'testkey'
hermes_chain_store_prefix: 'ibc'
hermes_chain_max_gas: 2000000
hermes_chain_fee_granter: ''
gas_price: "{ price = 0.001, denom = 'uatom' }"
hermes_chain_gas_multiplier: 1.1
hermes_chain_clock_drift: '5s'
hermes_chain_trusting_period: '60seconds'
hermes_chain_trust_threshold: "{ numerator = '1', denominator = '3' }"
```
- Create a connection, client, and channel
```
hermes create connection --a-chain chain-a --b-chain chain-b
hermes create channel --a-chain chain-a --a-port transfer --b-port transfer --order unordered --a-connection connection-0
022-10-25T18:47:29.102257Z ERROR ThreadId(01) [chain-a -> chain-b:07-tendermint-0] client state is not valid: latest height is outside of trusting period! latest_height=0-152 network_timestmap=Timestamp(2022-10-25T18:47:24.020163694Z) consensus_state_timestamp=Timestamp(2022-10-25T18:45:38.622427593Z) elapsed=105.397736101s
2022-10-25T18:47:29.102328Z ERROR ThreadId(01) failed ChanOpenTry ChannelSide { chain: BaseChainHandle { chain_id: ChainId { id: "chain-b", version: 0 }, runtime_sender: Sender { .. } }, client_id: ClientId("07-tendermint-0"), connection_id: ConnectionId("connection-0"), port_id: PortId("transfer"), channel_id: None, version: None }: failed during an operation on client '07-tendermint-0' hosted by chain 'chain-b': client 07-tendermint-0 on chain id chain-b is expired or frozen: expired: time elapsed since last client update: 105.397736101s
2022-10-25T18:47:29.102353Z ERROR ThreadId(01) failed to open channel after 2 retries
ERROR Error after maximum retry of 2 and total delay of 3s: failed to finish channel handshake for Channel { ordering: Unordered, a_side: ChannelSide { chain: BaseChainHandle { chain_id: ChainId { id: "chain-a", version: 0 }, runtime_sender: Sender { .. } }, client_id: ClientId("07-tendermint-0"), connection_id: ConnectionId("connection-0"), port_id: PortId("transfer"), channel_id: Some(ChannelId("channel-0")), version: None }, b_side: ChannelSide { chain: BaseChainHandle { chain_id: ChainId { id: "chain-b", version: 0 }, runtime_sender: Sender { .. } }, client_id: ClientId("07-tendermint-0"), connection_id: ConnectionId("connection-0"), port_id: PortId("transfer"), channel_id: None, version: None }, connection_delay: 0ns }
```
It looks like the amount of time it took to create the channel pushed us over the 1m trusting period.
**Try setting unbonding time to 10 minutes and trusting period to 5m instead.**
Polling the client on `chain-a` results in an "expired" status.
We create a new client on both chains:
```
hermes create client --host-chain chain-a --reference-chain chain-b
2022-10-25T18:58:54.866241Z INFO ThreadId(01) using default configuration from '/home/hermes/.hermes/config.toml'
2022-10-25T18:58:54.942770Z INFO ThreadId(13) wait_for_block_commits: waiting for commit of tx hashes(s) 25BEE62CFD3FE93911D0AE5C67D0F969C7AEB06EEA6E08DBEA47F1033B3324E8 id=chain-a
SUCCESS CreateClient(
CreateClient(
Attributes {
client_id: ClientId(
"07-tendermint-1",
),
client_type: Tendermint,
consensus_height: Height {
revision: 0,
height: 338,
},
},
),
)
hermes create client --host-chain chain-b --reference-chain chain-a
2022-10-25T18:59:12.539103Z INFO ThreadId(01) using default configuration from '/home/hermes/.hermes/config.toml'
2022-10-25T18:59:12.574193Z INFO ThreadId(13) wait_for_block_commits: waiting for commit of tx hashes(s) F0208F9417E6E5FC5081FA28DDB7EE8CF626E812BBB3270DB43D9B602B5BEA82 id=chain-b
SUCCESS CreateClient(
CreateClient(
Attributes {
client_id: ClientId(
"07-tendermint-1",
),
client_type: Tendermint,
consensus_height: Height {
revision: 0,
height: 310,
},
},
),
)
```
Then we create a connection between those new `07-tendermint-1` clients:
```
hermes create connection --a-chain chain-a --a-client 07-tendermint-1 --b-client 07-tendermint-1
...
SUCCESS Connection {
delay_period: 0ns,
a_side: ConnectionSide {
chain: BaseChainHandle {
chain_id: ChainId {
id: "chain-a",
version: 0,
},
runtime_sender: Sender { .. },
},
client_id: ClientId(
"07-tendermint-1",
),
connection_id: Some(
ConnectionId(
"connection-1",
),
),
},
b_side: ConnectionSide {
chain: BaseChainHandle {
chain_id: ChainId {
id: "chain-b",
version: 0,
},
runtime_sender: Sender { .. },
},
client_id: ClientId(
"07-tendermint-1",
),
connection_id: Some(
ConnectionId(
"connection-1",
),
),
},
}
```
Lastly, we create a channel on the new `connection-0`.
```
hermes create channel --a-chain chain-a --a-connection connection-0 --a-port transfer --b-port transfer
SUCCESS Channel {
ordering: Unordered,
a_side: ChannelSide {
chain: BaseChainHandle {
chain_id: ChainId {
id: "chain-a",
version: 0,
},
runtime_sender: Sender { .. },
},
client_id: ClientId(
"07-tendermint-1",
),
connection_id: ConnectionId(
"connection-1",
),
port_id: PortId(
"transfer",
),
channel_id: Some(
ChannelId(
"channel-3",
),
),
version: None,
},
b_side: ChannelSide {
chain: BaseChainHandle {
chain_id: ChainId {
id: "chain-b",
version: 0,
},
runtime_sender: Sender { .. },
},
client_id: ClientId(
"07-tendermint-1",
),
connection_id: ConnectionId(
"connection-1",
),
port_id: PortId(
"transfer",
),
channel_id: Some(
ChannelId(
"channel-0",
),
),
version: None,
},
connection_delay: 0ns,
}
```
- Start hermes
- Poll the client on chain a, make sure it's online/active
```
gaiad q ibc client status 07-tendermint-1
status: Active
```
- Stop hermes and start the timer
- Poll the client on chain a before the trusting period is over
It was active up until minute 4, and it expired before reaching minute 5. It's likely hermes had not sent a refresh message recently when we stopped the hermes service.
- Poll the client on chain a after the trusting period has elapsed
- Poll the client on chain a after the unbonding period has elapsed
- Start hermes again
- Poll the client on chain a
The client should be expired.
How do we bring back the client?
See this doc: https://hermes.informal.systems/commands/upgrade/test.html
#### Bringing back an expired client
- Submit an upgrade chain command from hermes
```
hermes tx upgrade-chain --reference-chain chain-a --host-chain chain-b --host-client 07-tendermint-0 --amount 1 --denom uatom --height-offset 60 --upgrade-name v7-Theta
```
- Query the resulting gov proposal on chain a
```
gaiad --node tcp://localhost:26651 query gov proposal 1
ontent:
'@type': /ibc.core.client.v1.UpgradeProposal
description: upgrade the chain software and unbonding period
plan:
height: "13751"
info: ""
name: plan
time: "0001-01-01T00:00:00Z"
upgraded_client_state: null
title: proposal 0
upgraded_client_state:
'@type': /ibc.lightclients.tendermint.v1.ClientState
allow_update_after_expiry: false
allow_update_after_misbehaviour: false
chain_id: chain-a
frozen_height:
revision_height: "0"
revision_number: "0"
latest_height:
revision_height: "13752"
revision_number: "0"
max_clock_drift: 0s
proof_specs:
- inner_spec:
child_order:
- 0
- 1
child_size: 33
empty_child: null
hash: SHA256
max_prefix_length: 12
min_prefix_length: 4
leaf_spec:
hash: SHA256
length: VAR_PROTO
prefix: AA==
prehash_key: NO_HASH
prehash_value: SHA256
max_depth: 0
min_depth: 0
- inner_spec:
child_order:
- 0
- 1
child_size: 32
empty_child: null
hash: SHA256
max_prefix_length: 1
min_prefix_length: 1
leaf_spec:
hash: SHA256
length: VAR_PROTO
prefix: AA==
prehash_key: NO_HASH
prehash_value: SHA256
max_depth: 0
min_depth: 0
trust_level:
denominator: "0"
numerator: "0"
trusting_period: 0s
unbonding_period: 600s
upgrade_path:
- upgrade
- upgradedIBCState
deposit_end_time: "2022-10-28T14:28:39.818469616Z"
final_tally_result:
abstain: "0"
"no": "0"
no_with_veto: "0"
"yes": "8000000000"
proposal_id: "4"
status: PROPOSAL_STATUS_PASSED
submit_time: "2022-10-26T14:28:39.818469616Z"
total_deposit:
- amount: "1"
denom: uatom
voting_end_time: "2022-10-26T14:28:49.818469616Z"
voting_start_time: "2022-10-26T14:28:39.818469616Z"
```
- Vote on the resulting gov proposal on chain a
```
gaiad --node tcp://localhost:26651 tx gov vote 2 yes --from validator -y
```
Collect the upgrade height from the gov proposal and list it in the next command: 409
- Submit the upgrade client command from hermes
```
hermes upgrade client --host-chain chain-b --client 07-tendermint-0 --upgrade-height 1578
```
The above workflow hasn't worked because we are not upgrading the binary.
We'll try submitting a proposal manually rather than using hermes to do it.
#### Manual IBC client upgrade
1. Create a new client to replace the expired one.
On the hermes machine:
```
hermes create client --host-chain chain-a --reference-chain chain-b
hermes create client --host-chain chain-b --reference-chain chain-a
```
2. Submit the gov proposal
```json
{
"messages": [
{
"@type": "/ibc.core.client.v1.ClientUpdateProposal",
"title": "Client revival",
"description": "Update 07-tendermint-0 client",
"subject_client_id": "07-tendermint-0",
"substitute_client_id": "07-tendermint-1"
}
],
"metadata": "<metadata>",
"deposit": "1uatom"
}
```
Proposal submission (the above file is not used with gaia v7):
```
gaiad tx gov submit-proposal update-client 07-tendermint-0 07-tendermint-1 --from validator --title "Replace expired client" --description "New client"
```
Vote yes on proposal and make sure it passes
Restart hermes
Submit and pass proposal on the other chain
Restart hermes
Test ibc transfers
### Test 2: Different timeout periods on the chains
- Set the unbonding period to 10 minutes in chain a and 20 minutes in chain b
- Start both chains (a and b)
- Set the trusting period to Hermes config to 5 minutes in chain a and 10 minutes in chain b
- Create a connection and channel
- Start hermes
- Poll the client on chain a, make sure it's online/active
- Stop hermes and start the timer
- Poll the client on chain a before the trusting period is over in chain a
- Poll the client on chain a after the trusting period has elapsed in chain a
- Poll the client on chain a after the unbonding period has elapsed in chain a
- Start hermes again
- Poll the client on chain a
- Poll the client on chain b
The client should be expired.
Notes:
Sign in with Wallet
Connect another wallet