---
# System prepended metadata

title: 'Interchain Security: IBC Timeouts'

---

# Interchain Security: IBC Timeouts

This document will summarize how IBC timeouts can affect consumer chains in an Interchain Security environment.

- How to query that a client has expired?
```
interchain-security-pd q ibc client status 07-tendermint-0 --home ~/.isp
status: Expired
```

## Scenario 1: IBC Relayer is turned off longer than the timeout period

Check:
- [X] The timeout period is set by the trusting period in the Hermes config file.
- [X] The trusting period must be shorter than the unbonding period

In order to test this, we only need a single-validator environment with two chains connected to each other. We can test this in a non-Interchain Security setup first to simplify things.


### Test 1: Same trusting period and unbonding time on both chains

- Start both chains (a and b)

```yaml
---
# yamllint disable rule:line-length
all:
  vars:
    chain_home_clear: true
    chain_version: v7.0.0
    chain_create_validator: true
    ansible_user: root
    reboot: false
    pruning: nothing
    p2p_allow_duplicate_ip: true
    chain_gov_testing: true
    chain_voting_period: 60s
    api_enabled: true
    grpc_web_enabled: false
  children:
    node:
      hosts:
        ibc-test.stg.earthball.xyz:
          fast_sync: false
          chain_id: chain-a
          chain_moniker: chain-a
          node_user: chain-a
          chain_home: "{{node_user_home}}/.gaia"
          cosmovisor_service_name: cv-chain-a
          node_user: chain-a
          api_port: 26650
          rpc_port: 26651
          p2p_port: 26652
          grpc_port: 26653
          rpc_pprof_port: 26654
          config_proxy_app: 'tcp://127.0.0.1:26655'
        ibc-test-2.stg.earthball.xyz:
          fast_sync: false
          chain_id: chain-b
          chain_moniker: chain-b
          node_user: chain-b
          chain_home: "{{node_user_home}}/.gaia"
          cosmovisor_service_name: cv-chain-b
          node_user: chain-b
          api_port: 26660
          rpc_port: 26661
          p2p_port: 26662
          grpc_port: 26663
          rpc_pprof_port: 26664
          config_proxy_app: 'tcp://127.0.0.1:26665'
```

- Set the unbonding period to 2 minutes in both chains

Proposal to set the unbonding period to 2minutes:
```json
{
   "title":"Unbonding period - 2 minutes",
   "description":"Unbonding period update",
   "changes":[
      {
         "subspace":"staking",
         "key":"UnbondingTime",
         "value":"120000000000"
      }
   ],
   "deposit":"1uatom"
}
```

To submit and vote on the proposal:
```
gaiad tx gov  submit-proposal param-change unbonding-2m.json  --home ~/.gaia --from validator -y
gaiad tx gov vote 1 yes --from validator -y
```

- Set the trusting period to Hermes config to 1 minute (if it lets us) for both chains

Inventory file:
```
---
# yamllint disable rule:line-length
all:
  vars:
    chain_home_clear: true
    chain_version: v7.0.2
    hermes_version: v1.0.0
    ansible_user: root
  children:
    node:
      hosts:
        my-chain-1.dev.testnet.com:
          chain_id: my-chain-1
          chain_create_validator: true
          chain_airdrop: true
          chain_airdrop_accounts:
            - cosmos1r5v5srda7xfth3hn2s26txvrcrntldjumt8mhl  # relayer account
        my-chain-2.dev.testnet.com:
          chain_id: my-chain-2
          chain_create_validator: true
          chain_airdrop: true
          chain_airdrop_accounts:
            - cosmos1r5v5srda7xfth3hn2s26txvrcrntldjumt8mhl  # relayer account
    hermes:
      hosts:
        ibc-test.stg.earthball.xyz:
          hermes_relayer_mnemonics: true
          hermes_chains:
            chain-a:
              hermes_relayer_mnemonic: 'chaina-mnemonic.txt'
              hermes_port_name: transfer
              hermes_chain_rpc_url_schema: http
              hermes_chain_rpc_hostname: ibc-test.stg.earthball.xyz
              hermes_chain_rpc_port: 26651
              hermes_chain_grpc_url_schema: http
              hermes_chain_grpc_hostname: ibc-test.stg.earthball.xyz
              hermes_chain_grpc_port: 26653
              hermes_chain_websocket_url_schema: ws
              hermes_chain_rpc_timeout: '10s'
              hermes_chain_account_prefix: 'cosmos'
              hermes_chain_key_name: 'testkey'
              hermes_chain_store_prefix: 'ibc'
              hermes_chain_max_gas: 2000000
              hermes_chain_fee_granter: ''
              gas_price: "{ price = 0.001, denom = 'uatom' }"
              hermes_chain_gas_multiplier: 1.1
              hermes_chain_clock_drift: '5s'
              hermes_chain_trusting_period: '60seconds'
              hermes_chain_trust_threshold: "{ numerator = '1', denominator = '3' }"
            chain-b:
              hermes_relayer_mnemonic: 'chainb-mnemonic.txt'
              hermes_port_name: transfer
              hermes_chain_rpc_url_schema: http
              hermes_chain_rpc_hostname: ibc-test.stg.earthball.xyz
              hermes_chain_rpc_port: 26661
              hermes_chain_grpc_url_schema: http
              hermes_chain_grpc_hostname: ibc-test.stg.earthball.xyz
              hermes_chain_grpc_port: 26663
              hermes_chain_websocket_url_schema: ws
              hermes_chain_rpc_timeout: '10s'
              hermes_chain_account_prefix: 'cosmos'
              hermes_chain_key_name: 'testkey'
              hermes_chain_store_prefix: 'ibc'
              hermes_chain_max_gas: 2000000
              hermes_chain_fee_granter: ''
              gas_price: "{ price = 0.001, denom = 'uatom' }"
              hermes_chain_gas_multiplier: 1.1
              hermes_chain_clock_drift: '5s'
              hermes_chain_trusting_period: '60seconds'
              hermes_chain_trust_threshold: "{ numerator = '1', denominator = '3' }"
```

- Create a connection, client, and channel


```
hermes create connection --a-chain chain-a --b-chain chain-b
hermes create channel --a-chain chain-a --a-port transfer --b-port transfer --order unordered --a-connection connection-0
  
022-10-25T18:47:29.102257Z ERROR ThreadId(01) [chain-a -> chain-b:07-tendermint-0] client state is not valid: latest height is outside of trusting period! latest_height=0-152 network_timestmap=Timestamp(2022-10-25T18:47:24.020163694Z) consensus_state_timestamp=Timestamp(2022-10-25T18:45:38.622427593Z) elapsed=105.397736101s
2022-10-25T18:47:29.102328Z ERROR ThreadId(01) failed ChanOpenTry ChannelSide { chain: BaseChainHandle { chain_id: ChainId { id: "chain-b", version: 0 }, runtime_sender: Sender { .. } }, client_id: ClientId("07-tendermint-0"), connection_id: ConnectionId("connection-0"), port_id: PortId("transfer"), channel_id: None, version: None }: failed during an operation on client '07-tendermint-0' hosted by chain 'chain-b': client 07-tendermint-0 on chain id chain-b is expired or frozen: expired: time elapsed since last client update: 105.397736101s
2022-10-25T18:47:29.102353Z ERROR ThreadId(01) failed to open channel after 2 retries
ERROR Error after maximum retry of 2 and total delay of 3s: failed to finish channel handshake for Channel { ordering: Unordered, a_side: ChannelSide { chain: BaseChainHandle { chain_id: ChainId { id: "chain-a", version: 0 }, runtime_sender: Sender { .. } }, client_id: ClientId("07-tendermint-0"), connection_id: ConnectionId("connection-0"), port_id: PortId("transfer"), channel_id: Some(ChannelId("channel-0")), version: None }, b_side: ChannelSide { chain: BaseChainHandle { chain_id: ChainId { id: "chain-b", version: 0 }, runtime_sender: Sender { .. } }, client_id: ClientId("07-tendermint-0"), connection_id: ConnectionId("connection-0"), port_id: PortId("transfer"), channel_id: None, version: None }, connection_delay: 0ns }
```

It looks like the amount of time it took to create the channel pushed us over the 1m trusting period.

**Try setting unbonding time to 10 minutes and trusting period to 5m instead.**

Polling the client on `chain-a` results in an "expired" status.

We create a new client on both chains:
```
hermes create client --host-chain chain-a --reference-chain chain-b
2022-10-25T18:58:54.866241Z  INFO ThreadId(01) using default configuration from '/home/hermes/.hermes/config.toml'
2022-10-25T18:58:54.942770Z  INFO ThreadId(13) wait_for_block_commits: waiting for commit of tx hashes(s) 25BEE62CFD3FE93911D0AE5C67D0F969C7AEB06EEA6E08DBEA47F1033B3324E8 id=chain-a
SUCCESS CreateClient(
    CreateClient(
        Attributes {
            client_id: ClientId(
                "07-tendermint-1",
            ),
            client_type: Tendermint,
            consensus_height: Height {
                revision: 0,
                height: 338,
            },
        },
    ),
)

hermes create client --host-chain chain-b --reference-chain chain-a
2022-10-25T18:59:12.539103Z  INFO ThreadId(01) using default configuration from '/home/hermes/.hermes/config.toml'
2022-10-25T18:59:12.574193Z  INFO ThreadId(13) wait_for_block_commits: waiting for commit of tx hashes(s) F0208F9417E6E5FC5081FA28DDB7EE8CF626E812BBB3270DB43D9B602B5BEA82 id=chain-b
SUCCESS CreateClient(
    CreateClient(
        Attributes {
            client_id: ClientId(
                "07-tendermint-1",
            ),
            client_type: Tendermint,
            consensus_height: Height {
                revision: 0,
                height: 310,
            },
        },
    ),
)

```

Then we create a connection between those new `07-tendermint-1` clients:
```
hermes create connection --a-chain chain-a --a-client 07-tendermint-1 --b-client 07-tendermint-1
...
SUCCESS Connection {
    delay_period: 0ns,
    a_side: ConnectionSide {
        chain: BaseChainHandle {
            chain_id: ChainId {
                id: "chain-a",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-1",
        ),
        connection_id: Some(
            ConnectionId(
                "connection-1",
            ),
        ),
    },
    b_side: ConnectionSide {
        chain: BaseChainHandle {
            chain_id: ChainId {
                id: "chain-b",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-1",
        ),
        connection_id: Some(
            ConnectionId(
                "connection-1",
            ),
        ),
    },
}

```

Lastly, we create a channel on the new `connection-0`.

```
hermes create channel --a-chain chain-a --a-connection connection-0 --a-port transfer --b-port transfer

SUCCESS Channel {
    ordering: Unordered,
    a_side: ChannelSide {
        chain: BaseChainHandle {
            chain_id: ChainId {
                id: "chain-a",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-1",
        ),
        connection_id: ConnectionId(
            "connection-1",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: Some(
            ChannelId(
                "channel-3",
            ),
        ),
        version: None,
    },
    b_side: ChannelSide {
        chain: BaseChainHandle {
            chain_id: ChainId {
                id: "chain-b",
                version: 0,
            },
            runtime_sender: Sender { .. },
        },
        client_id: ClientId(
            "07-tendermint-1",
        ),
        connection_id: ConnectionId(
            "connection-1",
        ),
        port_id: PortId(
            "transfer",
        ),
        channel_id: Some(
            ChannelId(
                "channel-0",
            ),
        ),
        version: None,
    },
    connection_delay: 0ns,
}
```

- Start hermes
- Poll the client on chain a, make sure it's online/active

```
gaiad q ibc client status 07-tendermint-1
status: Active
```

- Stop hermes and start the timer
- Poll the client on chain a before the trusting period is over

It was active up until minute 4, and it expired before reaching minute 5. It's likely hermes had not sent a refresh message recently when we stopped the hermes service.

- Poll the client on chain a after the trusting period has elapsed
- Poll the client on chain a after the unbonding period has elapsed
- Start hermes again
- Poll the client on chain a

The client should be expired.

How do we bring back the client?
See this doc: https://hermes.informal.systems/commands/upgrade/test.html

#### Bringing back an expired client

- Submit an upgrade chain command from hermes

```
hermes tx upgrade-chain --reference-chain chain-a --host-chain chain-b --host-client 07-tendermint-0 --amount 1 --denom uatom --height-offset 60 --upgrade-name v7-Theta
```

- Query the resulting gov proposal on chain a

```
gaiad --node tcp://localhost:26651 query gov proposal 1
ontent:
  '@type': /ibc.core.client.v1.UpgradeProposal
  description: upgrade the chain software and unbonding period
  plan:
    height: "13751"
    info: ""
    name: plan
    time: "0001-01-01T00:00:00Z"
    upgraded_client_state: null
  title: proposal 0
  upgraded_client_state:
    '@type': /ibc.lightclients.tendermint.v1.ClientState
    allow_update_after_expiry: false
    allow_update_after_misbehaviour: false
    chain_id: chain-a
    frozen_height:
      revision_height: "0"
      revision_number: "0"
    latest_height:
      revision_height: "13752"
      revision_number: "0"
    max_clock_drift: 0s
    proof_specs:
    - inner_spec:
        child_order:
        - 0
        - 1
        child_size: 33
        empty_child: null
        hash: SHA256
        max_prefix_length: 12
        min_prefix_length: 4
      leaf_spec:
        hash: SHA256
        length: VAR_PROTO
        prefix: AA==
        prehash_key: NO_HASH
        prehash_value: SHA256
      max_depth: 0
      min_depth: 0
    - inner_spec:
        child_order:
        - 0
        - 1
        child_size: 32
        empty_child: null
        hash: SHA256
        max_prefix_length: 1
        min_prefix_length: 1
      leaf_spec:
        hash: SHA256
        length: VAR_PROTO
        prefix: AA==
        prehash_key: NO_HASH
        prehash_value: SHA256
      max_depth: 0
      min_depth: 0
    trust_level:
      denominator: "0"
      numerator: "0"
    trusting_period: 0s
    unbonding_period: 600s
    upgrade_path:
    - upgrade
    - upgradedIBCState
deposit_end_time: "2022-10-28T14:28:39.818469616Z"
final_tally_result:
  abstain: "0"
  "no": "0"
  no_with_veto: "0"
  "yes": "8000000000"
proposal_id: "4"
status: PROPOSAL_STATUS_PASSED
submit_time: "2022-10-26T14:28:39.818469616Z"
total_deposit:
- amount: "1"
  denom: uatom
voting_end_time: "2022-10-26T14:28:49.818469616Z"
voting_start_time: "2022-10-26T14:28:39.818469616Z"

```

- Vote on the resulting gov proposal on chain a

```
gaiad --node tcp://localhost:26651 tx gov vote 2 yes --from validator -y
```

Collect the upgrade height from the gov proposal and list it in the next command: 409

- Submit the upgrade client command from hermes 
```
hermes upgrade client --host-chain chain-b --client 07-tendermint-0 --upgrade-height 1578
```

The above workflow hasn't worked because we are not upgrading the binary.

We'll try submitting a proposal manually rather than using hermes to do it.

#### Manual IBC client upgrade

1. Create a new client to replace the expired one.

On the hermes machine:
```
hermes create client --host-chain chain-a --reference-chain chain-b
hermes create client --host-chain chain-b --reference-chain chain-a
```

2. Submit the gov proposal

```json
{
  "messages": [
    {
      "@type": "/ibc.core.client.v1.ClientUpdateProposal",
      "title": "Client revival",
      "description": "Update 07-tendermint-0 client",
      "subject_client_id": "07-tendermint-0",
      "substitute_client_id": "07-tendermint-1"
    }
  ],
  "metadata": "<metadata>",
  "deposit": "1uatom"
}
```

Proposal submission (the above file is not used with gaia v7):
```
gaiad tx gov submit-proposal update-client 07-tendermint-0 07-tendermint-1 --from validator --title "Replace expired client" --description "New client"
```

Vote yes on proposal and make sure it passes

Restart hermes
Submit and pass proposal on the other chain

Restart hermes

Test ibc transfers


### Test 2:  Different timeout periods on the chains

- Set the unbonding period to 10 minutes in chain a and 20 minutes in chain b
- Start both chains (a and b)
- Set the trusting period to Hermes config to 5 minutes in chain a and 10 minutes in chain b
- Create a connection and channel
- Start hermes
- Poll the client on chain a, make sure it's online/active
- Stop hermes and start the timer
- Poll the client on chain a before the trusting period is over in chain a
- Poll the client on chain a after the trusting period has elapsed in chain a
- Poll the client on chain a after the unbonding period has elapsed in chain a
- Start hermes again
- Poll the client on chain a
- Poll the client on chain b

The client should be expired.

Notes:

