---
tags: game-of-chains
---
# Consumer Chains Validator Set Updates
Status:
- `2022-11-14 2:40 UTC`
- The `apollo` consumer chain is not receiving validator set updates from the provider chain due to a VSC packet not being handled properly.
- The `sputnik` consumer chain seems to be receiving validator set updates from the provider chain.
Provider consensus monitor: https://provider-consensus.goc.earthball.xyz/
Sputnik consensus monitor: http://sputnik-consensus.goc.earthball.xyz/
Apollo consensus monitor: http://apollo-consensus.goc.earthball.xyz/
**Latest update at the top:**
### 2022-Nov-14
---
We try clearing the IBC packets in the apollo relayer:
```
hermes clear packets --chain provider --port provider --channel channel-2
2022-11-14T13:42:55.648345Z INFO ThreadId(01) using default configuration from '/home/hermes/.hermes/config.toml'
2022-11-14T13:42:56.845441Z INFO ThreadId(01) PacketRecvCmd{src_chain=provider src_port=provider src_channel=channel-2 dst_chain=apollo}: unreceived packets found: 260
2022-11-14T13:43:11.692351Z INFO ThreadId(01) PacketRecvCmd{src_chain=provider src_port=provider src_channel=channel-2 dst_chain=apollo}: pulled packet data for 0 events; events_total=260 events_left=210
2022-11-14T13:43:26.141233Z INFO ThreadId(01) PacketRecvCmd{src_chain=provider src_port=provider src_channel=channel-2 dst_chain=apollo}: pulled packet data for 0 events; events_total=260 events_left=160
2022-11-14T13:43:40.682975Z INFO ThreadId(01) PacketRecvCmd{src_chain=provider src_port=provider src_channel=channel-2 dst_chain=apollo}: pulled packet data for 0 events; events_total=260 events_left=110
2022-11-14T13:43:55.130085Z INFO ThreadId(01) PacketRecvCmd{src_chain=provider src_port=provider src_channel=channel-2 dst_chain=apollo}: pulled packet data for 0 events; events_total=260 events_left=60
2022-11-14T13:44:09.622661Z INFO ThreadId(01) PacketRecvCmd{src_chain=provider src_port=provider src_channel=channel-2 dst_chain=apollo}: pulled packet data for 0 events; events_total=260 events_left=10
2022-11-14T13:44:12.522770Z INFO ThreadId(01) PacketRecvCmd{src_chain=provider src_port=provider src_channel=channel-2 dst_chain=apollo}: pulled packet data for 0 events; events_total=260 events_left=0
SUCCESS []
```
That did not fix the out-of-sequence errors:
```
Nov 14 08:48:08 apollo-relayer hermes[79283]: 2022-11-14T13:48:08.976018Z ERROR ThreadId(84) packet_cmd{src_chain=provider src_port=provider src_channel=channel-2 dst_chain=apollo}: will retry: schedule execution encountered error: link failed with underlying error: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 1: receive packet verification failed: packet sequence ≠ next receive sequence (654 ≠ 393): packet sequence is out of order", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "50842"} }
Nov 14 08:48:08 apollo-relayer hermes[79283]: 2022-11-14T13:48:08.976060Z WARN ThreadId(84) packet_cmd{src_chain=provider src_port=provider src_channel=channel-2 dst_chain=apollo}: task encountered ignorable error: link error: link failed with underlying error: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 1: receive packet verification failed: packet sequence ≠ next receive sequence (654 ≠ 393): packet sequence is out of order", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "50842"} }
```
---
### 2022-Nov-13
---
We restarted the following services without success:
- `provider` chain endpoint for the relayer
- `apollo` chain endpoint for the relayer
- Hermes service in `provider`<->`apollo` relayer machine
It looks like this is where the packets stopped being relayed:
```
Nov 12 12:01:15 apollo-relayer hermes[3309]: 2022-11-12T17:01:15.473077Z ERROR ThreadId(23) send_tx_with_account_sequence_retry{id=apollo}:estimate_gas: failed to simulate tx. propagating error to caller: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 1: receive packet verification failed: packet sequence ≠ next receive sequence (394 ≠ 393): packet sequence is out of order", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "19235"} }
Nov 12 12:01:15 apollo-relayer hermes[3309]: 2022-11-12T17:01:15.473204Z ERROR ThreadId(67) packet_cmd{src_chain=provider src_port=provider src_channel=channel-2 dst_chain=apollo}: will retry: schedule execution encountered error: link failed with underlying error: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 1: receive packet verification failed: packet sequence ≠ next receive sequence (394 ≠ 393): packet sequence is out of order", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "19235"} }
Nov 12 12:01:15 apollo-relayer hermes[3309]: 2022-11-12T17:01:15.473229Z WARN ThreadId(67) packet_cmd{src_chain=provider src_port=provider src_channel=channel-2 dst_chain=apollo}: task encountered ignorable error: link error: link failed with underlying error: gRPC call failed with status: status: Unknown, message: "failed to execute message; message index: 1: receive packet verification failed: packet sequence ≠ next receive sequence (394 ≠ 393): packet sequence is out of order", details: [], metadata: MetadataMap { headers: {"content-type": "application/grpc", "x-cosmos-block-height": "19235"} }
```
We are seeing the following errors in the hermes log shortly after starting the service:
```log=
Nov 13 21:25:39 apollo-relayer hermes[79283]: 2022-11-14T02:25:39.928279Z INFO ThreadId(01) Hermes has started
Nov 13 21:25:43 apollo-relayer hermes[79283]: 2022-11-14T02:25:43.796261Z WARN ThreadId(83) DetectMisbehaviorWorker{client=07-tendermint-1 src_chain=apollo dst_chain=provider}: [apollo -> provider:07-tendermint-1] misbehaviour checking is being disabled: MisbehaviourExitSubdetail { reason: "could not extract header from update client event cs_h: 07-tendermint-1(0-42650) emitted by chain ChainId { id: \"provider\", version: 0 }" }
Nov 13 21:25:47 apollo-relayer hermes[79283]: 2022-11-14T02:25:47.497160Z WARN ThreadId(83) DetectMisbehaviorWorker{client=07-tendermint-1 src_chain=apollo dst_chain=provider}: [apollo -> provider:07-tendermint-1] misbehaviour checking is being disabled: MisbehaviourExitSubdetail { reason: "could not extract header from update client event cs_h: 07-tendermint-1(0-42769) emitted by chain ChainId { id: \"provider\", version: 0 }" }
Nov 13 21:26:12 apollo-relayer hermes[79283]: 2022-11-14T02:26:12.395398Z INFO ThreadId(84) packet_cmd{src_chain=provider src_port=provider src_channel=channel-2 dst_chain=apollo}: pulled packet data for 0 events; events_total=248 events_left=198
Nov 13 21:26:33 apollo-relayer hermes[79283]: 2022-11-14T02:26:33.733641Z INFO ThreadId(84) packet_cmd{src_chain=provider src_port=provider src_channel=channel-2 dst_chain=apollo}: pulled packet data for 0 events; events_total=248 events_left=148
Nov 13 21:26:37 apollo-relayer hermes[79283]: 2022-11-14T02:26:37.191107Z WARN ThreadId(26) DetectMisbehaviorWorker{client=07-tendermint-0 src_chain=provider dst_chain=apollo}: [provider -> apollo:07-tendermint-0] misbehaviour checking is being disabled: MisbehaviourExitSubdetail { reason: "could not extract header from update client event cs_h: 07-tendermint-0(0-96029) emitted by chain ChainId { id: \"apollo\", version: 0 }" }
```
And this one shows up regularly:
```log=
Nov 13 21:20:31 apollo-relayer hermes[72395]: 2022-11-14T02:20:31.409503Z ERROR ThreadId(23) send_tx_with_account_sequence_retry{id=apollo}: broadcast_tx_sync: Response { code: Err(22), data: Data([]), log: Log("packet messages are redundant"), hash: transaction::Hash(ECF974DB85D2051CC39675643C28CBB36FC36E90DD34D040D761EB85909BEA82) }: diagnostic: unknown TX sync response error: 22
```
Our Hermes config:
```toml=
[global]
log_level = 'info'
[mode]
[mode.clients]
enabled = true
refresh = true
misbehaviour = true
[mode.connections]
enabled = false
[mode.channels]
enabled = true
[mode.packets]
enabled = true
clear_interval = 100
clear_on_start = true
tx_confirmation = true
[rest]
enabled = true
host = '127.0.0.1'
port = 3000
[telemetry]
enabled = true
host = '127.0.0.1'
port = 3001
[[chains]]
id = 'apollo'
rpc_addr = 'http://apollo-backup.goc.earthball.xyz:26657'
grpc_addr = 'http://apollo-backup.goc.earthball.xyz:9090'
websocket_addr = 'ws://apollo-backup.goc.earthball.xyz:26657/websocket'
rpc_timeout = '10s'
account_prefix = 'cosmos'
key_name = 'testkey'
store_prefix = 'ibc'
max_gas = 2000000
fee_granter = ''
gas_price = { price = 0.001, denom = 'upol' }
gas_multiplier = 1.1
clock_drift = '30s'
trusting_period = '2days'
trust_threshold = { numerator = '1', denominator = '3' }
[[chains]]
id = 'provider'
rpc_addr = 'http://provider-sentry-01.goc.earthball.xyz:27001'
grpc_addr = 'http://provider-sentry-01.goc.earthball.xyz:27003'
websocket_addr = 'ws://provider-sentry-01.goc.earthball.xyz:27001/websocket'
rpc_timeout = '10s'
account_prefix = 'cosmos'
key_name = 'testkey'
store_prefix = 'ibc'
max_gas = 2000000
fee_granter = ''
gas_price = { price = 0.001, denom = 'uprov' }
gas_multiplier = 1.1
clock_drift = '30s'
trusting_period = '2days'
trust_threshold = { numerator = '1', denominator = '3' }
```
---
Claimens from Crypto Crew expects the `07-tendermint-1` client in the provider chain will expire around Nov 14 at 16:00 UTC if relayers aren't able to relay packet 399.
https://discord.com/channels/669268347736686612/1034957556038254612/1041303299246264320
They want to know if the coordinator relayer is in some weird state, but they think it could be any other relayer out there too.
---
### 2022-Nov-12
---
Claimens from Crypto Crew Validators reported VSC packets stuck between `provider` and `apollo`:
https://discord.com/channels/669268347736686612/1034957556038254612/1041039228810829854)
```log=
ThreadId(01) fetched channel from source chain: IdentifiedChannelEnd { port_id: PortId("consumer"), channel_id: ChannelId("channel-0"), channel_end: ChannelEnd { state: Open, ordering: Ordered, remote: Counterparty { port_id: PortId("provider"), channel_id: Some(ChannelId("channel-2")) }, connection_hops: [ConnectionId("connection-0")], version: Version("\n-cosmos17xpfvakm2amg962yls6f84z3kell8c5lserqta\u{12}\u{1}1") } } chain=apollo
SUCCESS Summary {
src: PendingPackets {
unreceived_packets: [],
unreceived_acks: [],
},
dst: PendingPackets {
unreceived_packets: [
Sequence(
393,
),
Sequence(
394,
),
Sequence(
395,
),
Sequence(
396,
),
Sequence(
397,
),
Sequence(
398,
),
Sequence(
399,
),
],
unreceived_acks: [],
},
}
```
---
### 2022-Nov-11
---
People started reporting validators not being synced properly in the Sputnik chain.
https://discord.com/channels/669268347736686612/1040649755912589343/1040653174408753282
---
Sign in with Wallet
Connect another wallet