---
tags: testnet,release,upgrade
---

# Lagging Nodes in Release Testnet

The logs of several nodes in the release testnet have been flooded with IBC client update messages. This may be impairing their ability to sync with other nodes on the network.

* We are running two block explorers on this testnet:
* [BigDipper](https://explorer.theta-testnet.polypore.xyz/blocks), which syncs to a full (non-validator) node
* [Ping.Pub](https://pingpub.stg.earthball.xyz/theta-testnet-001/blocks), which queries a sentry node
* All of the Hypha nodes run on 8 CPU/16GB virtual machines.
* All of the Hypha nodes in the release testnet upgraded from `v7.1.0` to `v8.0.0-rc3` on Jan 20.
* This is a log from the wind validator, it is representative of all the validator nodes: https://paste.c-net.org/SidewaysIndicate
* This is a log from the sentry-01 node, it is representative of all the non-validator nodes: https://paste.c-net.org/PosingKaufman

We queried the `/block` RPC endpoint of multiple nodes, one after the other, and these were the reported heights and associated times:

2023-01-23 20:22:20.196183
earth: 14229654 | 2023-01-24T01:22:06.851357653Z
fire: 14229654 | 2023-01-24T01:22:06.851357653Z
wind: 14229654 | 2023-01-24T01:22:06.851357653Z
water: 14229655 | 2023-01-24T01:22:12.178527789Z
heart: 14229655 | 2023-01-24T01:22:12.178527789Z
sentry-1: 14218510 | 2023-01-23T08:45:26.716048831Z
sentry-2: 14221729 | 2023-01-23T13:33:22.784094638Z
sync-1: 14217098 | 2023-01-23T06:39:30.972263883Z
sync-2: 14218107 | 2023-01-23T08:09:26.037867991Z
seed-1: 14228330 | 2023-01-23T23:24:03.350079943Z
seed-2: 14228839 | 2023-01-24T00:09:29.146781224Z

We used this Python script to collect the above data:

import json
import requests
from datetime import datetime, timezone

nodes=[
    {'node':'earth','rpc':'https://rpc.earth.theta-testnet.polypore.xyz:443/'},
    {'node':'fire','rpc':'https://rpc.fire.theta-testnet.polypore.xyz:443/'},
    {'node':'wind','rpc':'https://rpc.wind.theta-testnet.polypore.xyz:443/'},
    {'node':'water','rpc':'https://rpc.water.theta-testnet.polypore.xyz:443/'},
    {'node':'heart','rpc':'http://heart.stg.earthball.xyz:26657/'},
    {'node':'sentry-1','rpc':'https://rpc.sentry-01.theta-testnet.polypore.xyz:443/'},
    {'node':'sentry-2','rpc':'https://rpc.sentry-02.theta-testnet.polypore.xyz:443/'},
    {'node':'sync-1','rpc':'https://rpc.state-sync-01.theta-testnet.polypore.xyz:443/'},
    {'node':'sync-2','rpc':'https://rpc.state-sync-02.theta-testnet.polypore.xyz:443/'},
    {'node':'seed-1','rpc':'https://rpc.seed-01.theta-testnet.polypore.xyz:443/'},
    {'node':'seed-2','rpc':'https://rpc.seed-02.theta-testnet.polypore.xyz:443/'}
]

block_endpoint='block'
for node in nodes:
    node['result'] = requests.get(node['rpc']+block_endpoint).json()

print(datetime.now())
for node in nodes:
    node['height'] = node['result']['result']['block']['header']['height']
    node['time'] = node['result']['result']['block']['header']['time']
    print(f'{node["node"]}: {node["height"]} | {node["time"]}')

## Mitigation Options

* Set `recheck = false` under the mempool settings in config.toml: Not advisable
* Update the IBC client through Hermes: Unsuccessful
* Flush the mempool (set `unsafe = true` under the rpc settings in config.toml and use the RPC endpoint: `curl <node_address>:26657/unsafe_flush_mempool`): Unsuccessful

## Updates

2023-01-25

### Config Files

Genesis file (92MB download): https://github.com/cosmos/testnets/raw/master/public/genesis.json.gz

* (V) earth: [app.toml](https://paste.c-net.org/DariaClarke), [config.toml](https://paste.c-net.org/TrinketsAlcazar)
* (V) wind [app.toml](https://paste.c-net.org/MantleLocated), [config.toml](https://paste.c-net.org/SandalsWaste)
* sentry-01 [app.toml](https://paste.c-net.org/TrojanShredded), [config.toml](https://paste.c-net.org/SuddenlyPinata)
* state-sync-01 [app.toml](https://paste.c-net.org/CorinneInitial), [config.toml](https://paste.c-net.org/FriesAustin)
* hermes [config.toml](https://paste.c-net.org/ElevateSafford)