---
# System prepended metadata

title: Week 16

---

# Week 16

This time, I have benchamarked FCR for a day.

#### Slot-form latency (slots) over time
* median / p95
![Screenshot 2025-10-01 at 2.31.04 AM](https://hackmd.io/_uploads/rkxizSaFhxl.png)
![Screenshot 2025-10-01 at 2.31.32 AM](https://hackmd.io/_uploads/Hkd4BaFhee.png)

* confirmations per minute
![Screenshot 2025-10-01 at 2.31.53 AM](https://hackmd.io/_uploads/r13rSpt2xg.png)
* breakdown by delay buckets (0/1/2/ge3)
![Screenshot 2025-10-01 at 2.33.16 AM](https://hackmd.io/_uploads/BJkjH6theg.png)

For P50, we get : 
* Consistently 0.50-0.55 slots, which means most blocks confirm within the same slot or immediately next slot
* Brief increase to 0.65 slots during 16:00-18:00

For P95, I see : 
* Consistently ~1.00 slot, 95% of blocks confirm within 1 slot
* Brief increase to 2.00-2.50 slots during 16:00-18:00

For Confirmations Per Minute, I see : 
*  Regular sawtooth spikes from 0.05 to 0.18-0.20
*  ~1 confirmation per 12-second slot 

And when breaking down by delay buckets : 
* 0-slot delays: Regular spikes peaking at 0.10-0.12 (same-slot confirmation)
* 1-slot delays: Regular spikes peaking at 0.08-0.10 (next-slot confirmation)
* 2-slot delays: Almost flat at 0.00 (rare tail cases)
* 3+ slot delays: Completely flat at 0.00 (zero occurrences)

#### Head→confirmed gap (slots) over time
* median / p95
![Screenshot 2025-10-01 at 2.33.48 AM](https://hackmd.io/_uploads/SyJaBathle.png)
![Screenshot 2025-10-01 at 2.34.03 AM](https://hackmd.io/_uploads/HJ1CH6Fngx.png)

* outlier share (>8 slots)
![Screenshot 2025-10-01 at 2.34.26 AM](https://hackmd.io/_uploads/rkUyUpF2ex.png)

For  Median Head→Confirmed Gap (P50):
* In the initial state, a high gap of ~24-25 slots at startup (22:00)
* During transition state, a rapid decay to <5 slots by 02:00, ~0 slots by 04:00-06:00
* And in steady state, consistently ~0 slots for 14+ hours (06:00-20:00)

For P95 Head→Confirmed Gap : 
* Initial State: Even higher gap of ~32-33 slots at startup
* Transition Phase: Slower decay, <5 slots by 04:00, ~0 slots by 06:00-08:00
* Steady State: Consistently ~0 slots for 12+ hours (08:00-20:00)

And, Outlier Share (>8 Slots) : 
* Initial Burst: High outlier share (~0.015-0.018) during startup
* Quick Convergence: Drops to 0.00 within first hour
* Steady State: Zero outliers for most of 24-hour period
* Minor Spike: Brief spike to ~0.005 around 17:00-18:00 (correlates with previous events)

#### Restarts over time (with zero-line)
* rates per reason (stale / reorg)
![Screenshot 2025-10-01 at 2.34.44 AM](https://hackmd.io/_uploads/H1OlUTK2lg.png)

* cumulative 24h (stacked area)

![Screenshot 2025-10-01 at 2.35.01 AM](https://hackmd.io/_uploads/BJezUaF3eg.png)

For Rates per Reason (Stale / Reorg)
Stale Restarts:
* Initial Burst: High rate (~0.05-0.06) at 22:00 (startup)
* Quick Drop: Falls to 0.00 by 23:00
* Stable Period: Remains at 0.00 for ~21 hours
Reorg Restarts:
* Zero Activity: No visible line (confirmed by OR vector(0))
* Perfect Record: Zero reorg restarts throughout 22-hour period

For Cumulative 24h (Stacked Area)
Stale Restarts:
* Total Count: ~31.60 cumulative restarts
* Timing: All occurred in first hour (22:00-23:00)
* Stability: Flat line after initial hour
Reorg Restarts:
* Zero Cumulative: No reorg restarts accumulated
* Perfect Stability: Confirmed blocks never reorged


#### Confirmed reorgs and rollbacks (likely zero; zero-line included)
* confirmed-block reorg rate
![Screenshot 2025-10-01 at 2.35.32 AM](https://hackmd.io/_uploads/B1u7LpK2ll.png)

* P50 reorg size
![Screenshot 2025-10-01 at 2.40.46 AM](https://hackmd.io/_uploads/rkbwv6K3lx.png)

* rollbacks: count and size
![Screenshot 2025-10-01 at 2.36.57 AM](https://hackmd.io/_uploads/H1ZKI6Yhgg.png)

* stale restarts
![Screenshot 2025-10-01 at 2.38.29 AM](https://hackmd.io/_uploads/SytAL6K3lg.png)

Confirmed-Block Reorg Rate : 
* Flat line at 0.00 for entire 24-hour period
* Zero confirmed block reorgs throughout observation
* No confirmed blocks were reorged out

P50 Reorg Size : 
* No reorg data available because no reorgs occurred

Rollbacks: Count and Size :
* Flat line at 0.00 for entire 24-hour period
* Zero confirmed root rollbacks

Stale Restarts Analysis :
* Two brief spikes (22:00 and 10:00) reaching ~0.01, then flat at 0.00
* Minimal stale restart activity
* Likely node startup and epoch transitions

#### Tail-case analysis (delay ≥ 2) over time
* total tail rate
![Screenshot 2025-10-01 at 2.41.23 AM](https://hackmd.io/_uploads/BkLYwTt3eg.png)

* by delay bucket (2/3/ge4)
![Screenshot 2025-10-01 at 2.41.40 AM](https://hackmd.io/_uploads/By25D6Yhle.png)

* epoch boundary vs mid-epoch
![Screenshot 2025-10-01 at 2.42.00 AM](https://hackmd.io/_uploads/S1jswTY2le.png)
![Screenshot 2025-10-01 at 2.42.13 AM](https://hackmd.io/_uploads/BJq2DpF2lx.png)

* boundary share
![Screenshot 2025-10-01 at 2.42.29 AM](https://hackmd.io/_uploads/Byi6DTt3gx.png)

Total Tail Rate Over Time :
* Mostly 0.00 with small intermittent spikes
* Sharp spike to ~0.02 between 16:00-18:00
* Tail cases are infrequent

Tail Cases by Delay Bucket :
* 2-slot delays: Dominant pattern, mirrors total tail rate
* 3-slot delays: Minimal activity (barely visible spike at 17:00)
* 4+ slot delays: Zero occurrences throughout 24-hour period

Epoch Boundary vs Mid-Epoch Analysis
Epoch Boundaries:
* Rare but severe when they occur (spikes to 3.00m)
* Two major spikes around 16:30 and 17:00
Mid-Epoch:
* More frequent but minor (small spikes <0.01)
* Consistent low-level activity throughout

Boundary Share Analysis
* Baseline: 0% boundary share (no tail cases)
* Spikes: 70% and 115% boundary share during 16:00-18:00
* When tail cases occur, they're heavily concentrated at epoch boundaries

#### In-sync segmentation (observational)
* in-sync fraction per 10m window (0..1), graph over 24h
![Screenshot 2025-10-01 at 2.42.58 AM](https://hackmd.io/_uploads/Hyq1O6Khxl.png)

* split any rate by in-sync/catch-up
![Screenshot 2025-10-01 at 2.44.05 AM](https://hackmd.io/_uploads/Sk5SuTYhex.png)

In-Sync Fraction (0..1)
* Flat line at 0.00 for entire 24-hour period as node was never in sync (head_slot != current_slot)

Tail Cases During Catch-Up
* Empty query result (no tail cases recorded)
* Zero tail cases even during prolonged catch-up


#### Validator support and confirmation-time (seconds)
* validator support % (median)
![Screenshot 2025-10-01 at 2.45.02 AM](https://hackmd.io/_uploads/BkHPd6Yhlg.png)

* confirmation time (seconds) median / p95
![Screenshot 2025-10-01 at 2.45.25 AM](https://hackmd.io/_uploads/SJAdu6K3ge.png)
![Screenshot 2025-10-01 at 2.45.48 AM](https://hackmd.io/_uploads/HyQq_6Ynge.png)

Validator Support % (Median)
* Flat line at exactly 10.00% for entire 24-hour period, consistent validator participation - no fluctuations

Median Confirmation Time
* Stable at ~11.00 seconds
* Brief increase to 11.70 seconds during 17:00-18:00

P95 Confirmation Time
* 0.00 seconds
* Sharp increase to 40-50 seconds during 17:00-18:00
* Returns to 0.00 after spike

#### Safe-head reorg hygiene
* safe head reorg count
![Screenshot 2025-10-01 at 2.46.07 AM](https://hackmd.io/_uploads/rkwoOaK2xl.png)

Analysis : 
* Consistent 0.08-0.10 rate (not zero as title suggests)
* Sharp increase to 0.13-0.14 between 16:00-18:00
* Overall Low but persistent reorg activity

#### Epoch boundary activity and late attestations
* epoch boundary transition rate
![Screenshot 2025-10-01 at 2.46.42 AM](https://hackmd.io/_uploads/B1upOaY3el.png)

* late attestation rate
![Screenshot 2025-10-01 at 2.47.08 AM](https://hackmd.io/_uploads/HkfJKpt3le.png)

Epoch Boundary Transition Rate :
* Sharp, regular sawtooth spikes every few hours (0.00 → 2.5-2.75 → 0.00)
* FCR correctly identifies epoch boundaries
* Consistent ~2.5-2.75 rate during active periods

Late Attestation Rate :
* Stable, noisy line between 0.08-0.14
* Healthy network conditions - low, consistent late attestations
* No extreme spikes or drops to zero
* Continuous low-level activity (8-14% late attestations)

#### Housekeeping/overheads
* FCR metadata cache size (gauge)
![Screenshot 2025-10-01 at 2.47.24 AM](https://hackmd.io/_uploads/ry-lK6Y2gx.png)

* committee weight / FFG support calc time p95
![Screenshot 2025-10-01 at 2.47.50 AM](https://hackmd.io/_uploads/S15-KpF3ll.png)
![Screenshot 2025-10-01 at 2.48.09 AM](https://hackmd.io/_uploads/r1AftpY3ee.png)

FCR Metadata Cache Size (Gauge) :
* Sawtooth pattern cycling every ~1 hour (10→90→10). Cache grows to ~90 entries (reasonable size), Regular pruning prevents memory leaks and the Clean cycles indicate proper DAG-aligned pruning
* Perfect cache behavior - fills up over time, then gets pruned

Committee Weight Calculation Time P95 :
* Stable line between 0.06-0.09 seconds
* sub-100ms calculation time, stable timing indicates consistent performance and no spikes or degradation over 22 hours

#### Confirmation throughput views
* confirmations per 12s slot (≈1 when steady)
![Screenshot 2025-10-01 at 2.48.35 AM](https://hackmd.io/_uploads/S1hNYpt3lx.png)

* per-minute confirmations
![Screenshot 2025-10-01 at 2.48.54 AM](https://hackmd.io/_uploads/SyTHKTKnlg.png)

Confirmations per 12s slot :
* Sharp, regular spikes from 0.00 to ~0.20-0.22 every 12 seconds, so exactly 1 confirmation per slot (0.20-0.22 ≈ 1/5 = 0.2, accounting for 5 slots per minute)

confirmations per minute : 
* Smooth, undulating line staying above 0.05, peaking around 0.18-0.20
* ~5 confirmations per minute (0.18-0.20 × 60 ≈ 10-12, but accounting for rate calculation)
* Performance is Consistent, steady confirmation rate without gaps

#### Confirmation Path Analysis

* LMD vs FFG confirmation rates
![Screenshot 2025-10-01 at 3.34.53 AM](https://hackmd.io/_uploads/BJZMV0Yngl.png)

* FFG dependency ratio
![Screenshot 2025-10-01 at 3.35.39 AM](https://hackmd.io/_uploads/BkREN0t3gl.png)

The Q-indicator formula `2 * S > W + W/50 * β + proposer_score` is working well with my 25% Byzantine threshold, so blocks are confirming via LMD-GHOST alone! 

FFG validation is used in these scenarios:
* Epoch boundary advancement: When confirming blocks from previous epoch
* Current epoch advancement: When crossing into current epoch
* Safety checks: For `will_checkpoint_be_justified()` and `will_no_conflicting_checkpoint_be_justified()`
* Since the blocks are confirming within 1 slot (same epoch), they don't trigger the epoch boundary logic that would require FFG validation.

