# Week 16

This time, I have benchamarked FCR for a day.

#### Slot-form latency (slots) over time
* median / p95

* confirmations per minute
* breakdown by delay buckets (0/1/2/ge3)

For P50, we get :
* Consistently 0.50-0.55 slots, which means most blocks confirm within the same slot or immediately next slot
* Brief increase to 0.65 slots during 16:00-18:00

For P95, I see :
* Consistently ~1.00 slot, 95% of blocks confirm within 1 slot
* Brief increase to 2.00-2.50 slots during 16:00-18:00

For Confirmations Per Minute, I see :
* Regular sawtooth spikes from 0.05 to 0.18-0.20
* ~1 confirmation per 12-second slot

And when breaking down by delay buckets :
* 0-slot delays: Regular spikes peaking at 0.10-0.12 (same-slot confirmation)
* 1-slot delays: Regular spikes peaking at 0.08-0.10 (next-slot confirmation)
* 2-slot delays: Almost flat at 0.00 (rare tail cases)
* 3+ slot delays: Completely flat at 0.00 (zero occurrences)

#### Head→confirmed gap (slots) over time
* median / p95

* outlier share (>8 slots)

For Median Head→Confirmed Gap (P50):
* In the initial state, a high gap of ~24-25 slots at startup (22:00)
* During transition state, a rapid decay to <5 slots by 02:00, ~0 slots by 04:00-06:00
* And in steady state, consistently ~0 slots for 14+ hours (06:00-20:00)

For P95 Head→Confirmed Gap :
* Initial State: Even higher gap of ~32-33 slots at startup
* Transition Phase: Slower decay, <5 slots by 04:00, ~0 slots by 06:00-08:00
* Steady State: Consistently ~0 slots for 12+ hours (08:00-20:00)

And, Outlier Share (>8 Slots) :
* Initial Burst: High outlier share (~0.015-0.018) during startup
* Quick Convergence: Drops to 0.00 within first hour
* Steady State: Zero outliers for most of 24-hour period
* Minor Spike: Brief spike to ~0.005 around 17:00-18:00 (correlates with previous events)

#### Restarts over time (with zero-line)
* rates per reason (stale / reorg)

* cumulative 24h (stacked area)


For Rates per Reason (Stale / Reorg)
Stale Restarts:
* Initial Burst: High rate (~0.05-0.06) at 22:00 (startup)
* Quick Drop: Falls to 0.00 by 23:00
* Stable Period: Remains at 0.00 for ~21 hours
Reorg Restarts:
* Zero Activity: No visible line (confirmed by OR vector(0))
* Perfect Record: Zero reorg restarts throughout 22-hour period

For Cumulative 24h (Stacked Area)
Stale Restarts:
* Total Count: ~31.60 cumulative restarts
* Timing: All occurred in first hour (22:00-23:00)
* Stability: Flat line after initial hour
Reorg Restarts:
* Zero Cumulative: No reorg restarts accumulated
* Perfect Stability: Confirmed blocks never reorged

#### Confirmed reorgs and rollbacks (likely zero; zero-line included)
* confirmed-block reorg rate

* P50 reorg size

* rollbacks: count and size

* stale restarts

Confirmed-Block Reorg Rate :
* Flat line at 0.00 for entire 24-hour period
* Zero confirmed block reorgs throughout observation
* No confirmed blocks were reorged out

P50 Reorg Size :
* No reorg data available because no reorgs occurred

Rollbacks: Count and Size :
* Flat line at 0.00 for entire 24-hour period
* Zero confirmed root rollbacks

Stale Restarts Analysis :
* Two brief spikes (22:00 and 10:00) reaching ~0.01, then flat at 0.00
* Minimal stale restart activity
* Likely node startup and epoch transitions

#### Tail-case analysis (delay ≥ 2) over time
* total tail rate

* by delay bucket (2/3/ge4)

* epoch boundary vs mid-epoch

* boundary share

Total Tail Rate Over Time :
* Mostly 0.00 with small intermittent spikes
* Sharp spike to ~0.02 between 16:00-18:00
* Tail cases are infrequent

Tail Cases by Delay Bucket :
* 2-slot delays: Dominant pattern, mirrors total tail rate
* 3-slot delays: Minimal activity (barely visible spike at 17:00)
* 4+ slot delays: Zero occurrences throughout 24-hour period

Epoch Boundary vs Mid-Epoch Analysis
Epoch Boundaries:
* Rare but severe when they occur (spikes to 3.00m)
* Two major spikes around 16:30 and 17:00
Mid-Epoch:
* More frequent but minor (small spikes <0.01)
* Consistent low-level activity throughout

Boundary Share Analysis
* Baseline: 0% boundary share (no tail cases)
* Spikes: 70% and 115% boundary share during 16:00-18:00
* When tail cases occur, they're heavily concentrated at epoch boundaries

#### In-sync segmentation (observational)
* in-sync fraction per 10m window (0..1), graph over 24h

* split any rate by in-sync/catch-up

In-Sync Fraction (0..1)
* Flat line at 0.00 for entire 24-hour period as node was never in sync (head_slot != current_slot)

Tail Cases During Catch-Up
* Empty query result (no tail cases recorded)
* Zero tail cases even during prolonged catch-up

#### Validator support and confirmation-time (seconds)
* validator support % (median)

* confirmation time (seconds) median / p95

Validator Support % (Median)
* Flat line at exactly 10.00% for entire 24-hour period, consistent validator participation - no fluctuations

Median Confirmation Time
* Stable at ~11.00 seconds
* Brief increase to 11.70 seconds during 17:00-18:00

P95 Confirmation Time
* 0.00 seconds
* Sharp increase to 40-50 seconds during 17:00-18:00
* Returns to 0.00 after spike

#### Safe-head reorg hygiene
* safe head reorg count

Analysis :
* Consistent 0.08-0.10 rate (not zero as title suggests)
* Sharp increase to 0.13-0.14 between 16:00-18:00
* Overall Low but persistent reorg activity

#### Epoch boundary activity and late attestations
* epoch boundary transition rate

* late attestation rate

Epoch Boundary Transition Rate :
* Sharp, regular sawtooth spikes every few hours (0.00 → 2.5-2.75 → 0.00)
* FCR correctly identifies epoch boundaries
* Consistent ~2.5-2.75 rate during active periods

Late Attestation Rate :
* Stable, noisy line between 0.08-0.14
* Healthy network conditions - low, consistent late attestations
* No extreme spikes or drops to zero
* Continuous low-level activity (8-14% late attestations)

#### Housekeeping/overheads
* FCR metadata cache size (gauge)

* committee weight / FFG support calc time p95

FCR Metadata Cache Size (Gauge) :
* Sawtooth pattern cycling every ~1 hour (10→90→10). Cache grows to ~90 entries (reasonable size), Regular pruning prevents memory leaks and the Clean cycles indicate proper DAG-aligned pruning
* Perfect cache behavior - fills up over time, then gets pruned

Committee Weight Calculation Time P95 :
* Stable line between 0.06-0.09 seconds
* sub-100ms calculation time, stable timing indicates consistent performance and no spikes or degradation over 22 hours

#### Confirmation throughput views
* confirmations per 12s slot (≈1 when steady)

* per-minute confirmations

Confirmations per 12s slot :
* Sharp, regular spikes from 0.00 to ~0.20-0.22 every 12 seconds, so exactly 1 confirmation per slot (0.20-0.22 ≈ 1/5 = 0.2, accounting for 5 slots per minute)

confirmations per minute :
* Smooth, undulating line staying above 0.05, peaking around 0.18-0.20
* ~5 confirmations per minute (0.18-0.20 × 60 ≈ 10-12, but accounting for rate calculation)
* Performance is Consistent, steady confirmation rate without gaps

#### Confirmation Path Analysis

* LMD vs FFG confirmation rates

* FFG dependency ratio

The Q-indicator formula `2 * S > W + W/50 * β + proposer_score` is working well with my 25% Byzantine threshold, so blocks are confirming via LMD-GHOST alone!

FFG validation is used in these scenarios:
* Epoch boundary advancement: When confirming blocks from previous epoch
* Current epoch advancement: When crossing into current epoch
* Safety checks: For `will_checkpoint_be_justified()` and `will_no_conflicting_checkpoint_be_justified()`
* Since the blocks are confirming within 1 slot (same epoch), they don't trigger the epoch boundary logic that would require FFG validation.