# Final Report - Ethereum Protocol Fellowship - Cohort 6
## ✷ tl;dr
- Lodestar testing framework migrating to [Kurtosis](https://docs.kurtosis.com/advanced-concepts/why-kurtosis/): [GitHub repo](https://github.com/IreneBa26/lodestar/tree/feature/kurtosis-sim-tests)
- [Kick-off presentation](https://youtu.be/UqkQdlyEwdA?t=6723) @ EthCC Cannes
- [Project Prosposal](https://github.com/eth-protocol-fellows/cohort-six/blob/master/projects/lodestar-sim-tests-framework-to-kurtosis.md) & My Lodestar [PR](https://github.com/ChainSafe/lodestar/pull/8296)
- [Updates & Write-ups](https://hackmd.io/@IB26/categories/epf6)
- [Final presentation](https://youtu.be/1ZktKW1NBz0?t=4184) @ Devconnect Buenos Aires
## ✷ Abstract
This project explored whether **Lodestar's simulation test framework** could migrate to **Kurtosis** without disrupting existing test logic or developer workflows.
The project investigated running multi-fork simulations inside isolated Kurtosis enclaves while **preserving** Lodestar's assertions, tracker system, and test structure.
**Kurtosis migration in one line**: Decouple testnet orchestration from `Docker Job` to `Kurtosis SDK`, enabling declarative YAML-based network configurations and integration with ethpandaops/ethereum-package, while preserving all existing simulation tests and assertions **without modification**.
The migration touches core **testing infrastructure**: simulation lifecycle management, node service discovery and mapping, network configuration parsing, and the adapter layer bridging Kurtosis services to Lodestar's NodePair abstraction.
---
### Overview of work:
**- KurtosisSDKRunner and IRunner interface**: Implemented a runner that manages the full Kurtosis enclave lifecycle (create, start, stop, destroy), deploys Ethereum networks via `ethpandaops/ethereum-package` through Starlark execution, infers service roles (beacon/execution/validator) from service names and port signatures, and extracts node indices for deterministic node pairing.
**Simulation class adapter layer**: Extended Simulation with `initWithKurtosisConfig()`, adding support for loading YAML configurations, converting Kurtosis services into Crucible-compatible NodePair structures, handling flexible port discovery (http, rpc, http-beacon, http-validator), and preserving the existing simulation lifecycle, including genesis detection via API polling, tracker registration, and node health checks.
**Service-to-NodePair conversion**: Implemented mapping functions that extract node indices from service names, pair beacon/execution/validator services by nodeIndex, construct API clients from discovered ports, and preserve all existing interfaces so assertions run unchanged.
**YAML-based network configuration**: Integrated a configuration loader that parses participant definitions (client types, counts, extra params), network parameters (fork epochs, genesis delay, validator keys), and prefunded accounts, enabling fully declarative testnet setup.
---
### Goals and Hypotheses
The core hypothesis:
> *Can Lodestar's testing framework, built around locally orchestrated Docker jobs, migrate to Kurtosis without changing test logic, assertions, or developer workflows and maintaining backward compatibility?*
Specifically:
- The multi-fork test simulation could run inside isolated Kurtosis enclaves
- The NodePair abstraction could be preserved, with Kurtosis services mapped to existing interfaces
- Genesis and network initialization could shift from internal state creation to external API readiness detection
- YAML-based configuration could replace programmatic node creation
This work aims to lower the barrier for the Lodestar team who could leverage it as an alternative testing option alongside their existing testing framework. It could also provides an entry point for the team to continue the transition.
## ✷ What Was Shipped
### Phase 1: SDK Integration
I started by understanding the fundamentals of the Kurtosis SDK: enclave lifecycle, Starlark package execution, and service context management.
I then built `KurtosisSDKRunner`, implementing the `IRunner` interface, handling enclave creation/destruction, connecting to the local Kurtosis engine, and executing `ethpandaops/ethereum-package` with serialized parameters.
**Key challenge**: structuring the Kurtosis SDK integration inside the Runner without breaking existing simulation patterns. The Runner needed to abstract away Kurtosis specifics while still providing the same lifecycle hooks that the original assertions tests expected.
**Migration entry point:**
- Running the simulations without `cli/test/utils/crucible/clients` folder
- Redesigning the current IRunner interface:
```typescript
export interface IRunner {
create: (jobOptions: JobOptions[]) => Job;
on(event: RunnerEvent, cb: (id: string) => void | Promise<void>): void;
start(): Promise<void>;
stop(): Promise<void>;
getNextIp(): string;
}
```
to this new Kurtosis-based version:
```typescript
export interface IRunner {
create: (config: KurtosisNetworkConfig) => Promise<KurtosisServicesMap>;
start(enclaveName: string): Promise<void>;
stop(): Promise<void>;
on(event: RunnerEvent, cb: (id: string) => void | Promise<void>): void;
}
```
- Redesigning the `multifork.test.ts` structure, from:
```typescript
// Lodestar multiFork.test.ts
const env = await Simulation.initWithDefaults(
{
id: "multi-fork",
logsDir: path.join(logFilesDir, "multi-fork"),
forkConfig,
},
[
{ id: "node-1", beacon: Lodestar, execution: Geth, ... },
{ id: "node-2", beacon: Lodestar, execution: Geth, ... },
{ id: "node-3", ... },
{ id: "node-4", ... },
{ id: "node-5", beacon: Lighthouse, execution: Geth },
]
);
```
To:
```typescript
// Kurtosis multiFork.test.ts
const env = await Simulation.initWithKurtosisConfig(
{
id: "multi-fork",
logsDir: path.join(logFilesDir, "multi-fork"),
forkConfig,
},
"multi-fork.yml" // 🟢 <-- Kurtosis YAML network configuration 🟢
);
```
### 1.1 The KurtosisSDKRunner
The ==KurtosisSDKRunner== implements the `IRunner` interface and manages the three-phase lifecycle of Kurtosis enclaves: initialization (`start()`), network deployment (`create()`), and cleanup (`stop()`).
Each method handles a distinct responsibility in the migration to the Kurtosis-based test infrastructure.
:::spoiler **start(): Enclave Initialization**
```typescript
async start(enclaveName: string): Promise<void>
```
**Purpose**: Establishes connection to Kurtosis engine and creates isolated enclave environment
**What it does**: It connects to local Kurtosis engine, creates enclave and stores context for later use
**Why it exists**: Docker approach didn't need explicit "start", containers were created on-demand.
Kurtosis requires explicit enclave creation before deploying services. The enclave provides isolation: each test run gets its own network namespace, preventing port conflicts and resource collisions.
**Usage**: Called once during `Simulation.initWithKurtosisConfig()` before loading network configuration.
The enclave exists empty at this point: no services deployed yet.
:::
:::spoiler **create(): Network Deployment**
```typescript
async create(config: KurtosisNetworkConfig): Promise<KurtosisServicesMap>
```
**Purpose**: Deploys Ethereum network inside enclave using ethpandaops/ethereum-package, then maps returned services to NodeService objects.
**What it does:** It validates enclave, serializes config by converting the `KurtosisNetworkConfig` to JSON string for Starlark package. Retrieves services: Calls `enclaveContext.getServices()` to get list of deployed services.
Maps services: Iterates through services, filtering out utility services. For each service it constructs `NodeService` object with id, role, serviceContext, apiUrl (if http port exists), and metadata containing nodeIndex
**Why it exists**: Docker approach created nodes programmatically, instantiated Job objects, configured options, started containers. Kurtosis approach is declarative, describe network in YAML, let package deploy it. The create() method bridges this gap: takes declarative config, executes package, maps returned services to programmatic NodeService objects.
**Usage**: Called once during `Simulation.initWithKurtosisConfig()` after `start()`. Takes YAML-loaded config, returns services ready for NodePair conversion.
:::
:::spoiler **stop(): Enclave Cleanup**
```typescript
async stop(): Promise<void>
```
Purpose: Destroys enclave and all contained services, freeing resources.
**What it does**: Verifies both kurtosisContext and enclaveName are set. It stops all services, removes containers, cleans up network namespaces, frees ports.
**Why it exists**: Docker approach had explicit container stop/remove. Kurtosis enclaves are self-contained, destroying enclave cleans up everything. No need to stop individual services.
**Usage**: Called during Simulation.stop() when test completes.
:::
### Phase 2: Adapter Layer
Extended `Simulation` class with `initWithKurtosisConfig()` method. This became the entry point for Kurtosis-based simulations, replacing the Docker-based `initWithDefaults()`.
**The adapter's job**: load YAML config, create enclave, deploy services, then convert Kurtosis services to NodePair structures.
**- What is a ==YAML file==?**
A **declarative network description** consumed by the EthPandaOps `ethereum-package`. Example:
```yaml
# multifork.yaml
participants:
el_type: geth
cl_type: lodestar
count: 1
vc_extra_params: ["--builder.selection=default"]
network_params:
preset: minimal
deneb_fork_epoch: 0
electra_fork_epoch: 0
genesis_delay: 120
```
It specifies:
- how many nodes to deploy
- which clients to use (Lodestar, Lighthouse, Geth, Nethermind…)
- validator settings (builder selections, blinded/local paths, remote keys, etc.)
- prefunded accounts
- fork epochs (Altair → Electra)
- any extra EL/CL parameters
**Critical design decision**: keep the NodePair abstraction intact.
That required mapping Kurtosis services, with their `ServiceContext` and port definitions, onto the existing `BeaconNode`, `ExecutionNode`, and `ValidatorNode` interfaces.
Port discovery needed to be flexible: services can expose `"http"`, `"rpc"`, `"http-beacon"`, or `"http-validator"`, so the adapter checks several names before wiring the clients.
### Phase 3: Service Mapping
Built `createNodePairsFromKurtosis()` to convert Kurtosis services to NodePair structures. It emulates `createNodePair()` function.
Kurtosis exposes all services as generic `ServiceContext` objects with no embedded type information. The runner infers each service’s role from its name (cl-1, el-2, vc-3) and from its port signatures—engine ports map to execution nodes, validator HTTP ports to validators, and discovery ports to beacon nodes.
**Node pairing** had to be deterministic, so the adapter extracts the node index from each service name and pairs services across roles by nodeIndex. That lets the system rebuild the same NodePair structure used in the Docker-based tests—just sourced from Kurtosis services instead of container definitions.
### Phase 4: Configuration Translation
Created `loadKurtosisConfig()` to parse YAML files into `KurtosisNetworkConfig` objects. The loader tries multiple paths (absolute, cwd, test directory, module directory) so tests don't need to care about current working directory.
- What is the ==**KurtosisNetworkConfig**==?
Is the TypeScript representation of the YAML required by `ethereum-package`
:::spoiler KurtosisNetworkConfig structure
```typescript
export type KurtosisNetworkConfig = {
participants: Array<{
el_type: string;
cl_type: string;
cl_image?: string;
count?: number;
cl_extra_params?: string[];
el_extra_params?: string[];
vc_extra_params?: string[];
}>;
additional_services?: string[];
network_params: Record<string, string | number>;
};
```
:::
### Phase 5: Genesis Handling Shift
Biggest architectural change: genesis moved from internal creation to external detection.
The pre-migration approach had `initGenesisState()` creating beacon state programmatically, writing `genesis.ssz` files.
Kurtosis approach: the `ethereum-package` handles genesis internally based on YAML config, so the adapter polls `/genesis` endpoints and health checks until nodes are ready.
Implemented `waitForBeaconGenesis()` polling all beacon nodes in parallel, checking `/genesis` responds with genesisTime, then verifying health endpoints return 200/206. This replaced the synchronous file-based genesis initialization with asynchronous API readiness detection.
`waitForBeaconGenesis()` waits for external genesis completion before proceeding with the rest of the simulation setup.
### Phase 6: Testing and Integration
Migrated `multiFork.test.ts` to use Kurtosis config. The test itself didn't change: same assertions, same tracker registrations, same slot/epoch logic. Only the initialization switched from Docker node pairs to Kurtosis config loading.
Current state: multiFork test runs with Kurtosis. Sync assertions (range sync, checkpoint sync, unknown block sync) remain commented out and planned for future migration. Other tests like `deneb.test.ts` are on the roadmap.
As observed in [EPF6 - Week 21](/HnIQXNz-RgOeZ5gPZqP-1Q) report, this is the comparison between Lodestar simulation:
and Kurtosis simulation:
---
## ✷ Design Decisions and Challenges
### 1. Architectural uncertainty
The migration path was intentionally open-ended, which made many decisions exploratory. There was no proof that the existing simulation framework would cleanly map to a declarative runtime. Some design choices may prove suboptimal later, but they were necessary to validate feasibility.
### 2. Keeping NodePair Abstraction
**Lodestar requirement**: assertions had to remain untouched. That meant keeping the NodePair abstraction intact, even though Kurtosis services come in with a completely different structure. The adapter effectively became a translation layer:
`Kurtosis ServiceContext → NodePair`
pulling out ports and IDs, then wiring up the API clients the tests expect.
**Trade-off**: the adapter gets more complex, but the test code stays clean. The assertions don’t know or care whether nodes come from Docker or Kurtosis, they just receive NodePair objects with the same interfaces.
### 3. Genesis Timing and Synchronization
Genesis timing isn’t perfectly synced yet. The Docker workflow controlled genesis very precisely
This creates timing uncertainty. Nodes might initialize at slightly different times. `waitForBeaconGenesis()` compensates by polling each beacon until they report readiness, but there’s still room for race conditions in tests that assume perfect synchronization.
### 4. Port Discovery Flexibility
Services expose ports with varying names. Some use "http", others "rpc", validator services might use "http-validator". `getPort()` is a flexible method that probes multiple names and falls back cleanly. This keeps the adapter **resilient** to naming inconsistencies now and in the future.
### 5. Dependency on Starlark Package Stability
The entire simulation depends on ethereum-package output shapes.
Naming conventions (`cl-1`, `el-1`) must remain stable.
### 6. Still a Prototype
Behavior is promising but not validated for strict correctness.
More edge cases need benchmarking.
### 7. Removed Job and JobOption
The Docker approach relied on `Job` and `JobOption` abstractions for container management. These disappear entirely in Kurtosis, which handles containerization internally. The Runner now manages **only** enclaves and services, no direct container manipulation.
---
## ✷ Challengeblue, Trade-offs, Considerations
### Architectural design without clear blueprint
The **hardest challenge** was building the architecture without a clear reference output. This migration required designing the integration pattern from scratch, there was no existing example of how to bridge Kurtosis services into Lodestar’s NodePair abstraction.
A good example is the **hybrid lifecycle**: combining Kurtosis enclave management (`create`/`start`/`stop`) with Lodestar’s simulation lifecycle (`tracker`, `clock`, `assertions`). Getting those two models to cooperate cleanly had to be figured out **step by step**.
Each architectural decision carried downstream effects that only revealed themselves once the full simulation was **running**.
### Sync Assertions
Range sync, checkpoint sync, and unknown block sync assertions remain commented out in `multiFork.test.ts`. These require more complex node state manipulation that hasn't been migrated yet. **Planned for future work**.
### Genesis Time Synchronization
Genesis timing detection works but isn't perfectly synchronized, it needs refinement. Nodes might report ready at slightly different times. Need better coordination or explicit synchronization points if tests require precise timing.
### Compatibility Testing
Haven't tested all existing simulation tests with Kurtosis. Only `multiFork.test.ts` migrated so far. `deneb.test.ts` and others planned for future migration.
---
## ✷ Final Results and Conclusions
### What Works
The simulation now produces a full output: the Kurtosis-based setup can run the multi-fork test without changing **any test logic**.
In this proof of concept, the test layer and assertions remain **untouched**, the tracker system behaves as expected, and existing developer workflows stay intact. YAML configuration enables a declarative testnet setup, and genesis detection is handled through API polling instead of file-based initialization.
**- Adapter pattern works**: The NodePair recreation with Kurtosis allows simulation reproduction. The test code stays clean, and the infrastructure swap happens entirely underneath.
**Backward compatibility**: Keeping the existing assertions intact served as a useful architectural constraint and gave a clear baseline for layering in the Kurtosis integration.
---
## ✷ Impact and Future Trajectory
**Current Impact**: The migration remains experimental, only one simulation test has been ported so far, serving as a proof of concept that Kurtosis integration could be technically feasible without breaking existing assertions. Keeping those assertions untouched turned out to be a productive constraint: it anchored the architecture and offered a clear baseline to build on top of.
For broader ecosystem impact, it’s still too early to say. But if the migration succeeds, it could serve as a useful reference implementation for how Kurtosis services can be leveraged to expose endpoint APIs and integrate with existing client testing setups.
**Beyond Lodestar**: The YAML-based configuration is one of the most transferable outcomes due to its portability, easy to modify without touching other part of the code. It could be especially useful for testing different network compositions or fork configurations without requiring changes faster.
---
## ✷ Personal Reflection
The most challenging part was making architectural decisions while still coming up to speed on both systems, but it turned into a solid growth opportunity. Questions like “**What belongs in the Runner vs. the Simulation?**”, “**How should services map to assertions?**”, and “**Which abstractions are worth preserving?**” pushed me toward a more conservative approach that respected Lodestar’s original architecture. At the same time, it left enough room for creative experimentation where the design allowed it.
The most satisfying moment was seeing the simulation run with Kurtosis instead of Docker Jobs. When `yarn workspace @chainsafe/lodestar test:sim:multifork` executed `multiFork.test.ts`, the code ran cleanly, the nodes came up, and the only errors were from the assertions, not from the implementation itself. The adapter layer worked: Kurtosis services became NodePairs, and the test logic treated them just like the original setup.
The most satisfying moment was watching the simulation run on Kurtosis instead of Docker Jobs. When `yarn workspace @chainsafe/lodestar test:sim:multifork` executed `multiFork.test.ts`, the code ran cleanly, the nodes came up, and the only failures came from assertion logic, not from the implementation. The adapter layer held up: Kurtosis services mapped cleanly into **NodePair** objects, and the test logic treated them exactly like the original Job-based setup.
### Self-Evaluation
Working alongside protocol engineers and seeing how they approach architecture was invaluable. I joined their weekly standups and shared progress in the EPF fellows’ update slot throughout the cohort. In the Lodestar Discord channel, there was even a dedicated thread for the Kurtosis migration, great for everyone to follow and contribute:
Mentor PR reviews were particularly helpful. Clear feedback on what was a priority versus what was merely nice-to-have helped shape the integration logic.
I also leveled up my open-source contribution. I picked up a clearer sense of how to document technical decisions and how to structure code for reviewers. After working this closely with the Lodestar team, I’m looking forward to contributing even more across the Ethereum ecosystem.
The project doesn’t end with the fellowship. The early results are promising, I want to keep pushing the migration forward. I’ll do my best to carve out time after the cohort to extend the integration and see whether full 1:1 parity with the existing simulation framework is actually achievable.
Being comfortable shifting perspective became the guiding mindset. What first looked like a limitation, for example no clear blueprint, no fixed output, turned into an interesting challenge to solve and each gap became a learning opportunity.
**Biggest takeaway**: the constraints that looked limiting at first ended up being some of the most enjoyable parts of the process. It really was just a matter of perspective.
---
## EPF Fellowship thanks
Huge thanks to [Mario](https://x.com/TMIYChao) and [Josh](https://x.com/joshdavislight) for welcoming me into this cohort as a fellow and steering this cohort with the right mix of structure, freedom, and trust. You're doing invaluable work!
I still remember applying to Cohort 4, so getting accepted into Cohort 6, especially in a cycle that leaned heavily technical rather than research-oriented, was an even bigger and very welcome surprise. It changed the trajectory of my year in the best possible way.
Another mention to Mario, your invite to the Interop Day during the Berlin Blockchain Week led to meeting the Lodestar team. We discussed a possible Kurtosis integration for their testing framework, and what started as a conversation turned into a solid project proposal.
Thanks to my mentor [Nazar](https://github.com/nazarhussain) for the guidance, the feedback and the time! Every reviews played a crucial role in the project development. Thanks also to [Matthew](https://github.com/matthewkeil), Lodestar engineer, for the great advices during the fellowship and the whole Lodestar team.
And finally, thanks to [Solène](https://x.com/_SDAV), who gave me the decisive push to submit the application to this fellowship.
Big congratulation to my fellow fellows on everything you have accomplished, here's to an amazing post-fellowship chapter!
---
Best of luck to all the future EPF fellows.
Feel free to reach out if I can be of assistance or just want to connect!
Irene
Sign in with Wallet
Connect another wallet