---
tags: withdrawals, tooling, ejector
---


# Validator Ejector Setup

## Introduction

The validator ejector is a daemon service which monitors [ValidatorsExitBusOracle](https://github.com/lidofinance/lido-dao/blob/5fcedc6e9a9f3ec154e69cff47c2b9e25503a78a/contracts/0.8.9/oracle/ValidatorsExitBusOracle.sol) events and sends out stored exit messages when necessary. It allows node operators to generate and sign exit messages ahead of time, which will be sent out by the ejector when the protocol requests an exit to be made.
On start, it loads exit messages from a specified folder of individual `.json` files and validates their format, structure, and signature. Then, it loads events from a configurable amount of latest finalized blocks, checks if exits should be made and after that periodically fetches fresh events.

## Requirements

### Hardware

- 2-core CPU
- 1GB RAM

### Nodes

- Execution node - [full node required](https://ethereum.org/en/developers/docs/nodes-and-clients/#node-types)
- Consensus node

### Software

#### Using Docker:

Docker & docker-compose

#### Running directly or using for message encryption:

Node.js 16 or higher

## Exit Messages

The ejector loads and validates exit messages on start. This means that any changes to the messages folder (e.g. new exit messages) require a restart of the application to be picked up.

The ejector accepts messages in three formats:

### Generic Format

{
  "message": { "epoch": "123", "validator_index": "123" },
  "signature": "0x123"
}

### ethdo Output Format

{
  "exit": {
	"message": { "epoch": "123", "validator_index": "123" },
	"signature": "0x123"
  },
  "fork_version": "0x123"
}

### Encrypted Format

{
  "version": 4,
  "uuid": "123abc-123abc-123abc",
  "path": "",
  "pubkey": "",
  "crypto": {
	"kdf": {
  	"function": "pbkdf2",
  	"params": {
    	"dklen": 123,
    	"c": 123,
    	"prf": "hmac-sha256",
    	"salt": "123abc"
  	},
  	"message": ""
	},
	"checksum": {
  	"function": "sha256",
  	"params": {},
  	"message": "123abc"
	},
	"cipher": {
  	"function": "aes-128-ctr",
  	"params": { "iv": "123abc" },
  	"message": "123abc"
	}
  }
}

## Encrypting Messages

It is highly advised that after exit messages are generated and signed, they should be encrypted for storage safety. The ejector will decrypt files on start by looking up the password in the `MESSAGES_PASSWORD` environment variable.

Exit messages are encrypted and decrypted by the ejector according to the [EIP-2335](https://github.com/ethereum/ercs/blob/master/ERCS/erc-2335.md) specification.

The ejector is bundled with a small, easy to use encryption script.

### Encryption using the Ejector - Source Code

1. Clone repository:

git clone https://github.com/lidofinance/validator-ejector.git
cd validator-ejector

2. Create an `.env` file with your encryption password or pass it to the terminal before proceeding:

MESSAGES_PASSWORD=password

3. Copy your `.json` exit message files to directory `encryptor/input`
4. Run `yarn & yarn encrypt`
5. Your encrypted exit message files will be saved to `encryptor/output`

### Encryption using the Ejector - Docker

The ejector is bundled with an encryptor script inside, so you can run it using the same Docker image (the sha in the code must always be from the [latest production release](https://hub.docker.com/r/lidofinance/validator-ejector/tags)):

docker run \
	-e MESSAGES_PASSWORD=secret \
	-v /full/path/to/input:/app/encryptor/input/ \
	-v /full/path/to/output:/app/encryptor/output/ \
	lidofinance/validator-ejector@sha256:7b0c780d7d62afbf7ac7bdc6f7a35634e67a99ec733632c1fd6eed62388f4d19 \
	node /app/dist/encryptor/encrypt.js

For platforms with a different architecture but with emulation/transpilation support e.g. macOS on M-series processors, additionally specify:

--platform linux/amd64

## Environment Variables

### EXECUTION_NODE

Address of the execution node.

### CONSENSUS_NODE

Address of the consensus node.

### LOCATOR_ADDRESS

Address of the [LidoLocator](https://github.com/lidofinance/lido-dao/blob/5fcedc6e9a9f3ec154e69cff47c2b9e25503a78a/contracts/0.8.9/LidoLocator.sol) contract (can be looked up at the [respective network’s deployed contracts page](https://docs.lido.fi/deployed-contracts/)).

Holešky Testnet:

0x28FAB2059C713A7F9D8c86Db49f9bb0e96Af1ef8

Mainnet:

0xC1d0b3DE6792Bf6b4b37EccdcC24e45978Cfd2Eb

### STAKING_MODULE_ID

ID of the [StakingRouter](https://github.com/lidofinance/lido-dao/blob/5fcedc6e9a9f3ec154e69cff47c2b9e25503a78a/contracts/0.8.9/StakingRouter.sol) contract module.

Currently, the staking router’s [NodeOperatorsRegistry](https://github.com/lidofinance/lido-dao/blob/5fcedc6e9a9f3ec154e69cff47c2b9e25503a78a/contracts/0.4.24/nos/NodeOperatorsRegistry.sol) has two modules, the Curated one with ID `1` and Simple DVT with ID `2`.

### OPERATOR_ID

You can find this on the operators dashboard (e.g. #123 on the operator card): [Holešky Curated module](https://operators-holesky.testnet.fi/module/1), [Holešky Simple DVT module](https://operators-holesky.testnet.fi/module/2), [Mainnet Curated module](https://operators.lido.fi/module/1), [Mainnet Simple DVT module](https://operators.lido.fi/module/2)

### MESSAGES_LOCATION

Location from which to load `.json` exit messages.

When set, messages mode will be activated. This is not needed if you use the ejector in webhook mode.

For example, `/messages` in Docker or simply `messages` if running directly for local files.

External storage bucket URLs are supported for AWS S3 and Google Cloud Storage:

- `s3://` for S3
- `gs://` for GCS

Authentication setup: [S3](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/setting-credentials-node.html), [GCS](https://cloud.google.com/docs/authentication/application-default-credentials#attached-sa)

### VALIDATOR_EXIT_WEBHOOK

Endpoint to fetch when an exit has to be made. Allows to implement a just in time approach by offloading exiting logic to an external service and using the ejector as a secure exit events reader.

When set, webhook mode will be activated. This is not needed if you use the ejector in messages mode.

On the endpoint, JSON will be POSTed with the following structure:

{
	"validatorIndex": "123",
	"validatorPubkey": "0x123"
}

A 200 response will be counted as a successful exit, non-200 ones as failures.

### ORACLE_ADDRESSES_ALLOWLIST

JSON array of Lido Oracle addresses, from which only report transactions will be accepted.

For mainnet you can get a list from [Aragon](https://mainnet.lido.fi/#/lido-dao/0x442af784a788a5bd6f42a01ebe9f287a871243fb/).

Alternatively, and for Holešky, you can go to the “Read Contract” section of the ValidatorsExitBusOracle’s HashConsensus on Etherscan and query the `getMembers()` methode: [Holešky](https://holesky.etherscan.io/address/0xe77Cf1A027d7C10Ee6bb7Ede5E922a181FF40E8f#readContract), [Mainnet](https://etherscan.io/address/0x7FaDB6358950c5fAA66Cb5EB8eE5147De3df355a#readContract)

Example list for Holešky _(last updated 2024-04-10)_:

- 0x12A1D74F8697b9f4F1eEBb0a9d0FB6a751366399
- 0xD892c09b556b547c80B7d8c8cB8d75bf541B2284
- 0xf7aE520e99ed3C41180B5E12681d31Aa7302E4e5
- 0x31fa51343297FFce0CC1E67a50B2D3428057D1b1
- 0x81E411f1BFDa43493D7994F82fb61A415F6b8Fd4
- 0x4c75FA734a39f3a21C57e583c1c29942F021C6B7
- 0xD3b1e36A372Ca250eefF61f90E833Ca070559970

Example `.env` entry for Holešky:

["0x12A1D74F8697b9f4F1eEBb0a9d0FB6a751366399","0xD892c09b556b547c80B7d8c8cB8d75bf541B2284","0xf7aE520e99ed3C41180B5E12681d31Aa7302E4e5","0x31fa51343297FFce0CC1E67a50B2D3428057D1b1","0x81E411f1BFDa43493D7994F82fb61A415F6b8Fd4","0x4c75FA734a39f3a21C57e583c1c29942F021C6B7","0xD3b1e36A372Ca250eefF61f90E833Ca070559970"]

Example list for Mainnet _(last updated 2024-03-22)_:

- 0x140Bd8FbDc884f48dA7cb1c09bE8A2fAdfea776E
- 0xA7410857ABbf75043d61ea54e07D57A6EB6EF186
- 0x404335BcE530400a5814375E7Ec1FB55fAff3eA2
- 0x946D3b081ed19173dC83Cd974fC69e1e760B7d78
- 0x007DE4a5F7bc37E2F26c0cb2E8A95006EE9B89b5
- 0xEC4BfbAF681eb505B94E4a7849877DC6c600Ca3A
- 0x61c91ECd902EB56e314bB2D5c5C07785444Ea1c8
- 0x1Ca0fEC59b86F549e1F1184d97cb47794C8Af58d
- 0xc79F702202E3A6B0B6310B537E786B9ACAA19BAf

Example `.env` entry for Mainnet:

["0x140Bd8FbDc884f48dA7cb1c09bE8A2fAdfea776E","0xA7410857ABbf75043d61ea54e07D57A6EB6EF186","0x404335BcE530400a5814375E7Ec1FB55fAff3eA2","0x946D3b081ed19173dC83Cd974fC69e1e760B7d78","0x007DE4a5F7bc37E2F26c0cb2E8A95006EE9B89b5","0xEC4BfbAF681eb505B94E4a7849877DC6c600Ca3A","0x61c91ECd902EB56e314bB2D5c5C07785444Ea1c8","0x1Ca0fEC59b86F549e1F1184d97cb47794C8Af58d","0xc79F702202E3A6B0B6310B537E786B9ACAA19BAf"]

Note: make sure quotes are copied correctly if copying these examples.

### MESSAGES_PASSWORD

Password to decrypt encrypted exit messages with on application start.

### MESSAGES_PASSWORD_FILE

Alternative to `MESSAGES_PASSWORD`. Path to file containing the password to decrypt exit messages with. If used, MESSAGES_PASSWORD (not MESSAGES_PASSWORD_FILE) needs to be added to LOGGER_SECRETS to be sanitized.

### BLOCKS_PRELOAD

Amount of blocks to load events from on application start.

Suggested to include in your `.env` variables, but to be left at the default of 50000 (~7 days of blocks).

In case your ejector will be down due to an emergency, this value can be tweaked to let the ejector load a higher amount of blocks on start.

### HTTP_PORT

Port for serving metrics and a health check endpoint, default 8989.

### RUN_METRICS

Enable with `true` to serve Prometheus metrics: [full list](https://github.com/lidofinance/validator-ejector#metrics)

Will be served on `HOST:$HTTP_PORT/metrics`.

Highly advised for monitoring and alerting.

### RUN_HEALTH_CHECK

Enabled by default, disabled with `false`. Highly recommended to monitor this endpoint.

Will be served on `HOST:$HTTP_PORT/health`.

### LOGGER_LEVEL

Recommended to set to `info` (default), can be changed to `debug` in case of issues for easier debugging.

### LOGGER_FORMAT

Format of logs, `simple` by default, but can be set to `json` to be easily parsable, e.g. by [Loki](https://github.com/grafana/loki).

### LOGGER_SECRETS

Environment variable names or exact values which should be replaced in logs, in JSON array of strings format.

Advised to include your MESSAGES_PASSWORD, EXECUTION_NODE, and MESSAGES_PASSWORD:

["MESSAGES_PASSWORD","EXECUTION_NODE","CONSENSUS_NODE"]

Note: make sure quotes are copied correctly if copying this example.

### DRY_RUN

Allows to test the application with `true` without actually sending out exit messages.

Use with caution!

Make sure to set to `false` or completely leave it out in production.

### Advanced Parameters

Please do not use unless suggested by a Lido contributor.

- BLOCKS_LOOP - 900 (3 hours of blocks) - Amount of blocks the ejector looks behind on wake in polling jobs
- JOB_INTERVAL - 384000 (1 epoch) - Time for which the ejector sleeps between jobs
- DISABLE_SECURITY_DONT_USE_IN_PRODUCTION - false - Set to `true` to skip security checks, for example if the ValidatorsExitBusOracle’s HashConsensus contract was changed after the ejector was unable to exit validators e.g. because it was switched off

## Running

### Source Code

1. Clone repository:

git clone https://github.com/lidofinance/validator-ejector.git
cd validator-ejector

2. Create an exit messages directory, for example locally `mkdir messages`
3. Put your exit message files in the messages folder
4. Copy the env sample file `cp sample.env .env`
5. Fill the environment variables in the `.env` file
6. Run

yarn
yarn build
yarn start

### Docker with docker-compose

1. Create a root directory for the ejector, `cd` into that folder
2. Create an exit messages directory `mkdir messages`
3. Put your exit message files in the messages folder
4. Copy the env sample file `cp sample.env .env`
5. Fill the environment variables in the `.env` file
6. Create a `docker-compose.yml` file using [this template](https://github.com/lidofinance/validator-ejector/blob/develop/docker-compose.yml)
7. Run `docker-compose up` or `docker-compose up -d` to start in detached mode (in the background)

## Check the Ejector is working

1. Ensure that there are no errors in the logs and no container restarts occuring
2. Verify that the config logged on start is correctly represented in the logs
3. If you have put presigned messages in the messages folder, make sure the `loadedMessages` count is greater than `0`
4. Ensure you can see the `Job started` and `Job finished` lines in the logs

An example of a correct operation log:

info: Application started, version 1.0.0 {"EXECUTION_NODE":"<secret>","CONSENSUS_NODE":"<secret>","LOCATOR_ADDRESS":"0x123","STAKING_MODULE_ID":"1","OPERATOR_ID":"0","MESSAGES_LOCATION":"messages","ORACLE_ADDRESSES_ALLOWLIST":["0x123"],"MESSAGES_PASSWORD":"<secret>","BLOCKS_PRELOAD":190000,"BLOCKS_LOOP":64,"JOB_INTERVAL":384000,"HTTP_PORT":8989,"RUN_METRICS":true,"RUN_HEALTH_CHECK":true,"DRY_RUN":false}
info: Loading messages from messages
info: Loaded 123 messages
info: Validating messages
info: Starting, searching only for requests for operator 0
info: Loading initial events for 190000 last blocks
info: Job started {"operatorId":"0","stakingModuleId":"1","loadedMessages":123}
info: Resolved Exit Bus contract address using the Locator {"exitBusAddress":"0x123"}
info: Resolved Consensus contract address {"consensusAddress":"0x123"}
info: Fetched the latest block from EL {"latestBlock":12345}
info: Fetching request events from the Exit Bus {"eventsNumber":190000,"fromBlock":12345,"toBlock":12345}
info: Loaded ValidatorExitRequest events {"amount":0}
info: Handling ejection requests {"amount":0}
info: Job finished
info: Starting 384 seconds polling for 64 last blocks

## What if something is wrong?

1. Make sure your configuration is correct
2. Make sure you are on the recommended Docker image sha hash or version if running directly
3. Check if your nodes are synced and working correctly
4. Restart the application
5. Start the application with the LOGGER_LEVEL=debug environment variable and contact the Lido devs with logs to investigate the problem

## Additional Resources

Validator Ejector GitHub Repository (Open Source)
https://github.com/lidofinance/validator-ejector

What's Changing for Node Operators in Lido V2 - parts can be outdated
https://hackmd.io/@lido/Byue6SQxh

Lido Withdrawals: Automating Validator Exits - parts can be outdated
https://hackmd.io/@lido/BkxRxAr-o

Ejector Logic Spec - parts can be outdated
https://hackmd.io/@lido/r1KZ4YNdj