Validator Ejector Setup

Introduction

The validator ejector is a daemon service which monitors ValidatorsExitBusOracle events and sends out stored exit messages when necessary. It allows node operators to generate and sign exit messages ahead of time, which will be sent out by the ejector when the protocol requests an exit to be made.
On start, it loads exit messages from a specified folder of individual .json files and validates their format, structure, and signature. Then, it loads events from a configurable amount of latest finalized blocks, checks if exits should be made and after that periodically fetches fresh events.

Requirements

Hardware

• 2-core CPU
• 1GB RAM

Nodes

• Execution node - full node required
• Consensus node

Software

Using Docker:

Docker & docker-compose

Running directly or using for message encryption:

Node.js 16 or higher

Exit Messages

The ejector loads and validates exit messages on start. This means that any changes to the messages folder (e.g. new exit messages) require a restart of the application to be picked up.

The ejector accepts messages in three formats:

Generic Format

{
  "message": { "epoch": "123", "validator_index": "123" },
  "signature": "0x123"
}

ethdo Output Format

{
  "exit": {
	"message": { "epoch": "123", "validator_index": "123" },
	"signature": "0x123"
  },
  "fork_version": "0x123"
}

Encrypted Format

{
  "version": 4,
  "uuid": "123abc-123abc-123abc",
  "path": "",
  "pubkey": "",
  "crypto": {
	"kdf": {
  	"function": "pbkdf2",
  	"params": {
    	"dklen": 123,
    	"c": 123,
    	"prf": "hmac-sha256",
    	"salt": "123abc"
  	},
  	"message": ""
	},
	"checksum": {
  	"function": "sha256",
  	"params": {},
  	"message": "123abc"
	},
	"cipher": {
  	"function": "aes-128-ctr",
  	"params": { "iv": "123abc" },
  	"message": "123abc"
	}
  }
}

Encrypting Messages

It is highly advised that after exit messages are generated and signed, they should be encrypted for storage safety. The ejector will decrypt files on start by looking up the password in the MESSAGES_PASSWORD environment variable.

Exit messages are encrypted and decrypted by the ejector according to the EIP-2335 specification.

The ejector is bundled with a small, easy to use encryption script.

Encryption using the Ejector - Source Code

1. Clone repository:

git clone https://github.com/lidofinance/validator-ejector.git
cd validator-ejector

2. Create an .env file with your encryption password or pass it to the terminal before proceeding:

MESSAGES_PASSWORD=password

3. Copy your .json exit message files to directory encryptor/input
4. Run yarn & yarn encrypt
5. Your encrypted exit message files will be saved to encryptor/output

Encryption using the Ejector - Docker

The ejector is bundled with an encryptor script inside, so you can run it using the same Docker image (the sha in the code must always be from the latest production release):

docker run \
	-e MESSAGES_PASSWORD=secret \
	-v /full/path/to/input:/app/encryptor/input/ \
	-v /full/path/to/output:/app/encryptor/output/ \
	lidofinance/validator-ejector@sha256:7b0c780d7d62afbf7ac7bdc6f7a35634e67a99ec733632c1fd6eed62388f4d19 \
	node /app/dist/encryptor/encrypt.js

For platforms with a different architecture but with emulation/transpilation support e.g. macOS on M-series processors, additionally specify:

--platform linux/amd64

Environment Variables

EXECUTION_NODE

Address of the execution node.

CONSENSUS_NODE

Address of the consensus node.

LOCATOR_ADDRESS

Address of the LidoLocator contract (can be looked up at the respective network’s deployed contracts page).

Hoodi Testnet:

0xe2EF9536DAAAEBFf5b1c130957AB3E80056b06D8

Holešky Testnet:

0x28FAB2059C713A7F9D8c86Db49f9bb0e96Af1ef8

Mainnet:

0xC1d0b3DE6792Bf6b4b37EccdcC24e45978Cfd2Eb

STAKING_MODULE_ID

ID of the StakingRouter contract module.

Currently, the staking router’s NodeOperatorsRegistry has three modules, the Curated one with ID 1 and Simple DVT with ID 2, and the Community Staking Module (CSM) with ID 3 (or 4 on testnets)

OPERATOR_ID

You can find this on the operators dashboard (e.g. #123 on the operator card):
Hoodi Curated module,
Hoodi Simple DVT module,
Holešky Curated module,
Holešky Simple DVT module,
Mainnet Curated module,
Mainnet Simple DVT module

MESSAGES_LOCATION

Location from which to load .json exit messages.

When set, messages mode will be activated. This is not needed if you use the ejector in webhook mode.

For example, /messages in Docker or simply messages if running directly for local files.

External storage bucket URLs are supported for AWS S3 and Google Cloud Storage:

• s3:// for S3
• gs:// for GCS

Authentication setup: S3, GCS

VALIDATOR_EXIT_WEBHOOK

Endpoint to fetch when an exit has to be made. Allows to implement a just in time approach by offloading exiting logic to an external service and using the ejector as a secure exit events reader.

When set, webhook mode will be activated. This is not needed if you use the ejector in messages mode.

On the endpoint, JSON will be POSTed with the following structure:

{
	"validatorIndex": "123",
	"validatorPubkey": "0x123"
}

A 200 response will be counted as a successful exit, non-200 ones as failures.

ORACLE_ADDRESSES_ALLOWLIST

JSON array of Lido Oracle addresses, from which only report transactions will be accepted.

To get the list of members you can go to the “Read Contract” section of the ValidatorsExitBusOracle’s HashConsensus on Etherscan and query the getMembers() method: Mainnet, Holešky, Hoodi

Example list for Mainnet (last updated 2024-10-25):

• 0x140Bd8FbDc884f48dA7cb1c09bE8A2fAdfea776E
• 0xA7410857ABbf75043d61ea54e07D57A6EB6EF186
• 0x404335BcE530400a5814375E7Ec1FB55fAff3eA2
• 0x946D3b081ed19173dC83Cd974fC69e1e760B7d78
• 0x007DE4a5F7bc37E2F26c0cb2E8A95006EE9B89b5
• 0xc79F702202E3A6B0B6310B537E786B9ACAA19BAf
• 0x61c91ECd902EB56e314bB2D5c5C07785444Ea1c8
• 0x73181107c8D9ED4ce0bbeF7A0b4ccf3320C41d12
• 0xe57B3792aDCc5da47EF4fF588883F0ee0c9835C9

Example .env entry for Mainnet (last updated 2024-10-25):

["0x140Bd8FbDc884f48dA7cb1c09bE8A2fAdfea776E","0xA7410857ABbf75043d61ea54e07D57A6EB6EF186","0x404335BcE530400a5814375E7Ec1FB55fAff3eA2","0x946D3b081ed19173dC83Cd974fC69e1e760B7d78","0x007DE4a5F7bc37E2F26c0cb2E8A95006EE9B89b5","0xc79F702202E3A6B0B6310B537E786B9ACAA19BAf","0x61c91ECd902EB56e314bB2D5c5C07785444Ea1c8","0x73181107c8D9ED4ce0bbeF7A0b4ccf3320C41d12","0xe57B3792aDCc5da47EF4fF588883F0ee0c9835C9"]

Note: make sure quotes are copied correctly if copying these examples.

Example list for Holešky (last updated 2024-09-30):

• 0x12A1D74F8697b9f4F1eEBb0a9d0FB6a751366399
• 0xD892c09b556b547c80B7d8c8cB8d75bf541B2284
• 0xf7aE520e99ed3C41180B5E12681d31Aa7302E4e5
• 0x31fa51343297FFce0CC1E67a50B2D3428057D1b1
• 0x81E411f1BFDa43493D7994F82fb61A415F6b8Fd4
• 0x4c75FA734a39f3a21C57e583c1c29942F021C6B7
• 0xD3b1e36A372Ca250eefF61f90E833Ca070559970
• 0xF0F23944EfC5A63c53632C571E7377b85d5E6B6f
• 0xb29dD2f6672C0DFF2d2f173087739A42877A5172
• 0xfe43A8B0b481Ae9fB1862d31826532047d2d538c

Example .env entry for Holešky (last updated 2024-09-30):

["0x12A1D74F8697b9f4F1eEBb0a9d0FB6a751366399","0xD892c09b556b547c80B7d8c8cB8d75bf541B2284","0xf7aE520e99ed3C41180B5E12681d31Aa7302E4e5","0x31fa51343297FFce0CC1E67a50B2D3428057D1b1","0x81E411f1BFDa43493D7994F82fb61A415F6b8Fd4","0x4c75FA734a39f3a21C57e583c1c29942F021C6B7","0xD3b1e36A372Ca250eefF61f90E833Ca070559970", "0xF0F23944EfC5A63c53632C571E7377b85d5E6B6f", "0xb29dD2f6672C0DFF2d2f173087739A42877A5172","0xfe43A8B0b481Ae9fB1862d31826532047d2d538c"]

Example list for Hoodi (last updated 2025-04-22):

1 0xcA80ee7313A315879f326105134F938676Cfd7a9
2 0xf03B8DC8762B97F13Ac82e6F94bE3Ed002FF7459
3 0x1932f53B1457a5987791a40Ba91f71c5Efd5788F
4 0xf7aE520e99ed3C41180B5E12681d31Aa7302E4e5
5 0x99B2B75F490fFC9A29E4E1f5987BE8e30E690aDF
6 0x219743f1911d84B32599BdC2Df21fC8Dba6F81a2
7 0xD3b1e36A372Ca250eefF61f90E833Ca070559970
8 0x4c75FA734a39f3a21C57e583c1c29942F021C6B7
9 0xB1cC91878c1831893D39C2Bb0988404ca5Fa7918
10 0xfe43A8B0b481Ae9fB1862d31826532047d2d538c
11 0x43C45C2455C49eed320F463fF4f1Ece3D2BF5aE2

Example .env entry for Hoodi (last updated 2025-04-22):

["0xcA80ee7313A315879f326105134F938676Cfd7a9","0xf03B8DC8762B97F13Ac82e6F94bE3Ed002FF7459","0x1932f53B1457a5987791a40Ba91f71c5Efd5788F","0xf7aE520e99ed3C41180B5E12681d31Aa7302E4e5","0x99B2B75F490fFC9A29E4E1f5987BE8e30E690aDF","0x219743f1911d84B32599BdC2Df21fC8Dba6F81a2","0xD3b1e36A372Ca250eefF61f90E833Ca070559970","0x4c75FA734a39f3a21C57e583c1c29942F021C6B7","0xB1cC91878c1831893D39C2Bb0988404ca5Fa7918","0xfe43A8B0b481Ae9fB1862d31826532047d2d538c","0x43C45C2455C49eed320F463fF4f1Ece3D2BF5aE2"]

Note: make sure quotes are copied correctly if copying these examples.

MESSAGES_PASSWORD

Password to decrypt encrypted exit messages with on application start.

MESSAGES_PASSWORD_FILE

Alternative to MESSAGES_PASSWORD. Path to file containing the password to decrypt exit messages with. If used, MESSAGES_PASSWORD (not MESSAGES_PASSWORD_FILE) needs to be added to LOGGER_SECRETS to be sanitized.

BLOCKS_PRELOAD

Amount of blocks to load events from on application start.

Suggested to include in your .env variables, but to be left at the default of 50000 (~7 days of blocks).

In case your ejector will be down due to an emergency, this value can be tweaked to let the ejector load a higher amount of blocks on start.

HTTP_PORT

Port for serving metrics and a health check endpoint, default 8989.

RUN_METRICS

Enable with true to serve Prometheus metrics: full list

Will be served on HOST:$HTTP_PORT/metrics.

Highly advised for monitoring and alerting.

RUN_HEALTH_CHECK

Enabled by default, disabled with false. Highly recommended to monitor this endpoint.

Will be served on HOST:$HTTP_PORT/health.

LOGGER_LEVEL

Recommended to set to info (default), can be changed to debug in case of issues for easier debugging.

LOGGER_FORMAT

Format of logs, simple by default, but can be set to json to be easily parsable, e.g. by Loki.

LOGGER_SECRETS

Environment variable names or exact values which should be replaced in logs, in JSON array of strings format.

Advised to include your MESSAGES_PASSWORD, EXECUTION_NODE, and MESSAGES_PASSWORD:

["MESSAGES_PASSWORD","EXECUTION_NODE","CONSENSUS_NODE"]

Note: make sure quotes are copied correctly if copying this example.

DRY_RUN

Allows to test the application with true without actually sending out exit messages.

Use with caution!

Make sure to set to false or completely leave it out in production.

Advanced Parameters

Please do not use unless suggested by a Lido contributor.

• BLOCKS_LOOP - 900 (3 hours of blocks) - Amount of blocks the ejector looks behind on wake in polling jobs
• JOB_INTERVAL - 384000 (1 epoch) - Time for which the ejector sleeps between jobs
• DISABLE_SECURITY_DONT_USE_IN_PRODUCTION - false - Set to true to skip security checks, for example if the ValidatorsExitBusOracle’s HashConsensus contract was changed after the ejector was unable to exit validators e.g. because it was switched off

Running

Source Code

1. Clone repository:

git clone https://github.com/lidofinance/validator-ejector.git
cd validator-ejector

2. Create an exit messages directory, for example locally mkdir messages
3. Put your exit message files in the messages folder
4. Copy the env sample file cp sample.env .env
5. Fill the environment variables in the .env file
6. Run

yarn
yarn build
yarn start

Docker with docker-compose

1. Create a root directory for the ejector, cd into that folder
2. Create an exit messages directory mkdir messages
3. Put your exit message files in the messages folder
4. Copy the env sample file cp sample.env .env
5. Fill the environment variables in the .env file
6. Create a docker-compose.yml file using this template
7. Run docker-compose up or docker-compose up -d to start in detached mode (in the background)

Check the Ejector is working

1. Ensure that there are no errors in the logs and no container restarts occuring
2. Verify that the config logged on start is correctly represented in the logs
3. If you have put presigned messages in the messages folder, make sure the loadedMessages count is greater than 0
4. Ensure you can see the Job started and Job finished lines in the logs

An example of a correct operation log:

info: Application started, version 1.0.0 {"EXECUTION_NODE":"<secret>","CONSENSUS_NODE":"<secret>","LOCATOR_ADDRESS":"0x123","STAKING_MODULE_ID":"1","OPERATOR_ID":"0","MESSAGES_LOCATION":"messages","ORACLE_ADDRESSES_ALLOWLIST":["0x123"],"MESSAGES_PASSWORD":"<secret>","BLOCKS_PRELOAD":190000,"BLOCKS_LOOP":64,"JOB_INTERVAL":384000,"HTTP_PORT":8989,"RUN_METRICS":true,"RUN_HEALTH_CHECK":true,"DRY_RUN":false}
info: Loading messages from messages
info: Loaded 123 messages
info: Validating messages
info: Starting, searching only for requests for operator 0
info: Loading initial events for 190000 last blocks
info: Job started {"operatorId":"0","stakingModuleId":"1","loadedMessages":123}
info: Resolved Exit Bus contract address using the Locator {"exitBusAddress":"0x123"}
info: Resolved Consensus contract address {"consensusAddress":"0x123"}
info: Fetched the latest block from EL {"latestBlock":12345}
info: Fetching request events from the Exit Bus {"eventsNumber":190000,"fromBlock":12345,"toBlock":12345}
info: Loaded ValidatorExitRequest events {"amount":0}
info: Handling ejection requests {"amount":0}
info: Job finished
info: Starting 384 seconds polling for 64 last blocks

What if something is wrong?

1. Make sure your configuration is correct
2. Make sure you are on the recommended Docker image sha hash or version if running directly
3. Check if your nodes are synced and working correctly
4. Restart the application
5. Start the application with the LOGGER_LEVEL=debug environment variable and contact the Lido devs with logs to investigate the problem

Additional Resources

Validator Ejector GitHub Repository (Open Source)
https://github.com/lidofinance/validator-ejector

What's Changing for Node Operators in Lido V2 - parts can be outdated
https://hackmd.io/@lido/Byue6SQxh

Lido Withdrawals: Automating Validator Exits - parts can be outdated
https://hackmd.io/@lido/BkxRxAr-o

Ejector Logic Spec - parts can be outdated
https://hackmd.io/@lido/r1KZ4YNdj