Validator Ejector Setup

Introduction

The validator ejector is a daemon service which monitors ValidatorsExitBusOracle events and sends out stored exit messages when necessary. It allows node operators to generate and sign exit messages ahead of time, which will be sent out by the ejector when the protocol requests an exit to be made.
On start, it loads exit messages from a specified folder of individual .json files and validates their format, structure, and signature. Then, it loads events from a configurable amount of latest finalized blocks, checks if exits should be made and after that periodically fetches fresh events.

Requirements

Hardware

2-core CPU
1GB RAM

Nodes

Execution node - full node required
Consensus node

Software

Using Docker:

Docker & docker-compose

Running directly or using for message encryption:

Node.js 16 or higher

Exit Messages

The ejector loads and validates exit messages on start. This means that any changes to the messages folder (e.g. new exit messages) require a restart of the application to be picked up.

The ejector accepts messages in three formats:

Generic Format

{
  "message": { "epoch": "123", "validator_index": "123" },
  "signature": "0x123"
}

ethdo Output Format

{
  "exit": {
	"message": { "epoch": "123", "validator_index": "123" },
	"signature": "0x123"
  },
  "fork_version": "0x123"
}

Encrypted Format

{
  "version": 4,
  "uuid": "123abc-123abc-123abc",
  "path": "",
  "pubkey": "",
  "crypto": {
	"kdf": {
  	"function": "pbkdf2",
  	"params": {
    	"dklen": 123,
    	"c": 123,
    	"prf": "hmac-sha256",
    	"salt": "123abc"
  	},
  	"message": ""
	},
	"checksum": {
  	"function": "sha256",
  	"params": {},
  	"message": "123abc"
	},
	"cipher": {
  	"function": "aes-128-ctr",
  	"params": { "iv": "123abc" },
  	"message": "123abc"
	}
  }
}

Encrypting Messages

It is highly advised that after exit messages are generated and signed, they should be encrypted for storage safety. The ejector will decrypt files on start by looking up the password in the MESSAGES_PASSWORD environment variable.

Exit messages are encrypted and decrypted by the ejector according to the EIP-2335 specification.

The ejector is bundled with a small, easy to use encryption script.

Encryption using the Ejector - Source Code

Clone repository:

git clone https://github.com/lidofinance/validator-ejector.git
cd validator-ejector

Create an .env file with your encryption password or pass it to the terminal before proceeding:

MESSAGES_PASSWORD=password

Copy your .json exit message files to directory encryptor/input
Run yarn & yarn encrypt
Your encrypted exit message files will be saved to encryptor/output

Encryption using the Ejector - Docker

The ejector is bundled with an encryptor script inside, so you can run it using the same Docker image (the sha in the code must always be from the latest production release):

docker run \
	-e MESSAGES_PASSWORD=secret \
	-v /full/path/to/input:/app/encryptor/input/ \
	-v /full/path/to/output:/app/encryptor/output/ \
	lidofinance/validator-ejector@sha256:7b0c780d7d62afbf7ac7bdc6f7a35634e67a99ec733632c1fd6eed62388f4d19 \
	node /app/dist/encryptor/encrypt.js

For platforms with a different architecture but with emulation/transpilation support e.g. macOS on M-series processors, additionally specify:

--platform linux/amd64

Environment Variables

EXECUTION_NODE

Address of the execution node.

CONSENSUS_NODE

Address of the consensus node.

LOCATOR_ADDRESS

Address of the LidoLocator contract (can be looked up at the respective network’s deployed contracts page).

Hoodi Testnet:

0xe2EF9536DAAAEBFf5b1c130957AB3E80056b06D8

Holešky Testnet:

0x28FAB2059C713A7F9D8c86Db49f9bb0e96Af1ef8

Mainnet:

0xC1d0b3DE6792Bf6b4b37EccdcC24e45978Cfd2Eb

STAKING_MODULE_ID

ID of the StakingRouter contract module.

Currently, the staking router’s NodeOperatorsRegistry has three modules, the Curated one with ID 1 and Simple DVT with ID 2, and the Community Staking Module (CSM) with ID 3 (or 4 on testnets)

OPERATOR_ID

You can find this on the operators dashboard (e.g. #123 on the operator card):
Hoodi Curated module,
Hoodi Simple DVT module,
Holešky Curated module,
Holešky Simple DVT module,
Mainnet Curated module,
Mainnet Simple DVT module

MESSAGES_LOCATION

Location from which to load .json exit messages.

When set, messages mode will be activated. This is not needed if you use the ejector in webhook mode.

For example, /messages in Docker or simply messages if running directly for local files.

External storage bucket URLs are supported for AWS S3 and Google Cloud Storage:

s3:// for S3
gs:// for GCS

Authentication setup: S3, GCS

VALIDATOR_EXIT_WEBHOOK

Endpoint to fetch when an exit has to be made. Allows to implement a just in time approach by offloading exiting logic to an external service and using the ejector as a secure exit events reader.

When set, webhook mode will be activated. This is not needed if you use the ejector in messages mode.

On the endpoint, JSON will be POSTed with the following structure:

{
	"validatorIndex": "123",
	"validatorPubkey": "0x123"
}

A 200 response will be counted as a successful exit, non-200 ones as failures.

ORACLE_ADDRESSES_ALLOWLIST

JSON array of Lido Oracle addresses, from which only report transactions will be accepted.

To get the list of members you can go to the “Read Contract” section of the ValidatorsExitBusOracle’s HashConsensus on Etherscan and query the getMembers() method: Mainnet, Holešky, Hoodi

Example list for Mainnet (last updated 2024-10-25):

0x140Bd8FbDc884f48dA7cb1c09bE8A2fAdfea776E
0xA7410857ABbf75043d61ea54e07D57A6EB6EF186
0x404335BcE530400a5814375E7Ec1FB55fAff3eA2
0x946D3b081ed19173dC83Cd974fC69e1e760B7d78
0x007DE4a5F7bc37E2F26c0cb2E8A95006EE9B89b5
0xc79F702202E3A6B0B6310B537E786B9ACAA19BAf
0x61c91ECd902EB56e314bB2D5c5C07785444Ea1c8
0x73181107c8D9ED4ce0bbeF7A0b4ccf3320C41d12
0xe57B3792aDCc5da47EF4fF588883F0ee0c9835C9

Example .env entry for Mainnet (last updated 2024-10-25):

["0x140Bd8FbDc884f48dA7cb1c09bE8A2fAdfea776E","0xA7410857ABbf75043d61ea54e07D57A6EB6EF186","0x404335BcE530400a5814375E7Ec1FB55fAff3eA2","0x946D3b081ed19173dC83Cd974fC69e1e760B7d78","0x007DE4a5F7bc37E2F26c0cb2E8A95006EE9B89b5","0xc79F702202E3A6B0B6310B537E786B9ACAA19BAf","0x61c91ECd902EB56e314bB2D5c5C07785444Ea1c8","0x73181107c8D9ED4ce0bbeF7A0b4ccf3320C41d12","0xe57B3792aDCc5da47EF4fF588883F0ee0c9835C9"]

Note: make sure quotes are copied correctly if copying these examples.

Example list for Holešky (last updated 2024-09-30):

0x12A1D74F8697b9f4F1eEBb0a9d0FB6a751366399
0xD892c09b556b547c80B7d8c8cB8d75bf541B2284
0xf7aE520e99ed3C41180B5E12681d31Aa7302E4e5
0x31fa51343297FFce0CC1E67a50B2D3428057D1b1
0x81E411f1BFDa43493D7994F82fb61A415F6b8Fd4
0x4c75FA734a39f3a21C57e583c1c29942F021C6B7
0xD3b1e36A372Ca250eefF61f90E833Ca070559970
0xF0F23944EfC5A63c53632C571E7377b85d5E6B6f
0xb29dD2f6672C0DFF2d2f173087739A42877A5172
0xfe43A8B0b481Ae9fB1862d31826532047d2d538c

Example .env entry for Holešky (last updated 2024-09-30):

["0x12A1D74F8697b9f4F1eEBb0a9d0FB6a751366399","0xD892c09b556b547c80B7d8c8cB8d75bf541B2284","0xf7aE520e99ed3C41180B5E12681d31Aa7302E4e5","0x31fa51343297FFce0CC1E67a50B2D3428057D1b1","0x81E411f1BFDa43493D7994F82fb61A415F6b8Fd4","0x4c75FA734a39f3a21C57e583c1c29942F021C6B7","0xD3b1e36A372Ca250eefF61f90E833Ca070559970", "0xF0F23944EfC5A63c53632C571E7377b85d5E6B6f", "0xb29dD2f6672C0DFF2d2f173087739A42877A5172","0xfe43A8B0b481Ae9fB1862d31826532047d2d538c"]

Example list for Hoodi (last updated 2025-04-22):

1 0xcA80ee7313A315879f326105134F938676Cfd7a9
2 0xf03B8DC8762B97F13Ac82e6F94bE3Ed002FF7459
3 0x1932f53B1457a5987791a40Ba91f71c5Efd5788F
4 0xf7aE520e99ed3C41180B5E12681d31Aa7302E4e5
5 0x99B2B75F490fFC9A29E4E1f5987BE8e30E690aDF
6 0x219743f1911d84B32599BdC2Df21fC8Dba6F81a2
7 0xD3b1e36A372Ca250eefF61f90E833Ca070559970
8 0x4c75FA734a39f3a21C57e583c1c29942F021C6B7
9 0xB1cC91878c1831893D39C2Bb0988404ca5Fa7918
10 0xfe43A8B0b481Ae9fB1862d31826532047d2d538c
11 0x43C45C2455C49eed320F463fF4f1Ece3D2BF5aE2

Example .env entry for Hoodi (last updated 2025-04-22):

["0xcA80ee7313A315879f326105134F938676Cfd7a9","0xf03B8DC8762B97F13Ac82e6F94bE3Ed002FF7459","0x1932f53B1457a5987791a40Ba91f71c5Efd5788F","0xf7aE520e99ed3C41180B5E12681d31Aa7302E4e5","0x99B2B75F490fFC9A29E4E1f5987BE8e30E690aDF","0x219743f1911d84B32599BdC2Df21fC8Dba6F81a2","0xD3b1e36A372Ca250eefF61f90E833Ca070559970","0x4c75FA734a39f3a21C57e583c1c29942F021C6B7","0xB1cC91878c1831893D39C2Bb0988404ca5Fa7918","0xfe43A8B0b481Ae9fB1862d31826532047d2d538c","0x43C45C2455C49eed320F463fF4f1Ece3D2BF5aE2"]

Note: make sure quotes are copied correctly if copying these examples.

MESSAGES_PASSWORD

Password to decrypt encrypted exit messages with on application start.

MESSAGES_PASSWORD_FILE

Alternative to MESSAGES_PASSWORD. Path to file containing the password to decrypt exit messages with. If used, MESSAGES_PASSWORD (not MESSAGES_PASSWORD_FILE) needs to be added to LOGGER_SECRETS to be sanitized.

BLOCKS_PRELOAD

Amount of blocks to load events from on application start.

Suggested to include in your .env variables, but to be left at the default of 50000 (~7 days of blocks).

In case your ejector will be down due to an emergency, this value can be tweaked to let the ejector load a higher amount of blocks on start.

HTTP_PORT

Port for serving metrics and a health check endpoint, default 8989.

RUN_METRICS

Enable with true to serve Prometheus metrics: full list

Will be served on HOST:$HTTP_PORT/metrics.

Highly advised for monitoring and alerting.

RUN_HEALTH_CHECK

Enabled by default, disabled with false. Highly recommended to monitor this endpoint.

Will be served on HOST:$HTTP_PORT/health.

LOGGER_LEVEL

Recommended to set to info (default), can be changed to debug in case of issues for easier debugging.

LOGGER_FORMAT

Format of logs, simple by default, but can be set to json to be easily parsable, e.g. by Loki.

LOGGER_SECRETS

Environment variable names or exact values which should be replaced in logs, in JSON array of strings format.

Advised to include your MESSAGES_PASSWORD, EXECUTION_NODE, and MESSAGES_PASSWORD:

["MESSAGES_PASSWORD","EXECUTION_NODE","CONSENSUS_NODE"]

Note: make sure quotes are copied correctly if copying this example.

DRY_RUN

Allows to test the application with true without actually sending out exit messages.

Use with caution!

Make sure to set to false or completely leave it out in production.

Advanced Parameters

Please do not use unless suggested by a Lido contributor.

BLOCKS_LOOP - 900 (3 hours of blocks) - Amount of blocks the ejector looks behind on wake in polling jobs
JOB_INTERVAL - 384000 (1 epoch) - Time for which the ejector sleeps between jobs
DISABLE_SECURITY_DONT_USE_IN_PRODUCTION - false - Set to true to skip security checks, for example if the ValidatorsExitBusOracle’s HashConsensus contract was changed after the ejector was unable to exit validators e.g. because it was switched off

Running

Source Code

Clone repository:

git clone https://github.com/lidofinance/validator-ejector.git
cd validator-ejector

Create an exit messages directory, for example locally mkdir messages
Put your exit message files in the messages folder
Copy the env sample file cp sample.env .env
Fill the environment variables in the .env file
Run

yarn
yarn build
yarn start

Docker with docker-compose

Create a root directory for the ejector, cd into that folder
Create an exit messages directory mkdir messages
Put your exit message files in the messages folder
Copy the env sample file cp sample.env .env
Fill the environment variables in the .env file
Create a docker-compose.yml file using this template
Run docker-compose up or docker-compose up -d to start in detached mode (in the background)

Check the Ejector is working

Ensure that there are no errors in the logs and no container restarts occuring
Verify that the config logged on start is correctly represented in the logs
If you have put presigned messages in the messages folder, make sure the loadedMessages count is greater than 0
Ensure you can see the Job started and Job finished lines in the logs

An example of a correct operation log:

info: Application started, version 1.0.0 {"EXECUTION_NODE":"<secret>","CONSENSUS_NODE":"<secret>","LOCATOR_ADDRESS":"0x123","STAKING_MODULE_ID":"1","OPERATOR_ID":"0","MESSAGES_LOCATION":"messages","ORACLE_ADDRESSES_ALLOWLIST":["0x123"],"MESSAGES_PASSWORD":"<secret>","BLOCKS_PRELOAD":190000,"BLOCKS_LOOP":64,"JOB_INTERVAL":384000,"HTTP_PORT":8989,"RUN_METRICS":true,"RUN_HEALTH_CHECK":true,"DRY_RUN":false}
info: Loading messages from messages
info: Loaded 123 messages
info: Validating messages
info: Starting, searching only for requests for operator 0
info: Loading initial events for 190000 last blocks
info: Job started {"operatorId":"0","stakingModuleId":"1","loadedMessages":123}
info: Resolved Exit Bus contract address using the Locator {"exitBusAddress":"0x123"}
info: Resolved Consensus contract address {"consensusAddress":"0x123"}
info: Fetched the latest block from EL {"latestBlock":12345}
info: Fetching request events from the Exit Bus {"eventsNumber":190000,"fromBlock":12345,"toBlock":12345}
info: Loaded ValidatorExitRequest events {"amount":0}
info: Handling ejection requests {"amount":0}
info: Job finished
info: Starting 384 seconds polling for 64 last blocks

What if something is wrong?

Make sure your configuration is correct
Make sure you are on the recommended Docker image sha hash or version if running directly
Check if your nodes are synced and working correctly
Restart the application
Start the application with the LOGGER_LEVEL=debug environment variable and contact the Lido devs with logs to investigate the problem

Additional Resources

Validator Ejector GitHub Repository (Open Source)
https://github.com/lidofinance/validator-ejector

What's Changing for Node Operators in Lido V2 - parts can be outdated
https://hackmd.io/@lido/Byue6SQxh

Lido Withdrawals: Automating Validator Exits - parts can be outdated
https://hackmd.io/@lido/BkxRxAr-o

Ejector Logic Spec - parts can be outdated
https://hackmd.io/@lido/r1KZ4YNdj

Sven

2023/10/18 14:49:53

You can get a list from the Aragon ap

can we add or replace this with a generic call to the getMembers() function? Aragon ui is currently running into rpc issues and we need an alternative.

sssngth

2024/03/24 10:30:02

Added description for getMembers()

no3cm

2024/05/18 22:42:51

any chance to have it working on ARM devices? I'm a part of the obol/lido testnet, the ejector docker image isn't working for me