The validator ejector is a daemon service which monitors ValidatorsExitBusOracle events and sends out stored exit messages when necessary. It allows node operators to generate and sign exit messages ahead of time, which will be sent out by the ejector when the protocol requests an exit to be made.
On start, it loads exit messages from a specified folder of individual .json
files and validates their format, structure, and signature. Then, it loads events from a configurable amount of latest finalized blocks, checks if exits should be made and after that periodically fetches fresh events.
Docker & docker-compose
Node.js 16 or higher
The ejector loads and validates exit messages on start. This means that any changes to the messages folder (e.g. new exit messages) require a restart of the application to be picked up.
The ejector accepts messages in three formats:
{
"message": { "epoch": "123", "validator_index": "123" },
"signature": "0x123"
}
{
"exit": {
"message": { "epoch": "123", "validator_index": "123" },
"signature": "0x123"
},
"fork_version": "0x123"
}
{
"version": 4,
"uuid": "123abc-123abc-123abc",
"path": "",
"pubkey": "",
"crypto": {
"kdf": {
"function": "pbkdf2",
"params": {
"dklen": 123,
"c": 123,
"prf": "hmac-sha256",
"salt": "123abc"
},
"message": ""
},
"checksum": {
"function": "sha256",
"params": {},
"message": "123abc"
},
"cipher": {
"function": "aes-128-ctr",
"params": { "iv": "123abc" },
"message": "123abc"
}
}
}
It is highly advised that after exit messages are generated and signed, they should be encrypted for storage safety. The ejector will decrypt files on start by looking up the password in the MESSAGES_PASSWORD
environment variable.
Exit messages are encrypted and decrypted by the ejector according to the EIP-2335 specification.
The ejector is bundled with a small, easy to use encryption script.
git clone https://github.com/lidofinance/validator-ejector.git
cd validator-ejector
.env
file with your encryption password or pass it to the terminal before proceeding:MESSAGES_PASSWORD=password
.json
exit message files to directory encryptor/input
yarn & yarn encrypt
encryptor/output
The ejector is bundled with an encryptor script inside, so you can run it using the same Docker image (the sha in the code must always be from the latest production release):
docker run \
-e MESSAGES_PASSWORD=secret \
-v /full/path/to/input:/app/encryptor/input/ \
-v /full/path/to/output:/app/encryptor/output/ \
lidofinance/validator-ejector@sha256:7b0c780d7d62afbf7ac7bdc6f7a35634e67a99ec733632c1fd6eed62388f4d19 \
node /app/dist/encryptor/encrypt.js
For platforms with a different architecture but with emulation/transpilation support e.g. macOS on M-series processors, additionally specify:
--platform linux/amd64
Address of the execution node.
Address of the consensus node.
Address of the LidoLocator contract (can be looked up at the respective network’s deployed contracts page).
Holešky Testnet:
0x28FAB2059C713A7F9D8c86Db49f9bb0e96Af1ef8
Mainnet:
0xC1d0b3DE6792Bf6b4b37EccdcC24e45978Cfd2Eb
ID of the StakingRouter contract module.
Currently, the staking router’s NodeOperatorsRegistry has two modules, the Curated one with ID 1
and Simple DVT with ID 2
.
You can find this on the operators dashboard (e.g. #123 on the operator card): Holešky Curated module, Holešky Simple DVT module, Mainnet Curated module, Mainnet Simple DVT module
Location from which to load .json
exit messages.
When set, messages mode will be activated. This is not needed if you use the ejector in webhook mode.
For example, /messages
in Docker or simply messages
if running directly for local files.
External storage bucket URLs are supported for AWS S3 and Google Cloud Storage:
s3://
for S3gs://
for GCSEndpoint to fetch when an exit has to be made. Allows to implement a just in time approach by offloading exiting logic to an external service and using the ejector as a secure exit events reader.
When set, webhook mode will be activated. This is not needed if you use the ejector in messages mode.
On the endpoint, JSON will be POSTed with the following structure:
{
"validatorIndex": "123",
"validatorPubkey": "0x123"
}
A 200 response will be counted as a successful exit, non-200 ones as failures.
JSON array of Lido Oracle addresses, from which only report transactions will be accepted.
To get the list of members you can go to the “Read Contract” section of the ValidatorsExitBusOracle’s HashConsensus on Etherscan and query the getMembers()
method: Mainnet, Holešky
Example .env
entry for Mainnet (last updated 2024-10-25):
["0x140Bd8FbDc884f48dA7cb1c09bE8A2fAdfea776E","0xA7410857ABbf75043d61ea54e07D57A6EB6EF186","0x404335BcE530400a5814375E7Ec1FB55fAff3eA2","0x946D3b081ed19173dC83Cd974fC69e1e760B7d78","0x007DE4a5F7bc37E2F26c0cb2E8A95006EE9B89b5","0xc79F702202E3A6B0B6310B537E786B9ACAA19BAf","0x61c91ECd902EB56e314bB2D5c5C07785444Ea1c8","0x73181107c8D9ED4ce0bbeF7A0b4ccf3320C41d12","0xe57B3792aDCc5da47EF4fF588883F0ee0c9835C9"]
Note: make sure quotes are copied correctly if copying these examples.
Example .env
entry for Holešky (last updated 2024-09-30):
["0x12A1D74F8697b9f4F1eEBb0a9d0FB6a751366399","0xD892c09b556b547c80B7d8c8cB8d75bf541B2284","0xf7aE520e99ed3C41180B5E12681d31Aa7302E4e5","0x31fa51343297FFce0CC1E67a50B2D3428057D1b1","0x81E411f1BFDa43493D7994F82fb61A415F6b8Fd4","0x4c75FA734a39f3a21C57e583c1c29942F021C6B7","0xD3b1e36A372Ca250eefF61f90E833Ca070559970", "0xF0F23944EfC5A63c53632C571E7377b85d5E6B6f", "0xb29dD2f6672C0DFF2d2f173087739A42877A5172","0xfe43A8B0b481Ae9fB1862d31826532047d2d538c"]
Note: make sure quotes are copied correctly if copying these examples.
Password to decrypt encrypted exit messages with on application start.
Alternative to MESSAGES_PASSWORD
. Path to file containing the password to decrypt exit messages with. If used, MESSAGES_PASSWORD (not MESSAGES_PASSWORD_FILE) needs to be added to LOGGER_SECRETS to be sanitized.
Amount of blocks to load events from on application start.
Suggested to include in your .env
variables, but to be left at the default of 50000 (~7 days of blocks).
In case your ejector will be down due to an emergency, this value can be tweaked to let the ejector load a higher amount of blocks on start.
Port for serving metrics and a health check endpoint, default 8989.
Enable with true
to serve Prometheus metrics: full list
Will be served on HOST:$HTTP_PORT/metrics
.
Highly advised for monitoring and alerting.
Enabled by default, disabled with false
. Highly recommended to monitor this endpoint.
Will be served on HOST:$HTTP_PORT/health
.
Recommended to set to info
(default), can be changed to debug
in case of issues for easier debugging.
Format of logs, simple
by default, but can be set to json
to be easily parsable, e.g. by Loki.
Environment variable names or exact values which should be replaced in logs, in JSON array of strings format.
Advised to include your MESSAGES_PASSWORD, EXECUTION_NODE, and MESSAGES_PASSWORD:
["MESSAGES_PASSWORD","EXECUTION_NODE","CONSENSUS_NODE"]
Note: make sure quotes are copied correctly if copying this example.
Allows to test the application with true
without actually sending out exit messages.
Use with caution!
Make sure to set to false
or completely leave it out in production.
Please do not use unless suggested by a Lido contributor.
true
to skip security checks, for example if the ValidatorsExitBusOracle’s HashConsensus contract was changed after the ejector was unable to exit validators e.g. because it was switched offgit clone https://github.com/lidofinance/validator-ejector.git
cd validator-ejector
mkdir messages
cp sample.env .env
.env
fileyarn
yarn build
yarn start
cd
into that foldermkdir messages
cp sample.env .env
.env
filedocker-compose.yml
file using this templatedocker-compose up
or docker-compose up -d
to start in detached mode (in the background)loadedMessages
count is greater than 0
Job started
and Job finished
lines in the logsAn example of a correct operation log:
info: Application started, version 1.0.0 {"EXECUTION_NODE":"<secret>","CONSENSUS_NODE":"<secret>","LOCATOR_ADDRESS":"0x123","STAKING_MODULE_ID":"1","OPERATOR_ID":"0","MESSAGES_LOCATION":"messages","ORACLE_ADDRESSES_ALLOWLIST":["0x123"],"MESSAGES_PASSWORD":"<secret>","BLOCKS_PRELOAD":190000,"BLOCKS_LOOP":64,"JOB_INTERVAL":384000,"HTTP_PORT":8989,"RUN_METRICS":true,"RUN_HEALTH_CHECK":true,"DRY_RUN":false}
info: Loading messages from messages
info: Loaded 123 messages
info: Validating messages
info: Starting, searching only for requests for operator 0
info: Loading initial events for 190000 last blocks
info: Job started {"operatorId":"0","stakingModuleId":"1","loadedMessages":123}
info: Resolved Exit Bus contract address using the Locator {"exitBusAddress":"0x123"}
info: Resolved Consensus contract address {"consensusAddress":"0x123"}
info: Fetched the latest block from EL {"latestBlock":12345}
info: Fetching request events from the Exit Bus {"eventsNumber":190000,"fromBlock":12345,"toBlock":12345}
info: Loaded ValidatorExitRequest events {"amount":0}
info: Handling ejection requests {"amount":0}
info: Job finished
info: Starting 384 seconds polling for 64 last blocks
Validator Ejector GitHub Repository (Open Source)
https://github.com/lidofinance/validator-ejector
What's Changing for Node Operators in Lido V2 - parts can be outdated
https://hackmd.io/@lido/Byue6SQxh
Lido Withdrawals: Automating Validator Exits - parts can be outdated
https://hackmd.io/@lido/BkxRxAr-o
Ejector Logic Spec - parts can be outdated
https://hackmd.io/@lido/r1KZ4YNdj