Zurich Reproduction

# Zurich Reproduction ## Steps 0. Start the honest peers. 1. S3I1-Honest 2. S2I1-HonestMiner, S2I2-Honest 3. S4I1-HonestVictim 2. `orch` needs to be ready; 1. `serveDb.py` 2. `orch --fraction FRACTION --time TIME --honest-hashrate RATE` 3. > Wait for the orchestrator to print `"Attack ready to go"`. 3. `debug.setHead(2 million = 0x1e8480)` on S2I1-HonestMiner 4. ... simultaneously (approximately) launch the malicious peers. ## Infrastructure - `geth` is an honest node running `ethereum/go-ethereum@v1.10.23` - with `MinimumDifficulty` patched to `0x10000` - `mgeth` is a malicious node running `malicious-go-ethereum`. All servers have directory trees like: ```shell [servers 2..4] /root/go-ethereum # go-ethereum source (built for honest nodes) [server 1] /root/mgeth # go-ethereum source (built for malicious nodes) /root/go # GOROOT (1.17.13) /root/zurich # testdata, scripts /root/zurich/datadirs/<node_name> # chaindata (via `geth --datadir`) /root/zurich/logs # logs get written file/instantiation here /root/zurich/genesis.json /root/zurich/exported_blockchain /root/zurich/start_*.sh # startup script(s) for geth/mgeth /root/zurich_original.tar.gz # backup of original data given by ETH Zurich ``` ### Server 1 Running tmux sessions (eg. `tmux a -t orch`): - orch (/root/zurich/start_orch.sh) - malicious-node-1 (/root/zurich/start_malicious_1.sh) - malicious-node-2 (/root/zurich/start_malicious_2.sh) - seed-server (/root/zurich/start_seed_server.sh) ``` > find /root/zurich -type f -name '*static*' zurich/datadirs/malicious1/static-peers.json zurich/datadirs/malicious2/static-peers.json ``` - [x] `mgeth` - [x] static peers - [x] `mgeth` - [x] static peers - [x] `orch` - [x] seeds table - [x] python seeds server ### Server 2 Runs in tmux: - honest-mine-node-1 (/root/zurich/start_honest-mine-node-1.sh) - honest-node-2 (/root/zurich/start_honest-node-2.sh) - [x] `geth` __miner__ (`honest_mining_1`) - [x] `--mine --miner.etherbase=0xab...cd --miner.threads=$(numcpu / 2)` - [x] `datadirs/honest_mining_1/nodekey` = `954835a2bca35dd5aca02b219e82fd0dc55d2a44d2cf2dd42b1750cb79764901` - [ ] `geth` (`honest_2`) - [x] `datadirs/honest_2/nodekey` = `b3e0ccd0f7019ea4233a5e69ec4cd6449cd3f81f8d0c700bf0ecc02e35ddaebf` ### Server 3 Runs in tmux: - honest-node-1 (/root/zurich/start_honest-node-1.sh) - honest-node-2 (/root/zurich/start_honest-node-2.sh) - [ ] `geth` (`honest_1`) - [x] `datadirs/honest_1/nodekey` = `39a1fa9420e1caf1131ee41bbdb72068906761563ce5938983f40e6b8c433338` - [ ] `geth` (`honest_2`) - [x] `datadirs/honest_2/nodekey` = `a9253bc0bb7e3ff416df25760cf16a6fca4c9f1e51dfe255335a6aae305731bd` ### Server 4 Runs in tmux: - victim-node (/root/zurich/start_victim-node.sh) - [ ] `geth` __victim__ - [x] `datadirs/victim_1/nodekey` = `de5adf2df2e6ee7f08c003058f2aa7ea9d820a066fc300cc3b17c35a604074b9` --- ### Experiment Notes > ## Start the victim node > 1. Download the `core-geth` source code. > 2. You may want to change `MinimumDifficulty` in `params/protocol_params.go` according to your needs. - :bug: `mgeth` uses tag **1.10.23**. Based on **go.mod**, it uses **go1.15**; but does not compile with this Go version. I used **go1.17.13** to successfully compile it instead. - :construction: We had to change the `geth` `MinimumDifficulty` to the value from the provided `genesis.json` file: `65536` (`0x10000`). Isaac 20230314 - [ ] We need to make sure the honest nodes - [ ] are communicating with one another, - [ ] and are in sync; - [ ] use `static-peers` to connect them to each other. - [ ] We need to estimate honest miner's hashrate. - [ ] and use it to configure the mgeth nodes accordingly. **TODO(meowsbits): THIS IS RUNNING CURRENTLY** S2I1(miner): Number of machine CPUs=8; geth CPU miner running with `miner.start(4)`. _Update:_ Going with `150000`; seems like a reasonable average. Isaac 20230315 Had a hard time getting the honest peers to network. Using `--bootnodes` in combination with changing (adding) the `discport` query value as a non-zero value equivalent to the `:port`, eg. `168.142.145.498:30323?discport=0` becomes `168.142.145.498:30323?discport=30323`. With this pattern, I configured all honest, non-mining, non-victim nodes to use the single honest miner as the bootnode. But I'd rather rely only on a working `static-peers.json` file instead, so I'm going to try changing (adding/aforementioned fix) to the ports in their static peers lists and see if that works too. - [x] Got a bug (a panic) when adding the victim as a peer to the malicious geth. Have screenshot of logs. ``` ATCK [03-15|19:05:37.942] Message handling done INFO [03-15|19:19:15.198] Looking for peers peercount=0 tried=0 static=1 INFO [03-15|19:19:15.304] Peer announced its TD peer=a8ed5d30 td=65536 ATCK [03-15|19:19:15.305] Prediction TD in database: 65536 ATCK [03-15|19:19:15.305] Cheating to peer a8ed5d30 if necessary at this point panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x9e702f] goroutine 226 [running]: github.com/ethereum/go-ethereum/attack/bridge.latest(0x1) /root/mgeth/attack/bridge/internals.go:103 +0x16f github.com/ethereum/go-ethereum/attack/bridge.CheatAboutTd({0xc00266dac0, 0x2}, 0xc000665950) /root/mgeth/attack/bridge/bridge.go:576 +0x206 github.com/ethereum/go-ethereum/eth/protocols/eth.(*Peer).Handshake(0xc0061e84e0, 0x1, 0xc00266a5e0, {0x66, 0x56, 0xf9, 0x50, 0x5f, 0xd8, 0xb9, ...}, ...) /root/mgeth/eth/protocols/eth/handshake.go:121 +0x62e github.com/ethereum/go-ethereum/eth.(*handler).runEthPeer(0xc0026e42d0, 0xc0061e84e0, 0xc00634e198) /root/mgeth/eth/handler.go:327 +0x4f1 github.com/ethereum/go-ethereum/eth.(*ethHandler).RunPeer(0x42, 0xc00635a100, 0x17993e0) /root/mgeth/eth/handler_eth.go:46 +0x19 github.com/ethereum/go-ethereum/eth/protocols/eth.MakeProtocols.func1(0x0, {0x17993e0, 0xc006808000}) /root/mgeth/eth/protocols/eth/handler.go:111 +0x122 github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1() /root/mgeth/p2p/peer.go:450 +0x8c created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols /root/mgeth/p2p/peer.go:448 +0xb3 ``` Still having a hard time getting static-peers.json to work consistently, even with the fixed discport=xxx option. Nodes are getting usually only 0,1,or 2 peers (expected 4+). Adding them manually via console works ok. - [x] Got another bug (panic) at `mgeth`. This seems out of the blue. ``` INFO [03-15|19:05:08.859] Started goroutine for importing fake batches ATCK [03-15|19:05:37.940] Handling new message ATCK [03-15|19:05:37.940] Code: 9 ATCK [03-15|19:05:37.940] Attack phase switched to ready ATCK [03-15|19:05:37.940] Message handling done ATCK [03-15|19:05:37.961] Handling new message ATCK [03-15|19:05:37.961] Code: 13 ATCK [03-15|19:05:37.961] Set avoidVictim = true ATCK [03-15|19:05:37.961] Message handling done ATCK [03-15|19:21:06.551] Handling new message ATCK [03-15|19:21:06.551] Code: 12 INFO [03-15|19:21:06.551] Stopping goroutine for importing fake batches INFO [03-15|19:21:06.551] IPC endpoint closed url=/root/zurich/datadirs/malicious2/geth.ipc ATCK [03-15|19:21:06.551] Quitting read loop ATCK [03-15|19:21:06.551] Message handling done ATCK [03-15|19:21:06.551] Handling new message panic: runtime error: slice bounds out of range [:4] with capacity 0 goroutine 44 [running]: github.com/ethereum/go-ethereum/attack/msg.Decode({0x0, 0x1, 0x1}) /root/mgeth/attack/msg/messages.go:191 +0x155 github.com/ethereum/go-ethereum/attack/bridge.handleMessages() /root/mgeth/attack/bridge/bridge.go:1224 +0x16b created by github.com/ethereum/go-ethereum/attack/bridge.Initialize /root/mgeth/attack/bridge/bridge.go:153 +0x91d root@zurich-8vcpu-16gb-fra1-01:~/zurich# ``` - [x] Another `mgeth` panic at `mgeth-1`. Restarted; did not occur with `mgeth-2` and victim is syncing. ``` ATCK [03-15|19:23:35.499] Attack phase switched to ready ATCK [03-15|19:23:35.499] Message handling done INFO [03-15|19:28:27.475] Looking for peers peercount=1 tried=0 static=0 INFO [03-15|19:28:27.475] Peer announced its TD peer=a8ed5d30 td=65536 ATCK [03-15|19:28:27.475] Prediction TD in database: 65536 ATCK [03-15|19:28:27.475] Cheating to peer a8ed5d30 if necessary at this point panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x9e702f] goroutine 57 [running]: github.com/ethereum/go-ethereum/attack/bridge.latest(0x1) /root/mgeth/attack/bridge/internals.go:103 +0x16f github.com/ethereum/go-ethereum/attack/bridge.CheatAboutTd({0xc006a120c0, 0x2}, 0xc007a49950) /root/mgeth/attack/bridge/bridge.go:576 +0x206 github.com/ethereum/go-ethereum/eth/protocols/eth.(*Peer).Handshake(0xc0003f5110, 0x1, 0xc002d09da0, {0x66, 0x56, 0xf9, 0x50, 0x5f, 0xd8, 0xb9, ...}, ...) /root/mgeth/eth/protocols/eth/handshake.go:121 +0x62e github.com/ethereum/go-ethereum/eth.(*handler).runEthPeer(0xc002d842d0, 0xc0003f5110, 0xc00247c060) /root/mgeth/eth/handler.go:327 +0x4f1 github.com/ethereum/go-ethereum/eth.(*ethHandler).RunPeer(0x42, 0xc0001e2900, 0x17993e0) /root/mgeth/eth/handler_eth.go:46 +0x19 github.com/ethereum/go-ethereum/eth/protocols/eth.MakeProtocols.func1(0xc0029e8370, {0x17993e0, 0xc0069a81e0}) /root/mgeth/eth/protocols/eth/handler.go:111 +0x122 github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1() /root/mgeth/p2p/peer.go:450 +0x8c created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols /root/mgeth/p2p/peer.go:448 +0xb3 ``` - [x] Getting lots of these logs from `mgeth-1` ``` ERROR[03-15|19:31:31.914] Write ERROR[03-15|19:31:31.927] Write ERROR[03-15|19:31:32.022] Write ERROR[03-15|19:31:32.071] Write ERROR[03-15|19:31:32.659] Write ERROR[03-15|19:31:32.672] Write ERROR[03-15|19:31:32.850] Write ERROR[03-15|19:31:32.858] Write ERROR[03-15|19:31:33.040] Write ERROR[03-15|19:31:33.051] Write ERROR[03-15|19:31:33.214] Write ERROR[03-15|19:31:33.225] Write ERROR[03-15|19:31:33.316] Write ERROR[03-15|19:31:33.383] Write ERROR[03-15|19:31:33.459] Write ERROR[03-15|19:31:33.838] Write ERROR[03-15|19:31:33.945] Write ERROR[03-15|19:31:33.953] Write ERROR[03-15|19:31:34.113] Write ERROR[03-15|19:31:34.125] Write ERROR[03-15|19:31:34.283] Write ERROR[03-15|19:31:34.291] Write ERROR[03-15|19:31:34.482] Write ERROR[03-15|19:31:34.491] Write ERROR[03-15|19:31:34.578] Write ERROR[03-15|19:31:34.622] Write ... INFO [03-15|19:31:59.264] Using honest chain to fulfil it ERROR[03-15|19:31:59.264] Write INFO [03-15|19:31:59.264] Got headers for query amount=192 from=413,377 len_headers=192 peer=a8ed5d30 INFO [03-15|19:31:59.365] Replying to query id=6,413,064,872,273,108,398 ERROR[03-15|19:31:59.365] Write INFO [03-15|19:31:59.377] Received query query="&{Origin:{Hash:0x0000000000000000000000000000000000000000000000000000000000000000 Number:414145} Amount:192 Skip:0 Reverse:false}" id=334,592,203,248,864,406 INFO [03-15|19:31:59.377] Using honest chain to fulfil it ERROR[03-15|19:31:59.377] Write INFO [03-15|19:31:59.377] Got headers for query amount=192 from=414,145 len_headers=192 peer=a8ed5d30 INFO [03-15|19:31:59.477] Replying to query id=334,592,203,248,864,406 ERROR[03-15|19:31:59.478] Write INFO [03-15|19:31:59.492] Received query query="&{Origin:{Hash:0x0000000000000000000000000000000000000000000000000000000000000000 Number:417217} Amount:192 Skip:0 Reverse:false}" id=13,192,938,265,651,319,816 INFO [03-15|19:31:59.492] Using honest chain to fulfil it INFO [03-15|19:31:59.492] Got headers for query amount=192 from=417,217 len_headers=192 peer=a8ed5d30 INFO [03-15|19:31:59.594] Replying to query id=13,192,938,265,651,319,816 ERROR[03-15|19:31:59.594] Write ERROR[03-15|19:31:59.767] Write ERROR[03-15|19:31:59.872] Write ERROR[03-15|19:31:59.880] Write ERROR[03-15|19:32:00.042] Write ERROR[03-15|19:32:00.051] Write ERROR[03-15|19:32:00.220] Write ``` So I'm going to leave it as-is (using only `static-peers.json`, no `--bootnodes`) and once the honest peers (incl. miner) are networked, I'll 1. ~~add the victim node a peer to the malicious nodes 1 and 2.~~ This causes the aforementioned panic. Bummer. Instead: add the malicious peers to the victim node. 3. add the victim node to the honest miner 4. add the victim node to S3I1 (an honest node) 5. start the honest miner. Let the attack run. - [x] All nodes are at block 2000000. - [x] I've made sure that the malicious peers were started after the latest `orch` instantiation and that `orch` has printed `Attack ready`. - [x] I've made sure that the victim node was started after orch and the malicious peers. Add the malicious nodes to the victim via the console: ```js // mgeth-1 admin.addPeer("enode://ca0110925670cc2108787a4559993fe49347dbaed61154d93f532b390f8af1aa35ef228ca7feaa2165ef361cc24ac88c8ce1a051420e95bb4720f2a8d82c0a0c@134.122.82.71:30303?discport=30303"); // mgeth-2 admin.addPeer("enode://9f86dabd4f8fbc5b756fa0e9e5917f70f45c03a5ab2fd5782c93a99cbf72ad86976a1b3b1e66a852ec2532914330e4f4e9a83a7851d272de5a323ed38ebb8bd8@134.122.82.71:30304?discport=30304"); ``` Add the victim node to the honest miner via the console: (only works with honest nodes; mgeth panics on `admin.addPeer`): ``` admin.addPeer("enode://c14cea56b96baa60554890e59d95c0c24f1853dd4ce56a6d1d131dd3e8277861a33d5b4f7973ba1cf95f8a7c37fc66ff5ae9184e42b63ab8df6ca1295168896b@46.101.219.187:30343?discport=30343") ``` Start the miner (4 is half of the 8 total CPUs on the machine): ```js miner.start(4) ``` 20230316 Isaac - [x] s/`static-peers.json`/`static-nodes.json`/g. No wonder I couldn't get em to peer right. Just another couple data points for the honest miner hashrate this morn: ``` > console.log(new Date(), eth.mining, ethash.getHashrate(), eth.syncing, eth.blockNumber, admin.peers.length); Thu Mar 16 2023 14:28:42 GMT+0000 (UTC) true 149644 false 2011294 4 null > console.log(new Date(), eth.mining, ethash.getHashrate(), eth.syncing, eth.blockNumber, admin.peers.length); Thu Mar 16 2023 15:33:24 GMT+0000 (UTC) true 153668 false 2011617 5 null ``` I have now renamed all honest nodes `static-peers.json` files to `static-nodes.json`. I plan to restart honest nodes and network, allowing the non-mining honest peers to sync with the additional 11k blocks produced by the miner (on top of the original 2M block import). Once the honest network is in sync and the honest miner is mining at a plateau rate -- which is currently; the miner has produced 11k blocks -- then I'll restart the orch and mgeth instances and allow them to connect to the network. Once the mgeth's are seen to be well connected (and synced? - I assume so), then I'll nuke the victim's datadir and restart her. This will allow the mgeths to identify and attempt to lure her to a bifurcated fate. :bug: Bug: `WARN [03-16|15:29:00.775] Head state missing, repairing number=2,011,598 hash=222495..13cff6 snaproot=ea4f9c..80fc82` I just restarted the miner. The head state is missing for blocks which this very instance supposedly mined. But on restart it rolled all 11k blocks back and started mining again at block 2M. Logs below. _Update (1 minute later)_: The node re-synced the 11k blocks and is now mining on top of 2,011,610. I'm chalking this up to an (unfortunately common) ungraceful shutdown. ```sh INFO [03-16|15:29:00.636] - Petersburg: 0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md) INFO [03-16|15:29:00.636] - Istanbul: 0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md) INFO [03-16|15:29:00.636] - Muir Glacier: 0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md) INFO [03-16|15:29:00.636] - Berlin: 0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md) INFO [03-16|15:29:00.636] - London: 0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md) INFO [03-16|15:29:00.636] - Arrow Glacier: 0 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md) INFO [03-16|15:29:00.636] INFO [03-16|15:29:00.636] The Merge is not yet available for this network! INFO [03-16|15:29:00.636] - Hard-fork specification: https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md INFO [03-16|15:29:00.636] --------------------------------------------------------------------------------------------------------------------------------------------------------- INFO [03-16|15:29:00.636] INFO [03-16|15:29:00.721] Disk storage enabled for ethash caches dir=/root/zurich/datadirs/honest_mining_1/geth/ethash count=3 INFO [03-16|15:29:00.721] Disk storage enabled for ethash DAGs dir=/root/.ethash count=2 INFO [03-16|15:29:00.722] Initialising Ethereum protocol network=1 dbversion=8 INFO [03-16|15:29:00.774] Loaded most recent local header number=2,011,598 hash=222495..13cff6 td=142,779,778,411 age=42s INFO [03-16|15:29:00.774] Loaded most recent local full block number=2,011,598 hash=222495..13cff6 td=142,779,778,411 age=42s INFO [03-16|15:29:00.774] Loaded most recent local fast block number=2,011,598 hash=222495..13cff6 td=142,779,778,411 age=42s WARN [03-16|15:29:00.775] Head state missing, repairing number=2,011,598 hash=222495..13cff6 snaproot=ea4f9c..80fc82 INFO [03-16|15:29:06.732] Loaded most recent local header number=2,011,598 hash=222495..13cff6 td=142,779,778,411 age=48s INFO [03-16|15:29:06.732] Loaded most recent local full block number=2,000,000 hash=6656f9..9a3623 td=131,072,065,536 age=53y1mo3w INFO [03-16|15:29:06.732] Loaded most recent local fast block number=2,011,598 hash=222495..13cff6 td=142,779,778,411 age=48s WARN [03-16|15:29:06.732] Enabling snapshot recovery chainhead=2,000,000 diskbase=2,000,096 WARN [03-16|15:29:06.742] Loaded snapshot journal diskroot=ea4f9c..80fc82 diffs=unmatched WARN [03-16|15:29:06.742] Snapshot is not continuous with chain snaproot=ea4f9c..80fc82 chainroot=5d663e..46ee90 INFO [03-16|15:29:06.742] Loaded local transaction journal transactions=0 dropped=0 INFO [03-16|15:29:06.742] Regenerated local transaction journal transactions=0 accounts=0 WARN [03-16|15:29:06.742] Switch sync mode from snap sync to full sync INFO [03-16|15:29:06.743] Gasprice oracle is ignoring threshold set threshold=2 WARN [03-16|15:29:06.743] Unclean shutdown detected booted=2023-03-14T17:06:39+0000 age=1d22h22m WARN [03-16|15:29:06.743] Unclean shutdown detected booted=2023-03-15T15:31:31+0000 age=23h57m35s WARN [03-16|15:29:06.743] Unclean shutdown detected booted=2023-03-15T17:23:06+0000 age=22h6m WARN [03-16|15:29:06.743] Unclean shutdown detected booted=2023-03-15T18:42:09+0000 age=20h46m57s WARN [03-16|15:29:06.743] Unclean shutdown detected booted=2023-03-15T18:44:34+0000 age=20h44m32s WARN [03-16|15:29:06.743] Unclean shutdown detected booted=2023-03-16T15:25:46+0000 age=3m20s WARN [03-16|15:29:06.743] Engine API enabled protocol=eth WARN [03-16|15:29:06.743] Engine API started but chain not configured for merge yet INFO [03-16|15:29:06.744] Starting peer-to-peer node instance=Geth/v1.10.23-stable/linux-amd64/go1.17.13 INFO [03-16|15:29:06.768] New local node record seq=1,678,980,546,767 id=6cf172019e8979f2 ip=127.0.0.1 udp=30323 tcp=30323 INFO [03-16|15:29:06.769] Started P2P networking self=enode://910da3fcffa54cc9dccc15827dae54acb23d9c30ace7369b8e344b0e8531ece2424fc1cc5ecde024e1976b48258f2926fa388b4803fbdd69c0bfee9eb126fe37@127.0.0.1:30323 INFO [03-16|15:29:06.771] IPC endpoint opened url=/root/zurich/datadirs/honest_mining_1/geth.ipc INFO [03-16|15:29:06.771] Loaded JWT secret file path=/root/zurich/datadirs/honest_mining_1/geth/jwtsecret crc32=0x796c7a7e INFO [03-16|15:29:06.773] WebSocket enabled url=ws://127.0.0.1:8551 INFO [03-16|15:29:06.773] HTTP server started endpoint=127.0.0.1:8551 auth=true prefix= cors=localhost vhosts=localhost INFO [03-16|15:29:06.773] Transaction pool price threshold updated price=0 INFO [03-16|15:29:06.773] Updated mining threads threads=4 INFO [03-16|15:29:06.774] Transaction pool price threshold updated price=1,000,000,000 INFO [03-16|15:29:06.774] Commit new sealing work number=2,000,001 sealhash=c7c151..7755fd uncles=0 txs=0 gas=0 fees=0 elapsed="243.067µs" INFO [03-16|15:29:06.774] Commit new sealing work number=2,000,001 sealhash=c7c151..7755fd uncles=0 txs=0 gas=0 fees=0 elapsed="601.761µs" INFO [03-16|15:29:07.105] Successfully sealed new block number=2,000,001 sealhash=c7c151..7755fd hash=1e495c..5047c0 elapsed=331.069ms INFO [03-16|15:29:07.105] 🔨 mined potential block number=2,000,001 hash=1e495c..5047c0 INFO [03-16|15:29:07.106] Commit new sealing work number=2,000,002 sealhash=01ae32..5a6b22 uncles=0 txs=0 gas=0 fees=0 elapsed="408.452µs" INFO [03-16|15:29:07.106] Commit new sealing work number=2,000,002 sealhash=01ae32..5a6b22 uncles=0 txs=0 gas=0 fees=0 elapsed="722.307µs" INFO [03-16|15:29:09.167] Successfully sealed new block number=2,000,002 sealhash=01ae32..5a6b22 hash=369c8d..a716e1 elapsed=2.061s INFO [03-16|15:29:09.167] 🔨 mined potential block number=2,000,002 hash=369c8d..a716e1 INFO [03-16|15:29:09.167] Commit new sealing work number=2,000,003 sealhash=3be565..1fcd06 uncles=0 txs=0 gas=0 fees=0 elapsed="218µs" INFO [03-16|15:29:09.167] Commit new sealing work number=2,000,003 sealhash=3be565..1fcd06 uncles=0 txs=0 gas=0 fees=0 elapsed="379.362µs" INFO [03-16|15:29:09.300] Successfully sealed new block number=2,000,003 sealhash=3be565..1fcd06 hash=dea81e..649603 elapsed=132.925ms INFO [03-16|15:29:09.300] 🔨 mined potential block number=2,000,003 hash=dea81e..649603 INFO [03-16|15:29:09.300] Commit new sealing work number=2,000,004 sealhash=f1553b..1e84f1 uncles=0 txs=0 gas=0 fees=0 elapsed="248.315µs" INFO [03-16|15:29:09.301] Commit new sealing work number=2,000,004 sealhash=f1553b..1e84f1 uncles=0 txs=0 gas=0 fees=0 elapsed="685.185µs" INFO [03-16|15:29:09.695] Successfully sealed new block number=2,000,004 sealhash=f1553b..1e84f1 hash=768e9a..8dd9e6 elapsed=394.955ms INFO [03-16|15:29:09.695] 🔨 mined potential block number=2,000,004 hash=768e9a..8dd9e6 INFO [03-16|15:29:09.696] Commit new sealing work number=2,000,005 sealhash=d7967e..4f6a8c uncles=0 txs=0 gas=0 fees=0 elapsed="401.798µs" INFO [03-16|15:29:09.696] Commit new sealing work number=2,000,005 sealhash=d7967e..4f6a8c uncles=0 txs=0 gas=0 fees=0 elapsed="741.963µs" INFO [03-16|15:29:09.715] Successfully sealed new block number=2,000,005 sealhash=d7967e..4f6a8c hash=d30faa..af458e elapsed=19.042ms INFO [03-16|15:29:09.715] 🔨 mined potential block number=2,000,005 hash=d30faa..af458e ``` Later. Unclean shutdown strikes again. Both Server 3 nodes (S3N1, S3N2) are rewinding to `0` after an unclean shutdown (via CTRL+c). God damn it. This is going to take a long time. ``` INFO [03-16|15:35:49.179] - Hard-fork specification: https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md INFO [03-16|15:35:49.179] --------------------------------------------------------------------------------------------------------------------------------------------------------- INFO [03-16|15:35:49.179] INFO [03-16|15:35:49.271] Disk storage enabled for ethash caches dir=/root/zurich/datadirs/honest_1/geth/ethash count=3 INFO [03-16|15:35:49.271] Disk storage enabled for ethash DAGs dir=/root/.ethash count=2 INFO [03-16|15:35:49.271] Initialising Ethereum protocol network=1 dbversion=8 INFO [03-16|15:35:49.273] Loaded most recent local header number=2,011,629 hash=7f2b2b..9909fb td=142,841,385,885 age=32s INFO [03-16|15:35:49.273] Loaded most recent local full block number=2,011,629 hash=7f2b2b..9909fb td=142,841,385,885 age=32s INFO [03-16|15:35:49.273] Loaded most recent local fast block number=2,011,629 hash=7f2b2b..9909fb td=142,841,385,885 age=32s WARN [03-16|15:35:49.275] Head state missing, repairing number=2,011,629 hash=7f2b2b..9909fb snaproot=abc257..05cae9 WARN [03-16|15:47:48.278] Force rewinding till ancient limit head=0 ``` Follow up from this. Annoying. I'll run the recommended command (making sure to use `-datadir` appropriately) and see what happens. ```shell INFO [03-16|15:39:27.305] INFO [03-16|15:39:27.394] Disk storage enabled for ethash caches dir=/root/zurich/datadirs/honest_2/geth/ethash count=3 INFO [03-16|15:39:27.394] Disk storage enabled for ethash DAGs dir=/root/.ethash count=2 INFO [03-16|15:39:27.394] Initialising Ethereum protocol network=1 dbversion=8 INFO [03-16|15:39:27.396] Loaded most recent local header number=2,011,645 hash=ce3c85..86ad2e td=142,873,225,798 age=41s INFO [03-16|15:39:27.396] Loaded most recent local full block number=2,011,645 hash=ce3c85..86ad2e td=142,873,225,798 age=41s INFO [03-16|15:39:27.396] Loaded most recent local fast block number=2,011,645 hash=ce3c85..86ad2e td=142,873,225,798 age=41s WARN [03-16|15:39:27.397] Head state missing, repairing number=2,011,645 hash=ce3c85..86ad2e snaproot=abc257..05cae9 WARN [03-16|15:51:55.254] Force rewinding till ancient limit head=0 INFO [03-16|16:11:56.671] Loaded most recent local header number=0 hash=2289e1..c346e7 td=65536 age=53y11mo3w INFO [03-16|16:11:56.671] Loaded most recent local full block number=0 hash=2289e1..c346e7 td=65536 age=53y11mo3w INFO [03-16|16:11:56.671] Loaded most recent local fast block number=0 hash=2289e1..c346e7 td=65536 age=53y11mo3w WARN [03-16|16:11:56.672] Enabling snapshot recovery chainhead=0 diskbase=1,997,301 WARN [03-16|16:11:56.774] Snapshot is not continuous with chain snaproot=5d663e..46ee90 chainroot=56e81f..63b421 INFO [03-16|16:11:56.774] Loaded local transaction journal transactions=0 dropped=0 INFO [03-16|16:11:56.775] Regenerated local transaction journal transactions=0 accounts=0 INFO [03-16|16:11:56.776] Gasprice oracle is ignoring threshold set threshold=2 WARN [03-16|16:11:56.776] Unclean shutdown detected booted=2023-03-15T15:21:12+0000 age=1d50m44s WARN [03-16|16:11:56.776] Unclean shutdown detected booted=2023-03-15T17:23:39+0000 age=22h48m17s WARN [03-16|16:11:56.776] Unclean shutdown detected booted=2023-03-15T18:26:05+0000 age=21h45m51s WARN [03-16|16:11:56.776] Unclean shutdown detected booted=2023-03-15T18:40:36+0000 age=21h31m20s WARN [03-16|16:11:56.776] Unclean shutdown detected booted=2023-03-16T15:36:12+0000 age=35m44s WARN [03-16|16:11:56.776] Engine API enabled protocol=eth WARN [03-16|16:11:56.776] Engine API started but chain not configured for merge yet Fatal: Database has receipts with a legacy format. Please run `geth db freezer-migrate`. ``` Running: ```shell $GOPATH/bin/geth --datadir /root/zurich/datadirs/honest_2 db freezer-migrate ``` This worked occasionally. Twice I was prompted on Server 3 for this continuously, even after running the command. To resolve the issue, I nuked the database and allowed the node to sync via p2p normally. New issue at malicious geth logged below. `0x10000 = 65536` is noted above as a "patch" for the honest miner. ``` ERROR[03-16|19:08:45.669] Write ERROR[03-16|19:08:45.671] Write ERROR[03-16|19:08:45.673] Write ERROR[03-16|19:08:45.674] Write ERROR[03-16|19:08:45.675] Write ERROR[03-16|19:08:45.676] Write ERROR[03-16|19:08:45.688] Write ERROR[03-16|19:08:45.696] Write WARN [03-16|19:08:45.697] Blocks undergoing full processing firstnum=2,000,001 lastnum=2,000,192 INFO [03-16|19:08:45.699] Skip duplicated bad block number=2,000,001 hash=63beb4..d4b7bb ERROR[03-16|19:08:45.699] ########## BAD BLOCK ######### Chain config: {ChainID: 1 Homestead: 0 DAO: 0 DAOSupport: true EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: 0, Berlin: 0, London: 0, Arrow Glacier: 0, Gray Glacier: <nil>, MergeFork: <nil>, Terminal TD: <nil>, Engine: unknown} Number: 2000001 Hash: 0x63beb40ec15ebdf2ccd65bf44d3759a6d22a62f775545d038833995049d4b7bb Error: invalid difficulty: have 65536, want 62368 ############################## WARN [03-16|19:08:45.699] Synchronisation failed, dropping peer peer=6cf172019e8979f290271665820421e998332e4fe86ee61c8ad7f16f8aaf369c err="retrieved hash chain is invalid: invalid difficulty: have 65536, want 62368" INFO [03-16|19:08:45.699] Disconnecting peer="Peer 6cf172019e8979f2 164.92.140.229:30323" INFO [03-16|19:08:45.699] Disconnected peer="Peer 6cf172019e8979f2 164.92.140.229:30323" ERROR[03-16|19:08:45.699] Disc INFO [03-16|19:08:45.700] Looking for peers peercount=0 tried=0 static=5 ``` _Fixed_. I made the same patch at `mgeth`; adjusting the `GenesisDifficulty` and `MinimumDifficulty` values (params/protocol_params.go) to `65536`, in accordance with the honest miner's values. The malicious nodes have now synced to 2M + 12k. - [ ] Wait (again) on Server3's nodes to sync the chain because of the nuke because of the invalid database bug. mgeth: Another bug. ```shell INFO [03-16|20:02:34.160] Deep froze chain segment blocks=2 elapsed=207.032ms number=1,922,870 hash=5c7e7e..d837fd INFO [03-16|20:03:01.975] Imported new chain segment blocks=1 txs=0 mgas=0.000 elapsed=8.640ms mgasps=0.000 number=2,012,871 hash=4a9385..09bafc dirty=562.75KiB INFO [03-16|20:03:06.216] Imported new chain segment blocks=1 txs=0 mgas=0.000 elapsed=9.859ms mgasps=0.000 number=2,012,872 hash=39ae25..2a8059 dirty=562.75KiB INFO [03-16|20:03:08.877] Looking for peers peercount=2 tried=3 static=5 INFO [03-16|20:03:08.984] Peer announced its TD peer=3c4b10ed td=3,221,291,008 ATCK [03-16|20:03:08.985] Prediction TD in database: 65536 INFO [03-16|20:03:08.985] Not cheating about TD peer=3c4b10ed td=145,298,818,024 ERROR[03-16|20:03:08.985] Write INFO [03-16|20:03:08.986] Received query query="&{Origin:{Hash:0x0000000000000000000000000000000000000000000000000000000000000000 Number:55873} Amount:192 Skip:0 Reverse:false}" id=9,041,821,870,070,044,330 INFO [03-16|20:03:08.986] Creating attackChain ATCK [03-16|20:03:08.987] Latest() returning latest block of prediction chain even if we are in ready panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x9e702f] goroutine 41265 [running]: github.com/ethereum/go-ethereum/attack/bridge.latest(0x1) /root/mgeth/attack/bridge/internals.go:103 +0x16f github.com/ethereum/go-ethereum/attack/bridge.Latest({0x17dd138, 0xc0068dee60, 0x0}) /root/mgeth/attack/bridge/bridge.go:759 +0xef github.com/ethereum/go-ethereum/eth/protocols/eth.handleGetBlockHeaders66({0x17b6280, 0xc0027f8960}, {0x179a920, 0xc00b1a3b00}, 0xc00847c410) /root/mgeth/eth/protocols/eth/handlers.go:58 +0x3f5 github.com/ethereum/go-ethereum/eth/protocols/eth.handleMessage({0x17b6280, 0xc0027f8960}, 0xc00847c410) /root/mgeth/eth/protocols/eth/handler.go:246 +0x5bf github.com/ethereum/go-ethereum/eth/protocols/eth.Handle.func1() /root/mgeth/eth/protocols/eth/handler.go:163 +0x3b created by github.com/ethereum/go-ethereum/eth/protocols/eth.Handle /root/mgeth/eth/protocols/eth/handler.go:159 +0x172 ``` Ok its 16:00 Thursday 20230316 and I've got the network up. This includes the malicious nodes as well as the honest ones, expect for the victim. I write this pending of the final step of turning the victim on. Once I do that, we expect the victim to then be lured in to sync with one of the malicious nodes as its master/origin snap sync peer, and then to fall victim to the attacks `mgeth` has in store. All nodes are posting metrics to https://dash.metrics.etccore.in/d/QC1Arp5Wk/geth-dashboard?orgId=1&refresh=15s, and I've just made 2 new Zurich-filtered panels so we can keep an eye on the cattle. For example: ![](https://i.imgur.com/Fum6p8Y.png) We see a spike of 2 coincidental with finally turning on the two malicious nodes; there are 6 nodes, each with 5 peers. I confirm the attack is ready to go: ```shell= root@zurich-8vcpu-16gb-fra1-01:~/zurich# ./start_orch.sh Orchestrator started The chain is already built and ready to use Peer a8b25ec5 joined (numPeers: 1) Peer 202cca9f joined (numPeers: 2) Attack ready to go ``` Thursday 20230323 I am trying with different values for `AdversarialSyncDelay`, yesterday I tried with `58` and both malicious nodes failed with a panic, I believe it had to do whie trying to import the fake batches faster than it should be, because of no real delay during sync. Today I am giving this a try with `AdversarialSyncDelay =1200` just to verify my assumption with a not so small delay. --- ### Honest miner hashrate > Start the honest nodes and start mining on at least one. Estimate the total honest mining hashrate as you'll need it below. > [...] `orch --honest-hashrate 600000` Honest miner hashrate samples: ```js > console.log(new Date(), eth.blockNumber, ethash.getHashrate()); Tue Mar 14 2023 15:35:08 GMT+0000 (UTC) 2000189 97180 null > console.log(new Date(), eth.blockNumber, ethash.getHashrate()); Tue Mar 14 2023 15:35:50 GMT+0000 (UTC) 2000278 120663 null > admin.nodeInfo.enode "enode://910da3fcffa54cc9dccc15827dae54acb23d9c30ace7369b8e344b0e8531ece2424fc1cc5ecde024e1976b48258f2926fa388b4803fbdd69c0bfee9eb126fe37@127.0.0.1:30323?discport=0" > console.log(new Date(), eth.blockNumber, ethash.getHashrate()); Tue Mar 14 2023 15:54:59 GMT+0000 (UTC) 2001939 146174 null > console.log(new Date(), eth.blockNumber, ethash.getHashrate()); Tue Mar 14 2023 15:59:41 GMT+0000 (UTC) 2002180 145835 > console.log(new Date(), eth.blockNumber, ethash.getHashrate()); Tue Mar 14 2023 16:51:57 GMT+0000 (UTC) 2003883 157502 null ``` #### Enodes ``` cat /root/zurich/datadirs/malicious1/static-peers.json [ "enode://910da3fcffa54cc9dccc15827dae54acb23d9c30ace7369b8e344b0e8531ece2424fc1cc5ecde024e1976b48258f2926fa388b4803fbdd69c0bfee9eb126fe37@164.92.140.229:30323?discport=0", "enode://64dcfc24fba2fbafea6f5c184647c0b8815f2d47de4550c76b65627447d30075d18c243b8fb7db41685451be28ba1069fab70ee0a788f2b49b9070f1586b68d2@164.92.140.229:30324?discport=0", "enode://43f1ced822c854a3c4d05348053a0de1fb106eec0d3c70eabd0b9887313318fd19637336254367278a74c1497932fc68b926bbcf50c8fc780a6096b18ee4beab@164.92.130.171:30333?discport=0", "enode://a7ff9226d8baf5d12ba08cdd5e9d524e4713bc16499796dc3b2fc3b652b081569cdc39442f779eb11fd96ee80aecdd8301b8653b00cb28e7b91f4b03bf464fba@164.92.130.171:30334?discport=0", "enode://c14cea56b96baa60554890e59d95c0c24f1853dd4ce56a6d1d131dd3e8277861a33d5b4f7973ba1cf95f8a7c37fc66ff5ae9184e42b63ab8df6ca1295168896b@46.101.219.187:30343?discport=0" ] ``` __All honest__ ``` [ "enode://910da3fcffa54cc9dccc15827dae54acb23d9c30ace7369b8e344b0e8531ece2424fc1cc5ecde024e1976b48258f2926fa388b4803fbdd69c0bfee9eb126fe37@164.92.140.229:30323?discport=0", "enode://64dcfc24fba2fbafea6f5c184647c0b8815f2d47de4550c76b65627447d30075d18c243b8fb7db41685451be28ba1069fab70ee0a788f2b49b9070f1586b68d2@164.92.140.229:30324?discport=0", "enode://43f1ced822c854a3c4d05348053a0de1fb106eec0d3c70eabd0b9887313318fd19637336254367278a74c1497932fc68b926bbcf50c8fc780a6096b18ee4beab@164.92.130.171:30333?discport=0", "enode://a7ff9226d8baf5d12ba08cdd5e9d524e4713bc16499796dc3b2fc3b652b081569cdc39442f779eb11fd96ee80aecdd8301b8653b00cb28e7b91f4b03bf464fba@164.92.130.171:30334?discport=0", "enode://c14cea56b96baa60554890e59d95c0c24f1853dd4ce56a6d1d131dd3e8277861a33d5b4f7973ba1cf95f8a7c37fc66ff5ae9184e42b63ab8df6ca1295168896b@46.101.219.187:30343?discport=0" ] ``` - S1 - M1 - M2 > Add the malicious nodes to the victim via the console: ```js // mgeth-1 admin.addPeer("enode://ca0110925670cc2108787a4559993fe49347dbaed61154d93f532b390f8af1aa35ef228ca7feaa2165ef361cc24ac88c8ce1a051420e95bb4720f2a8d82c0a0c@134.122.82.71:30303?discport=30303"); // mgeth-2 admin.addPeer("enode://9f86dabd4f8fbc5b756fa0e9e5917f70f45c03a5ab2fd5782c93a99cbf72ad86976a1b3b1e66a852ec2532914330e4f4e9a83a7851d272de5a323ed38ebb8bd8@134.122.82.71:30304?discport=30304"); ``` - S2 - [x] S2I1(miner) `"enode://910da3fcffa54cc9dccc15827dae54acb23d9c30ace7369b8e344b0e8531ece2424fc1cc5ecde024e1976b48258f2926fa388b4803fbdd69c0bfee9eb126fe37@127.0.0.1:30323?discport=0"` ``` <datadir>/static-peers.json [ "enode://64dcfc24fba2fbafea6f5c184647c0b8815f2d47de4550c76b65627447d30075d18c243b8fb7db41685451be28ba1069fab70ee0a788f2b49b9070f1586b68d2@164.92.140.229:30324?discport=0", "enode://43f1ced822c854a3c4d05348053a0de1fb106eec0d3c70eabd0b9887313318fd19637336254367278a74c1497932fc68b926bbcf50c8fc780a6096b18ee4beab@164.92.130.171:30333?discport=0", "enode://a7ff9226d8baf5d12ba08cdd5e9d524e4713bc16499796dc3b2fc3b652b081569cdc39442f779eb11fd96ee80aecdd8301b8653b00cb28e7b91f4b03bf464fba@164.92.130.171:30334?discport=0", "enode://c14cea56b96baa60554890e59d95c0c24f1853dd4ce56a6d1d131dd3e8277861a33d5b4f7973ba1cf95f8a7c37fc66ff5ae9184e42b63ab8df6ca1295168896b@46.101.219.187:30343?discport=0" ] ``` - [x] S2I2 `"enode://64dcfc24fba2fbafea6f5c184647c0b8815f2d47de4550c76b65627447d30075d18c243b8fb7db41685451be28ba1069fab70ee0a788f2b49b9070f1586b68d2@127.0.0.1:30324?discport=0"` ``` <datadir>/static-peers.json [ "enode://910da3fcffa54cc9dccc15827dae54acb23d9c30ace7369b8e344b0e8531ece2424fc1cc5ecde024e1976b48258f2926fa388b4803fbdd69c0bfee9eb126fe37@164.92.140.229:30323?discport=0", "enode://43f1ced822c854a3c4d05348053a0de1fb106eec0d3c70eabd0b9887313318fd19637336254367278a74c1497932fc68b926bbcf50c8fc780a6096b18ee4beab@164.92.130.171:30333?discport=0", "enode://a7ff9226d8baf5d12ba08cdd5e9d524e4713bc16499796dc3b2fc3b652b081569cdc39442f779eb11fd96ee80aecdd8301b8653b00cb28e7b91f4b03bf464fba@164.92.130.171:30334?discport=0", "enode://c14cea56b96baa60554890e59d95c0c24f1853dd4ce56a6d1d131dd3e8277861a33d5b4f7973ba1cf95f8a7c37fc66ff5ae9184e42b63ab8df6ca1295168896b@46.101.219.187:30343?discport=0" ] ``` - S3 - [x] S3I1 `"enode://43f1ced822c854a3c4d05348053a0de1fb106eec0d3c70eabd0b9887313318fd19637336254367278a74c1497932fc68b926bbcf50c8fc780a6096b18ee4beab@127.0.0.1:30333?discport=0"` ``` <datadir>/static-peers.json [ "enode://910da3fcffa54cc9dccc15827dae54acb23d9c30ace7369b8e344b0e8531ece2424fc1cc5ecde024e1976b48258f2926fa388b4803fbdd69c0bfee9eb126fe37@164.92.140.229:30323?discport=0", "enode://64dcfc24fba2fbafea6f5c184647c0b8815f2d47de4550c76b65627447d30075d18c243b8fb7db41685451be28ba1069fab70ee0a788f2b49b9070f1586b68d2@164.92.140.229:30324?discport=0", "enode://a7ff9226d8baf5d12ba08cdd5e9d524e4713bc16499796dc3b2fc3b652b081569cdc39442f779eb11fd96ee80aecdd8301b8653b00cb28e7b91f4b03bf464fba@164.92.130.171:30334?discport=0", "enode://c14cea56b96baa60554890e59d95c0c24f1853dd4ce56a6d1d131dd3e8277861a33d5b4f7973ba1cf95f8a7c37fc66ff5ae9184e42b63ab8df6ca1295168896b@46.101.219.187:30343?discport=0" ] ``` - [x] S3I2 `"enode://a7ff9226d8baf5d12ba08cdd5e9d524e4713bc16499796dc3b2fc3b652b081569cdc39442f779eb11fd96ee80aecdd8301b8653b00cb28e7b91f4b03bf464fba@127.0.0.1:30334?discport=0"` ``` <datadir>/static-peers.json [ "enode://910da3fcffa54cc9dccc15827dae54acb23d9c30ace7369b8e344b0e8531ece2424fc1cc5ecde024e1976b48258f2926fa388b4803fbdd69c0bfee9eb126fe37@164.92.140.229:30323?discport=0", "enode://64dcfc24fba2fbafea6f5c184647c0b8815f2d47de4550c76b65627447d30075d18c243b8fb7db41685451be28ba1069fab70ee0a788f2b49b9070f1586b68d2@164.92.140.229:30324?discport=0", "enode://43f1ced822c854a3c4d05348053a0de1fb106eec0d3c70eabd0b9887313318fd19637336254367278a74c1497932fc68b926bbcf50c8fc780a6096b18ee4beab@164.92.130.171:30333?discport=0", "enode://c14cea56b96baa60554890e59d95c0c24f1853dd4ce56a6d1d131dd3e8277861a33d5b4f7973ba1cf95f8a7c37fc66ff5ae9184e42b63ab8df6ca1295168896b@46.101.219.187:30343?discport=0" ] ``` - S4 - [x] S4I1(victim) `"enode://c14cea56b96baa60554890e59d95c0c24f1853dd4ce56a6d1d131dd3e8277861a33d5b4f7973ba1cf95f8a7c37fc66ff5ae9184e42b63ab8df6ca1295168896b@127.0.0.1:30343?discport=0"` ``` <datadir>/static-peers.json [ "enode://910da3fcffa54cc9dccc15827dae54acb23d9c30ace7369b8e344b0e8531ece2424fc1cc5ecde024e1976b48258f2926fa388b4803fbdd69c0bfee9eb126fe37@164.92.140.229:30323?discport=0", "enode://64dcfc24fba2fbafea6f5c184647c0b8815f2d47de4550c76b65627447d30075d18c243b8fb7db41685451be28ba1069fab70ee0a788f2b49b9070f1586b68d2@164.92.140.229:30324?discport=0", "enode://43f1ced822c854a3c4d05348053a0de1fb106eec0d3c70eabd0b9887313318fd19637336254367278a74c1497932fc68b926bbcf50c8fc780a6096b18ee4beab@164.92.130.171:30333?discport=0", "enode://a7ff9226d8baf5d12ba08cdd5e9d524e4713bc16499796dc3b2fc3b652b081569cdc39442f779eb11fd96ee80aecdd8301b8653b00cb28e7b91f4b03bf464fba@164.92.130.171:30334?discport=0", ] ```

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.