# 24.5.0 Besu Release Performance Regression ## 24.5.0-RC1 vs 23.3.0 Equivalent versions were used as exact versions weren't available: :::spoiler 24.5.0-RC1 24.5.0-RC1 is currently still burning in, but it is functionally equivalent to [the test run on these nodes last week](https://grafana.o11y.web3factory.consensys.net/d/_Mqt4ksnz/besu-full?orgId=19&var-system=dev-elc-bu-tk-mainnet-jframe-halt-1&var-system=dev-elc-bu-tk-mainnet-jframe-halt-10&var-system=dev-elc-bu-tk-mainnet-jframe-halt-2&var-system=dev-elc-bu-tk-mainnet-jframe-halt-3&var-system=dev-elc-bu-tk-mainnet-jframe-halt-4&var-system=dev-elc-bu-tk-mainnet-jframe-halt-5&var-system=dev-elc-bu-tk-mainnet-jframe-halt-6&var-system=dev-elc-bu-tk-mainnet-jframe-halt-7&var-system=dev-elc-bu-tk-mainnet-jframe-halt-8&var-system=dev-elc-bu-tk-mainnet-jframe-halt-9&var-sync_task_quantile=All&var-executor=All&var-vertx_pool_name=All&var-protocol_inbound=All&var-protocol_outbound=All&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B&var-overview_engine_percentiles=All&var-txpool_layer=All&from=1714462487150&to=1715067287150) (24.4.0-RC3 + haltfix2) Version string: `besu/v24.5-develop-c5b97d2/linux-x86_64/openjdk-java-21` - commit sha is from https://github.com/hyperledger/besu/pull/7059 These nodes have been deleted now, but here is an historical Grafana link for the **most recent two day time range** following all nodes getting into sync: https://grafana.o11y.web3factory.consensys.net/d/_Mqt4ksnz/besu-full?orgId=19&var-system=dev-elc-bu-tk-mainnet-jframe-halt-1&var-system=dev-elc-bu-tk-mainnet-jframe-halt-10&var-system=dev-elc-bu-tk-mainnet-jframe-halt-2&var-system=dev-elc-bu-tk-mainnet-jframe-halt-3&var-system=dev-elc-bu-tk-mainnet-jframe-halt-4&var-system=dev-elc-bu-tk-mainnet-jframe-halt-5&var-system=dev-elc-bu-tk-mainnet-jframe-halt-6&var-system=dev-elc-bu-tk-mainnet-jframe-halt-7&var-system=dev-elc-bu-tk-mainnet-jframe-halt-8&var-system=dev-elc-bu-tk-mainnet-jframe-halt-9&var-sync_task_quantile=All&var-executor=All&var-vertx_pool_name=All&var-protocol_inbound=All&var-protocol_outbound=All&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B&var-overview_engine_percentiles=All&var-txpool_layer=All&from=1714867200000&to=1715039999000 ::: :::spoiler 24.3.0 I can't find any 24.3.3 candidates however the changes since 24.3.0 are minor: [https://github.com/hyperledger/besu/compare/24.3.0...24.3.3](https://github.com/hyperledger/besu/compare/24.3.0...24.3.3) https://github.com/hyperledger/besu/releases/tag/24.3.3 24.3.0 Candidates: - **dev-elc-besu-teku-mainnet-dev-simon-RC4-teku-1** - **dev-elc-besu-teku-mainnet-dev-simon-RC4-teku-2** - version string `besu/v24.2.0-dev-698ec420/linux-x86_64/openjdk-java-21` - https://github.com/hyperledger/besu/releases/tag/24.2.0-RC4 - === https://github.com/hyperledger/besu/releases/tag/24.3.0 - **Same two day time range**: https://grafana.o11y.web3factory.consensys.net/d/_Mqt4ksnz/besu-full?orgId=19&var-system=dev-elc-besu-teku-mainnet-dev-simon-RC4-teku-1&var-system=dev-elc-besu-teku-mainnet-dev-simon-RC4-teku-2&var-sync_task_quantile=All&var-executor=All&var-vertx_pool_name=All&var-protocol_inbound=All&var-protocol_outbound=All&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B&var-overview_engine_percentiles=All&var-txpool_layer=All&from=1714867200000&to=1715039999000 :::spoiler Another candidate is 24.2.0-RC3 - dev-elc-besu-teku-mainnet-simon-24.2.0-RC3-1 - dev-elc-besu-teku-mainnet-simon-24.2.0-RC3-2 - dev-elc-besu-teku-mainnet-simon-24.2.0-RC3-3 - version string `besu/v24.2.0-RC3/linux-x86_64/openjdk-java-21` - https://github.com/hyperledger/besu/releases/tag/24.2.0-RC3 - Only minor difference with 24.3.0 - [https://github.com/hyperledger/besu/compare/24.2.0-RC3...24.3.0](https://github.com/hyperledger/besu/compare/24.2.0-RC3...24.3.0) ::: Both nodes are running Java 21 on Standard_D4s_v5 with Standard SSD LRS (500 Max IOPS, 100 MBps Max Throughput) and read-only host caching enabled. [24.5.0-RC1 Grafana Link for all nodes](https://grafana.o11y.web3factory.consensys.net/d/_Mqt4ksnz/besu-full?orgId=19&var-system=dev-elc-bu-tk-mainnet-jframe-halt-1&var-system=dev-elc-bu-tk-mainnet-jframe-halt-10&var-system=dev-elc-bu-tk-mainnet-jframe-halt-2&var-system=dev-elc-bu-tk-mainnet-jframe-halt-3&var-system=dev-elc-bu-tk-mainnet-jframe-halt-4&var-system=dev-elc-bu-tk-mainnet-jframe-halt-5&var-system=dev-elc-bu-tk-mainnet-jframe-halt-6&var-system=dev-elc-bu-tk-mainnet-jframe-halt-7&var-system=dev-elc-bu-tk-mainnet-jframe-halt-8&var-system=dev-elc-bu-tk-mainnet-jframe-halt-9&var-sync_task_quantile=All&var-executor=All&var-vertx_pool_name=All&var-protocol_inbound=All&var-protocol_outbound=All&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B&var-overview_engine_percentiles=All&var-txpool_layer=All&from=1714867200000&to=1715039999000) [24.3.3 Grafana Link for all nodes](https://grafana.o11y.web3factory.consensys.net/d/_Mqt4ksnz/besu-full?orgId=19&var-system=dev-elc-besu-teku-mainnet-dev-simon-RC4-teku-1&var-system=dev-elc-besu-teku-mainnet-dev-simon-RC4-teku-2&var-sync_task_quantile=All&var-executor=All&var-vertx_pool_name=All&var-protocol_inbound=All&var-protocol_outbound=All&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B&var-overview_engine_percentiles=All&var-txpool_layer=All&from=1714867200000&to=1715039999000) ## Comparison [dev-elc-bu-tk-mainnet-jframe-halt-1 vs dev-elc-besu-teku-mainnet-dev-simon-RC4-teku-1 Grafana Link](https://grafana.o11y.web3factory.consensys.net/d/_Mqt4ksnz/besu-full?orgId=19&var-system=dev-elc-besu-teku-mainnet-dev-simon-RC4-teku-1&var-system=dev-elc-bu-tk-mainnet-jframe-halt-1&var-sync_task_quantile=All&var-executor=All&var-vertx_pool_name=All&var-protocol_inbound=All&var-protocol_outbound=All&var-diskdevices=%5Ba-z%5D%2B%7Cnvme%5B0-9%5D%2Bn%5B0-9%5D%2B&var-overview_engine_percentiles=All&var-txpool_layer=All&from=1714867200000&to=1715039999000) ### Sync time (CHECKPOINT) #### 24.5.0-RC1 * 40 hours - dev-elc-bu-tk-mainnet-jframe-halt-1 * 31 hours - dev-elc-bu-tk-mainnet-jframe-halt-2 #### 24.3.0 * 33 hours - dev-elc-besu-teku-mainnet-dev-simon-RC4-teku-1 * 29 hours - dev-elc-besu-teku-mainnet-dev-simon-RC4-teku-2 ### Blocktime ![Screenshot 2024-05-08 at 1.27.03 PM](https://hackmd.io/_uploads/BJBzZ_uzA.png) ### Block processing and FCU timings ![Screenshot 2024-05-08 at 1.47.09 PM](https://hackmd.io/_uploads/H1naB_ufA.png) ### Preload caches ![Screenshot 2024-05-08 at 1.47.47 PM](https://hackmd.io/_uploads/SJGxI__zA.png) ### Transaction Pool ![Screenshot 2024-05-08 at 1.49.32 PM](https://hackmd.io/_uploads/SylP8OuM0.png) ### GC activity ![Screenshot 2024-05-08 at 1.51.30 PM](https://hackmd.io/_uploads/BkyAId_zR.png) ### Number of threads ![Screenshot 2024-05-08 at 1.52.51 PM](https://hackmd.io/_uploads/r1u7DudGC.png) ### CPU usage ![Screenshot 2024-05-08 at 1.53.45 PM](https://hackmd.io/_uploads/r1uUDudMR.png) ### Memory usage ![Screenshot 2024-05-08 at 1.54.07 PM](https://hackmd.io/_uploads/ryRPDdOMR.png) ### Disk Usage ![Screenshot 2024-05-08 at 1.54.50 PM](https://hackmd.io/_uploads/Bkj5P_uGR.png) ### Disk IOPS MB/S #### 24.5.0-RC1 ![Screenshot 2024-05-08 at 1.58.27 PM](https://hackmd.io/_uploads/SkLO_d_GR.png) #### 24.3.3 ![Screenshot 2024-05-08 at 1.59.04 PM](https://hackmd.io/_uploads/SJ5quuOzA.png) ### Network activity ![Screenshot 2024-05-08 at 1.56.57 PM](https://hackmd.io/_uploads/HytzOd_zA.png)