HackMD
dev-elc-bu-tk-holesky-burn-24.5.0-RC1-full-forest
Java heap OOMs are bottom red arrows…
Besu Starts/Restarts
grep "Starting Besu"
2024-05-07T02:03:06 <-- node launch
2024-05-08T01:16:13 <-- maintenance restart
2024-05-08T11:04:40 <-- manual restart?
Application Analysis
Heap Dump shows vast majority of objects are CompletableFuture held by the ForkJoinPool thread:
The thread java.util.concurrent.ForkJoinWorkerThread @ 0x60f403ec0 ForkJoinPool.commonPool-worker-1 keeps local variables with total size 7,768.83 MB (97.92%) bytes.
Significant stack frames and local variables
java.util.concurrent.CompletableFuture$UniApply.tryFire(I)Ljava/util/concurrent/CompletableFuture; (CompletableFuture.java:646)
java.util.concurrent.CompletableFuture @ 0x7f7f1dec0 retains 7,768.81 MB (97.92%) bytes
and gives the following stack as a memory leak suspect:
ForkJoinPool.commonPool-worker-1
at org.hyperledger.besu.datatypes.Hash.hash(Lorg/apache/tuweni/bytes/Bytes;)Lorg/hyperledger/besu/datatypes/Hash; (Hash.java:69)
at org.hyperledger.besu.ethereum.mainnet.MainnetBlockHeaderFunctions.createHash(Lorg/hyperledger/besu/ethereum/core/BlockHeader;)Lorg/hyperledger/besu/datatypes/Hash; (MainnetBlockHeaderFunctions.java:35)
at org.hyperledger.besu.ethereum.mainnet.MainnetBlockHeaderFunctions.hash(Lorg/hyperledger/besu/ethereum/core/BlockHeader;)Lorg/hyperledger/besu/datatypes/Hash; (MainnetBlockHeaderFunctions.java:30)
at org.hyperledger.besu.ethereum.mainnet.ScheduleBasedBlockHeaderFunctions.hash(Lorg/hyperledger/besu/ethereum/core/BlockHeader;)Lorg/hyperledger/besu/datatypes/Hash; (ScheduleBasedBlockHeaderFunctions.java:41)
at org.hyperledger.besu.ethereum.core.BlockHeader.lambda$new$0(Lorg/hyperledger/besu/ethereum/core/BlockHeaderFunctions;)Lorg/hyperledger/besu/datatypes/Hash; (BlockHeader.java:93)
at org.hyperledger.besu.ethereum.core.BlockHeader$$Lambda+0x00007b2b72499000.get()Ljava/lang/Object; ()
at com.google.common.base.Suppliers$NonSerializableMemoizingSupplier.get()Ljava/lang/Object; (Suppliers.java:181)
at org.hyperledger.besu.ethereum.core.BlockHeader.getHash()Lorg/hyperledger/besu/datatypes/Hash; (BlockHeader.java:132)
at org.hyperledger.besu.ethereum.eth.sync.backwardsync.BackwardChain.prependAncestorsHeader(Lorg/hyperledger/besu/ethereum/core/BlockHeader;Z)V (BackwardChain.java:139)
at org.hyperledger.besu.ethereum.eth.sync.backwardsync.BackwardSyncStep.possibleRestoreOldNodes(Lorg/hyperledger/besu/ethereum/core/BlockHeader;)Lorg/hyperledger/besu/datatypes/Hash; (BackwardSyncStep.java:51)
at org.hyperledger.besu.ethereum.eth.sync.backwardsync.BackwardSyncStep$$Lambda+0x00007b2b72771578.apply(Ljava/lang/Object;)Ljava/lang/Object; ()
Which involves line 51 in BackwardSyncStep.java:
L47: protected Hash possibleRestoreOldNodes(final BlockHeader firstAncestor) {
L48: Hash lastHash = firstAncestor.getParentHash();
L49: Optional<BlockHeader> iterator = backwardChain.getHeader(lastHash);
L50: while (iterator.isPresent()) {
L51: backwardChain.prependAncestorsHeader(iterator.get(), true);
L52: lastHash = iterator.get().getParentHash();
L53: iterator = backwardChain.getHeader(lastHash);
L54: }
L55: return lastHash;
L56: }
probably going on to create and cache a Hash object without the method completing
Log analysis
Slack conversation
I am adding the following debug logging…
$ more debugBesu.sh
#!/bin/bash
set -euo pipefail
curl --retry-delay 50 --retry-connrefused --location \
--request POST 'http://127.0.0.1:8545' \
--header 'Content-Type: application/json' \
--data-raw '{
"jsonrpc": "2.0",
"method": "admin_changeLogLevel",
"params": [
"DEBUG", [
"org.hyperledger.besu.ethereum.eth.sync.backwardsync",
"org.hyperledger.besu.consensus.merge",
"org.hyperledger.besu.ethereum.api.jsonrpc.internal.methods.engine"
]
],
"id": 1
}'
Simon Dudley 6 hours ago
After restarting the logs absolutely spam with:
{"@timestamp":"2024-05-09T00:17:58,598","level":"DEBUG","thread":"ForkJoinPool.commonPool-worker-1","class":"BackwardChain","message":"Added header 1305398 (0xec3e7e91daefa4e40be7005d960536fb5047d44304bfb5e8ea353543275ffec4) to backward chain led by pivot 1503950 (0xd66eb0280943fa74aac2e10c4e29a4c8b00cf38a61dc0f0a14a2889c796c4075) on height 1305399","throwable":""}
{"@timestamp":"2024-05-09T00:17:58,598","level":"DEBUG","thread":"ForkJoinPool.commonPool-worker-1","class":"BackwardChain","message":"Added header 1305397 (0x5348fbd8ee09125fb3bd2ee1fec18767bf0950ff0bf81a43de3c1f82006f007f) to backward chain led by pivot 1503950 (0xd66eb0280943fa74aac2e10c4e29a4c8b00cf38a61dc0f0a14a2889c796c4075) on height 1305398","throwable":""}
{"@timestamp":"2024-05-09T00:17:58,598","level":"DEBUG","thread":"ForkJoinPool.commonPool-worker-1","class":"BackwardChain","message":"Added header 1305396 (0x7711ee6cfbf150fb951728a417a12d811c5b1c92fe00660c3e0e61d1dd9acde8) to backward chain led by pivot 1503950 (0xd66eb0280943fa74aac2e10c4e29a4c8b00cf38a61dc0f0a14a2889c796c4075) on height 1305397","throwable":""}
Simon Dudley 6 hours ago
^ Guess that was during BWS phase 1 so maybe expected
The INFO logs then switched to phase 2…
{"@timestamp":"2024-05-09T00:18:27,965","level":"INFO","thread":"ForkJoinPool.commonPool-worker-1","class":"BackwardSyncStep","message":"Backward sync phase 1 of 2 completed, downloaded a total of 604564 headers. Peers: 2","throwable":""}
{"@timestamp":"2024-05-09T00:18:29,679","level":"INFO","thread":"nioEventLoopGroup-3-2","class":"TransactionPoolFactory","message":"Node out of sync, disabling transaction handling","throwable":""}
{"@timestamp":"2024-05-09T00:18:29,692","level":"INFO","thread":"nioEventLoopGroup-3-2","class":"BackwardSyncContext","message":"Backward sync phase 2 of 2, 0.00% completed, imported 1 blocks of at least 604564 (current head 899390, target head 1503953). Peers: 2","throwable":""}
Simon Dudley 5 hours ago
The pattern from running relatively fine to memory spiking is in > the range of 5-12 hours so not expecting to get anything interesting in the logs for a few hours.
Simon Dudley 1 hour ago
The memory appears stable so far but found something interesting…
Not sure if this is expected but there is periodic "BWS Phase 1"-style Added header "spam". These periodic phases appear to be proceeded by Re-pivoting to new target block
$ grep "Added header" -B 100 besu.log | grep "Re-pivoting\|Added header" | grep -C 1 "Re-pivoting"
...
{"@timestamp":"2024-05-09T02:06:14,344","level":"DEBUG","thread":"vert.x-worker-thread-0","class":"BackwardChain","message":"Re-pivoting to new target block 1506619 (0x750893f58e28201898d9967fb0a5e8061668777b8c3aeb9700607cb2558d7f13)","throwable":""}
{"@timestamp":"2024-05-09T02:06:16,508","level":"DEBUG","thread":"ForkJoinPool.commonPool-worker-4","class":"BackwardChain","message":"Added header 1506618 (0x673554f1921b012386f7dc947f49ee553886ecc593d74f97a61ff83e17e390fa) to backward chain led by pivot 1506619 (0x750893f58e28201898d9967fb0a5e8061668777b8c3aeb9700607cb2558d7f13) on height 1506619","throwable":""}
--
{"@timestamp":"2024-05-09T02:07:11,236","level":"DEBUG","thread":"ForkJoinPool.commonPool-worker-4","class":"BackwardChain","message":"Added header 928251 (0x5d143100dc7440cc36770039535b30f9fb1f2ba1f1081ecab9eec6ac20000fc3) to backward chain led by pivot 1506623 (0x5ca15e940179bbae046374d3ccaf76445c7775e4871cb061d22e90de8fcec28c) on height 928252","throwable":""}
// 20 mins later
{"@timestamp":"2024-05-09T02:27:37,851","level":"DEBUG","thread":"vert.x-worker-thread-0","class":"BackwardChain","message":"Re-pivoting to new target block 1506718 (0xfae16a20a5ac9c87c1058246922e45cad9fcf89f0397b75c2c0f39b46fe65f7b)","throwable":""}
{"@timestamp":"2024-05-09T02:27:38,612","level":"DEBUG","thread":"ForkJoinPool.commonPool-worker-3","class":"BackwardChain","message":"Added header 1506717 (0x772915197fcdb09d053424e7e2e5b2462a25caade083a43f0d4201a2ef9d1d6f) to backward chain led by pivot 1506718 (0xfae16a20a5ac9c87c1058246922e45cad9fcf89f0397b75c2c0f39b46fe65f7b) on height 1506718","throwable":""}
--
{"@timestamp":"2024-05-09T02:28:33,878","level":"DEBUG","thread":"ForkJoinPool.commonPool-worker-3","class":"BackwardChain","message":"Added header 935092 (0x669dc1aa6bad6bbc7dbc0835bea4af46eeab7af3b60bb03c8dbdc4f9aac88f43) to backward chain led by pivot 1506721 (0x5d86cbfe267bd11a2032745000c7cd9f169100bf42d0ca7f9eec1d7114fd0cf8) on height 935093","throwable":""}
// 42 mins later
{"@timestamp":"2024-05-09T03:10:37,526","level":"DEBUG","thread":"vert.x-worker-thread-0","class":"BackwardChain","message":"Re-pivoting to new target block 1506917 (0x6994edfe3a58e43213404d9f156041b754e8298a3c8400e3224cd2f53a83777a)","throwable":""}
{"@timestamp":"2024-05-09T03:10:40,645","level":"DEBUG","thread":"ForkJoinPool.commonPool-worker-3","class":"BackwardChain","message":"Added header 1506916 (0x6e22ee097b0ad8c3f3f72b0b8133d3e7419ff05ef75001b6ef77ef7c4b2aba87) to backward chain led by pivot 1506917 (0x6994edfe3a58e43213404d9f156041b754e8298a3c8400e3224cd2f53a83777a) on height 1506917","throwable":""}
...
Performance Analysis
Box is Standard_D8as_v5
/dev/sda1 1.4T 160G 1.3T 12% /data
Linux 60-second analysis @ Wed 2024-05-08 06:56:55 UTC …
simon.dudley@dev-elc-bu-tk-holesky-burn-24.5.0-RC1-full-forest:/var/log/besu$ uptime
06:56:55 up 5:40, 2 users, load average: 4.28, 4.43, 4.16
$ sudo dmesg -T | tail
[Wed May 8 01:16:06 2024] sr 0:0:0:2: [sr0] tag#113 unaligned transfer
[Wed May 8 01:16:06 2024] sr 0:0:0:2: [sr0] tag#114 unaligned transfer
[Wed May 8 01:16:06 2024] sr 0:0:0:2: [sr0] tag#115 unaligned transfer
[Wed May 8 01:16:06 2024] sr 0:0:0:2: [sr0] tag#116 unaligned transfer
[Wed May 8 01:16:06 2024] sr 0:0:0:2: [sr0] tag#117 unaligned transfer
[Wed May 8 01:16:09 2024] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[Wed May 8 01:16:09 2024] Bridge firewalling registered
[Wed May 8 01:16:09 2024] Initializing XFRM netlink socket
[Wed May 8 01:16:10 2024] loop6: detected capacity change from 0 to 8
[Wed May 8 01:16:49 2024] hv_balloon: Max. dynamic memory size: 32768 MB
$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
9 1 0 708880 86192 21006080 0 0 469 1006 160 32 37 2 60 1 0
10 0 0 878212 86192 20839424 0 0 82536 69712 8223 10718 91 4 4 1 0
9 1 0 692916 86192 21021456 0 0 85708 66632 7177 7586 94 4 2 0 0
9 1 0 594868 86192 21168016 0 0 84556 66240 6767 6694 90 3 6 1 0
8 1 0 587404 86192 21172688 0 0 59356 132016 4961 5468 96 3 2 0 0
9 1 0 606780 86192 21180752 0 0 67932 66400 7065 8116 92 3 3 1 0
11 0 0 930328 86192 20854996 0 0 31388 24732 8182 11049 93 3 3 0 0
10 0 0 890760 86192 20883672 0 0 24336 1888 4873 5463 93 2 5 0 0
9 0 0 856412 86192 20909096 0 0 23124 0 5389 7270 96 2 2 0 0
9 0 0 815188 86192 20939372 0 0 25112 488 6253 11535 94 2 3 0 0
9 0 0 765656 86192 20985824 0 0 26380 59092 7530 10462 92 3 4 0 0
9 0 0 730388 86192 21021124 0 0 30324 424 6854 12092 93 2 5 0 0
14 0 0 678224 86192 21054692 0 0 29320 1388 5727 7686 95 2 3 0 0
9 0 0 572384 86192 21088568 0 0 31072 412 5576 5713 94 2 4 0 0
9 0 0 588332 86192 21074284 0 0 20432 208 5119 8769 90 2 8 0 0
9 0 0 629272 86192 21081532 0 0 5556 1224 4269 8886 96 1 3 0 0
9 0 0 590780 86192 21118468 0 0 32608 216 6239 8608 95 2 3 0 0
9 0 0 587552 86192 21128840 0 0 30700 1192 5645 5186 93 1 5 0 0
9 0 0 601764 86192 21113900 0 0 32044 0 5629 10145 91 2 6 1 0
9 1 0 591288 86192 21128660 0 0 35604 35188 6113 10414 92 2 5 1 0
9 0 0 576932 86192 21156952 0 0 26540 51712 6294 12071 89 2 7 1 0
9 0 0 592608 86192 21142000 0 0 28796 776 5672 8524 94 2 4 1 0
9 0 0 562368 86192 21172408 0 0 26668 588 4760 5007 94 2 4 1 0
9 0 0 574520 86192 21155880 0 0 28900 600 5502 7629 94 2 4 0 0
9 0 0 624988 86192 21188684 0 0 29768 596 5439 5924 94 2 3 0 0
9 0 0 599312 86192 21219832 0 0 26924 812 5639 7687 93 2 5 1 0
9 0 0 588948 86192 21226868 0 0 5488 844 5806 11790 93 2 5 0 0
9 0 0 565324 86192 21260776 0 0 30512 420 5823 5602 94 1 4 0 0
8 0 0 972620 86152 21124056 0 0 30156 636 5412 5138 90 5 5 0 0
6 0 0 936336 86152 21159544 0 0 30804 76 5626 8557 91 2 7 1 0
8 1 0 852888 86152 21188712 0 0 24616 1460 5333 7128 94 2 3 0 0
9 0 0 818380 86152 21219072 0 0 27016 824 5860 8405 95 2 3 0 0
9 0 0 779604 86152 21251904 0 0 29760 676 5642 7848 93 2 5 1 0
9 0 0 742060 86152 21286244 0 0 29304 996 5314 5678 94 1 5 0 0
8 0 0 697652 86152 21322944 0 0 31388 788 6463 7873 93 2 4 0 0
11 1 0 662932 86152 21357592 0 0 30492 332 6266 6965 91 2 7 1 0
8 1 0 648520 86152 21381700 0 0 32836 916 6739 9828 92 2 5 1 0
$ mpstat -P ALL 1
Linux 6.5.0-1019-azure (dev-elc-bu-tk-holesky-burn-24) 05/08/24 _x86_64_ (8 CPU)
07:09:56 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
07:09:57 all 93.34 0.00 1.88 0.63 0.00 0.25 0.00 0.00 0.00 3.89
07:09:57 0 92.93 0.00 1.01 1.01 0.00 0.00 0.00 0.00 0.00 5.05
07:09:57 1 97.96 0.00 0.00 1.02 0.00 0.00 0.00 0.00 0.00 1.02
07:09:57 2 95.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 5.00
07:09:57 3 88.89 0.00 6.06 0.00 0.00 1.01 0.00 0.00 0.00 4.04
07:09:57 4 89.80 0.00 4.08 1.02 0.00 1.02 0.00 0.00 0.00 4.08
07:09:57 5 93.07 0.00 0.99 0.99 0.00 0.00 0.00 0.00 0.00 4.95
07:09:57 6 96.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00
07:09:57 7 93.07 0.00 1.98 0.99 0.00 0.00 0.00 0.00 0.00 3.96
07:09:57 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
07:09:58 all 94.60 0.00 1.38 0.25 0.00 0.13 0.00 0.00 0.00 3.64
07:09:58 0 92.93 0.00 2.02 0.00 0.00 0.00 0.00 0.00 0.00 5.05
07:09:58 1 94.00 0.00 0.00 1.00 0.00 1.00 0.00 0.00 0.00 4.00
07:09:58 2 98.02 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 0.99
07:09:58 3 92.93 0.00 1.01 0.00 0.00 0.00 0.00 0.00 0.00 6.06
07:09:58 4 95.05 0.00 2.97 0.99 0.00 0.00 0.00 0.00 0.00 0.99
07:09:58 5 93.88 0.00 1.02 0.00 0.00 0.00 0.00 0.00 0.00 5.10
07:09:58 6 97.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00
07:09:58 7 92.93 0.00 3.03 0.00 0.00 0.00 0.00 0.00 0.00 4.04
07:09:58 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
07:09:59 all 89.18 0.00 3.14 0.63 0.00 0.13 0.00 0.00 0.00 6.92
07:09:59 0 87.00 0.00 4.00 1.00 0.00 0.00 0.00 0.00 0.00 8.00
07:09:59 1 90.72 0.00 2.06 2.06 0.00 1.03 0.00 0.00 0.00 4.12
07:09:59 2 87.76 0.00 2.04 0.00 0.00 0.00 0.00 0.00 0.00 10.20
07:09:59 3 87.00 0.00 4.00 0.00 0.00 0.00 0.00 0.00 0.00 9.00
07:09:59 4 93.07 0.00 0.99 0.99 0.00 0.00 0.00 0.00 0.00 4.95
07:09:59 5 93.94 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 6.06
07:09:59 6 81.00 0.00 11.00 1.00 0.00 0.00 0.00 0.00 0.00 7.00
07:09:59 7 93.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 6.00
07:09:59 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
07:10:00 all 91.44 0.00 2.02 0.50 0.00 0.38 0.00 0.00 0.00 5.67
07:10:00 0 92.93 0.00 0.00 1.01 0.00 0.00 0.00 0.00 0.00 6.06
07:10:00 1 92.86 0.00 1.02 3.06 0.00 0.00 0.00 0.00 0.00 3.06
07:10:00 2 92.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 7.00
07:10:00 3 77.78 0.00 13.13 0.00 0.00 0.00 0.00 0.00 0.00 9.09
07:10:00 4 89.00 0.00 1.00 0.00 0.00 3.00 0.00 0.00 0.00 7.00
07:10:00 5 99.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00
07:10:00 6 94.95 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 5.05
07:10:00 7 92.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 7.07
07:10:00 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
07:10:01 all 89.53 0.00 3.15 0.00 0.00 0.13 0.00 0.00 0.00 7.19
07:10:01 0 93.94 0.00 1.01 0.00 0.00 0.00 0.00 0.00 0.00 5.05
07:10:01 1 91.84 0.00 1.02 0.00 0.00 0.00 0.00 0.00 0.00 7.14
07:10:01 2 79.59 0.00 12.24 0.00 0.00 0.00 0.00 0.00 0.00 8.16
07:10:01 3 94.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00
07:10:01 4 87.76 0.00 2.04 0.00 0.00 1.02 0.00 0.00 0.00 9.18
07:10:01 5 88.89 0.00 2.02 0.00 0.00 0.00 0.00 0.00 0.00 9.09
07:10:01 6 93.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 5.00
07:10:01 7 87.13 0.00 1.98 0.00 0.00 0.00 0.00 0.00 0.00 10.89
07:10:01 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
07:10:02 all 96.52 0.00 0.87 0.25 0.00 0.12 0.00 0.00 0.00 2.24
07:10:02 0 98.00 0.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00
07:10:02 1 96.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 3.00
07:10:02 2 97.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.97
07:10:02 3 97.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00
07:10:02 4 95.05 0.00 1.98 0.00 0.00 0.00 0.00 0.00 0.00 2.97
07:10:02 5 98.04 0.00 0.98 0.00 0.00 0.00 0.00 0.00 0.00 0.98
07:10:02 6 93.07 0.00 3.96 0.00 0.00 0.00 0.00 0.00 0.00 2.97
07:10:02 7 97.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.02
^C
Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
Average: all 92.45 0.00 2.07 0.38 0.00 0.19 0.00 0.00 0.00 4.92
Average: 0 92.95 0.00 1.34 0.84 0.00 0.00 0.00 0.00 0.00 4.87
Average: 1 93.91 0.00 0.68 1.18 0.00 0.51 0.00 0.00 0.00 3.72
Average: 2 91.64 0.00 2.68 0.00 0.00 0.00 0.00 0.00 0.00 5.69
Average: 3 89.61 0.00 4.52 0.00 0.00 0.17 0.00 0.00 0.00 5.70
Average: 4 91.65 0.00 2.17 0.50 0.00 0.83 0.00 0.00 0.00 4.84
Average: 5 94.49 0.00 0.83 0.17 0.00 0.00 0.00 0.00 0.00 4.51
Average: 6 92.50 0.00 3.00 0.17 0.00 0.00 0.00 0.00 0.00 4.33
Average: 7 92.82 0.00 1.34 0.17 0.00 0.00 0.00 0.00 0.00 5.68
$ pidstat 1
Linux 6.5.0-1019-azure (dev-elc-bu-tk-holesky-burn-24) 05/08/24 _x86_64_ (8 CPU)
07:10:10 UID PID %usr %system %guest %wait %CPU CPU Command
07:10:11 0 77 0.00 0.99 0.00 0.00 0.99 3 kworker/3:1-events
07:10:11 997 638 0.00 0.99 0.00 0.00 0.99 5 promtail
07:10:11 1024 678 78.22 4.95 0.00 0.00 83.17 3 java
07:10:11 1025 679 660.40 2.97 0.00 0.00 663.37 0 java
07:10:11 104 946 0.99 0.99 0.00 0.00 1.98 7 mdsd
07:10:11 0 29705 0.00 0.99 0.00 0.00 0.99 0 kworker/u16:2-events_power_efficient
07:10:11 0 29917 0.00 0.99 0.00 0.99 0.99 0 kworker/u16:3-writeback
07:10:11 1007 31129 0.00 0.99 0.00 0.00 0.99 1 pidstat
07:10:11 UID PID %usr %system %guest %wait %CPU CPU Command
07:10:12 0 1 1.00 1.00 0.00 0.00 2.00 0 systemd
07:10:12 0 127 0.00 1.00 0.00 0.00 1.00 0 jbd2/sdb1-8
07:10:12 102 619 1.00 1.00 0.00 1.00 2.00 0 dbus-daemon
07:10:12 998 630 6.00 1.00 0.00 0.00 7.00 6 node_exporter
07:10:12 1024 678 70.00 7.00 0.00 0.00 77.00 3 java
07:10:12 1025 679 649.00 3.00 0.00 0.00 652.00 0 java
07:10:12 104 946 1.00 0.00 0.00 0.00 1.00 7 mdsd
07:10:12 UID PID %usr %system %guest %wait %CPU CPU Command
07:10:13 0 625 1.00 0.00 0.00 0.00 1.00 0 wdavdaemon
07:10:13 1024 678 70.00 8.00 0.00 0.00 78.00 3 java
07:10:13 1025 679 656.00 8.00 0.00 0.00 664.00 0 java
07:10:13 0 923 1.00 0.00 0.00 0.00 1.00 4 python3
07:10:13 0 930 1.00 0.00 0.00 0.00 1.00 6 wdavdaemon
07:10:13 1007 31129 0.00 1.00 0.00 0.00 1.00 1 pidstat
07:10:13 UID PID %usr %system %guest %wait %CPU CPU Command
07:10:14 0 670 1.00 0.00 0.00 0.00 1.00 0 containerd
07:10:14 1024 678 67.00 7.00 0.00 0.00 74.00 3 java
07:10:14 1025 679 700.00 6.00 0.00 0.00 706.00 0 java
07:10:14 104 946 0.00 1.00 0.00 0.00 1.00 7 mdsd
07:10:14 0 30510 0.00 1.00 0.00 0.00 1.00 3 kworker/u16:0-ext4-rsv-conversion
07:10:14 1007 31129 1.00 1.00 0.00 0.00 2.00 1 pidstat
07:10:14 UID PID %usr %system %guest %wait %CPU CPU Command
07:10:15 0 40 0.00 1.00 0.00 0.00 1.00 1 ksoftirqd/1
07:10:15 0 625 0.00 1.00 0.00 0.00 1.00 0 wdavdaemon
07:10:15 1024 678 80.00 3.00 0.00 0.00 83.00 3 java
07:10:15 1025 679 683.00 4.00 0.00 0.00 687.00 0 java
07:10:15 1007 31129 0.00 1.00 0.00 0.00 1.00 1 pidstat
07:10:15 UID PID %usr %system %guest %wait %CPU CPU Command
07:10:16 0 625 0.00 1.00 0.00 0.00 1.00 0 wdavdaemon
07:10:16 997 638 0.00 2.00 0.00 0.00 2.00 5 promtail
07:10:16 1024 678 104.00 12.00 0.00 0.00 116.00 3 java
07:10:16 1025 679 645.00 3.00 0.00 0.00 648.00 0 java
07:10:16 0 14440 0.00 2.00 0.00 0.00 2.00 1 kworker/1:0-xfs-conv/sda1
07:10:16 1007 31129 0.00 1.00 0.00 0.00 1.00 1 pidstat
07:10:16 UID PID %usr %system %guest %wait %CPU CPU Command
07:10:17 0 15 0.00 1.00 0.00 0.00 1.00 4 rcu_sched
07:10:17 102 619 2.00 0.00 0.00 1.00 2.00 3 dbus-daemon
07:10:17 998 630 5.00 2.00 0.00 0.00 7.00 6 node_exporter
07:10:17 1024 678 70.00 7.00 0.00 0.00 77.00 3 java
07:10:17 1025 679 679.00 4.00 0.00 0.00 683.00 0 java
07:10:17 104 946 1.00 1.00 0.00 0.00 2.00 7 mdsd
07:10:17 1007 31129 0.00 1.00 0.00 0.00 1.00 2 pidstat
^C
Average: UID PID %usr %system %guest %wait %CPU CPU Command
Average: 0 1 0.14 0.14 0.00 0.00 0.29 - systemd
Average: 0 15 0.00 0.14 0.00 0.14 0.14 - rcu_sched
Average: 0 40 0.00 0.14 0.00 0.00 0.14 - ksoftirqd/1
Average: 0 77 0.00 0.14 0.00 0.00 0.14 - kworker/3:1-mm_percpu_wq
Average: 0 127 0.00 0.14 0.00 0.00 0.14 - jbd2/sdb1-8
Average: 102 619 0.43 0.14 0.00 0.29 0.57 - dbus-daemon
Average: 0 625 0.14 0.29 0.00 0.00 0.43 - wdavdaemon
Average: 998 630 1.57 0.43 0.00 0.00 2.00 - node_exporter
Average: 997 638 0.00 0.43 0.00 0.00 0.43 - promtail
Average: 0 670 0.14 0.00 0.00 0.00 0.14 - containerd
Average: 1024 678 77.03 6.99 0.00 0.00 84.02 - java
Average: 1025 679 667.48 4.42 0.00 0.00 671.90 - java
Average: 0 923 0.14 0.00 0.00 0.00 0.14 - python3
Average: 0 930 0.14 0.00 0.00 0.00 0.14 - wdavdaemon
Average: 104 946 0.43 0.43 0.00 0.00 0.86 - mdsd
Average: 0 14440 0.00 0.29 0.00 0.00 0.29 - kworker/1:0-mm_percpu_wq
Average: 0 29705 0.00 0.14 0.00 0.00 0.14 - kworker/u16:2-events_power_efficient
Average: 0 29917 0.00 0.14 0.00 0.14 0.14 - kworker/u16:3-events_unbound
Average: 0 30510 0.00 0.14 0.00 0.00 0.14 - kworker/u16:0-ext4-rsv-conversion
Average: 1007 31129 0.14 0.86 0.00 0.00 1.00 - pidstat
$ iostat -xz 1
Linux 6.5.0-1019-azure (dev-elc-bu-tk-holesky-burn-24) 05/08/24 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
37.57 0.00 1.64 1.13 0.00 59.66
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
loop0 0.00 0.02 0.00 0.00 0.21 8.05 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop1 0.01 0.11 0.00 0.00 0.15 9.40 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop2 0.00 0.05 0.00 0.00 0.27 19.27 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop3 0.00 0.05 0.00 0.00 0.15 15.56 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop4 0.00 0.02 0.00 0.00 0.17 8.37 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop5 0.03 1.19 0.00 0.00 0.37 36.34 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
loop6 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 97.30 3996.37 0.00 0.00 0.31 41.07 7.67 7582.41 0.14 1.84 79.15 988.59 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.64 8.97
sdb 1.46 56.75 0.30 17.00 0.30 38.76 47.38 707.76 84.58 64.10 2.12 14.94 8.99 51406.97 2.90 24.37 2.18 5716.19 0.00 0.00 0.12 7.50
sr0 0.23 0.12 0.00 0.00 0.34 0.53 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01
avg-cpu: %user %nice %system %iowait %steal %idle
91.21 0.00 4.15 0.50 0.00 4.15
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 1028.00 31264.00 0.00 0.00 0.25 30.41 36.00 29336.00 0.00 0.00 19.75 814.89 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.97 81.60
sdb 0.00 0.00 0.00 0.00 0.00 0.00 229.00 2292.00 309.00 57.43 5.59 10.01 15.00 90300.00 5.00 25.00 2.33 6020.00 0.00 0.00 1.31 14.40
avg-cpu: %user %nice %system %iowait %steal %idle
93.38 0.00 2.25 0.12 0.00 4.25
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 913.00 28144.00 0.00 0.00 0.24 30.83 14.00 256.00 5.00 26.32 2.43 18.29 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.25 79.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 15.00 200.00 28.00 65.12 1.93 13.33 3.00 18060.00 1.00 25.00 2.00 6020.00 0.00 0.00 0.04 3.20
avg-cpu: %user %nice %system %iowait %steal %idle
96.12 0.00 1.63 0.25 0.00 2.01
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 997.00 29844.00 0.00 0.00 0.24 29.93 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.24 76.40
sdb 0.00 0.00 0.00 0.00 0.00 0.00 17.00 220.00 30.00 63.83 1.35 12.94 4.00 18068.00 1.00 20.00 2.00 4517.00 0.00 0.00 0.03 4.00
avg-cpu: %user %nice %system %iowait %steal %idle
95.26 0.00 1.37 0.25 0.00 3.12
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 1074.00 32344.00 0.00 0.00 0.26 30.12 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.28 82.40
sdb 0.00 0.00 0.00 0.00 0.00 0.00 46.00 612.00 86.00 65.15 1.50 13.30 10.00 54184.00 3.00 23.08 2.20 5418.40 0.00 0.00 0.09 9.60
avg-cpu: %user %nice %system %iowait %steal %idle
91.41 0.00 1.74 0.50 0.00 6.35
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 992.00 30436.00 0.00 0.00 0.25 30.68 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.25 77.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 35.00 440.00 54.00 60.67 1.23 12.57 6.00 36120.00 2.00 25.00 2.33 6020.00 0.00 0.00 0.06 4.80
avg-cpu: %user %nice %system %iowait %steal %idle
94.01 0.00 1.62 0.25 0.00 4.11
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 1051.00 31332.00 0.00 0.00 0.24 29.81 3.00 40.00 1.00 25.00 2.00 13.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.26 79.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 56.00 772.00 109.00 66.06 1.07 13.79 12.00 72240.00 4.00 25.00 2.00 6020.00 0.00 0.00 0.08 8.40
avg-cpu: %user %nice %system %iowait %steal %idle
96.99 0.00 1.38 0.13 0.00 1.50
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 1004.00 30524.00 0.00 0.00 0.25 30.40 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.25 76.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 31.00 396.00 54.00 63.53 1.39 12.77 6.00 36120.00 2.00 25.00 2.00 6020.00 0.00 0.00 0.06 6.00
avg-cpu: %user %nice %system %iowait %steal %idle
90.73 0.00 2.63 0.50 0.00 6.14
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 1080.00 32612.00 0.00 0.00 0.24 30.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.26 79.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 89.00 1284.00 190.00 68.10 1.91 14.43 19.00 108364.00 6.00 24.00 2.21 5703.37 0.00 0.00 0.21 19.60
avg-cpu: %user %nice %system %iowait %steal %idle
95.37 0.00 1.63 0.13 0.00 2.88
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 1003.00 30164.00 0.00 0.00 0.24 30.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.24 79.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 2.00 16.00 1.00 33.33 1.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.20
avg-cpu: %user %nice %system %iowait %steal %idle
92.86 0.00 2.63 0.38 0.00 4.14
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 1003.00 30284.00 0.00 0.00 0.24 30.19 14.00 14336.00 0.00 0.00 24.43 1024.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.59 78.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 15.00 212.00 31.00 67.39 1.80 14.13 4.00 18068.00 1.00 20.00 2.00 4517.00 0.00 0.00 0.04 3.20
$ free -m
total used free shared buff/cache available
Mem: 32083 10957 651 4 20475 20650
Swap: 0 0 0
simon.dudley@dev-elc-bu-tk-holesky-burn-24.5.0-RC1-full-forest:/var/log/besu$ sar -n DEV 1
Linux 6.5.0-1019-azure (dev-elc-bu-tk-holesky-burn-24) 05/08/24 _x86_64_ (8 CPU)
07:10:43 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
07:10:44 lo 3.00 3.00 2.88 2.88 0.00 0.00 0.00 0.00
07:10:44 eth0 53.00 59.00 25.71 4.78 0.00 0.00 0.00 0.00
07:10:44 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:10:44 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
07:10:45 lo 4.00 4.00 0.64 0.64 0.00 0.00 0.00 0.00
07:10:45 eth0 75.00 71.00 11.71 11.86 0.00 0.00 0.00 0.00
07:10:45 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:10:45 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
07:10:46 lo 3.00 3.00 2.36 2.36 0.00 0.00 0.00 0.00
07:10:46 eth0 102.00 115.00 23.21 21.28 0.00 0.00 0.00 0.00
07:10:46 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:10:46 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
07:10:47 lo 7.00 7.00 2.88 2.88 0.00 0.00 0.00 0.00
07:10:47 eth0 164.00 170.00 40.23 50.48 0.00 0.00 0.00 0.00
07:10:47 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:10:47 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
07:10:48 lo 6.00 6.00 4.11 4.11 0.00 0.00 0.00 0.00
07:10:48 eth0 100.00 112.00 17.33 25.48 0.00 0.00 0.00 0.00
07:10:48 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:10:48 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
07:10:49 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:10:49 eth0 122.00 102.00 48.01 15.95 0.00 0.00 0.00 0.00
07:10:49 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:10:49 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
07:10:50 lo 3.00 3.00 2.40 2.40 0.00 0.00 0.00 0.00
07:10:50 eth0 2244.00 654.00 3030.40 57.76 0.00 0.00 0.00 0.05
07:10:50 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:10:50 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
07:10:51 lo 3.00 3.00 2.46 2.46 0.00 0.00 0.00 0.00
07:10:51 eth0 153.00 170.00 24.63 18.68 0.00 0.00 0.00 0.00
07:10:51 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:10:51 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
07:10:52 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:10:52 eth0 129.00 130.00 23.61 35.68 0.00 0.00 0.00 0.00
07:10:52 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
^C
Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
Average: lo 3.22 3.22 1.97 1.97 0.00 0.00 0.00 0.00
Average: eth0 349.11 175.89 360.54 26.88 0.00 0.00 0.00 0.01
Average: docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
$ sar -n TCP,ETCP 1
Linux 6.5.0-1019-azure (dev-elc-bu-tk-holesky-burn-24) 05/08/24 _x86_64_ (8 CPU)
07:10:54 active/s passive/s iseg/s oseg/s
07:10:55 15.00 0.00 164.00 117.00
07:10:54 atmptf/s estres/s retrans/s isegerr/s orsts/s
07:10:55 0.00 14.00 0.00 0.00 0.00
07:10:55 active/s passive/s iseg/s oseg/s
07:10:56 8.00 1.00 235.00 184.00
07:10:55 atmptf/s estres/s retrans/s isegerr/s orsts/s
07:10:56 0.00 10.00 0.00 0.00 0.00
07:10:56 active/s passive/s iseg/s oseg/s
07:10:57 18.00 0.00 274.00 199.00
07:10:56 atmptf/s estres/s retrans/s isegerr/s orsts/s
07:10:57 0.00 15.00 0.00 0.00 0.00
07:10:57 active/s passive/s iseg/s oseg/s
07:10:58 15.00 3.00 90.00 71.00
07:10:57 atmptf/s estres/s retrans/s isegerr/s orsts/s
07:10:58 0.00 12.00 0.00 0.00 0.00
07:10:58 active/s passive/s iseg/s oseg/s
07:10:59 10.00 0.00 77.00 92.00
07:10:58 atmptf/s estres/s retrans/s isegerr/s orsts/s
07:10:59 0.00 8.00 0.00 0.00 0.00
07:10:59 active/s passive/s iseg/s oseg/s
07:11:00 14.00 0.00 149.00 195.00
07:10:59 atmptf/s estres/s retrans/s isegerr/s orsts/s
07:11:00 0.00 11.00 0.00 0.00 0.00
^C
Average: active/s passive/s iseg/s oseg/s
Average: 13.33 0.67 164.83 143.00
Average: atmptf/s estres/s retrans/s isegerr/s orsts/s
Average: 0.00 11.67 0.00 0.00 0.00
$ top
top - 07:11:17 up 5:55, 2 users, load average: 9.24, 9.02, 6.81
Tasks: 192 total, 2 running, 190 sleeping, 0 stopped, 0 zombie
%Cpu(s): 93.1 us, 3.8 sy, 0.0 ni, 2.3 id, 0.5 wa, 0.0 hi, 0.2 si, 0.0 st
MiB Mem : 32083.4 total, 600.8 free, 11034.4 used, 20448.2 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 20573.1 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
679 teku 20 0 12.6g 7.0g 1.0g S 635.5 22.4 290:52.11 java
678 besu 20 0 14.2g 3.1g 37204 S 128.2 9.9 813:42.43 java
94 root 20 0 0 0 0 S 4.3 0.0 0:16.15 kswapd0
79 root 20 0 0 0 0 I 2.0 0.0 0:15.35 kworker/4:1-xfs-conv/sda1
630 node-exp 20 0 726472 27444 11776 S 2.0 0.1 3:27.96 node_exporter
619 message+ 20 0 8776 5376 4096 S 0.7 0.0 1:07.34 dbus-daemon
625 root 20 0 1133924 147536 64128 S 0.7 0.4 1:34.36 wdavdaemon
1 root 20 0 167912 12840 7928 S 0.3 0.0 0:41.09 systemd
638 promtail 20 0 1693500 89416 53248 S 0.3 0.3 1:06.56 promtail
$ iostat -xz 1
Linux 6.5.0-1019-azure (dev-elc-bu-tk-holesky-burn-24) 05/08/24 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
44.33 0.00 1.43 1.59 0.00 52.65
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
loop0 0.00 0.00 0.00 0.00 0.21 8.05 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop1 0.00 0.03 0.00 0.00 0.15 9.37 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop2 0.00 0.01 0.00 0.00 0.27 19.27 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop3 0.00 0.01 0.00 0.00 0.17 15.39 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop4 0.00 0.00 0.00 0.00 0.17 8.37 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop5 0.01 0.32 0.00 0.00 0.37 36.09 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop6 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 265.17 8839.38 0.00 0.00 0.30 33.34 11.05 10606.14 0.18 1.59 70.61 960.24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.86 17.22
sdb 0.55 18.03 0.09 13.57 0.27 32.54 46.93 637.62 84.61 64.32 1.71 13.59 8.88 51231.91 2.66 23.03 2.16 5768.94 0.00 0.00 0.10 7.42
sr0 0.06 0.03 0.00 0.00 0.34 0.53 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
97.24 0.00 0.50 0.00 0.00 2.26
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 45.00 624.00 90.00 66.67 1.13 13.87 9.00 54180.00 3.00 25.00 2.22 6020.00 0.00 0.00 0.07 6.80
avg-cpu: %user %nice %system %iowait %steal %idle
98.50 0.00 0.38 0.13 0.00 1.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 48.00 644.00 92.00 65.71 1.69 13.42 9.00 54180.00 3.00 25.00 2.22 6020.00 0.00 0.00 0.10 9.20
avg-cpu: %user %nice %system %iowait %steal %idle
75.66 0.00 0.63 0.63 0.00 23.09
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 60.00 824.00 118.00 66.29 1.18 13.73 12.00 72240.00 5.00 29.41 2.08 6020.00 0.00 0.00 0.10 10.00
avg-cpu: %user %nice %system %iowait %steal %idle
88.41 0.00 0.88 0.00 0.00 10.71
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 61.00 844.00 121.00 66.48 1.66 13.84 13.00 72248.00 4.00 23.53 2.23 5557.54 0.00 0.00 0.13 12.80
avg-cpu: %user %nice %system %iowait %steal %idle
94.25 0.00 0.25 0.12 0.00 5.38
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 30.00 412.00 59.00 66.29 1.33 13.73 6.00 36120.00 2.00 25.00 2.00 6020.00 0.00 0.00 0.05 5.20
avg-cpu: %user %nice %system %iowait %steal %idle
93.34 0.00 0.38 0.00 0.00 6.28
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 15.00 208.00 30.00 66.67 1.27 13.87 3.00 18060.00 1.00 25.00 2.00 6020.00 0.00 0.00 0.03 2.40
avg-cpu: %user %nice %system %iowait %steal %idle
90.99 0.00 0.25 0.00 0.00 8.76
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 4.00 84.00 17.00 80.95 1.25 21.00 1.00 32.00 0.00 0.00 2.00 32.00 0.00 0.00 0.01 1.20
avg-cpu: %user %nice %system %iowait %steal %idle
87.59 0.00 0.38 0.63 0.00 11.40
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sdb 0.00 0.00 0.00 0.00 0.00 0.00 103.00 1452.00 211.00 67.20 1.21 14.10 21.00 126392.00 7.00 25.00 2.19 6018.67 0.00 0.00 0.17 16.40
avg-cpu: %user %nice %system %iowait %steal %idle
87.08 0.00 1.13 0.00 0.00 11.79
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 1.00 4.00 0.00 0.00 2.00 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.40
sdb 0.00 0.00 0.00 0.00 0.00 0.00 65.00 852.00 125.00 65.79 1.22 13.11 13.00 72244.00 4.00 23.53 2.15 5557.23 0.00 0.00 0.11 12.00
$ top
top - 23:57:10 up 22:41, 4 users, load average: 7.43, 7.66, 7.72
Tasks: 196 total, 1 running, 195 sleeping, 0 stopped, 0 zombie
%Cpu(s): 93.1 us, 0.2 sy, 0.0 ni, 6.6 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 32083.4 total, 7521.6 free, 12362.8 used, 12199.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 19244.9 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
31565 teku 20 0 12.9g 7.2g 1.1g S 743.2 23.0 2521:58 java
38829 besu 20 0 16.8g 4.7g 36528 S 1.0 14.8 1014:49 java
638 promtail 20 0 1693500 92040 53504 S 0.7 0.3 4:52.03 promtail
15 root 20 0 0 0 0 I 0.3 0.0 0:30.57 rcu_sched
494 root 20 0 0 0 0 S 0.3 0.0 0:12.88 xfsaild/sda1
625 root 20 0 1139800 153164 64384 S 0.3 0.5 6:36.31 wdavdaemon
946 syslog 20 0 884408 164840 24192 S 0.3 0.5 7:22.93 mdsd
1149 root 20 0 267732 24424 15232 S 0.3 0.1 0:55.32 telemetryd_v2
Linux 60-second analysis @ Thu 2024-05-09 05:09:39 UTC …
$ date
Thu May 9 05:09:39 UTC 2024
$ uptime
05:09:41 up 1 day, 3:53, 5 users, load average: 1.55, 1.75, 1.82
$ sudo dmesg | tail
[ 6.489797] sr 0:0:0:2: [sr0] tag#116 unaligned transfer
[ 6.489801] sr 0:0:0:2: [sr0] tag#117 unaligned transfer
[ 9.892221] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[ 9.894105] Bridge firewalling registered
[ 9.960166] Initializing XFRM netlink socket
[ 10.442181] loop6: detected capacity change from 0 to 8
[ 49.040915] hv_balloon: Max. dynamic memory size: 32768 MB
[21547.232613] workqueue: xfs_end_io [xfs] hogged CPU for >10000us 4 times, consider switching to WQ_UNBOUND
[35586.770240] workqueue: xfs_end_io [xfs] hogged CPU for >10000us 8 times, consider switching to WQ_UNBOUND
[85381.126472] workqueue: xfs_end_io [xfs] hogged CPU for >10000us 16 times, consider switching to WQ_UNBOUND
$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 0 291840 79312 20906152 0 0 1558 1832 12 21 43 2 54 2 0
2 0 0 276244 79312 20920944 0 0 11060 1232 4709 6584 15 1 82 2 0
4 0 0 367772 79312 20829212 0 0 38928 572 6128 5971 39 2 55 4 0
2 1 0 333752 79312 20864384 0 0 33908 576 5257 5189 35 1 60 4 0
0 1 0 280328 79312 20914060 0 0 47140 592 6492 6463 29 2 64 5 0
1 1 0 283352 79312 20912808 0 0 10820 158868 4749 6846 17 2 74 7 0
0 1 0 282408 79312 20917076 0 0 3580 133780 6236 11925 15 2 76 7 0
2 0 0 248932 79312 20950928 0 0 3380 137216 6049 10357 17 3 77 4 0
1 1 0 312060 79312 20903868 0 0 67576 133312 6694 9047 14 4 73 9 0
1 1 0 250836 79312 20970384 0 0 59216 132152 7350 9693 13 4 73 10 0
1 1 0 583608 79312 20648816 0 0 96968 170924 9408 11671 12 5 70 13 0
3 0 0 360500 79312 20873144 0 0 90376 76104 9493 12791 19 4 68 9 0
0 2 0 270084 79312 20955388 0 0 82400 188152 8947 10615 20 4 67 9 0
1 2 0 438660 79312 20786692 0 0 90116 87856 8704 12312 15 4 71 10 0
1 1 0 323680 79312 20905044 0 0 121088 156012 9648 10343 19 5 65 11 0
0 2 0 247068 79312 20977796 0 0 69324 133616 7187 9309 17 3 73 7 0
2 1 0 915420 79312 20306140 0 0 97560 114544 10593 14696 13 5 72 11 0
1 1 0 667300 79312 20548956 0 0 115332 111536 6993 7798 17 3 72 8 0
0 1 0 749980 79312 20468320 0 0 49536 88244 6130 8131 10 2 83 5 0
2 1 0 523728 79312 20692432 0 0 103624 136552 8985 12038 23 3 65 9 0
5 0 0 319612 79312 20895356 0 0 105012 141324 8020 10034 55 4 34 7 0
13 0 0 256748 79312 20953184 0 0 85512 73944 7903 10164 72 4 20 3 0
7 0 0 787076 79312 20417564 0 0 96676 109856 8799 9843 72 6 16 7 0
11 1 0 420592 79312 20627468 0 0 94464 132620 7372 8774 70 4 22 5 0
15 0 0 367576 79312 19780408 0 0 114380 62228 8628 10139 71 13 11 5 0
10 1 0 321524 79312 19672384 0 0 85748 104396 6409 6609 93 5 1 0 0
7 0 0 310068 79312 19720560 0 0 113208 61720 9247 8682 83 11 5 2 0
4 2 0 883956 79312 19607664 0 0 49968 262324 7544 9980 55 15 23 7 0
5 0 0 502640 79312 19725048 0 0 89552 61072 9319 11814 46 13 32 9 0
^C
simon.dudley@dev-elc-bu-tk-holesky-burn-24.5.0-RC1-full-forest:~$ mpstat -P ALL 1
Linux 6.5.0-1019-azure (dev-elc-bu-tk-holesky-burn-24) 05/09/24 _x86_64_ (8 CPU)
05:10:32 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
05:10:33 all 17.99 0.00 1.89 4.28 0.00 0.38 0.00 0.00 0.00 75.47
05:10:33 0 4.04 0.00 7.07 0.00 0.00 0.00 0.00 0.00 0.00 88.89
05:10:33 1 4.12 0.00 3.09 9.28 0.00 1.03 0.00 0.00 0.00 82.47
05:10:33 2 2.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 97.00
05:10:33 3 1.01 0.00 1.01 1.01 0.00 0.00 0.00 0.00 0.00 96.97
05:10:33 4 15.15 0.00 3.03 13.13 0.00 0.00 0.00 0.00 0.00 68.69
05:10:33 5 10.78 0.00 0.00 0.98 0.00 0.00 0.00 0.00 0.00 88.24
05:10:33 6 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:10:33 7 6.06 0.00 1.01 10.10 0.00 1.01 0.00 0.00 0.00 81.82
05:10:33 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
05:10:34 all 13.57 0.00 3.74 6.72 0.00 0.25 0.00 0.00 0.00 75.72
05:10:34 0 1.94 0.00 0.00 4.85 0.00 1.94 0.00 0.00 0.00 91.26
05:10:34 1 1.96 0.00 2.94 3.92 0.00 0.00 0.00 0.00 0.00 91.18
05:10:34 2 0.99 0.00 0.99 1.98 0.00 0.00 0.00 0.00 0.00 96.04
05:10:34 3 42.00 0.00 1.00 8.00 0.00 0.00 0.00 0.00 0.00 49.00
05:10:34 4 0.00 0.00 0.00 3.09 0.00 0.00 0.00 0.00 0.00 96.91
05:10:34 5 12.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 87.00
05:10:34 6 34.00 0.00 20.00 19.00 0.00 0.00 0.00 0.00 0.00 27.00
05:10:34 7 16.00 0.00 5.00 12.00 0.00 0.00 0.00 0.00 0.00 67.00
05:10:34 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
05:10:35 all 14.25 0.00 7.06 8.20 0.00 0.13 0.00 0.00 0.00 70.37
05:10:35 0 2.00 0.00 2.00 5.00 0.00 0.00 0.00 0.00 0.00 91.00
05:10:35 1 27.27 0.00 6.06 29.29 0.00 0.00 0.00 0.00 0.00 37.37
05:10:35 2 5.05 0.00 0.00 4.04 0.00 0.00 0.00 0.00 0.00 90.91
05:10:35 3 4.08 0.00 1.02 4.08 0.00 0.00 0.00 0.00 0.00 90.82
05:10:35 4 13.00 0.00 3.00 19.00 0.00 1.00 0.00 0.00 0.00 64.00
05:10:35 5 3.06 0.00 0.00 2.04 0.00 0.00 0.00 0.00 0.00 94.90
05:10:35 6 4.00 0.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 94.00
05:10:35 7 55.56 0.00 44.44 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:10:35 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
05:10:36 all 19.92 0.00 1.13 6.39 0.00 0.50 0.00 0.00 0.00 72.06
05:10:36 0 1.01 0.00 1.01 2.02 0.00 0.00 0.00 0.00 0.00 95.96
05:10:36 1 34.00 0.00 4.00 20.00 0.00 1.00 0.00 0.00 0.00 41.00
05:10:36 2 6.00 0.00 1.00 1.00 0.00 0.00 0.00 0.00 0.00 92.00
05:10:36 3 4.95 0.00 0.00 5.94 0.00 0.99 0.00 0.00 0.00 88.12
05:10:36 4 12.12 0.00 2.02 20.20 0.00 1.01 0.00 0.00 0.00 64.65
05:10:36 5 1.00 0.00 1.00 1.00 0.00 0.00 0.00 0.00 0.00 97.00
05:10:36 6 1.01 0.00 0.00 1.01 0.00 0.00 0.00 0.00 0.00 97.98
05:10:36 7 99.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00
05:10:36 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
05:10:37 all 18.00 0.00 4.75 9.25 0.00 0.12 0.00 0.00 0.00 67.88
05:10:37 0 16.00 0.00 1.00 1.00 0.00 0.00 0.00 0.00 0.00 82.00
05:10:37 1 31.00 0.00 6.00 33.00 0.00 0.00 0.00 0.00 0.00 30.00
05:10:37 2 2.04 0.00 0.00 2.04 0.00 0.00 0.00 0.00 0.00 95.92
05:10:37 3 0.99 0.00 0.00 4.95 0.00 0.99 0.00 0.00 0.00 93.07
05:10:37 4 35.64 0.00 2.97 4.95 0.00 0.00 0.00 0.00 0.00 56.44
05:10:37 5 3.96 0.00 0.99 0.99 0.00 0.00 0.00 0.00 0.00 94.06
05:10:37 6 4.00 0.00 0.00 5.00 0.00 0.00 0.00 0.00 0.00 91.00
05:10:37 7 50.51 0.00 27.27 22.22 0.00 0.00 0.00 0.00 0.00 0.00
05:10:37 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
05:10:38 all 19.32 0.00 5.40 5.02 0.00 0.25 0.00 0.00 0.00 70.01
05:10:38 0 8.91 0.00 0.99 0.99 0.00 0.00 0.00 0.00 0.00 89.11
05:10:38 1 36.46 0.00 2.08 23.96 0.00 1.04 0.00 0.00 0.00 36.46
05:10:38 2 4.95 0.00 0.00 0.99 0.00 0.00 0.00 0.00 0.00 94.06
05:10:38 3 17.00 0.00 1.00 2.00 0.00 0.00 0.00 0.00 0.00 80.00
05:10:38 4 17.17 0.00 3.03 13.13 0.00 1.01 0.00 0.00 0.00 65.66
05:10:38 5 3.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 96.97
05:10:38 6 41.00 0.00 23.00 0.00 0.00 0.00 0.00 0.00 0.00 36.00
05:10:38 7 26.73 0.00 12.87 0.00 0.00 0.00 0.00 0.00 0.00 60.40
05:10:38 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
05:10:39 all 51.01 0.00 3.04 6.96 0.00 0.38 0.00 0.00 0.00 38.61
05:10:39 0 30.21 0.00 5.21 9.38 0.00 0.00 0.00 0.00 0.00 55.21
05:10:39 1 46.00 0.00 4.00 7.00 0.00 3.00 0.00 0.00 0.00 40.00
05:10:39 2 17.17 0.00 0.00 3.03 0.00 0.00 0.00 0.00 0.00 79.80
05:10:39 3 89.11 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 10.89
05:10:39 4 26.80 0.00 8.25 25.77 0.00 0.00 0.00 0.00 0.00 39.18
05:10:39 5 88.89 0.00 1.01 1.01 0.00 0.00 0.00 0.00 0.00 9.09
05:10:39 6 67.68 0.00 2.02 0.00 0.00 0.00 0.00 0.00 0.00 30.30
05:10:39 7 40.40 0.00 4.04 10.10 0.00 0.00 0.00 0.00 0.00 45.45
05:10:39 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
05:10:40 all 39.02 0.00 6.27 7.65 0.00 0.13 0.00 0.00 0.00 46.93
05:10:40 0 14.29 0.00 1.02 7.14 0.00 0.00 0.00 0.00 0.00 77.55
05:10:40 1 84.85 0.00 2.02 1.01 0.00 0.00 0.00 0.00 0.00 12.12
05:10:40 2 23.76 0.00 1.98 1.98 0.00 0.00 0.00 0.00 0.00 72.28
05:10:40 3 46.46 0.00 32.32 20.20 0.00 0.00 0.00 0.00 0.00 1.01
05:10:40 4 36.00 0.00 6.00 11.00 0.00 0.00 0.00 0.00 0.00 47.00
05:10:40 5 36.00 0.00 3.00 8.00 0.00 0.00 0.00 0.00 0.00 53.00
05:10:40 6 14.14 0.00 1.01 6.06 0.00 0.00 0.00 0.00 0.00 78.79
05:10:40 7 56.44 0.00 2.97 5.94 0.00 0.99 0.00 0.00 0.00 33.66
^C
Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
Average: all 24.10 0.00 4.16 6.81 0.00 0.27 0.00 0.00 0.00 64.66
Average: 0 9.67 0.00 2.26 3.77 0.00 0.25 0.00 0.00 0.00 84.05
Average: 1 33.17 0.00 3.78 15.89 0.00 0.76 0.00 0.00 0.00 46.41
Average: 2 7.76 0.00 0.50 1.88 0.00 0.13 0.00 0.00 0.00 89.74
Average: 3 25.78 0.00 4.51 5.76 0.00 0.25 0.00 0.00 0.00 63.70
Average: 4 19.57 0.00 3.54 13.76 0.00 0.38 0.00 0.00 0.00 62.75
Average: 5 19.77 0.00 0.75 1.88 0.00 0.00 0.00 0.00 0.00 77.60
Average: 6 33.25 0.00 5.77 4.14 0.00 0.00 0.00 0.00 0.00 56.84
Average: 7 43.86 0.00 12.16 7.52 0.00 0.38 0.00 0.00 0.00 36.09
$ pidstat 1
Linux 6.5.0-1019-azure (dev-elc-bu-tk-holesky-burn-24) 05/09/24 _x86_64_ (8 CPU)
05:10:45 UID PID %usr %system %guest %wait %CPU CPU Command
05:10:46 0 1 0.99 0.00 0.00 0.00 0.99 6 systemd
05:10:46 102 619 0.99 0.99 0.00 0.00 1.98 0 dbus-daemon
05:10:46 998 630 2.97 0.99 0.00 0.00 3.96 0 node_exporter
05:10:46 997 638 0.00 0.99 0.00 0.00 0.99 5 promtail
05:10:46 1024 63238 63.37 4.95 0.00 0.00 68.32 1 java
05:10:46 1025 64747 87.13 14.85 0.00 0.00 101.98 2 java
05:10:46 1007 74552 0.00 0.99 0.00 0.00 0.99 6 pidstat
05:10:46 UID PID %usr %system %guest %wait %CPU CPU Command
05:10:47 0 94 0.00 3.00 0.00 0.00 3.00 4 kswapd0
05:10:47 0 625 1.00 0.00 0.00 0.00 1.00 7 wdavdaemon
05:10:47 997 638 1.00 0.00 0.00 0.00 1.00 5 promtail
05:10:47 104 946 0.00 1.00 0.00 0.00 1.00 7 mdsd
05:10:47 0 1149 1.00 0.00 0.00 0.00 1.00 4 telemetryd_v2
05:10:47 1007 62271 1.00 0.00 0.00 0.00 1.00 0 sshd
05:10:47 1024 63238 36.00 11.00 0.00 0.00 47.00 1 java
05:10:47 1025 64747 76.00 12.00 0.00 0.00 88.00 2 java
05:10:47 0 66238 0.00 1.00 0.00 0.00 1.00 1 kworker/1:0-xfs-conv/sda1
05:10:47 1007 74552 0.00 1.00 0.00 0.00 1.00 6 pidstat
05:10:47 UID PID %usr %system %guest %wait %CPU CPU Command
05:10:48 0 625 0.00 1.00 0.00 0.00 1.00 7 wdavdaemon
05:10:48 997 638 1.00 1.00 0.00 0.00 2.00 5 promtail
05:10:48 1007 61965 0.00 1.00 0.00 0.00 1.00 4 sshd
05:10:48 1024 63238 44.00 4.00 0.00 0.00 48.00 1 java
05:10:48 1025 64747 116.00 49.00 0.00 0.00 165.00 2 java
05:10:48 0 66238 0.00 1.00 0.00 0.00 1.00 1 kworker/1:0-events
05:10:48 1007 74552 0.00 1.00 0.00 0.00 1.00 6 pidstat
05:10:48 UID PID %usr %system %guest %wait %CPU CPU Command
05:10:49 997 638 1.00 0.00 0.00 0.00 1.00 5 promtail
05:10:49 0 930 1.00 0.00 0.00 0.00 1.00 4 wdavdaemon
05:10:49 0 10855 0.00 1.00 0.00 0.00 1.00 4 hv_kvp_daemon
05:10:49 1024 63238 48.00 1.00 0.00 0.00 49.00 1 java
05:10:49 1025 64747 132.00 1.00 0.00 0.00 133.00 2 java
05:10:49 0 65745 0.00 1.00 0.00 0.00 1.00 2 kworker/2:0-events
05:10:49 1007 74552 1.00 0.00 0.00 0.00 1.00 6 pidstat
05:10:49 UID PID %usr %system %guest %wait %CPU CPU Command
05:10:50 0 15 0.00 1.00 0.00 0.00 1.00 3 rcu_sched
05:10:50 997 638 0.00 1.00 0.00 0.00 1.00 5 promtail
05:10:50 104 946 1.00 0.00 0.00 0.00 1.00 7 mdsd
05:10:50 0 10855 0.00 4.00 0.00 0.00 4.00 2 hv_kvp_daemon
05:10:50 1024 63238 63.00 8.00 0.00 0.00 71.00 1 java
05:10:50 1007 64631 0.00 1.00 0.00 0.00 1.00 5 tail
05:10:50 1025 64747 87.00 20.00 0.00 0.00 107.00 2 java
05:10:50 0 66422 0.00 1.00 0.00 0.00 1.00 4 kworker/4:0-xfs-conv/sda1
05:10:50 1007 74552 0.00 1.00 0.00 0.00 1.00 6 pidstat
05:10:50 UID PID %usr %system %guest %wait %CPU CPU Command
05:10:51 0 1 0.00 1.00 0.00 0.00 1.00 7 systemd
05:10:51 102 619 0.00 1.00 0.00 0.00 1.00 2 dbus-daemon
05:10:51 0 625 1.00 0.00 0.00 0.00 1.00 7 wdavdaemon
05:10:51 998 630 3.00 1.00 0.00 0.00 4.00 0 node_exporter
05:10:51 0 930 1.00 0.00 0.00 0.00 1.00 4 wdavdaemon
05:10:51 1024 63238 68.00 8.00 0.00 0.00 76.00 1 java
05:10:51 1025 64747 77.00 25.00 0.00 0.00 102.00 2 java
05:10:51 0 74093 0.00 1.00 0.00 0.00 1.00 3 kworker/u16:2-events_unbound
05:10:51 UID PID %usr %system %guest %wait %CPU CPU Command
05:10:52 0 94 0.00 2.00 0.00 0.00 2.00 7 kswapd0
05:10:52 0 127 0.00 1.00 0.00 0.00 1.00 0 jbd2/sdb1-8
05:10:52 104 946 0.00 1.00 0.00 0.00 1.00 7 mdsd
05:10:52 1024 63238 67.00 9.00 0.00 0.00 76.00 1 java
05:10:52 1025 64747 83.00 29.00 0.00 0.00 112.00 2 java
05:10:52 0 66422 0.00 1.00 0.00 0.00 1.00 4 kworker/4:0-xfs-conv/sda1
05:10:52 1007 74552 0.00 1.00 0.00 0.00 1.00 6 pidstat
05:10:52 UID PID %usr %system %guest %wait %CPU CPU Command
05:10:53 0 94 0.00 3.00 0.00 0.00 3.00 7 kswapd0
05:10:53 0 930 1.00 0.00 0.00 0.00 1.00 4 wdavdaemon
05:10:53 1024 63238 57.00 5.00 0.00 0.00 62.00 1 java
05:10:53 1025 64747 131.00 2.00 0.00 0.00 133.00 2 java
05:10:53 1007 74552 0.00 1.00 0.00 0.00 1.00 0 pidstat
05:10:53 UID PID %usr %system %guest %wait %CPU CPU Command
05:10:54 0 94 0.00 2.00 0.00 0.00 2.00 7 kswapd0
05:10:54 997 638 1.00 0.00 0.00 0.00 1.00 5 promtail
05:10:54 1024 63238 53.00 6.00 0.00 0.00 59.00 1 java
05:10:54 1025 64747 5.00 21.00 0.00 0.00 26.00 2 java
05:10:54 0 66238 0.00 1.00 0.00 0.00 1.00 1 kworker/1:0-xfs-conv/sda1
05:10:54 0 66422 0.00 1.00 0.00 0.00 1.00 4 kworker/4:0-events
05:10:54 0 74500 0.00 1.00 0.00 0.00 1.00 2 kworker/u16:4-events_power_efficient
^C
Average: UID PID %usr %system %guest %wait %CPU CPU Command
Average: 0 1 0.11 0.11 0.00 0.00 0.22 - systemd
Average: 0 15 0.00 0.11 0.00 0.00 0.11 - rcu_sched
Average: 0 94 0.00 1.11 0.00 0.00 1.11 - kswapd0
Average: 0 127 0.00 0.11 0.00 0.00 0.11 - jbd2/sdb1-8
Average: 102 619 0.11 0.22 0.00 0.00 0.33 - dbus-daemon
Average: 0 625 0.22 0.11 0.00 0.00 0.33 - wdavdaemon
Average: 998 630 0.67 0.22 0.00 0.00 0.89 - node_exporter
Average: 997 638 0.44 0.33 0.00 0.00 0.78 - promtail
Average: 0 930 0.33 0.00 0.00 0.00 0.33 - wdavdaemon
Average: 104 946 0.11 0.22 0.00 0.00 0.33 - mdsd
Average: 0 1149 0.11 0.00 0.00 0.00 0.11 - telemetryd_v2
Average: 0 10855 0.00 0.55 0.00 0.00 0.55 - hv_kvp_daemon
Average: 1007 61965 0.00 0.11 0.00 0.00 0.11 - sshd
Average: 1007 62271 0.11 0.00 0.00 0.00 0.11 - sshd
Average: 1024 63238 55.49 6.33 0.00 0.00 61.82 - java
Average: 1007 64631 0.00 0.11 0.00 0.00 0.11 - tail
Average: 1025 64747 88.24 19.31 0.00 0.00 107.55 - java
Average: 0 65745 0.00 0.11 0.00 0.00 0.11 - kworker/2:0-events
Average: 0 66238 0.00 0.33 0.00 0.00 0.33 - kworker/1:0-xfs-conv/sda1
Average: 0 66422 0.00 0.33 0.00 0.00 0.33 - kworker/4:0-events
Average: 0 74093 0.00 0.11 0.00 0.00 0.11 - kworker/u16:2-events_unbound
Average: 0 74500 0.00 0.11 0.00 0.00 0.11 - kworker/u16:4-events_power_efficient
Average: 1007 74552 0.11 0.67 0.00 0.00 0.78 - pidstat
$ iostat -xz 1
Linux 6.5.0-1019-azure (dev-elc-bu-tk-holesky-burn-24) 05/09/24 _x86_64_ (8 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
42.80 0.00 1.55 1.93 0.00 53.73
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
loop0 0.00 0.00 0.00 0.00 0.21 8.05 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop1 0.00 0.02 0.00 0.00 0.15 9.37 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop2 0.00 0.01 0.00 0.00 0.27 19.27 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop3 0.00 0.01 0.00 0.00 0.17 15.39 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop4 0.00 0.00 0.00 0.00 0.17 8.37 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop5 0.01 0.26 0.00 0.00 0.37 36.09 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
loop6 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sda 368.87 12376.01 0.00 0.00 0.30 33.55 14.57 13940.43 0.23 1.56 68.56 956.73 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.11 23.23
sdb 0.67 32.03 0.08 10.43 0.30 47.77 47.09 655.45 85.75 64.55 1.77 13.92 8.87 51223.66 2.69 23.27 2.18 5772.00 0.00 0.00 0.10 7.53
sr0 0.05 0.03 0.00 0.00 0.34 0.53 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
avg-cpu: %user %nice %system %iowait %steal %idle
56.73 0.00 6.79 4.40 0.00 32.08
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 1195.00 39296.00 0.00 0.00 0.27 32.88 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.32 63.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 30.00 408.00 58.00 65.91 1.10 13.60 6.00 36120.00 2.00 25.00 2.00 6020.00 0.00 0.00 0.05 4.00
avg-cpu: %user %nice %system %iowait %steal %idle
23.71 0.00 1.38 3.64 0.00 71.27
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 811.00 26616.00 0.00 0.00 0.26 32.82 2.00 8.00 0.00 0.00 1.50 4.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.21 40.40
sdb 0.00 0.00 0.00 0.00 0.00 0.00 58.00 796.00 106.00 64.63 1.36 13.72 11.00 54216.00 3.00 21.43 2.27 4928.73 0.00 0.00 0.10 10.80
avg-cpu: %user %nice %system %iowait %steal %idle
6.39 0.00 1.38 6.39 0.00 85.84
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 1956.00 65020.00 0.00 0.00 0.25 33.24 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.49 88.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 20.00 284.00 37.00 64.91 1.15 14.20 4.00 36084.00 2.00 33.33 2.25 9021.00 0.00 0.00 0.03 3.20
avg-cpu: %user %nice %system %iowait %steal %idle
5.06 0.00 1.27 7.72 0.00 85.95
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 1969.00 64660.00 0.00 0.00 0.26 32.84 1.00 8.00 1.00 50.00 2.00 8.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.51 90.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 60.00 804.00 113.00 65.32 1.40 13.40 12.00 72240.00 4.00 25.00 2.83 6020.00 0.00 0.00 0.12 10.40
avg-cpu: %user %nice %system %iowait %steal %idle
8.35 0.00 1.52 6.71 0.00 83.42
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 1612.00 52704.00 0.00 0.00 0.26 32.69 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.42 77.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 75.00 996.00 139.00 64.95 1.32 13.28 15.00 90300.00 5.00 25.00 2.27 6020.00 0.00 0.00 0.13 12.80
avg-cpu: %user %nice %system %iowait %steal %idle
7.72 0.00 1.99 9.46 0.00 80.82
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 1939.00 63848.00 0.00 0.00 0.30 32.93 100.00 94380.00 0.00 0.00 29.75 943.80 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.55 95.60
sdb 0.00 0.00 0.00 0.00 0.00 0.00 31.00 396.00 53.00 63.10 1.58 12.77 6.00 36120.00 2.00 25.00 2.00 6020.00 0.00 0.00 0.06 6.80
avg-cpu: %user %nice %system %iowait %steal %idle
7.07 0.00 1.14 4.42 0.00 87.37
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 1313.00 43384.00 0.00 0.00 0.26 33.04 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.34 64.80
sdb 0.00 0.00 0.00 0.00 0.00 0.00 17.00 248.00 31.00 64.58 1.12 14.59 4.00 18068.00 1.00 20.00 2.25 4517.00 0.00 0.00 0.03 2.40
avg-cpu: %user %nice %system %iowait %steal %idle
13.66 0.00 0.50 1.38 0.00 84.46
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 321.00 10552.00 0.00 0.00 0.26 32.87 3.00 20.00 1.00 25.00 2.33 6.67 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.09 25.60
sdb 0.00 0.00 0.00 0.00 0.00 0.00 18.00 300.00 50.00 73.53 1.39 16.67 4.00 18092.00 1.00 20.00 2.00 4523.00 0.00 0.00 0.03 4.00
avg-cpu: %user %nice %system %iowait %steal %idle
6.31 0.00 0.63 6.44 0.00 86.62
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 1642.00 54128.00 0.00 0.00 0.27 32.96 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.45 78.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 43.00 596.00 85.00 66.41 1.16 13.86 9.00 54152.00 3.00 25.00 2.33 6016.89 0.00 0.00 0.07 6.40
avg-cpu: %user %nice %system %iowait %steal %idle
7.13 0.00 2.13 6.76 0.00 83.98
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 1652.00 54516.00 0.00 0.00 0.26 33.00 2.00 1340.00 0.00 0.00 8.00 670.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.44 77.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 77.00 1000.00 138.00 64.19 1.44 12.99 15.00 90300.00 5.00 25.00 2.13 6020.00 0.00 0.00 0.14 14.00
avg-cpu: %user %nice %system %iowait %steal %idle
20.73 0.00 1.01 6.03 0.00 72.24
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 1281.00 42284.00 0.00 0.00 0.29 33.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.37 66.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 74.00 996.00 140.00 65.42 1.35 13.46 16.00 90304.00 5.00 23.81 2.56 5644.00 0.00 0.00 0.14 12.80
avg-cpu: %user %nice %system %iowait %steal %idle
14.12 0.00 1.38 4.12 0.00 80.38
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
sda 1245.00 41068.00 0.00 0.00 0.27 32.99 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.33 65.20
sdb 0.00 0.00 0.00 0.00 0.00 0.00 109.00 996.00 122.00 52.81 3.69 9.14 3.00 18060.00 1.00 25.00 2.00 6020.00 0.00 0.00 0.41 4.40
$ free -m
total used free shared buff/cache available
Mem: 32083 11443 265 4 20374 20164
Swap: 0 0 0
$ sar -n DEV 1
Linux 6.5.0-1019-azure (dev-elc-bu-tk-holesky-burn-24) 05/09/24 _x86_64_ (8 CPU)
05:11:24 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:11:25 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:11:25 eth0 217.00 303.00 22.11 38.75 0.00 0.00 0.00 0.00
05:11:25 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:11:25 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:11:26 lo 2.00 2.00 0.17 0.17 0.00 0.00 0.00 0.00
05:11:26 eth0 337.00 361.00 136.20 192.64 0.00 0.00 0.00 0.00
05:11:26 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:11:26 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:11:27 lo 11.00 11.00 90.28 90.28 0.00 0.00 0.00 0.00
05:11:27 eth0 1757.00 1369.00 1582.62 3027.43 0.00 0.00 0.00 0.05
05:11:27 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:11:27 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:11:28 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:11:28 eth0 311.00 244.00 165.43 40.83 0.00 0.00 0.00 0.00
05:11:28 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:11:28 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:11:29 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:11:29 eth0 887.00 503.00 935.17 81.65 0.00 0.00 0.00 0.02
05:11:29 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:11:29 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:11:30 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:11:30 eth0 282.00 143.00 274.08 16.08 0.00 0.00 0.00 0.00
05:11:30 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:11:30 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:11:31 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:11:31 eth0 331.00 289.00 70.32 37.92 0.00 0.00 0.00 0.00
05:11:31 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:11:31 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:11:32 lo 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:11:32 eth0 172.00 191.00 30.45 64.28 0.00 0.00 0.00 0.00
05:11:32 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
05:11:32 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
05:11:33 lo 138.00 138.00 162.98 162.98 0.00 0.00 0.00 0.00
05:11:33 eth0 184.00 160.00 48.85 28.65 0.00 0.00 0.00 0.00
05:11:33 docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
^C
Average: IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s %ifutil
Average: lo 16.78 16.78 28.16 28.16 0.00 0.00 0.00 0.00
Average: eth0 497.56 395.89 362.80 392.03 0.00 0.00 0.00 0.01
Average: docker0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
$ sar -n TCP,ETCP 1
Linux 6.5.0-1019-azure (dev-elc-bu-tk-holesky-burn-24) 05/09/24 _x86_64_ (8 CPU)
05:11:35 active/s passive/s iseg/s oseg/s
05:11:36 0.00 0.00 246.00 194.00
05:11:35 atmptf/s estres/s retrans/s isegerr/s orsts/s
05:11:36 0.00 0.00 1.00 0.00 0.00
05:11:36 active/s passive/s iseg/s oseg/s
05:11:37 0.00 0.00 96.00 116.00
05:11:36 atmptf/s estres/s retrans/s isegerr/s orsts/s
05:11:37 0.00 0.00 0.00 0.00 0.00
05:11:37 active/s passive/s iseg/s oseg/s
05:11:38 5.00 0.00 80.00 94.00
05:11:37 atmptf/s estres/s retrans/s isegerr/s orsts/s
05:11:38 0.00 0.00 0.00 0.00 0.00
05:11:38 active/s passive/s iseg/s oseg/s
05:11:39 0.00 0.00 528.00 737.00
05:11:38 atmptf/s estres/s retrans/s isegerr/s orsts/s
05:11:39 0.00 0.00 0.00 0.00 0.00
05:11:39 active/s passive/s iseg/s oseg/s
05:11:40 0.00 2.00 142.00 170.00
05:11:39 atmptf/s estres/s retrans/s isegerr/s orsts/s
05:11:40 0.00 0.00 0.00 0.00 0.00
05:11:40 active/s passive/s iseg/s oseg/s
05:11:41 0.00 2.00 332.00 277.00
05:11:40 atmptf/s estres/s retrans/s isegerr/s orsts/s
05:11:41 0.00 1.00 1.00 0.00 0.00
05:11:41 active/s passive/s iseg/s oseg/s
05:11:42 0.00 2.00 154.00 179.00
05:11:41 atmptf/s estres/s retrans/s isegerr/s orsts/s
05:11:42 0.00 0.00 0.00 0.00 0.00
05:11:42 active/s passive/s iseg/s oseg/s
05:11:43 0.00 2.00 122.00 104.00
05:11:42 atmptf/s estres/s retrans/s isegerr/s orsts/s
05:11:43 0.00 1.00 0.00 0.00 1.00
05:11:43 active/s passive/s iseg/s oseg/s
05:11:44 5.00 0.00 189.00 234.00
05:11:43 atmptf/s estres/s retrans/s isegerr/s orsts/s
05:11:44 0.00 0.00 0.00 0.00 0.00
^C
Average: active/s passive/s iseg/s oseg/s
Average: 1.11 0.89 209.89 233.89
Average: atmptf/s estres/s retrans/s isegerr/s orsts/s
Average: 0.00 0.22 0.22 0.00 0.11
$ top
top - 05:11:52 up 1 day, 3:55, 5 users, load average: 1.69, 1.82, 1.84
Tasks: 199 total, 1 running, 198 sleeping, 0 stopped, 0 zombie
%Cpu(s): 12.3 us, 1.3 sy, 0.0 ni, 83.8 id, 2.5 wa, 0.0 hi, 0.1 si, 0.0 st
MiB Mem : 32083.4 total, 311.2 free, 11371.1 used, 20401.0 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 20236.5 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
63238 besu 20 0 22.5g 3.0g 36640 S 96.3 9.7 274:09.72 java
64747 teku 20 0 12.9g 6.7g 730632 S 2.3 21.4 257:21.46 java
69 root 20 0 0 0 0 S 1.7 0.0 2:55.72 kcompactd0
630 node-exp 20 0 726728 27284 11776 S 1.3 0.1 16:40.39 node_exporter
10855 root 20 0 3812 2560 1920 S 1.3 0.0 1:26.85 hv_kvp_daemon
94 root 20 0 0 0 0 S 1.0 0.0 4:45.95 kswapd0
638 promtail 20 0 1841540 109212 54016 S 1.0 0.3 12:50.51 promtail
619 message+ 20 0 8796 5412 4096 S 0.7 0.0 5:19.00 dbus-daemon
930 root 20 0 1013648 89612 54272 S 0.7 0.3 7:06.33 wdavdaemon
1 root 20 0 167912 12968 7928 S 0.3 0.0 2:42.15 systemd
625 root 20 0 1138776 156480 64384 S 0.3 0.5 8:37.14 wdavdaemon
