Monitoring Lighthouse Backfill Processing CPU Usage

Issue: https://github.com/sigp/lighthouse/issues/3212
Pull Request: https://github.com/sigp/lighthouse/pull/3936

Rate-limiting on backfill sync processing has been introduced to solve the CPU consumption issue on lower spec machines. To figure out a good default rate for processing, it is helpful to understand:

• the processing time for each backfill batch, so we don't schedule the processing on critical time window and impact validator performance, e.g. slot start
• the cpu usage improvement when rate-limit is applied to backfill processing
• the impact to overall time to complete backfill sync

Things to consider include:

• whether have a fixed default rate (e.g. 3 batches per slot) or a rate based on the number of work threads / CPU cores avaialble.
• the timing on when to schedule processing

Batch processing times

Mainnet

Setup: Docker, CPU limit = 2

Summary:

• The batch processing time generally fluctuates between 180ms - 250ms
• When NOT rate limited, the BeaconProcessor can process approx. up to 8 batches per slot

Rate limit disabled

Rate limit enabled (6,7,9 secs after slot start)