## Sidecar Performance Overview
Some performance-related issues have been reported in the Sidecar repository. While two of these issues are connected to Kusama's Spammening event, it is important to note that Sidecar should be robust enough to handle such events without degradation.

### Related Issues
- [Performance issue with kusama #1554](https://github.com/paritytech/substrate-api-sidecar/issues/1554): Open but will be marked as Resolved soon
- Root Cause : [Spammening](https://polkadot.com/spammening/) which resulted in High Volume of Extrinsics in Blocks
- [Accessing API interfaces is very slow #1555](https://github.com/paritytech/substrate-api-sidecar/issues/1555): Resolved
- Root Cause : [Spammening](https://polkadot.com/spammening/) which resulted in High Volume of Extrinsics in Blocks
- [Sidecar fails after reaching 4GB RSS Memory #1485](https://github.com/paritytech/substrate-api-sidecar/issues/1485): Open
- Root Cause : Looks like it is not relevant to the Spammening event.

Also, some teams reported degradation in specific endpoints:
- `/staking-info` : JavaScript heap out of memory
- `/blocks/head`

Last, we have cases of `out of heap memory` errors without querying any endpoints.

### Meta/Umbrella Issue
[[Meta] Performance #1361](https://github.com/paritytech/substrate-api-sidecar/issues/1361)

## Performance Improvement Initiative
To address these challenges, we started a performance analysis work which falls under the broader performance initiative tracked in Meta Issue [#1361](https://github.com/paritytech/substrate-api-sidecar/issues/1361) and includes:
- Function-level execution time analysis
- Version-by-version heap allocation tracking

The goal is to identify specific performance bottlenecks and implement improvements to enhance Sidecar's overall reliability and efficiency.

### Ideas / Suggested Solutions
Apart the profiling, some improvement ideas that were suggested were the following:
- Empty cache memory more efficiently between subsequent blocks.
- Sanitize values way more efficiently.
- Decode the extrinsics and event data cleaner. Currently it uses recursion, but it can definitely be optimized.
- Split the process in `blocks` endpoint into two distinct steps:
- First, retrieve all extrinsics in their raw, undecoded form during the "get block info" phase
- Then, create a separate function/step that takes these raw extrinsics as input and handles the decoding process
- Implement pagination: already mentioned in [#562](https://github.com/paritytech/substrate-api-sidecar/issues/562). If the block has more than a specific threshold of extrinsics it splits it into pages.

Regarding the `staking-info` endpoint:
- increased latency was reported
- Root Cause
- the latest breaking changes (related [PR](https://github.com/paritytech/substrate-api-sidecar/pull/1445)).
- to retrieve same information now requires a much more complex process/code.
- significant optimization potential with code refactoring.

## Sidecar Profiling Process
How/What to measure :
A. Measure Execution Time of Different Functions in Sidecar (mostly `blocks` endpoint)
B. Measure Memory usage (Chrome -> inspect)

### A. Measure Execution Time

#### How to Measure
Using the below repositories:
- [Forked Sidecar](https://github.com/Imod7/substrate-api-sidecar-with-measuring/pull/1) repo (branch `domi-add-execution-times`) with measuring statements (`performance.now`)
- [Script](https://github.com/Imod7/testing-sidecar/tree/main/sidecar-performance) that queries the above Forked Sidecar instance

#### Example Blocks to Query
- Polkadot
- [23527940-23527986](https://polkadot.subscan.io/block?page=9691&page_size=25)
- [23528139-23528171](https://polkadot.subscan.io/block?page=9686&page_size=25)
- [23528401-23528418](https://polkadot.subscan.io/block?page=9677&page_size=25)
- [23528461-23528626](https://polkadot.subscan.io/block?page=9668&page_size=25)
- [23528882-23528945](https://polkadot.subscan.io/block?page=9934&page_size=25)
- [23565143-23565307](https://polkadot.subscan.io/block?page=8205&page_size=25)
- [23565320-23565494](https://polkadot.subscan.io/block?page=8196&page_size=25)
- [23565511-23565694](https://polkadot.subscan.io/block?page=8190&page_size=25)

#### Measurement Results
- [Google Sheet - SidecarPerformance-ExecutionTimes](https://docs.google.com/spreadsheets/d/1aAA-eXnzM7WqoH0rlBLNlroGC4XG7a-L2xdAhu22k8Q/edit?usp=sharing)

### B. Measure Memory Usage (Heap)

#### How to Measure
- Clone the [Sidecar](https://github.com/paritytech/substrate-api-sidecar) repository locally
- Check out in the version you would like to test, e.g. `git checkout v19.3.1`
- Open `package.json` > add `inspect` in `main`

	"scripts": {
		"main": "node --inspect ./build/src/main.js",

- `yarn build`
- Run Sidecar with `NODE_ENV=polkadot yarn start` to test with the Polkadot chain
- Open the Chrome browser and put `chrome://inspect/#devices` in the address bar
- Port forwarding :
- 8080 : localhost:8080
- Configure
- localhost:9222
- localhost:9229
- localhost:8081
- Click on the link `Open dedicated DevTools for Node`
- It will open the `DevTools` window and in the console section you should see
- In the top menu click `Memory`
- Go back to `Chrome`, in another tab request this endpoint
`http://127.0.0.1:8080/blocks/?range=18705456-18705470`
- Then again back in `DevTools` window and click to get:
- a `Heap Snapshot` and wait until it is done
- a `Allocations on Timeline` and leave it to run as long as the request has not ended yet and then click `Stop`
- Save and analyse the results
- To be able to exit the running instance of Sidecar, first close the `DevTools` window and then `Ctr+C` in Sidecar

#### Measurement Results
- [Google Sheet - SidecarPerformance-ExecutionTimesPerBlockRange](https://docs.google.com/spreadsheets/d/1aAA-eXnzM7WqoH0rlBLNlroGC4XG7a-L2xdAhu22k8Q/edit?usp=sharing)
- [Google Sheet - SidecarPerformance-HeapAllocationPerVersion](https://docs.google.com/spreadsheets/d/1dBdPLDv2s3hRyNpSNnb2pNu1Nu-3jQVhncO68fxemOQ/edit?usp=sharing)

## Related Documents
- [Diffs between Sidecar versions](https://hackmd.io/@LwMsxe3-SFmNXxugAXOKgg/BkE7ly4IJe)
- Spammening [Referenda](https://kusama.subsquare.io/referenda/456)
- Subscan - [Daily Signed Extrinsic Chart for Nov24](https://polkadot.subscan.io/tools/charts?type=extrinsic)