In the process of research and analyzing the problems that we and our customers faced, I identified three groups of problems:

- [Problem with high, non-linear memory consumption during data update/writing process](#Problem-with-high-non-linear-memory-consumption-during-data-updatewriting-process)
- [Problem with high, non-linear memory consumption in the process of sending data from endpoints](#Problem-with-high-non-linear-memory-consumption-while-sending-data-from-endpoints)
- [Long key update process](#Long-process-of-updating-keys)

I suggest familiarizing yourself with each problem in order and exploring working prototypes that solve each problem.

### Problem with high, non-linear memory consumption during data update/writing process

### Problem

This problem is that during data processing, we process operators and operator keys in parallel. By describing the code in this way, we mean that we achieve the maximum possible performance. But it is not so: acting in this way, we really utilize all the resources of the network adapter, but at the same time we consume a very large amount of RAM. And what is most important - we do not control its consumption, and as a result of adding another SR Module - we will reach an even higher level of consumption. Also, because of maximum utilization and accumulation of requests on the stack, we can get into a situation when requests accumulating in the queue exceed the limit of the abort controller, and this will cause even greater memory consumption, because retries will be triggered.

Bottom line: the current solution is not scalable and the level of memory consumption is not controlled.

### Solution

As a solution, I have found a rather simple and easy-to-implement approach: refusing from parallel queries at the level of operator processing. Thus, we will make no more than X queries at a time, where X is the number of keys.

As a result of testing, the level of memory consumption fell by 10 times and became strictly predictable. The loading speed increased by 2 minutes, because we no longer go beyond the abort-controller in the process of queries, because each query has time to work within the allotted time.


I suggest to familiarize with PR. This is a prototype, please rate the idea, not the implementation/formatting of the code.

PR: https://github.com/lidofinance/lido-keys-api/pull/132

### Problem with high, non-linear memory consumption while sending data from endpoints

### Problem

In addition to the high write memory consumption, we have a similar problem with reads. This problem occurs when trying to request many keys by endpoint.

### Solution

The solution may be streaming/rendering in chunks. During testing, memory consumption went even beyond 4gb, causing reboots.

In the prototype, I used generators and implemented streaming by myself, since MikroORM does not have such functionality.

As a result, memory consumption is reduced by 7-10 times.

Also please pay attention to the StreamJSON library. This solution may seem controversial, I will be glad to your ideas, if you have any thoughts on how to do it differently.


PR: https://github.com/lidofinance/lido-keys-api/pull/136

### Long process of updating keys

### Problem

The cause of this problem was the lack of a method with batching in the contract, so we were requesting one key per request.

### Solution

Use a new method that will allow us to request many keys at once (depends on the node, 500 on average).

Current implementation makes loading 8 times more time-consuming. (17 times if not saved to the database)

### Questions
- keys are passed in the combined format: "key1key2". Parsing works by splitting strings by a fixed value. I have a question, is there validation at blockchain level for key length?

PR: https://github.com/lidofinance/lido-keys-api/pull/138