As someone who is implementing sth like this for the first time, 3 differnt approaches pops up, where none of them is optimal:
Indexer
Running a node
Scraping the mintscan website for data
Indexer
The V4 Indexer is a set of microservices which provides a set of rich APIs and websocket channels for a dYdX V4 application. It is the easiest way to query trader profile, and there are two ways to do it. Either by running a local v4 node or connecting to a remote v4 node. Unfortunatly, the instructions on running a local v4 node is an outdated link on dydx github page https://github.com/dydxprotocol/v4#running-the-chain-locally, which forces us to connect to a remote node to query the data. Additionally, to test our implementation, we can first use the https://indexer.dydx.trade/v4/perpetualPositions endpoint to query the data that we intend to get using the Indexer v4. Unfortunatly, this approach can't be used for the main product since there is a rate limit. Therefore one should set up an Indexer follwoing the link https://github.com/dydxprotocol/v4-chain/tree/main/indexer. Before discussing the possible implementation, here is the API calls that are available through the Indexer https://indexer.dydx.trade/docs/. Indexer also gives us access to off-chain data which we don't actually need currently, but can be used to create a heat map since limit orders are not stored on chain. Indexer does not give us an API that would help us to detect new events therefore using indexer will cause a big overhead that is proportioanl to number of active traders. Since we don't have an API for new events, we need to find a realtivly efficient algorithm to update the profiles with not that big of an overhead. One can initially query all the data for addresses and store the postiions as well as bids, the bids for each pair are stored in a list where we sort them based on prices. After that we have pairs many lists. Since, as mentioned, we don't have access to new events, what we do is we track the price of each pair and iterate over each list based on the price (in this case lists with the iterator serve as an outdated orderbook). We then set a integer threshold $k$ which will simply serve as an indicator. Whenever price movement causes an iteration over $k$ many orders or $\Delta t$ time has passed since the last refresh, we query each address one more time to refresh the data starting from the once with the most orders. Of course this is a very suboptimal approach, but the best using the indexer. PSA: A code tat can be used to set up the profiles https://gist.github.com/kingbaldwin-iv/7e7ff68081f75dae454e7cb76a119d0b. (Setting up the profiles can be done very easily using the indexer, the main issue is processing new transactions.)
Node/Validator