# Local DM creation and room joins on Matrix.org

DM creation seems to take consistently ~1-2+ seconds using Element web client to send the request with both users on Matrix.org

Looking at the Jaeger traces of requests to `/matrix/client/r0/createRoom` the processing time for just that endpoint appears to be about 1-2+ seconds. Here are some example times for this request: 1.17s, 2.9s, 1.19s, 1.08s (note that the "shorter" durations were tracked later in my work day, meaning after european business hours)

Digging deeper into the Jaeger traces, within the `createRoom` requests outgoing replication requests to the event persister, including persisting the event-appear to be the culprit for the slowness and are taking up significant chunks of the overall time (example: 575.66ms out of 2.9s), and happen several times within the overall request

The second longest call, which tends to be much shorter but is noticible, is `db._get_state_groups_from_groups`

I did notice that request processing times trend higher when there is higher traffic on Matrix.org (i.e. european business hours), which makes sense.

One suggestion that has been made (by Sean) for ameliorating this is to batch up these calls to the event persister rather than making several calls throughout the processing of the request.

----
# Local room joins

Local room joins apprear to be taking about 1-20s (20s being the longest join I saw). I used Jaeger and a test account that was always traced to compile these observations.

The top offenders for slowness in local room joins are:

- `bulk_get_push_rules` and `bulk_get_push_rules_enabled`
- `_get_joined_profiles_from_event_ids`
- `_get_state_groups_from_groups`
- runner-up: `get_subset_users_in_room_with_profiles`

Suggestions for speeding up these calls:

- calculating push rules: I am unsure if there is any way around this-looking at Grafana the cache hit rate for these calls don't seem terrible (i.e. they're not zero and often range up to 75-100%) so I am not sure that there's an immediate win there. I wish there was some way to filter calulating the push actions since I suspect not many people would have rules to be notified of room joins and thus there's a lot of wasted calculation, but I can't see any immediate way to do that.

- `_get_joined_profiles_from_event_ids`: This one is interesting. This function is called deep in a call chain originating with `cache_joined_hosts_for_event`, which is called when the event is persisted. It appears to be called as part of figuring out what servers are in the room. The cache hit ratio on this function is often zero and it is high on the list of top ten cache misses so I am wonering if there is some cache tuning that might be done to make these cache hit rate get better and thus the overall call go faster. Another mystery about this function call is that it is called many times in the course of one trace but originates from persisting the original join event, which should only happen once? I haven't dug into this, just noting it here.

- `_get_state_groups_from_groups`: Unsure how to speed this one up, although I did see a comment about missing indexes (something about de-duping state??) so getting to the bottom of that is an obvious first place to start.

I am also seeing weird mystery pauses, I have some screenshots here but these are not all of them by far. Often on the joins that took the longest you would see at least one of these, with the pauses often extending up to 6 or more seconds (the longest unexplained pause I saw was 13s). Weirdly a larger-than-random portion seem to occur on `get_local_users_in_room` but I have seen them in a large number of other calls as well. I checked the reactor tick times on Grafana for the `event_creator_users` workers and 99.9% were finishing in under ~400ms. Out of curiosity I turned it up to see what the time was for 100% of ticks and it was 5s, which I thought was notable but I am not sure how important it is. I do think something is going on here that it would be beneficial to get to the bottom of, I think adding more tracing would be the next logical step.

- get_auth_chain_difference_chain
looks like in this case we are recusively fetching event redactions for the events fetched for state res. These events are only being used in state res- seems like fetching the reaction events for them is unnecessary and it takes a very long time (seconds)

## DMR driveby


This is Shay's test account trying to join synapse-dev. It seems to spend ~430ms awaiting full state.
But Why: we haven't turned on faster joins; all rooms are full state as far as matrix.org is concerned.

I wonder if this call [here](https://github.com/matrix-org/synapse/blob/42b11d5565ed026c7d71f433c69e7b7007f45918/synapse/state/__init__.py#L209) should check if the room is considered partial-stated using [this cached function](https://github.com/matrix-org/synapse/blob/9d2823ab70e69a90f4a54256bd32cd85eab130d9/synapse/storage/databases/main/room.py#L1204) before asking if each individual event is partial stated.

I was surprised that the is_partial_state_room cache is so small and evicted from rapidly. https://grafana.matrix.org/d/000000012/synapse?orgId=1&var-datasource=default&var-bucket_size=$__auto_interval_bucket_size&var-instance=matrix.org&var-job=synapse_event_creator_users&var-index=4&from=now-2d&to=now&viewPanel=8

The is_partial_state_event cache seems like it would benefit from being bigger: https://grafana.matrix.org/d/000000012/synapse?orgId=1&var-datasource=default&var-bucket_size=$__auto_interval_bucket_size&var-instance=matrix.org&var-job=synapse_event_creator_users&var-index=4&from=now-7d&to=now&viewPanel=8

----

get_forgotten_rooms_for_user seemed like a slow query from the trace (100ms). Wait, why do we need this to join a single room? Looks like it's from https://github.com/matrix-org/synapse/blob/84ddcd7bbfe4100101741a408a91f283a8f742c7/synapse/handlers/room_member.py#L873

That calls https://github.com/matrix-org/synapse/blob/42b11d5565ed026c7d71f433c69e7b7007f45918/synapse/storage/databases/main/roommember.py#L444-L460 which seems like overkill. It ends up looking for all possible rooms when we only want to know about one. Why!?!

Why can't we look up the user's membership in the current state of that room to see if they're invited? (Out-of-band memberships?)

----