# UNTAP-1581, 1697
Date: Feb 8, 2022

Related to: Redis Caching

## Caching issue in survey module

The requirement of cache implementation:

1. Able to share redis cache between GET all response and GET particular response of role or submittedBy query.
2. If response is added/updated/deleted, we should only update single response in cache instead of invalidating the all responses of that particular survey.
3. All responses of particular survey must have same [TTL].(https://redis.io/commands/TTL) to avoid partial data of survey.
4. Don't make use of [KEYS](https://redis.io/commands/KEYS) to get cached value.

### 1. Partial survey responses data:

if `GET survey-response` by `survey-id` API is made
1. It check whether the cache exist or not by checking whether redis keys contain `survey_id`,
2. If at least one key exist, then it return value from cache
3. if not it query to get all survey-response and store in cache in following key pattern

survey-{survery-id}-res-Practitioner-{submitterId} 
survey-{survery-id}-res-Sponsor-{submitterId} 
survey-{survery-id}-res-Subject-{submitterId} 

As we can see from steps 2 it just check whether atleast one key exist which means if cache contain partial survey-response only it will return from cache.

### 2. TTL value not in sync after POST and PATCH request
if `PATCH survey-response API` is made
1. It first update the response in db then
2. It cache the updated response in cache. If cache already exist, then it overwrite the previous value, if not it create new cache.

Let's say we have all survey response in cache and all cache have
same TTL value e.g 1hr from T1 time


If PATCH request is made, it will update the cache with TTL value
1hrs from T2 time which means that new updated cache will have more
remaining time to live in cache then previous cache.


so in some point of time only new update cache will available in
cache others cache we be deleted. This will also result partial data

Similar issue with `POST survey-response API` as well

### Possible solution:
1. #### Make all role cache TTL value in sync:

This basically means either have all role cache or have none.However this solution
is hard to get right to sync the TTL value because if one response is updated then updated response cache will have different TTL value then previous responses which lead ot partial data in some point of time.
2. #### Avoid the partial survey-response cache:
Currently if we made the `GET single survey response` request with
queryParam is made then

1. First it check the cache if exist return from cache
2. If does not exist it will query from db and store single
survey-response to cache.(Now store will contain partial survey response)

To avoid the partial survey-response cache issue, if `single survey
response API` is made with `queryParam` we skip the Step.2 *(which means
dont store in cache if not exist)*.

With current implementataion we are using `KEY` command to get all
available keys from redis and filter out the key that is related to
requested `survey-id`. However using `KEY` command in production env
where there are thousands of key will might lead to `performance
bottleneck`. It is recommended not to use `KEY` command in
production.

Performance safer alternative to `KEY` in `SCAN` however it's just
a cursor it may not return all entries or return some entries multiple
times, so must be used carefully as well.
Reference : https://redis.io/commands/scan

3. #### Save all survey responses in single key instead of saving each survey response in its own key.


Pros:
1. We don't have to use `KEYS` command to get the particular survey responses cache
2. All survey responses will have same `TTL` value so no issue with partial data.

Cons:
1. If we want to get survey response of particular role, then we cannot simply get from cache. we have to perform extra steps like first get all responses related to survey and filter out from that.
2. If case of `POST, PATCH` request, we have first get all response related to survey and perform add or update operation and again set to cache. However we can make use of [LIST](https://redis.io/commands#list) data type of redis which makes [ADD/UPDATE/DELETE](https://) operation easier

**Conclusion:**
> Able to share redis cache between GET all response and GET particular response of role or submittedBy query.

This requirement makes the current redis implementation little complicated because
1. To avoid partial data and `TTL` in sync we must store all survey responses in single keys
2. To able to get particular `role` or `submittedBy` from cache we must store each survey response as individual cache with its own key.

#### Reason not to use KEYS in production env.
The time complexity of **KEYS** command in O(N) where N refer to total number of keys in the redis store.
And even worse, since only one command can run at the same time (Redis not being multi-threaded), everything else will have to wait for that KEYS to complete.

Reference:
https://redis.io/commands/KEYS
https://aws.amazon.com/caching/best-practices/
https://redis.com/blog/5-key-takeaways-for-developing-with-redis/