Analysis document

--- tags: covid --- # Analysis document ## Resources * [Github: COVID19 P2P Risk Prediction Model & Dataset](https://github.com/mila-iqia/covid_p2p_risk_prediction) * [Github: Simulator for COVID-19 spread](https://github.com/pg2455/covid_p2p_simulation) * [GSheet: Covid19 task force tracker](https://docs.google.com/spreadsheets/d/11t1T66AAVeR6P341nZYP1qwLdvhCkU_EwFwUkyLziLQ/edit?pli=1#gid=214477555) * [GDoc: Covid p2p Simulation top level doc](https://docs.google.com/document/d/1jn8dOXgmVRX62Ux-jBSuReayATrzrd5XZS2LJuQ2hLs/edit#heading=h.1rukwntwnqx) * [GDoc: 2020/05/08 Good Coding Practices](https://docs.google.com/document/d/1XXMp0un8LmffKMkjAHTzrAGKuBh_ZaXEfFnIsITfUdk/edit) ~ pure short functions and tests ## Code [Github: Simulator for COVID-19 spread](https://github.com/pg2455/covid_p2p_simulation) holds the code for the simulator. It has 3 main branches: * **master**: stable, optimized code. In general, don't touch that, unless it's for incredibly important (and very small) bug fixes. * **develop**: stable, currently being reviewed and optimized. In general, only contribute polished features that are designed in collaboration with others, well coded, reviewed, and tested. This branch will receive engineering updates from Satya/Olexa. * **experimental**: where the daily coding happens. Write `unittests`. :::warning When exploring simulations and features, you should checkout **`experimental`** ::: :::info ### Workflow   1. fork https://github.com/pg2455/covid_p2p_simulation 2. `git clone` your fork 3. add the base repo as a remote source `git remote add pg2455 https://github.com/pg2455/covid_p2p_simulation.git` 4. Update your branch to match `pg2455`'s a. fetch the branch `git fetch pg2455 experimental` b. check it out `git checkout pg2455/experimental` c. create a branch on your own fork from it `git switch -c <YOU>-experimental` 5. Do some work, commit, **create a PR** to pg2455/covid_p2p_simulation/experimental 6. Pull updates `git pull pg2455 experimental`   ::: Install requirements in `requirements.txt` in a virtual environment and install the `covid19` package locally with `pip install -e .` to play around with the simulation. ## Config * `src/covid19/configs/config.py` => epidemiology-sourced constants and simulator parameters, shouldn't change often and not for simulation results exploration * `src/covid19/configs/*.yaml` => simulation-specific parameters ```python # YML PARAMETERS P_HAS_APP: 1.0 # Probability that a human has the COVI app # ----------------------------------- ABSOLUTE_P_HAS_APP: True # Enforce the use P_HAS_APP *as is* in the simulation. # If False, P_HAS_APP is modulated with people's carefulness and age (see config.py) # ----------------------------------- INTERVENTION_DAY = 10 # day of the intervention (people get the app) # If < 0, no intervention # ----------------------------------- INTERVENTION = "Tracing" # ?? or? # ----------------------------------- PERCENT_FOLLOW = 1.0 # ?? play around or ignore? # ----------------------------------- RISK_MODEL = "transformer" # "naive" "manual", "digital", "transformer" | ?? description of each # naive: probabilistic contact-based model (cf Tristan) # manual: human contact tracers # digital: ?? # ----------------------------------- TRACE_SYMPTOMS = False # # ----------------------------------- TRACE_RISK_UPDATE = False # ----------------------------------- TRACING_ORDER = 1 # ----------------------------------- USE_INFERENCE_SERVER = True # ----------------------------------- TRANSFORMER_EXP_PATH = "./CTT-SHIPMENT-0/" # Path to the Transformer's experiment path. # Folder should contain Configuration/ Logs/ Plots and Weights subfolders # Train your own model or ask Prateek to get this. # ----------------------------------- ``` ## Output Parsing the prints of `run_simu()` ``` 2020-02-28 00:00:00 Ro: -1.00 S:494 E:6 I:0 T:6 P3: 6.00 RiskP:0.00 M:500.00 EM: 0.99 G:0 B:0 O:0 R:0 2020-02-28 00:00:00 => current day timestamp Ro: -1.00 => R0 for that day S:494 => number of symptomatic people E:6 => number of exposed people I:0 => number of infected people T:6 => number of ** P3: 6.00 => *** RiskP:0.00 => Average risk in the population M:500.00 => number of *** EM: 0.99 => *** G:0 => *** B:0 => *** O:0 => *** R:0 => *** ``` ## Commands ``` # Run a simulation # Outputs: ? python -m covid19sim.run sim --n_people 1000 --init_percent_sick 0.1 # Run a tune (?) # Outputs: ? python -m covid19sim.run tune --n_people 500 # Run the Transformer's inference server # Train a Transformer or ask ?? to get them python server_bootstrap.py -e CTT-SHIPMENT-0 -v 1 --mp-thread=1 -w 1 ``` :::warning `python server_bootstrap.py` Requires a folder (-e XXX, `./CTT-SHIPMENT-0` in this case) to hold the Transformer's weights ::: ## Tracker Docs Extract from `run.py/tune()` ### Main fields The following describes fields that have a simple type/structure (mainly, non-`dict` fields) ```python data['intervention_day'] = config.INTERVENTION_DAY # (int) day where the intervention occurs. # If < 0, no intervention. # ----------------------------------- data['intervention'] = config.INTERVENTION # (str) type of intervention that happened. # Can be one of ["Lockdown", "WearMask", "SocialDistancing", # "Quarantine", "Tracing", "WashHands", "Stand2M", "StayHome", "GetTested"] # ----------------------------------- data['expected_mobility'] = tracker.expected_mobility # (float) average of (1-human.risk) over all humans in the city. # See compute_mobility() # ----------------------------------- data['mobility'] = tracker.mobility # (float) weighted average of humans' mobility with a weight of: # - 1 for people with recommendation level (rec_level) of 0 # - 0.8 for people with rec_level of 1 # - 0.2 for people with rec_level of 2 # - 0.05 for people with rec_level of 3 # ----------------------------------- data['n_init_infected'] = tracker.n_infected_init # (int) Total number of infected humans at the start of the simulation: # sum([h.is_exposed for h in self.city.humans]) # /!\ Humans are created with a related quantity: init_percent_sick # where Human is created sick with pbty init_percent_sick # ----------------------------------- data['contacts'] = dict(tracker.contacts) # see dedicated doc # ----------------------------------- data['cases_per_day'] = tracker.cases_per_day # (list(int)) number of cases per day in the simulation # ----------------------------------- data['ei_per_day'] = tracker.ei_per_day # (list(int)) number of humans that are either # exposed or infections, per day # ----------------------------------- data['r_0'] = tracker.r_0 # (dict: str -> dict: {"infection_count" -> int, "humans" -> set}) # for each key in ["asymptomatic", "presymptomatic"], # store the number of infections and the names of humans transmitting it. # ----------------------------------- data['R'] = tracker.r # (list(float)) r per day, computed with tracker.get_R() # ?? add more # ----------------------------------- data['n_humans'] = tracker.n_humans # (int) number of humans in the city # ----------------------------------- data['s'] = tracker.s_per_day # (list(int)) total number of susceptible humans, per day # ----------------------------------- data['e'] = tracker.e_per_day # (list(int)) total number of exposed humans, per day # ----------------------------------- data['i'] = tracker.i_per_day # (list(int)) total number of infectious humans, per day # ----------------------------------- data['r'] = tracker.r_per_day # (list(int)) total number of removed humans, per day. # A human is removed if dead or immune. # Immunity is a parameter in the simulation # Death is a probability per age group. See P_NEVER_RECOVERS # ----------------------------------- data['avg_infectiousness_per_day'] = tracker.avg_infectiousness_per_day # list(float) Average infectiousness over all humans in the city, per day # mean([h.infectiousness for h in self.city.humans]) # ----------------------------------- data['risk_precision_global'] = tracker.compute_risk_precision(False) # see dedicated doc (TODO) # ----------------------------------- data['risk_precision'] = tracker.risk_precision_daily # list(tuple(float)) # For each day, what are the precision, lift and recall, # as per compute_risk_precision(True) # ----------------------------------- data['human_monitor'] = tracker.human_monitor # see dedicated doc # ----------------------------------- data['infection_monitor'] = tracker.infection_monitor # (list(tuple(Human | None, Human, Timestamp))) # Store all the infections from a human to another, # with their timestamp of occurance. # If a human is infected by being somewhere (the location infects them), # then the first Human in the tuple is None # ----------------------------------- data['dist_encounters'] = dict(tracker.dist_encounters) # (dict: int -> int) # maps a binned/bucketed distance (50 bins) to the number of encounters # which happened at that distance. # Last bin is INFECTION_RADIUS / 50 # ----------------------------------- data['time_encounters'] = dict(tracker.time_encounters) # (dict: int -> int) # maps a binned/bucketed time (5 15min bins) to the number of encounters # which happened at that distance. # Last bin is 4 = 60 / 15 for encounters longer than 1 hour # ----------------------------------- data['day_encounters'] = dict(tracker.day_encounters) # (dict: day -> [n, avg, count]) # ?? don't understand logic here # ----------------------------------- data['hour_encounters'] = dict(tracker.hour_encounters) # (dict: day -> [n, avg, count]) # ?? don't understand logic here # ----------------------------------- data['daily_age_group_encounters'] = dict(tracker.daily_age_group_encounters) # (dict: day -> [n, avg, count]) # ?? don't understand logic here # ----------------------------------- data['age_distribution'] = tracker.age_distribution # (list(int)) the age of all humans in the city # ----------------------------------- data['sex_distribution'] = tracker.sex_distribution # list(str) the sex/gender of all humans in the city. # sex ~ ["male", "female", "other"] # ----------------------------------- data['house_size'] = tracker.house_size # list(int) the list of number of residents for all households in the city # ----------------------------------- data['house_age'] = tracker.house_age # list(float) the average age per household # ----------------------------------- data['symptoms'] = dict(tracker.symptoms) # see dedicated doc # ----------------------------------- data['transition_probability'] = dict(tracker.transition_probability) # (dict(dict(dict(dict(int))))) # For each hour in the day, each age group, store the number of # transitions from one location to another # transition_probability[hour][bin][from_location][to_location] = n # ----------------------------------- ``` ### Specific fields #### contacts ```python data['contacts'] = dict(tracker.contacts) tracker.contacts = { "all_encounters":np.zeros((150,150)), # (np.array(150, 150)) # represents count of encounters between ages # self.contacts["all_encounters"][human1.age, human2.age] += 1 # ----------------------------------- "location_all_encounters": defaultdict(lambda: np.zeros((150,150))), # (dict: location_type -> np.array(150, 150)) # for each location, count per-age encounters # ----------------------------------- "human_infection": np.zeros((150,150)), # (np.array(150, 150)) # represents the total count of human -> human infections between ages # contacts["human_infection"][from_human.age, to_human.age] = n # ----------------------------------- "env_infection":get_nested_dict(1), # (dict: age-bin -> count) # Store the count of environmental infections per age bin # contacts["env_infection"][to_bin] = n # ----------------------------------- "location_env_infection": get_nested_dict(2), # (dict: location_type -> dict: age-bin -> count) # For each location type, count of environmental infections per age bin # contacts["location_env_infection"][location.location_type][to_bin] = n # ----------------------------------- "location_human_infection": defaultdict(lambda: np.zeros((150,150))), # (dict: location_type -> np.array(150, 150)) # As human_infection, per location_type # ----------------------------------- "duration": {"avg": (0, np.zeros((150,150))), "total": np.zeros((150,150)), "n": np.zeros((150,150))}, # (dict: str -> tuple | np.array(150, 150)) # "avg" -> (count, average_duration = total_duration / count) # "total" -> np.array(150, 150): total duration for encounter between age1 and age2 # "n" -> number of encounters between age1 and age2 # ----------------------------------- "histogram_duration": [0], # (list(int)) # for each bucketed duration (0:6 = 0min:60+min), # count the number of encounters of that duration # contacts["histogram_duration"][bin] = n # ----------------------------------- "location_duration":defaultdict(lambda : [0]), # (dict: location_type -> list(int)) # for each type of location, count the number of encounters for a given binned duration # (as histogram_duration) # contacts['location_duration'][location.location_type][bin] = n # ----------------------------------- "n_contacts": {"avg": (0, np.zeros((150,150))), "total": np.zeros((150,150))} # (dict: str -> tuple | np.array(150, 150)) # "avg": (count, average number of contacts per age pair # "total": total number of contacts per age pair # ----------------------------------- } ``` #### risk_precision_global ```python data['risk_precision_global'] = tracker.compute_risk_precision(False) # (tuple(list(list(float) | float))) # tuple[0]: top-k precision for data-type in [all, no_test, no_test_symptoms] and top_k in [0.01, 0.03, 0.05, 0.10] # e.g. data['risk_precision_global'][0][type_index][top_k_index] # tuple[1]: lift (idem) # tuple[2]: recall for each data-type # ----------------------------------- ``` #### human_monitor ```python data['human_monitor'] = tracker.human_monitor # (dict: day -> list(row)) # for each day, associate a list of rows, 1 row per human: row = { "infection_timestamp": h.infection_timestamp, # (datetime.datetime) when the infection occured # ----------------------------------- "n_infectious_contacts": h.n_infectious_contacts, # (int) number of other Humans one has infected # ----------------------------------- "risk": h.risk, # (float) h's risk of being infected, see compute_risk() # ----------------------------------- "risk_level": h.risk_level, # (int(0:16)) bucketed risk according to RISK_MAPPING # ----------------------------------- "rec_level": h.rec_level, # (int(0:4)) recommendation level according to rec_level. # See get_recommendations_level() # ----------------------------------- "state": h.state.index(1), # (int) finds the first True value in h.state: # h.state ~ [h.is_susceptible, h.is_exposed, h.is_infectious, h.is_removed] # row.state: # 0 -> h is susceptible # 1 -> h is exposed # 2 -> h is infectious # 3 -> h is removed # ----------------------------------------------- "test_result": h.test_result, # (str | None) whether h has been tested + result: # test_result ~ ["positive", "negative", None] # ----------------------------------------------- "n_symptoms": len(h.symptoms) # (int) total number of symptoms which may come from: # flu_symptoms, cold_symptoms, allergy_symptoms, covid_symptoms # see update_symptoms() # ----------------------------------------------- } ``` #### symptoms ```python data['symptoms'] = dict(tracker.symptoms) # (dict: "covid" -> dict, "all" -> dict) tracker.symptoms = { "covid": {symptom: count, "n": total} # (str -> int) for each covid-specific symptom, # the number of humans who have them # + extra field "n" for total count of symptoms (??) # ----------------------------------------------- "all": {symptom: count} # (str -> int) for each possible symptom, # the number of humans who have them (?? => covid.n > all.n) # ----------------------------------------------- } ``` ## Risk to recommendation-level mapping (WIP Prateek + Victor) :::danger ### Red * have been tested + * have symptoms * no symptoms but infectious * not infectious but infected * not infected but one-hop away from those who have been tested positive ::: :::warning ### Orange People who are not infected but they are one-hop (have had a contact) away from * People who are infectious * People who have no symptoms but are exposed ::: :::info ### Careful People who are not infected but one-hop away from exposed person ::: :::success ### OK Rest of the population ::: early metric: * early warning: how much time after exposure put in non-green bucket * false-positives = quarantine for nothing

Syntax	Example	Reference
# Header	Header	基本排版
- Unordered List	Unordered List
1. Ordered List	Ordered List
- [ ] Todo List	Todo List
> Blockquote	Blockquote
Bold font	Bold font
Italics font	Italics font
~~Strikethrough~~	~~Strikethrough~~
19^th^	19^th
H~2~O	H₂O
++Inserted text++	Inserted text
==Marked text==	Marked text
[link text](https:// "title")	Link
![image alt](https:// "title")	Image
`Code`	`Code`	在筆記中貼入程式碼
```javascript var i = 0; ```	`var i = 0;`
:smile:		Emoji list
{%youtube youtube_id %}	Externals
$L^aT_eX$	L^aT_eX
:::info This is a alert area. :::	This is a alert area.