# Documentation for SARA RL Algorithm
## The SARA App and Goals

On the domain knowledge side, our goal is to use the SARA framework to create an app that helps undergraduates develop healthy habits for 3 possible intervention components: 1) self-monitoring, 2) sleep, 3) time management. The final decision will depend on user studies.

On the reinforcement learning side, our goal is to have an online testbed for algorithms that maintain engagement in self-monitoring in mobile health. As long as we enable after study analyses (essentially we use some sort of micro-randomization, say via posterior sampling/Thompson sampling), then we should strive to develop new algorithmic/statistical ideas and try them out in real life. Our primary goal is to develop algorithmic/statistical ideas that hold promise in advancing the field of digital health. To achieve this goal we gain from the practical experience of seeing what the data streams actually look like (e.g. levels of missingness, problems due to not being able to sense the user's entire state and so on), we gain from learning about how challenging it is to do online learning in a stable manner.

Our primary goal **does not** involve solving this one single RL problem with its unique set of actions and unique set of sensors/self-report data.

To this end, our current directions include:
1. Provide a way for domain scientists to design their own RL algorithm
2. Allow black box predictors to prove themselves and potentially become acceptable to the domain science. Recall that a Bayesian regression model (either logistic or linear) is simply a GP model with a very simple dot product kernel. Domain scientists are very familiar with these simple GP models (they don't call them GP models--they have other names for them) so the GP framework both provides the opportunity to do (1) yet allow for more flexibility, if the data demands this, as we know that since the direct sum of kernels is a kernel (so the combination of the simple dot product kernel with a kernel that is more expressive is possible).
3. As for (2) as we have discussed, we would like to think about a variety of predictions that might be used as features in the GP model.

An advantage of the embedding within the GP framework is the large coding community associated with GPytorch. We do not need to reinvent many standard algorithms.

## The RL Setup
There are $N$ users in the study. The longitudinal data for a single user $\mathbf{x}_i$ where $i = 1, ..., N$ can be recorded by the following sequence:
\begin{align}
\mathbf{x}_i = \{S_{i,1}, A_{i,1}, ...S_{i,k}, A_{i,k}, R_{i,k}, ... S_{i,T_i}, A_{i,T_i}, R_{i,T_i}\}
\end{align}
Where $k$ indexes the decision $k$th decision time and $\mathbf{x}_{i, k}$ refers to the $k$th decision for user $i$. Because users arrive and exit at an asynchronous manner (not all user trajectories will be the same length), $T_i$ is the final decision time for user $i$, such that $k \in 1, ..., T_i$.
The problem will be use context and be formualted either as a bandit or as something in between a bandit and an MDP.
>MDP vs bandit depending on whether Sarah has early ideas about prediction distal rewards

>Possibly pooling across users depending on whether Eura has early ideas.

### Action Space
There will likely be one decision point per user per day. At a decision point, the RL algorithm will decide whether or not we send a notification for the user to self monitor.
> We might have 2 per day for the self-monitoring intervention component.
>
* When should the decision points be? Constant or varying times?
> [Susan] See [google doc](https://docs.google.com/document/d/1U8F91D1Mjw7_N2xguy2lfOpEk2w2_Ilxwb_p7RDY9-s/edit).
* *Should we do clipping to constrain the number of messages [Kelly]?*
* Would facilitate after-study analysis
* Entropy--clipping but "in a softer way", encourage probability towards 0.5
* Alternatives to discuss with Sonali (week of 9/28)
* Mutual information regularization
* Information bottlenekcks

Although the notification may take different forms from a behavioral perspective (positive, encouragement, statistic-based, personalized), from an RL perspective, we will only consider whether or not we have sent a notification at all. As a result, our action space is *binary*, where $A_{i,k} \in \{0,1\}$ for not sending or sending a notification respectively.

### Reward
Following a state and action pair, the reward will indicate whether or not the user engaged in self-monitoring that evening (or within 24 hours). As a result, the reward is also binary, where $R_{i,k} \in \{0, 1\}$ for whether the user did or did not complete the survey that evening.

> Participant may make a commitment intervention at baseline. For example request a committment to a certain number of times per week(set a goal)? We can explain, for example that if they do the self-monitoring x times per week then we can provide better life insights.

### State Space
We are interested in features that are indicative of whether or not a notification will prompt the user to self-engage. Some of the features we have considered so far:
**Contextual**
* Time of day for user
* Weekend or weekday?
* Type of phone, year of phone
* Differences between iOS/Android quality of data

**Past interaction based**
* App usage (distinguished between self-prompted and notification-prompted)
* Past responsivity in the last $n$ days
* Dosage/past interventions (how much and how)
* Notification feedback (thumbs up, thumb down, none)
* Prediction of impending disengagement
* Answers to prior self-monitoring questions
* We will select some questions that are more indicative of goal success
* Features from prediction models (RNN? CNN?) trained to predict:
* Impending disengagement
* Variability of engagement times

**Availability**
* We will initially assume it is ok to provide a treatment at any of that intervention component’s decision times.

## Consderations on Modeling the Reward
The reward function $r(s, a)$ in our model will ultimately let us determine whether or not to send a notification. At decision time, we will notify the user with a probability $P(a=1)$.
\begin{align}
P(a=1) = P(r(s, a=1) - r(s, a=0) > 0)
\end{align}

We will try to pool data unless Kelly needs it for a preliminary study for a grant.
### Should we consider much simpler model?
#### Beta Binomial
$R(s, a) \sim Bernoulli(p)$ with prior $p\sim Beta$

`1) Start R(s,a)=Bernoulli(p) with Beta prior (same p for all states and actions)`
`For k in 1...T:`
` 2) Update p as we observe more state-action pairs`
` 3) At certain intervals, adaptively bin so that state-action pairs with similar p are binned together`

Pros:
* Simple to code

Cons:
* Have to discretize states (binning)
* High noise

#### Bayesian Logistic Regression
To find the joint probability $p(y, \mathbf{x})$, we use Bayes' theorem:
\begin{align}
p(y, \mathbf{x}) = p(\mathbf{x})p(y|\mathbf{x})
\end{align}

Where $p(y|\mathbf{x})$ can be modeled as a probability over all possible classes:
\begin{align}
p(C_1|\mathbf{x}) = \lambda(\mathbf{x}^\mathrm{T}\mathbf{w})\textrm{, where }\lambda(z)=\frac{1}{1 + \textrm{exp}(-z)}
\end{align}

*How do we choose a prior?*

Pros:

Cons:

#### Random Forest
#### Bayesian Deep Learning
#### Decision Tree
* See Finale's work
* Can write as GP
* Not very smooth
#### Mixture of Experts
* "Experts"=different kernels
* See Sonali/Finale's work

### Gaussian Processes
We propose using a Gaussian Process regression model for the reward function.

\begin{align}
r(s, a) = f(x)
\end{align}

where $x = (s, a)$.

**Basic Model:Weight-Space View**
\begin{align}
s \rightarrow \phi(s) \rightarrow \textrm{linear regression}
\end{align}
* Bayesian linear regression with a kerney trick
* Kernel $\phi(s)$: projection of s to some higher feature space
* Domain knowledge is used to build kernel

**Basic Model:Function-Space View**
\begin{align}
f(x) \sim GP(m(x), k(x,x'))
\end{align}
* GP is a distribution over functions
* Kernel k(s,s') measures "similarity" between tuples
* Relationship to weight-space $\phi(s)$:
$k(s,s')=\phi(s)^T\sum_0\phi(s')$

### Questions
* *How often should we perform a posterior update? How do we appoach this in a principled way?*
* This will depend on the timescale of the intervention
* What assumptions should we make about nonstationarity?
* Assuming stationarity, can update as often as you want
* The data will be noisy so once a day may be too frequent (if we only have one decision point per day per user).
> Susan: Bigger issue is that how does one think about this from a principled point of view? update times are akin to hyperparameters. Need to consult with Anna Li.

**Additive Model (potential plan for long term)**
\begin{align}
f(x) \sim f^{(1)}(x)+f^{(2)}(x) + ... + f^{(D)}(x)
\end{align}

where $f^{(d)}(x) \sim GP(m^{(d)}(x), k^{(d)}(x, x'))$.

To help with interpretability, we may consider additive GP models as in [Chang 2019](https://www.nature.com/articles/s41467-019-09785-8) and [Dvenaud 2011](https://arxiv.org/abs/1112.4394). Instead of specifying one GP for the reward function, we can specify multiple GPs. Each GP may correspond to a different grouping of our features by meaning (clinician determined features vs. additional features) or type (numerical features vs. time-based features). This formulation lets us observe the *importance* of different groupings in determining the reward (e.g. do certain features make more difference in the reward than clinician determined ones?). It also allows us to specify different kernels per grouping as necessary (e.g. we could use a periodic kernel for a time-based feature, like week, vs. a standard RBF for numeric feature, like temperature outside).

**Semi-parametric Model (potential plan for long term)**
\begin{align}
f(x) \sim GP(m(x), k(x,x')) + (A - \pi)h(x)^T\beta
\end{align}

This formulation lets us consider random effects and action centering (the kernel is then the direct sum of a dot product kernel and a more complex kernel, like the squared exponential)

## Modular Algorithm Implementation
The number of modules is not fixed and is open for discussion. We can change the modules to allow for best software practices and the most intuitive compartmentalization.

### Modules

| Module | How often | Where | For who |
| -------- | -------- | -------- | -------- |
| 1: Action selection at decision time (poterior sampling) | Every decision point | On the phone or cloud | Every user individually |
| 2: Updating the reward function (calculating the posterior) | ? Needs discussion | In the cloud | Pooled or individual? |
| 3: Hyperparameter update | ? Needs discussion | In the cloud | Everyone at once|