#### What is the problem/reward signal/cost function  --- #### What is the problem/reward signal/cost function  --- #### What is the problem/reward signal/cost function   --- #### What is the problem/reward signal/cost function with optimal control...differentiate trough the environment==policy grandient.  some ref: https://michaelrzhang.github.io/model-based-rl <!-- .element: style="font-size: 10px;" --> ---   --- #### The optimal transport reprogramming data current results are ongoing...  [Schiebinger, Geoffrey, et al. "Optimal-transport analysis of single-cell gene expression identifies developmental trajectories in reprogramming." Cell 176.4 (2019): 928-943.](https://reader.elsevier.com/reader/sd/pii/S009286741930039X?token=CADB22D39F63F7E3D3F505F1D76D096E5DEFD0E16BDA93D3EB225C5FA46BD541A24CD091BDEB877D92918250A8861027&originRegion=us-east-1&originCreation=20220420200023) <!-- .element: style="font-size: 10px;" --> ---  #### What we needed to build to get to here... - build a simulator (fortunatly we sit on the shoulders of giants) - to be differentiable (for policy fradient) - use an oracle for the reward (a neural network classfier) - to be fast during simulation (very important!) - gym wrapping (to use already built RL policies) - bioinformatics (eh) <!-- .element: style="font-size: 23px;" --> Note: all of those makes a huge forest. none of the path is well known. time for sitting down and meditate. --- #### The Simulator  Note: we have a very good idea of how the system behaves in response to actions. do we? --- #### The input needed to do control  --- The underlying dynamics:  - A fixed GRN how genes interact with each other - Production rate $P_i$ is a non linear function of the previous state. - Decay rate - Noise (stochastic Wiener processes) ---  - We control the master regulator at t=0, our decision variable is the initial concetration $x_0$ = a, (actually a subset) - Loss function is on the last timestep $\ell(x_T, y)$ - Loss function is the loss of a classifier, the target state is not known and not even unique. <!-- .element: style="font-size: 23px;" --> --- ## Optimal Control approach - Single shooting - Jax Simulator --- ## Reinforcement Learning - Requires no gradient throught the environment. - $\gamma=0$, bandit problem - Will be extended to control intermediate steps --- #### Examples of control in our daily life: - the morning coffe and sometimes the evening coffe - sigaretes - alcohol - pills - ...for more [wiki Bioethics](https://en.wikipedia.org/wiki/Bioethics#:~:text=Bioethics%20is%20the%20study%20of,as%20environment%20and%20well%2Dbeing.) --- #### Examples of control in science **Yasmeen Hitti** uses **lights** to control the plant grow shape and directionality. **Micheal Levin** uses **bioelectricity** to change body parts of symbiotic bacteria. --- #### Some biologycal facts...  - all life originated from a single cell ancestor, A, T, G, C is the same in all species. - upregulation and downregualtion is a continous and non linear process. - because this microworld is so complex and difficult to make it hardware base some chose to work at the macrolevel but essentially: - **all life is encoded in the genome along with the epigenome** (sperm cell meets the egg cell) <!-- .element: style="font-size: 23px;" --> --- ### Contributions and Acknowledgements: \ **Pierre-Luc Bacon**: Supervision. **Payel Das**: Supervision and funding. **Manuel Del Verme**: For rewriting with me so many times the simulator, gym wrapping and daily support. **Simon Dufort-Labbé**: Adding JAX, fast simulator. **Youssef Mroueh**: Suggesting the optimal transport data. **David Kanaa**: Brainstorming with me about the math behind the simulator. **Ryan Jhon Cubero**: gave the right reccomendations for papers at the beginnning one year ago. **Rainer Kelz**: supervising my thesis and allowing me to have this project as my master thesis. <!-- .element: style="font-size: 23px; text-align:left;" --> \ I sincerely thank every single person who helped this project become so possible and I appreciate any small contribution which was made and it is yet to come. <!-- .element: style="font-size: 23px;" --> --- Thank you all for your attention. Time for discussion.