# Overview

## Core Terminology
* **Stochastic:** Involving random variables and uncertainty.
* **Optimization:** Finding the best solution given objectives and constraints.
* **Dynamic:** Adapting decisions to changing conditions.
* **Control:** Influencing a system to achieve desired outcomes.
* **Markov Decision Process (MDP):** A framework for modeling decision-making in uncertain environments where outcomes are partly random and partly under the control of the decision-maker.
* **Reinforcement Learning (RL):** Algorithms for learning optimal control strategies in MDPs through trial and error.

## Markov Decision Processes (MDPs)

* **Framework:** MDPs describe interactions between an agent and an environment over time.
* **Agent:** The decision-maker.
* **Environment:** The system the agent interacts with.
* **State $S$:** Information about the current situation of the environment.
* **Action $A$:** A decision made by the agent.
* **Reward $R$:** Feedback signal from the environment based on an action.
* **Policy $\pi$:** A rule the agent uses to choose actions.
* **Markov Property:** The next state depends only on the current state and action, not on the full history.
* **Goal:** Find the optimal policy that maximizes the expected sum of discounted future rewards.

### Applications of MDPs
MDPs have widespread applications across diverse fields:
* **Autonomous Driving:** Decision-making for self-driving cars in dynamic traffic conditions.
* **Strategic Games:** Developing AI players for complex games like chess and Go.
* **Robotics:** Optimizing robot movement and task execution in uncertain environments.
* **Financial Portfolio Management:** Designing adaptive asset allocation strategies.
* **Energy Systems:** Optimizing energy usage and distribution in smart grids.

### Challenges in Applying MDPs
* **Scalability:** Computational complexity increases dramatically with large state and action spaces.
* **Delayed Rewards:** Effects of actions may not be immediate, making it difficult to assign credit.
* **Model Uncertainty:** The agent may need to learn the environment's dynamics (transition probabilities and rewards) through experience.
* **Exploration vs. Exploitation:** Balancing the need to try new actions to improve the policy (exploration) with the need to use the best known actions to maximize reward (exploitation).

## Dynamic Programming (DP)
* **Assumptions:** The environment's model (including transition probabilities and rewards) is fully known in advance.
* **Approach:** DP decomposes complex problems into simpler overlapping subproblems, leveraging the **Bellman Equations** to find optimal policies without direct interaction with the environment.
* **Benefits:** Provides theoretically sound solutions when the environment model is accurate.

### Challenges with DP
* **Curse of Dimensionality:** Computational requirements explode as the number of state and action variables increase, making DP intractable for many larger-scale problems.
* **Curse of Modeling:** Even minor inaccuracies or uncertainties in the environment's model can significantly degrade the quality of solutions found by DP.

## Reinforcement Learning (RL)
* **Adapting to the Unknown:** In contrast to DP, RL doesn't require a pre-existing model of the environment. It learns directly through interaction, making it ideal for real-world scenarios where perfect models are rare.
* **Learning from Experience:** RL algorithms analyze each state-action-reward-next state transition, gathering data from either real-world or simulated environments.
* **Iterative Improvement:** Trial-and-error exploration lets the agent discover effective strategies. Over time, it refines its understanding of how actions lead to rewards, both immediate and delayed.
* **Value Functions:** RL learns to estimate the expected future reward associated with taking specific actions in different states. Deep neural networks often play a key role in approximating these value functions in complex environments.
* **Balancing Exploration and Exploitation:** A core challenge in RL is finding the right balance between trying new actions (exploration) and leveraging known successful strategies (exploitation). The Bellman Equations provide guidance for this decision-making process.

RL's ability to learn and adapt in complex, dynamic environments without predefined models underscores the remarkable potential of modern AI technologies.


## Overview of Topics
1. **Markov Decision Processes (MDPs):** The mathematical foundation for modeling sequential decision-making problems in stochastic environments.
2. **Dynamic Programming (DP):** Algorithms for solving MDPs when the environment's model (transition probabilities and rewards) is fully known. Emphasize the breakdown of complex problems into simpler subproblems.
3. **Approximate DP and Backward Induction:** Methods for making DP more computationally tractable in large-scale problems. Techniques include approximation, working backward from a terminal state, and related strategies.
4. **Reinforcement Learning (RL):** Algorithms for learning optimal control policies in MDPs without a predefined model. Focus on direct interaction with the environment and iterative learning through trial-and-error.
5. **Applications in Finance and Trading:**
* **Asset Allocation:** Optimizing portfolio composition over time to maximize expected utility.
* **Derivatives Pricing and Hedging:** Developing strategies for pricing options, futures, etc., and mitigating risk in incomplete markets.
* **Optimal Exercise of American Options:** Determining the best time to exercise options that can be exercised before their expiry date.
* **Optimal Trade Order Execution:** Minimizing the market impact of large trades by strategically breaking them into smaller orders.
* **Optimal Market-Making:** Setting bid and ask prices that balance profit potential against inventory risk.

## References
- Chapter 1 of the [RLForFinanceBook](https://stanford.edu/~ashlearn/RLForFinanceBook/book.pdf)
- [Course Overview](https://github.com/coverdrive/technical-documents/blob/master/finance/cme241/Stanford-CME241.pdf) slides for CME 241: Foundations of Reinforcement Learning with Applications in Finance