Portfolio Management Using Reinforcement Learning

Course:                  CSCI-GA 3033 Deep Reinforcement Learning
Authors:                 Di He (dh3171)
                         Congyun Jin(cj2164)
Project Duration:        November 2021 - December 2021

1. Introduction to Portfolio Management

Portfolio is a collection of financial assets. Portfolio Management is a decision making process of continuously re-allocating an amount of fund into a number of financial assets to maximize the return while restraining the risk.

Previous Works has done some RL experiments on single stock trading.

• Deng et al. 2016 applied deep RL to single-asset trading and verified its performance in the fund market.
• Snow et al. 2020 dealt with machine learning in asset management. It demonstrated how to apply three of the most popular methods, Q-learning, policy gradient, and actor–critic in single stock trading.
• Jiang et al. 2017 presented a financial-model-free Reinforcement Learning framework to provide a deep machine learning solution to the portfolio management problem. They also examined DPG in three back-test experiments in a cryptocurrency market.

We planned to do more experiments on stock portfolio management and explore the impact of observation window size.

1.1 Assets

Assets could be cash, bonds, stocks, options etc. In our project, the candidate assets only contain stocks and cash. And the portfolio consists of cash plus \(m\) stocks. The notations are listed below:

• \(t\): it means a time point related to an asset price or a time period related to an asset return or relative price. We assume the daily trading period in our project, which means we can adjust the assets weights each day, therefore \(t\) is at daily frequency.
• \(\pmb{v_t}\): The closing price vector for assets at time \(t\), where \(t \geq 0\). The first row in the vector stands for cash.
  \[\pmb{v_t} := (v_{0,t}, v_{1,t}, ..., v_{m,t} )^T \in R^{m+1}\]
• \(V_t\): The price and volume matrices for assets at time \(t\), where \(t \geq 0\). Each matrix contains 5 columns corresponding to open, low, high, close prices and daily volume. The first row in the matrix stands for cash. The prices and volume for cash are the same for each day.
  \[{V_t} \in R^{(m+1)\times 5} \]
• \(\pmb{y_t}\): The relative price vector of the trading period \(t\), where \(t \geq 1\). Because the price of cash is constant, the relative price is 1.
  \[\pmb{y_t} := \pmb{v_t} \oslash \pmb{v_{t-1}} = (1, \frac{v_{1,t}}{v_{1,t-1}}, ..., \frac{v_{m,t}}{v_{m,t-1}})^T \in R^{m+1} \tag{1}\]

1.2 Portfolio

In our project, assume we had \(p_0\) cash at the beginning \(t=0\). We will adjust the investment weights of those assets at the end of every trading day. The notations are listed below:

• \(\pmb{w_{t-1}}\): The weights on assets at the beginning of the trading period \(t\), where \(t-1\geq0\). The initial weight is \(w_0 = (1,0, ..., 0)^T\), meaning full weights on cash.
  \[\pmb{w_t} := (w_{0,t}, w_{1,t}, ..., w_{m,t} )^T \in R^{m+1}, \ s.t.\ \sum_{i=0}^{t}w_{i,t} = 1\]
• \(p_t\): The portfolio value at time \(t\), where \(t>0\).
  \[p_{t} := p_{t-1}\pmb{y_t} \cdot \pmb{w_{t-1}} \tag{2}\]
• \(r_t\): The log return of the portfolio of the trading period \(t\), where \(t>1\).
  \[r_t := log(\frac{p_t}{p_{t-1}}) = log(\pmb{y_t} \cdot \pmb{w_{t-1}}) \tag{3}\]

Based on above notation, we can get:
\[p_{t}=p_{0} e^{\sum_{i=1}^{t} r_{i}}=p_{0} \prod_{i=1}^{t} \pmb{y}_{i} \cdot \pmb{w}_{i-1} \tag{4}\]

1.3 Transaction Cost

Buying and selling stock will produce commission fees. Assume there is a constant commission fee rate \(\mu\) on the basis of the trading value, we can get:

• \(c_t\): The transaction cost at the trading time \(t\), where \(t \geq 1\).
  \[c_t := \mu\| \frac{\pmb{y_t} \odot \pmb{w_{t-1}}}{\pmb{y_t} \cdot \pmb{w_{t-1}}} - \pmb{w_t}\|_1 \tag{5}\]
• \(r_t\): The portfolio return at time period \(t\) considering transaction costs. It will change from the equation (3) to:
  \[r_{t} = log((1-c_t)\pmb{y}_{t} \cdot \pmb{w}_{t-1}) \tag{6}\]
• \(p_t\): The portfolio value at time \(t\) considering transaction costs. It will change from the equation (4) to:
  \[p_{t} = p_{0} \prod_{i=1}^{t} (1-c_t)\pmb{y}_{i} \cdot \pmb{w}_{i-1} \tag{7}\]

2 Reinforcement Learning Problem Formulation

2.1 Assumptions

• Market is Efficient at some extent. The efficient-market implies that asset prices reflect available information. This assumption lets us use historical prices as observations of the states.
• Small Volume Trading. The trading volume of our portfolio is too small to affect the market price. Therefore we will not consider any market impact and it simplifies our model.
• No Leverage Permitted. Assume we can't borrow money from the bank if we don't have enough cash to invest the stocks we want to invest.
• No Short-selling Permitted. Assume we can't short any stock. This assumption together with the assumption above enforce the weights for assets to be larger than 0.
• Zero Interest Rate. Assume no earnings for holding cash.
• We can trade at the end of the day using closing Price. It is not practical in the real world market. We assume this to make things simple.

2.2 Markov Decision Processes Formulation

As Figure 1 shows, Markov Decision Processes is a fundamental framework for reinforcement learning. We will transform our financial problem into a MDP problem. Figure 2 further illustrates the transformation. The notations of both figures are defined as follows:

• \(s_t\): The state at time point \(t\), unobservable.
• \(O_t\): The observation tensor of the state at time point \(t\), where \(t \geq 1\). It includes both the market information and the portfolio information. Assume recent \(n\)-days historical prices can represent necessary market information. And the latest portfolio weights \(\pmb{w_{t-1}}\) present the portfolio information. We can get:
  \[ O_t = (V_{t-n+1:t}, \pmb{w_{t-1}}), \ where\ V_{t-n+1:t} \in R^{\ n\times (m+1)\times 5} \]
• \(\pmb{a_t}\): The action we take at time t, where \(t \geq 1\). It is the portfolio weights we get after trading at the end of the market at time \(t\).
  \[ \pmb{a_t} = \pmb{w_{t}} \]
• \(r'_t\): The reward we get from the action we take at time period \(t\), where \(t \geq 1\). As the goal of our portfolio management is to maximize the profit. As the total portfolio trading days length \(T\) is given, we will use average log return as the whole target. And the reward is divided by the investment days as reward for one action.
  \[r'_t = \frac{1}{T}r_{t+1} = \frac{1}{T}log((1-c_{t+1})\pmb{y}_{t+1} \cdot \pmb{w}_{t}) \tag{8}\]

3 Method

Our problem is different with traditional reinforcement learning problem in the following ways:

• The reward of each timestamp is independent of actions of other timestamps. Based on our assumptions, our trading will not influence the markets, which means actions will not influence the market part of the state observation.
• The rewards at different timestamps are equally important to the final accumulated reward due to the zero interest rate assumption. Once we know an action is best for one observation, there is no need to do more exploration on other actions to increase the final accumulated reward because it has no influence on afterward rewards and they are equally important.

Because of the above characteristics, we choose to focus on exploitation rather than exploration. Hence, deterministic policy rather than a stochastic policy may be a better choice for us.

3.1 DDPG

DDPG Lillicrap, et al. 2015, short for Deep Deterministic Policy Gradient, is a model-free off-policy actor-critic algorithm, combining DPG with DQN. DQN (Deep Q-Network) stabilizes the learning of Q-function by experience replay and the frozen target network. The original DQN works in discrete space, and DDPG extends it to continuous space with the actor-critic framework while learning a deterministic policy. Figure 3 shows the pseudocodes for this algorithm.

• Considering our data is time series, we experimented DDPG with CNN and LSTM.
• We did some experiments on different window sizes (5, 20, 60) which define our observation.

4. Experiments

4.1 Data Preparation

4.1.1 Data Source and Stock Selection

We used the S&P 500 dataset provided by Yahoo. The dataset contains the historical data of stocks in the U.S. market from 2015-01-02 to 2020-01-01. The history data contains Open, High, Low, Close and Volume of every stock on each day.

The dataset for our project is formed by the historical price data of 17 stocks chosen from different S&P 500 sectors: Information Technology, Health Care, Financials, Consumer Discretionary, Communication Services (sorted by size in descending order). We chose stocks from Top5 sectors to reduce the risk from the industry and improve model generalization. To evaluate model performance, we will use both seen stocks and 17 more unseen stocks.

4.1.2 Data Visualization

To obtain an overview of data, we visualized the trend of the close price of 19 stocks in the training dataset, shown in Figure 4. We've noticed that the stock prices of Google and Amazon constantly outperformed, compared to the rest of the companies, which is a realistic phenomenon.

4.2 Evaluation Metrics

We use the following metrics to evaluate the performance of our agent in the test set.

• Cumulative portfolio return: A cumulative return on a portfolio is the aggregate amount that the investment has gained or lost over time, independent of the amount of time involved. This is the metric we care most about.
• Sharpe ratio: the ratio is the average return earned in excess of the risk-free rate per unit of volatility or total risk. It is used to help investors understand the return of an investment compared to its risk. In our project, we assume the risk-free rate is zero and use the standard deviation of the daily portfolio return to measure the volatility.
• Maximum Drawdown: maximum drawdown is the maximum observed loss from a peak to a trough of a portfolio, before a new peak is attained. Maximum drawdown is an indicator of downside risk over a specified time period. And it is important because investors are not willing to take large drawdown risks.

4.3 Baseline

We choose market value as the baseline of our series experiments. The market value is obtained by equally investing to all stocks in the pool.

4.4 DDPG Methods

Before analyzing our result, we clarify that:

• The window size is the number of the columns in each input price matrix.
• The reward value of our problem is defined as the equation (3)
• Qmax value is the max q value output by critic network in each episode.

4.4.1 DDPG with CNN

Training Performance

From Figure 5 and Figure 6 we notice that:

• The reward values in both two experiments don't converge for 500-epoch training. They both have rising trend.
• With a larger window size, the model performs better and less oscillates.

Testing Performance

In Figure 7, we show the results of trading on 17 seen stocks using our model. We use sharpe ratio and cumulative returns to evaluate the performance of our model in test periods (2019-01-01 to 2020-01-01). The Sharpe Ratio is used to take risk into account.

As the result shows, it turns out that larger window size works better. For larger window sizes, the portfolio value is better than market value, which means that the DDPG+CNN model can make more profits than equally-distributed strategy.

Generalization Test Performance

Figure 8 shows the results of trading on 17 unseen stocks to evaluate the model generalization ability. As the result shows, the generalization performance is fairly good. The agent with window size = 60 can win the baseline at some points in the testing period.

4.4.2 DDPG with LSTM

Training Performance

From two figures below, we see the training performance of two agents with window size=20 and window size=60, we notice that:

• The reward values in both two experiments don't converge for 500-epoch training. They both have rising trend, similar as DDPG+CNN.
• With a smaller window size, the model performs better and less oscillates.

Testing Performance

Figure 11 shows the results of trading on 17 seen stocks using DDPG+LSTM. As the result shows below, in contrast to the training performance, it turns out that the agent with larger window size performs better, which has higher portfolio return. And all agents can make more profit than the average distribution strategy (baseline model).

Generalization Test Performance

Figure 12 shows the results of trading on 17 unseen stocks to evaluate the model generalization ability. The generalization performance is not quite good. The agents can make profit at some points in the test period but the performance is not quite stable. In the last 50 steps, the agents with window size=20 and window size=50 make profits. It is still an optimistic trend.

5. Conclusion and future work

5.1. Conclusion

• In back test, both CNN and LSTM versions better performed in accumulated portfolio return than the equal weighted algorithm in comparison. Our agent is effective.
• DDPG with LSTM has a better performance than CNN. We think it is because our observations are time series, and the LSTM version has better performance.
• Longer window size has a better performance. Window size defines the observation space. Longer window size contains more information. Therefore our agent can make more accurate decisions.
• Our RL agent on other unseen stocks is not as profitable as seen stocks. The generalization ability of our model is not good. Our agent may be overfitting to see stocks.

5.2. Future Work

• Explore the generalization of our method on a longer economic cycle. The distribution of the returns of the markets has a fat tail distribution which is caused by events like Covid-19, Financial Crisis. In our experiments before, those periods weren't included. We want to do more research on this part to see whether our agent can do well when something unexpected happens.
• Explore the generalization of our method on a wider market of assets. We just take tens of stocks in our experiments. If we want to know whether it is profitable in other assets, we should do more research.
• Taking risk into the reward consideration. Because the goal of portfolio management is to maximize the return while restraining the risk. In our experiments before, we just used return as our reward. We can change the reward to Sharp Ratio in order to balance return and risk.
• Incorporate other information into our observation. Because the market is not fully efficient, information isn't captured by the prices. If we can combine information like financial ratios, news and social media comments with the share price as our observation, we can have a more comprehensive description of the state.

6. References

Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, Daan Wierstra. 2015. Continuous control with deep reinforcement learning
Yue Deng, Feng Bao, Youyong Kong, Zhiquan Ren, and Qionghai Dai, Senior Member, IEEE. 2017. Deep Direct Reinforcement Learning for Financial Signal Representation and Trading
Zhengyao Jiang, Dixing Xu, Jinjun Liang. 2017. A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem
Derek Snow. 2020. Machine Learning in Asset Management—Part 1: Portfolio
Construction—Trading Strategies

Appendix