# Algorithmic Trading
Authored by: Ernest Chan

## Chapter 1: Backtesting and Automated Execution
### Importance of Backtesting
- Backtesting is the process of testing a trading strategy using historical data.
- It is important to backtest strategies even if they are published or obtained from other sources.
- Implementation details, such as order types and timing, can significantly impact the profitability of a strategy.
- Backtesting helps to pinpoint and address potential pitfalls in the strategy or the backtesting process itself.
- Each market and strategy has its own specific pitfalls that can inflate backtest performance, leading to dangerous outcomes.
- Backtesting published strategies allows for true out-of-sample testing and helps determine if the strategy is robust or data-dependent.
- Backtesting also provides an opportunity to refine and improve strategies for better profitability and risk management.
- The backtesting process should follow the scientific method, starting with a hypothesis and modifying it based on backtest results.
- Small changes in strategy details can result in substantial improvements, and backtesting allows for experimentation with these details.

### Common Pitfalls of Backtesting
#### Look-ahead Bias
- Look-ahead bias refers to using future information or prices in a backtest program to determine current trading signals.
- An example of look-ahead bias is using the high or low price of a day during backtesting to determine entry signals on the same day.
- Look-ahead bias is a programming error that can only affect backtest programs, not live trading programs.
- To avoid look-ahead bias, it is recommended to use the same program for both backtesting and live trading, with the only difference being the type of data fed into the program.
- Platforms that allow the same source code to be used for both backtesting and live execution help eliminate look-ahead bias.

#### Data-Snooping Bias and the Beauty of Linearity
- Data-snooping bias refers to the misleading results that can arise from analyzing a large amount of data and discovering apparent patterns by chance.
- When conducting extensive data analysis, there is a higher likelihood of finding spurious correlations or patterns that do not hold true in future data.
- Data-snooping bias can lead to over-optimistic performance expectations and the creation of strategies that do not perform as expected in real-world trading.
- The beauty of linearity refers to the simplicity and robustness of linear models in mitigating data-snooping bias.
- Linear models, such as linear regression, provide a clear framework for analyzing relationships between variables without overly complex assumptions.
- By using linear models and focusing on robust statistical measures, traders can reduce the risk of data-snooping bias.
- It is essential to test strategies on out-of-sample data to verify their validity and guard against data-snooping bias.
- Proper validation techniques, such as cross-validation and walk-forward analysis, help identify strategies that are less susceptible to data-snooping bias.
- Understanding the limitations of data analysis and utilizing linear models effectively can enhance the reliability and performance of trading strategies.

#### Stock Splits and Dividend Adjustments
- Stock splits result in a reduction in stock price by a factor of N, where N is the split ratio. In backtesting, historical prices should be divided by N before the ex-date of the split to avoid erroneous trading signals.
- Cash or stock dividends cause a decrease in stock price by the dividend amount. Similar to stock splits, historical prices should be adjusted by subtracting the dividend amount before the ex-date.
- These adjustments should be made not only in backtesting but also in live trading to ensure accurate trading signals.
- Historical split and dividend information can be found on websites like earnings.com, which also provides future ex-dates and amounts. For pre-adjusted historical stock data, csidata.com is recommended as a reliable source.

#### Survivorship Bias in Stock Database
- Survivorship bias occurs when historical data used for backtesting a stock-trading model only includes stocks that are currently listed, ignoring delisted stocks.
- Ignoring delisted stocks can create a misleading picture of a strategy's performance.
- Survivorship bias can make long-only strategies that buy low and sell high appear more successful than they actually are.
- Short-only strategies that sell high and buy low can be deflated in performance due to survivorship bias.
- Mean-reverting long-short strategies are affected by survivorship bias, but the impact is less pronounced compared to long-only strategies.
- Momentum models can also be affected by survivorship bias, deflating the backtest return.
- To avoid survivorship bias, one can purchase historical data from sources that provide information on delisted stocks or collect their own data from daily stock prices of an index.
- Alternatively, limiting backtesting to more recent historical data can reduce the impact of survivorship bias.

#### Primary versus Consolidated Stock Prices
- Many stocks in the US are traded on multiple exchanges, ECNs, and dark pools.
- Historical daily closing and opening prices reflect the last or first execution price on any of these venues.
- Market-on-close (MOC) and market-on-open (MOO) orders are routed to the primary exchange only.
- To accurately backtest a strategy relying on MOC or MOO orders, historical prices from the primary exchange are needed.
- Using consolidated historical prices for backtesting can lead to unrealistic results, especially for mean-reverting models.
- Consolidated high and low prices may be unrepresentative and exaggerated due to trades on secondary exchanges.
- Access to historical prices from the primary exchanges is available through subscriptions like Bloomberg or direct live feeds.
- Retail brokers typically do not redistribute direct feeds, but some institutional brokers may offer them.
- Without access to such data, it's important to maintain a healthy skepticism of backtest results.

#### Venue Dependence of Currency Quotes
- Currency markets are fragmented, and trades executed on different venues may not reflect the best bid or ask across all venues.
- To conduct a realistic backtest, historical data from the same venue(s) where trading is expected to occur should be used.
- Quotes aggregators like Streambase consolidate data feeds from various venues into one order book, allowing the use of consolidated historical data for backtesting if trading can be executed on the corresponding venue.
- Currency trade prices and sizes are not widely available in real-time due to the absence of regulations requiring their reporting to all market participants.
- Using bid-ask quotes for backtesting forex strategies is recommended, considering the variation in bid-ask spreads and transaction costs across different venues.
- Venue dependence plays a crucial role in determining transaction costs, which need to be considered in a backtest.

#### Short-Sale Constraints
- Short-selling stocks is often hindered by various constraints.
- One of these constraints involves locating a stock to short. If a stock is heavily shorted or if the float of the stock is limited, it can become "hard to borrow."
- If a stock is hard to borrow, short sellers may have to pay interest to the stock lender. In extreme cases, the stock may not be available to borrow at all.
- Backtesting models that short hard-to-borrow stocks can show inflated returns as these stocks might not have been possible to short in reality.
- After the collapse of Lehman Brothers, the U.S. Securities and Exchange Commission (SEC) banned short sales in all financial industry stocks for several months.
- A historically accurate list of hard-to-borrow stocks is difficult to find as it depends on the broker used.
- Generally, small-cap stocks are more affected by short-sale constraints than large-cap stocks, making the returns on their short positions more suspect.
- Sometimes ETFs can be as hard to borrow as stocks. The author notes difficulty in borrowing SPY after the Lehman Brothers' collapse.
- Another constraint is the "uptick rule" imposed by the SEC. This rule, in effect from 1938 to 2007, required short sales to be executed at a price higher than the last traded price.
- The Alternative Uptick Rule, in effect since 2010, also requires a higher trade price for a short sale, but only when a stock's circuit breaker has been triggered (when a stock trades at 10% lower than its previous close).
- These constraints effectively prevent any short market orders from being filled.
- Accurate backtesting that involves short sales must take into account these constraints. Otherwise, the backtest performance will be inflated.

#### Futures Continuous Contracts
- Futures contracts have expiry dates, making a strategy on, for example, crude oil futures, a strategy on many different contracts. Usually, the strategy applies to front-month contracts.
- The "front month" contract depends on when you plan to roll over to the next month - selling the current front contract and buying the contract with the next nearest expiration date.
- Rollover dates can be decided based on various factors such as 10 days before the current contract expires or when the open interest of the next contract exceeds the current front contract.
- Rollover actions can complicate the trading strategy, as buying and selling independent of the strategy should result in minimal additional return or profit and loss (P&L).
- Most futures historical data vendors provide "continuous contract" data which concatenates the prices of the front-month contract together given a set of rollover dates. However, this creates a price series with significant price gaps and false returns or P&L on the rollover date.
- To correct for this, data vendors can back-adjust the data series to eliminate the price gap. However, this adjustment might lead to discrepancies in P&L and return calculations.
- As a result, the trader has to choose one performance measurement only, P&L or return when using a continuous contract series.
- An additional difficulty with price back-adjustment is that prices may turn negative in the past. A common solution is to add a constant to all prices to prevent negativity.
- The choice between price back-adjustment and return back-adjustment is important for strategies that involve trading spreads between different contracts. For strategies based on the price difference between contracts, the price back-adjustment method is appropriate. However, if the strategy is based on the ratio of prices between contracts, the return back-adjustment method is more suitable.
- When choosing a data vendor for historical futures prices, it's crucial to understand how they've dealt with the back-adjustment issue, as it significantly impacts backtest results. Different vendors offer different adjustment options.

#### Futures Close versus Settlement Prices
- The daily closing price of a futures contract provided by data vendors is usually the settlement price, not the last traded price of the day. A futures contract will have a daily settlement price even if it has not traded that day.
- If a contract has traded, the settlement price generally differs from the last traded price. However, some data vendors that provide tick-by-tick data may only provide the actual transaction price, making the closing price the last traded price.
- For most strategies, the settlement price should be used in backtests because it is closest to the transaction price for trades made near the close of trading. The last recorded trade price might have occurred hours earlier and not accurately reflect the price at close.
- Using settlement prices in pairs-trading strategies on futures ensures contemporaneous prices are used. If the last traded prices are used instead, they may reflect transactions occurring at very different times, leading to incorrect spread calculations and inflated backtest results.
- If an intraday spread strategy is being used, or if intraday futures prices are used in backtesting, historical data with bid and ask prices for both contracts is required. Otherwise, asynchronous last prices could result in unrealistically high backtest returns.
- One vendor that sells intraday historical calendar spread data (both quote and trade prices) is cqgdatafactory.com.
- It is important to note that contracts traded on different exchanges likely have different closing times. Therefore, it would be incorrect to form an intermarket spread using their closing prices. This also applies to forming a spread between a future and an ETF.
- A possible solution is to use intraday bid-ask data to ensure synchronicity, or trade an ETF that holds a future instead of the future itself. For instance, trade the gold trust GLD against the gold-miners ETF GDX instead of the gold future GC, because both GLD and GDX have their closing prices set at the same time.

### Statistical Significance of Backtesting: Hypothesis Testing
- Backtesting is affected by the issue of finite sample size, and the results, such as average returns or maximum drawdowns, might be due to chance.
- Hypothesis testing is used to address this issue:
1. A statistical measure (test statistic) is computed from a backtest on a limited data sample, for example, the average daily return of a trading strategy.
2. The null hypothesis is proposed, assuming that the true average daily return based on an infinite dataset is zero.
3. The probability distribution of daily returns is assumed to be known, with a zero mean as per the null hypothesis.
4. From this null hypothesis probability distribution, the p-value is computed, which is the probability that the average daily returns will be at least as large as the observed value in the backtest. If the p-value is very small (e.g., less than 0.01), the null hypothesis can be rejected, and the backtested average daily return is deemed statistically significant.
- Determining the probability distribution under the null hypothesis is a critical step. Three ways are suggested:
1. Assume that daily returns follow a standard parametric probability distribution like the Gaussian distribution, with a mean of zero and a standard deviation given by the sample standard deviation of the daily returns. High Sharpe ratio strategies are usually more statistically significant.
2. Use Monte Carlo methods to generate simulated historical price data and calculate the empirical probability distribution of profits. This method tests if the profitability of the strategy comes from subtle patterns or correlations in the price series, rather than just the first few moments of the price distributions.
3. Generate sets of simulated trades, with constraints matching the backtest's characteristics, and distribute these trades randomly over the actual historical price series. This method compares the average return of the strategy against a different benchmark of randomness.
- These three testing methods may yield different results due to the distinct probability distributions in each case.
- A null hypothesis is not unique and different null hypotheses can lead to different estimates of statistical significance, contributing to criticism of hypothesis testing as a flawed methodology (Gill, 1999).
- Critics also argue that we usually want to know the conditional probability of the null hypothesis being true given that we've observed the test statistic R (P(H0|R)), whereas the traditional testing procedure calculates the conditional probability of obtaining a test statistic R given that the null hypothesis is true (P(R|H0)).
- It's rare that P(R|H0) equals P(H0|R), adding further to the skepticism towards hypothesis testing.
- Despite criticism, not rejecting a null hypothesis can lead to valuable insights.
- Example 1.1 illustrates that a random returns distribution with high kurtosis can be advantageous for momentum strategies, providing an interesting insight generated from failure to reject a null hypothesis.

#### Example 1.1: Hypothesis Testing on a Futures Momentum Strategy
- Example 1.1 applies three versions of hypothesis testing to a Futures Momentum Strategy, with each test using a different probability distribution for the null hypothesis.
- The strategy: buy/sell the TU future if it has a positive/negative 12-month return and hold the position for 1 month.
- Hypothesis Test 1: Assumes a Gaussian distribution for daily returns, with a mean of zero (null hypothesis), and the standard deviation of the daily returns from the backtest. The test statistic equals `mean(ret)/std(ret)*sqrt(length(ret))`.
- Result of Test 1: The test statistic equals 2.93. The null hypothesis can be rejected with better than a 99% probability.
- Hypothesis Test 2: Involves generating a set of random, simulated daily return data for the TU future and runs the strategy on these simulated prices. The process is repeated 10,000 times to see how many times the strategy produces an average return greater or equal to that on the observed data set.
- Result of Test 2: Out of 10,000 random returns sets, 1,166 have an average strategy return greater or equal to the observed average return. So, the null hypothesis can be rejected with an 88% probability.
- Hypothesis Test 3: Involves randomizing the long and short entry dates, while maintaining the same number of long trades and short trades as the backtest.
- Result of Test 3: There is not a single sample out of 100,000 where the average strategy return is greater than or equal to the observed return.

Converting the MATLAB code to Python:

import numpy as np
from scipy.stats import skew, kurtosis, norm

# Hypothesis Test 1
ret = np.array(ret)  # ret is the daily returns of the strategy
test_statistic = ret.mean() / ret.std() * np.sqrt(len(ret))

# Hypothesis Test 2
moments = [ret.mean(), ret.std(), skew(ret), kurtosis(ret)]
numSampleAvgretBetterOrEqualObserved = 0
for sample in range(10000):
    marketRet_sim = norm.rvs(size=len(marketRet))
    cl_sim = np.cumprod(1 + marketRet_sim) - 1
    longs_sim = cl_sim > np.roll(cl_sim, lookback)
    shorts_sim = cl_sim < np.roll(cl_sim, lookback)
    pos_sim = np.zeros(len(cl_sim))
    for h in range(holddays):
        long_sim_lag = np.roll(longs_sim, h)
        short_sim_lag = np.roll(shorts_sim, h)
        pos_sim[long_sim_lag] += 1
        pos_sim[short_sim_lag] -= 1
    ret_sim = np.roll(pos_sim, 1) * marketRet_sim / holddays
    if ret_sim.mean() >= ret.mean():
        numSampleAvgretBetterOrEqualObserved += 1

# Hypothesis Test 3
numSampleAvgretBetterOrEqualObserved = 0
for sample in range(100000):
    P = np.random.permutation(len(longs))
    longs_sim = longs[P]
    shorts_sim = shorts[P]
    pos_sim = np.zeros(len(cl))
    for h in range(holddays):
        long_sim_lag = np.roll(longs_sim, h)
        short_sim_lag = np.roll(shorts_sim, h)
        pos_sim[long_sim_lag] += 1
        pos_sim[short_sim_lag] -= 1
    ret_sim = np.roll(pos

_sim, 1) * marketRet / holddays
    if ret_sim.mean() >= ret.mean():
        numSampleAvgretBetterOrEqualObserved += 1

Please replace the place holder variables such as `ret`, `marketRet`, `cl`, `longs`, `shorts`, `lookback`, and `holddays` with actual data.

### When Not to Backtest a Strategy
- There are certain situations where backtesting a strategy is not recommended due to obvious flaws or potential biases.

- Example 1: Avoid backtesting strategies with a high return but low Sharpe ratio, or those with a maximum drawdown duration longer than you or your investors can endure.
- A strategy with a 30 percent annualized return, a Sharpe ratio of 0.3, and a maximum drawdown duration of two years is an example of a potentially inconsistent strategy likely to fail in live trading due to long drawdown duration and data-snooping bias.

- Example 2: Choose the appropriate benchmark for comparing trading strategies.
- A long-only crude oil futures strategy returned 20 percent in 2007 with a Sharpe ratio of 1.5, but simply holding the front-month crude oil futures in 2007 yielded a return of 47 percent with a Sharpe ratio of 1.7, highlighting the importance of using appropriate benchmarks.

- Example 3: Be wary of strategies that may suffer from survivorship bias.
- A "buy-low-sell-high" strategy that picks the 10 lowest-priced stocks at the beginning of the year and holds them for a year showed a backtest return of 388 percent in 2001. If the data used does not include delisted stocks, the return is likely inflated due to survivorship bias.

- Example 4: Be skeptical of models with a high number of parameters, as they are prone to overfitting and data-snooping bias.
- A neural net trading model with about 100 nodes generating a backtest Sharpe ratio of 6 might be overfitting the training data, hence having little or no predictive power going forward.

- Example 5: Be very skeptical of backtests of high-frequency trading strategies.
- Performance of these strategies relies heavily on the market microstructure and reactions of other market participants. The act of placing or executing an order might alter other participants' behavior, introducing a kind of "Heisenberg uncertainty principle."

- Tips:
- Be aware of common pitfalls of backtesting to better select what strategies to backtest.
- Life is too short to backtest every single strategy, so focus on those with the most potential and least obvious flaws.

### Will a Backtest Be Predictive of Future Returns?
- The predictive power of a backtest rests on the central assumption that the statistical properties of the price series are unchanging. However, changes in economic prospects, management, and financial market structures often invalidate this assumption.

- Specific examples of changes affecting the predictive power of backtests include:

- Decimalization of U.S. stock quotes on April 9, 2001, which decreased the profitability of many statistical arbitrage strategies while increasing the profitability of many high-frequency strategies.

- The 2008 financial crisis resulted in a 50% collapse of average daily trading volumes, leading to decreased volatility but increased frequency of sudden outbursts. This situation generally decreased profits for mean-reverting strategies.

- The 2008 financial crisis also initiated a multiyear bear market in momentum strategies.

- The implementation of the SEC’s Regulation NMS in July 2007 contributed to a decrease in average trade sizes and the obsolescence of the NYSE block trade.

- Changes to rules for short sales, including the removal of the old uptick rule in June 2007 and the reinstatement of the new Alternative Uptick Rule in 2010, also affected strategies.

- Strategies that performed well before these "regime shifts" may stop performing afterward, and vice versa.

- Backtests using data prior to regime shifts may be worthless, while those using recent data may not indicate future profits if another regime shift is to occur.

- Algorithmic trading requires an understanding of algorithms, programming, and mathematics, but also an awareness of fundamental market and economic issues to determine whether a backtest is and will continue to be predictive.

## Chapter 2: The Basics of Mean Reversion
- Mean reversion is a concept observed across natural and social phenomena. For example, an athlete's performance can be thought of as randomly distributed around a mean, and an exceptional performance one year is likely followed by performances closer to the average.
- In the financial context, if price series were mean reverting, trading would be simplified: buy low (when the price is below the mean), wait for reversion to the mean price, and sell at the higher price.
- However, most price series are geometric random walks, not mean reverting. The returns, not the prices, usually randomly distribute around a mean of zero. Unfortunately, traders cannot trade on the mean reversion of returns.
- Some price series found to be mean reverting are called stationary. Statistical tests for stationarity include the Augmented Dickey-Fuller (ADF) test, the Hurst exponent, and the Variance Ratio test.
- It's possible to combine two or more individual price series that are not mean reverting into a portfolio whose net market value (i.e., price) is mean reverting. Such price series are called cointegrating, and the statistical tests for cointegration include the CADF test and the Johansen test.
- Through the Johansen test, the exact weightings of each asset in the portfolio to create a mean-reverting portfolio can be determined. This creates numerous opportunities for mean reversion traders.
- There's another kind of mean reversion, called "cross-sectional" mean reversion, which implies that the short-term relative returns of the instruments are serially anticorrelated. This phenomenon is often seen with stock baskets, and strategies to leverage it will be discussed in later chapters.

### Mean Reversion and Stationarity
- Mean reversion and stationarity are two different ways to view the same type of price series, leading to two different statistical tests.
- A mean-reverting price series is defined as one where the change in the price series in the next period is proportional to the difference between the mean price and the current price. The Augmented Dickey-Fuller (ADF) test is used to determine whether we can reject the null hypothesis that the proportionality constant is zero.
- A stationary price series is characterized by its variance of the log of prices increasing slower than that of a geometric random walk, i.e., the variance is a sublinear function of time. The Hurst exponent is used to characterize this sublinearity, where a value less than 0.5 signifies stationarity.
- The Variance Ratio test is used to determine whether we can reject the null hypothesis that the Hurst exponent is 0.5 (which would indicate a geometric random walk).
- The term 'stationarity' does not mean that prices are necessarily range-bound. It means that the variance of prices increases slower than normal diffusion.
- The mathematical details of the ADF and Variance Ratio tests can be found in Walter Beckert's course notes, but here the focus is on their application to practical trading strategies.

#### Augmented Dickery-Fuller Test
- The Augmented Dickey-Fuller (ADF) test is used to test if a price series is mean reverting.
- A price series is mean-reverting if its current price level indicates the direction of its next move: if the price is above the mean, it is expected to go down, if below, it is expected to go up.
- A linear model is used to describe the price changes: Δy(t) = λy(t − 1) + μ + βt + α1Δy(t − 1) + … + αkΔy(t − k) + ε(t), where Δy(t) is the difference in price at time t and time (t-1), and y(t - 1) is the price at time (t-1).
- The ADF test checks if λ = 0. If this hypothesis can be rejected, it suggests that the next move Δy(t) depends on the current level y(t − 1), indicating the price series is not a random walk.
- The test statistic is λ divided by the standard error of the regression fit: λ/SE(λ).
- The critical values for the test statistic that indicate whether the hypothesis can be rejected at a certain probability level have been tabulated by statisticians Dickey and Fuller.
- In the context of mean reversion, λ/SE(λ) needs to be negative, and more negative than the critical value for the hypothesis to be rejected.
- The critical values depend on the sample size and assumptions about the price series, including whether it has a non-zero mean -μ/λ or a steady drift -βt/λ.
- In practice, the constant drift in price (if any) is often much smaller than the daily fluctuations in price, so for simplicity, the drift term is typically assumed to be zero (β = 0).

#### Hurst Exponent and Variance Ratio Test
- A "stationary" price series suggests that prices move away from their starting point slower than a geometric random walk would.
- The speed of this movement can be characterized by the variance, represented as Var(τ) = 〈|z(t + τ) − z(t)|2〉, where z is the log of prices, τ is a time lag, and 〈…〉 represents an average over all t’s.
- For a geometric random walk, this variance relationship is roughly proportional to τ.
- However, if a (log) price series is mean-reverting or trending (i.e., has positive correlations between sequential price changes), the relationship differs and can be represented as 〈|z(t + τ) − z(t)|2〉 ∼ τ2H, where H is the Hurst exponent.
- The Hurst exponent, H, is 0.5 for a price series exhibiting geometric random walk. If the series is mean-reverting, H < 0.5, and for a trending series, H > 0.5. As H moves towards zero, the price series is more mean-reverting, and as it approaches 1, the series is increasingly trending.
- The Hurst exponent can therefore serve as an indicator of the degree of mean reversion or trendiness in a price series.

#### Half-Life of Mean Reversion
- Statistical tests for mean reversion or stationarity, like ADF, often demand high levels of certainty, typically at least 90%. In practical trading, however, we can often be profitable with less certainty.
- The coefficient λ in Equation 2.1 can be interpreted as a measure of how long it takes for a price to revert to the mean. Even if we cannot reject the null hypothesis that λ is zero with 90 percent certainty, if λ is sufficiently negative, a trading strategy can still be practical.
- Equation 2.1 can be transformed to a differential form, known as the Ornstein-Uhlenbeck formula, which allows an analytical solution for the expected value of y(t) to be obtained. This formula describes a mean-reverting process.
- If the regression coefficient λ is positive, it means the price series is not mean reverting, hence a mean-reverting trading strategy should not be attempted.
- If λ is very close to zero, the half-life of mean reversion will be very long. Consequently, a mean-reverting trading strategy will not be very profitable because fewer round-trip trades can be completed in a given time period.
- λ can be used to determine a natural time scale for many parameters in our trading strategy. For instance, if the half-life is 20 days, using a look-back of 5 days to compute a moving average or standard deviation for a mean-reversion strategy might not be optimal.
- Often, setting the look-back to equal a small multiple of the half-life is close to optimal. This avoids the need for brute-force optimization of a free parameter based on the performance of a trading strategy.
- As an example, the price series USD.CAD, despite not being stationary with at least 90 percent probability, could still be a candidate for mean reversion trading by determining its half-life of mean reversion.

### Cointegration
- Cointegration refers to a statistical property where a linear combination of nonstationary price series is stationary.
- This property is particularly beneficial in finance, where most price series are not stationary or mean reverting.
- Cointegration allows for the creation of a portfolio of individual price series so that the portfolio's market value (or price) series is stationary.
- A common application of cointegration is pairs trading, where one asset is longed and another is shorted, with a balanced allocation of capital to each.
- The concept of cointegration can also extend to portfolios comprising three or more assets.
- Two common tests for cointegration are the CADF (Cross-sectional Augmented Dickey-Fuller) and Johansen tests.
- The CADF test is applicable only for a pair of price series, whereas the Johansen test is applicable to any number of series.

#### Cointegrated Augmented Dickey-Fuller (CADF)
- The Cointegrated Augmented Dickey-Fuller (CADF) Test is used to determine the stationarity of portfolio price series.
- The need for the CADF arises as it is not always known what hedge ratios should be used to form a stationary portfolio from a given set of price series.
- Hedge ratios determine the number of units of an asset that should be long or short in a portfolio. A negative hedge ratio indicates a short position on the asset.
- The CADF test involves determining the optimal hedge ratio through a linear regression fit between two price series, forming a portfolio based on this hedge ratio, and then running a stationarity test on the portfolio price series.

#### Johansen Test
- The Johansen Test is used to test for cointegration of more than two variables or price series.
- In the Johansen Test, price variable y(t) and coefficients λ and α are considered as vectors and matrices, respectively, with βt assumed to be 0 for simplicity.
- The test generalizes an equation representing the relationship between price series and their lagged versions.
- If Λ equals 0, there is no cointegration.
- The rank (r) of Λ denotes the number of independent portfolios that can be formed by various linear combinations of the cointegrating price series.
- The Johansen Test calculates the rank in two ways, both based on eigenvector decomposition of Λ: the trace statistic and the eigen statistic.
- The test can confirm if all price series are cointegrated (r = n) when all these hypotheses are rejected.
- The eigenvectors found can be used as hedge ratios for the individual price series to form a stationary portfolio.
- The eigenvectors from the Johansen test are also used to form a stationary portfolio, and the half-life for mean reversion of this portfolio is determined.

#### Linear Mean-Reverting Trading on a Portfolio
- The portfolio was then used to backtest a simple linear mean-reverting strategy.
- This strategy involved accumulating units of the portfolio proportional to the negative Z-Score of the unit portfolio's price, with the unit portfolio having shares determined by the Johansen eigenvector.
- The share price of a unit portfolio is equivalent to its market value, similar to a mutual fund or ETF.
- When a unit portfolio involves only a long and a short position in two instruments, it is often called a spread.
- In this context, a "linear" strategy means the number of units invested is proportional to the Z-Score, not that the investment's market value is proportional.
- Despite its impracticalities, such as uncertainty about maximum capital required and the inability to enter and exit an infinitesimal number of shares with each price shift, the strategy is important for backtesting.
- The linear mean-reverting strategy shows that profits can be extracted without data-snooping bias, as there are no parameters to optimize.
- The look-back is set equal to the half-life, a value that depends on the properties of the price series, not the specific trading strategy.
- Given its continuous entering and exiting of positions, the strategy likely holds more statistical significance than strategies with more complex and selective entry and exit rules.

### Pros and Cons of Mean-Reverting Strategies
- Constructing mean-reverting strategies is often easy due to the ability to create custom portfolios from a variety of cointegrating stocks and ETFs.
- The increasing number of marginally different ETFs being created each year aids in this process.
- Mean-reverting pairs often have a strong fundamental rationale, e.g., EWA and EWC cointegrate due to both Canadian and Australian economies being dominated by commodities.
- When cointegrating pairs fall apart, it is often possible to discern a reason. For example, GDX and GLD stopped cointegrating in early 2008 due to high energy prices making gold mining abnormally expensive.
- Mean-reverting strategies have an advantage over momentum strategies because they can be justified with fundamental reasoning, while momentum strategies rely on the assumption that some investors react slower to news.
- Mean-reverting strategies can span a wide variety of time scales, benefiting traders who favor short time scales due to higher trade frequencies, leading to higher statistical confidence, higher Sharpe ratios, and ultimately higher compounded returns.
- However, the high consistency of mean-reverting strategies can lead to overconfidence and overleveraging, which can be catastrophic when the strategy unexpectedly breaks down, especially when trading at maximum leverage.
- Risk management for mean-reverting strategies is critical and challenging, as the traditional stop-loss tactics aren't logically applicable.

## Chapter 3: Implementing Mean Reversion Strategies
- Not all stationary series will yield significant profits, especially if the half-life for mean reversion is long, e.g., 10 years.
- There's no need for true stationarity or cointegration for a successful mean reversion strategy. Short-term or seasonal mean reversion can be captured, with positions liquidated before prices reach the next equilibrium level.
- A practical mean reversion strategy discussed is the Bollinger Bands, which has several variations including multiple entry and exit levels ("scaling-in"), and the use of the Kalman filter to estimate the hedge ratio and mean price.
- It's crucial to be cautious about data errors as they can pose significant risks to mean-reverting strategies.
- Backtests presented in the chapter do not include transaction costs and may have look-ahead bias, which are some of the pitfalls to be mindful of.
- It's necessary for readers to undertake the task of cleaning up such pitfalls when they are implementing these strategies in a real trading environment.

### Trading Pairs Using Price Spreads, Log Price Spreads, or Ratios
- The market value of a mean reversion portfolio is the weighted sum of its constituent price series, where weights are the hedge ratios.
- Mean reversion trading can be conducted using price spreads, log price spreads, or the ratio of prices.
- Trading using price spreads is simpler than using log price spreads, but both can be justified if both price and log price series are cointegrating.
- The ratio of prices y1/y2 doesn't necessarily form a stationary series, but it can be effective in certain situations where the spread is mean reverting on a short time frame, even if the two assets are not truly cointegrating.
- If the hedge ratio needs to be dynamically changed for non-cointegrating pairs, using the ratio as a signal might be more convenient.
- In the specific case of currency trading, ratios are often used. For instance, trading the currency pair EUR.GBP is the same as trading EUR.USD/GBP.USD.
- For currency pairs that do not have ready-made cross rates, such as MXN.NOK, the ratio USD.NOK/USD.MXN could be more effective than the spread, even if MXN.NOK is not truly stationary.
- Trading MXN.NOK directly and trading USD.NOK and USD.MXN to emulate MXN.NOK are not identical, as they generate profit and loss (P&L) denominated in different currencies.

### Bollinger Bands
- Mean-reversal strategy: Scale investment units based on deviation from moving average.
- Linear strategy is simple but not practical due to unknown maximum capital deployment.
- Bollinger bands provide practical trading approach.
- Enter position when price deviates by more than entryZscore standard deviations from mean.
- entryZscore is optimized parameter.
- Look-back period for standard deviation and mean computation is also an optimized parameter.
- Exit position when price mean-reverts to exitZscore standard deviations from mean (exitZscore < entryZscore).
- Different exitZscore values determine exit conditions (0 for current mean, -entryZscore for opposite band).
- Only one unit (long or short) invested at a time, easy capital allocation and risk management.
- Shorter look-back period, smaller entryZscore and exitZscore result in shorter holding period, more trades, and higher profits.

### Does Scaling-in Work?
- Scaling-in or averaging-in is a common strategy where you increase your investment as the price moves away from its average.
- Researchers found that entering or exiting at multiple Bollinger Bands is not the best approach.
- They discovered a single entry/exit level called "all-in" that generates higher average returns.
- They compared three methods: all-in at a certain price, all-in at a lower price, and averaging-in at the first price.
- The expected profits depend on the probability of the price dropping further before rebounding.
- Method I (all-in at the first price) is most profitable if the probability is zero, while Method II (all-in at the lower price) is most profitable if the probability is one.
- There is a transition probability where the profitability of the methods changes.
- Averaging-in is not the most profitable strategy in any situation.
- However, the assumption of constant probability may not hold in real life where volatility fluctuates.
- Scaling-in may result in a better realized Sharpe ratio or profits when volatility is not constant.
- Scaling-in may perform better in real-life situations compared to the "all-in" method tested in the research.

### Kalman Filter as Dynamic Linear Regression
- Kalman Filter is a tool used to estimate the relationship between two changing variables.
- It is particularly useful when the relationship between the variables can vary over time.
- Traditional methods like regression or moving averages have limitations in capturing dynamic changes.
- Kalman Filter uses an optimal linear algorithm that updates the expected value of a hidden variable based on the latest observation.
- It assumes that the relationship between the variables is linear with some random noise.
- By specifying variables and matrices, Kalman Filter can estimate the hedge ratio and other parameters of interest.
- It provides a dynamic hedge ratio that adjusts as the relationship between the variables changes.
- It also gives an estimate of the average and volatility of the relationship.
- The estimation process involves iterative calculations using the Kalman Filter algorithm.
- Kalman Filter is a powerful tool for traders and investors to make better decisions based on changing relationships between variables.

### Kalman Filter as Market-Making Model

- Kalman Filter can also be used as a market-making model for a mean-reverting strategy.
- In this case, we focus on a single mean-reverting price series and want to estimate the mean price and standard deviation.
- The mean price is the hidden variable, and the price is the observable variable.
- The measurement equation is simple, stating that the price equals the mean price plus some random noise.
- The state transition equation updates the mean price based on the previous value and noise.
- The state update equation calculates the current estimate of the mean price based on the observation and the previous estimate.
- The variance of the forecast error and the Kalman gain are also calculated.
- These equations are commonly used by market makers to estimate the mean price of an asset.
- Practitioners make further assumptions about the measurement error to make the model more practical.
- The uncertainty in the observed price can be interpreted based on the trade size, where larger trade sizes have less uncertainty.
- The trade size can be compared to a benchmark to determine the uncertainty in the observed price.
- The Kalman filter approach gives more weight to trades with larger sizes and more recent trade prices.
- It is similar to the volume-weighted average price (VWAP) approach that considers trade sizes and time in determining the mean price or fair value of an asset.

### The Dangers of Data Errors
- Data errors in historical data used for backtesting can inflate the performance of mean-reverting strategies, leading to fictitious profits.
- For example, if there are errors in recording trade prices, it can falsely indicate profitable trading opportunities.
- Data quality is crucial for intraday data as they present more opportunities for errors.
- Data errors can also affect live trading, leading to real-life losses.
- Erroneous bid/ask quotes are particularly dangerous for pairs or arbitrage strategies that rely on price differences.
- Errors in quotes can result in larger percentage errors in the spread and trigger incorrect trading orders.
- Using reliable data sources is important to avoid erroneous trades and losses.
- Bad ticks in live data can also cause mistakes in executing momentum strategies, resulting in losses.

## Mean Reversion of Stocks and ETFs
- Stocks and ETFs provide a fertile ground for mean-reverting strategies.
- Applying generic mean-reversion techniques to stocks and ETFs has some difficulties specific to these instruments.
- Short-term mean reversion can be observed in most stocks under normal circumstances, despite their long-term random walk behavior.
- Index arbitrage is a mean reversion strategy that relies on the cointegration of stocks with futures or ETFs.
- Cross-sectional mean reversion, where cumulative returns of instruments in a basket revert to the basket's cumulative return, is prevalent in stock baskets.
- The stock market attracts statistical arbitrageurs who exploit mean-reverting patterns, resulting in decreasing returns.
- Transaction costs and survivorship bias are important considerations in backtesting stock models.
- Historical prices used in backtests are consolidated opens and closes, but actual returns may be lower when using primary exchange open or close prices for implementation.

### The Difficulties of Trading Stock Pairs
- Trading stock pairs using mean-reversion strategies has become difficult and less profitable.
- Individual stocks rarely meet the criteria of stationarity and often exhibit a geometric random walk behavior, making it challenging to find consistently cointegrating pairs.
- Even if pairs are identified, they can lose cointegration in subsequent out-of-sample periods due to changes in company fortunes.
- Trading a large number of stock pairs does not guarantee profitability, as the small profits of successful pairs are outweighed by the losses of unsuccessful pairs.
- Short-sale constraints pose a risk, especially when shorting hard-to-borrow stocks, as a short squeeze can force liquidation at an unfavorable time.
- The uptick rule and circuit breakers limit short market orders and create uncertainty in backtesting and live executions of stock pairs strategies.
- Intraday trading of stock pairs faces challenges due to small national best bid and offer (NBBO) quote sizes, resulting from factors such as dark pools, high-frequency trading, and the reluctance of market makers to display large order sizes.
- Backtesting stock pair trading strategies using trade or quote prices is not realistic without considering transaction costs or trading small quantities.
- The profitability of stock pair trading has declined due to increased market efficiency, inefficiencies of past times, and the narrowing of bid-ask spreads after decimalization.
- Pair trading of ETFs can be a more profitable alternative to stock pair trading in the highly efficient U.S. markets.

### Trading ETF Pairs (and Triplets)
- ETF pairs have an advantage over stock pairs as they are less likely to fall apart in out-of-sample data due to the slower-changing fundamental economics of a basket of stocks.
- ETF pairs can be found by identifying ETFs that are exposed to common economic factors, such as country ETFs or sector ETFs.
- Pairing a commodity ETF with an ETF of companies that produce that commodity can also be a profitable strategy.
- It is important to test for cointegration between the ETF pairs to ensure their long-term relationship.
- Trading mechanics for ETF pairs are similar to stock pairs, but the short-sale constraint and the small size of NBBO quotes can pose challenges.
- Forming hypotheses and testing them empirically is a scientific approach to adapt trading strategies when they stop working.

### Intraday Mean Reversion: Buy-on-Gap Model
- Trading stock pairs can be challenging due to the difficulty in finding consistently cointegrating pairs and the potential for changes in company fortunes.
- ETF pairs provide a more stable option for mean reversion trading as their exposure to common economic factors leads to greater cointegration.
- Pairing ETFs from the same country, sector, or commodity can yield profitable mean reversion strategies.
- Intraday mean reversion strategies can be effective for stocks, particularly during special conditions or periods.
- The "Buy-on-Gap" model is an intraday mean reversion strategy that focuses on stocks that gap down at the market open and gradually appreciate throughout the day.
- The strategy incorporates rules such as selecting stocks with low returns from the previous day's lows, requiring open prices higher than the 20-day moving average, and liquidating positions at the market close.
- Backtesting the strategy shows promising results, but it's important to consider the limitations of using open prices for trading signals and to account for short-sale constraints.
- Short-only versions of the strategy are also possible but may have steeper drawdowns and face additional challenges.
- The strategy highlights the presence of intraday mean reversion patterns that may not be evident with daily bars and emphasizes the importance of seasonality at short time scales.

### Cross-Sectional Mean Reversion: A Linear Long-Short Model
- Cross-sectional mean reversion strategies focus on short-term relative returns of stocks within a particular universe.
- The strategy aims to capitalize on the serial anti-correlation of relative returns, expecting underperforming stocks to subsequently outperform and vice versa.
- The strategy involves investing in every stock from a selected index with different capital allocations per stock based on its relative return compared to the average return of all stocks in the index.
- The portfolio is normalized to have a total gross capital of $1 each day.
- The strategy is implemented using linear long-short models on stocks, with weights calculated based on daily returns.
- Intraday variations of the strategy can be implemented by using the return from the previous close to today's open to determine the weights for entry at the open.
- Other variables, such as price-earnings (P/E) ratio, can be used to rank stocks and potentially enhance the strategy's returns.

## Chapter 5: Mean Reversion of Currencies and Futures
- Conventional wisdom suggests that currencies and futures are more suitable for momentum trading strategies.
- Opportunities for mean reversion strategies in currencies and futures are limited but still exist in certain situations.
- Futures calendar spreads, such as volatility futures versus stock index futures, can present mean reversion opportunities.
- Understanding mathematical models of futures prices helps in analyzing concepts like spot versus roll returns and backwardation versus contango.
- Trading currencies requires careful consideration of cointegration and ensuring consistent dollar value across currency pairs.
- Rollover interests can impact total returns and should be taken into account.
- The chapter will cover these nuances and provide specific strategies for mean reversion in currencies and futures.

### Trading Currency Cross-Rates
- Trading currency cross-rates involves finding countries with similar economic fundamentals.
- Currency pairs have advantages such as higher liquidity, leverage, no short-sale constraints, and around-the-clock trading.
- Basic terminology includes base currency (B) and quote currency (Q) in a cross-rate.
- Synthetic pairs may be used when direct trading of a cross-rate is not available, using a liquid base currency like USD or EUR.
- Realized profits and losses need to be converted regularly into the local currency for accurate tracking and comparison to backtest results.
- Different weighting schemes may be employed, such as trading separate pairs instead of a ready-made cross-rate.
- The Johansen test can be used to determine the optimal hedge ratio or capital weights between currency pairs.
- The returns of a portfolio with different currencies can be computed using common quote currency pairs, such as B1.USD - B2.USD.

### Rollover Interests in Currency Trading
- Rollover interests play a significant role in currency trading when positions are held overnight.
- Rollover interest is the interest differential between the two currencies in a cross-rate, and it can be a debit or credit interest depending on the interest rates of the currencies.
- Rollover interests are multiplied by the number of non-trading days when positions are held over weekends or holidays.
- When calculating the Sharpe ratio for currency trading strategies, the excess return needs to be considered by subtracting the financing cost (including rollover interest) from the returns of the positions.
- In backtesting, the rollover interest is added to the percentage change of the cross-rate to accurately reflect the returns.

### Trading Futures Calendar Spread
#### Roll Returns, Backwardation, and Contango
#### Do Calendar Spreads Mean-Revert?
### Futures Intermarket Spreads
#### Volatility Futures versus Equity Index Futures