Betting on back-tests? Analysing alternative risk premia

Duncan Moir, Senior Investment Manager - Aberdeen Standard Investments

Academic studies have helped investors better understand the fundamental drivers of returns. Some of these sources of return can be isolated into alternative risk premia.

Now a number of investment banks offer products that replicate these alternative risk premia strategies. This gives investors access to low-cost, liquid alternatives to traditional asset classes. As such they offer diversification potential. Typically these products come with compelling back-tested track records.

In practice, however, these products have not delivered returns consistent with their respective back-tested track records. This paper analyses a comprehensive set of alternative risk premia products from 11 investment banks to answer the questions: why not, and have the diversification benefits persisted in the live environment? Our analysis leads us to question the performance of alternative risk premia in different economic conditions.

What is the investment issue?

Decades of academic studies have improved our understanding of the fundamental drivers of investment returns. We understand, for example, that equity investors can expect above-market returns over the long term through exposure to different factors such as value, quality, momentum, small size and low volatility.

Systematic investment strategies are able to isolate many of these sources of return. This allows investors to construct portfolios that are diversified across a number of alternative risk premia. Similar systematic strategies can be applied to other asset classes and multi-asset strategies. Many can be implemented using liquid, traded instruments.

In order to access a full suite of alternative risk premia, the investor must have:

  • a reasonable level of market knowledge
  • appropriate trading systems
  • the appropriate infrastructure for trade settlement
  • collateral management systems
  • access to borrowing; and
  • risk management tools.

Investment banks are well placed to meet these needs. They have sponsored a large number of products that have been launched in recent years.

Formulaic, non-discretionary sets of rules determine investment selection and portfolio construction for these products. This allows the investment banks to present back-tested performance statistics to potential clients. Most products have short-lived track records, but impressive back-tested returns.

However, investors have discovered that the performance in the live environment rarely lives up to the back-test. The author’s analysis aimed to understand why, and whether the diversification benefits persisted in the live environment.

Which existing studies have addressed this issue?

Several research papers convey scepticism over publicised results on the simulated performance of investment strategies. Three studies capture the key issues.

McLean and Pontiff [2016] reviewed the performance of 97 strategies identified in academic research as offering alternative returns to the stock market. They found that 26% produced lower returns out-of-sample and 58% produced lower returns post-publication. They suggested that the excess return was lower once investors learned about a mispricing from academic publications.

Harvey et al. [2016] argue that extensive data mining means it does not make sense to use the usual criteria to establish the statistical significance of factors that claim to explain expected returns. The authors argued a much higher hurdle is required, and that most claimed research findings in financial economics are likely false.

Suhonen et al. [2016] carried out a similar analysis to our own. They analysed 215 alternative risk premia products offered by investment banks. They found that the Sharpe ratio fell from 1.2 during their respective back-test periods to 0.31 during live performance. It was this study that led us to ask: is this result fully explained by the problems of data mining, or could there be a more fundamental explanation? Also, what happened to diversification in the live environment compared to the back-test?

How does the author tackle the issue?

The author created a database of 747 alternative risk premia strategies, offered by 11 global investment banks. He screened the database to remove duplicate indices; those driven by the same underlying factor and only different in the nuances of portfolio construction. Where duplicates existed, the author selected the most liquid market, the most liquid security and/or the oldest strategy. Any strategy that was a combination of other strategies was removed. The sample was reduced from 747 strategies to 218 independent strategies.

The author carried out four analyses: two to examine performance, two to examine diversification.

Firstly he calculated the Sharpe ratio for each strategy, separating each between the back-test period and the live environment. Secondly he examined the average Sharpe ratio of the overall sample over time, from 1995 to 2018.

Thirdly he used statistical factor analysis – principle component analysis – to decompose the statistical drivers for the basket of risk premia strategies offered by each of the banks. This allows him to compare these drivers in the back-test environment to the live environment.

Different investment banks offer different numbers of strategies. To compare the results for their basket of strategies, he calculated a concentration ratio that was adjusted for the number of strategies. This analysis produces a measure of diversification for the basket of strategies offered by the banks in back-test and live environments.

Fourthly he compared concentration measures of the overall universe over time. He calculated the Meucci Diversification Ratio between 1991 and 2018 (the weighted sum of each strategy’s volatility divided by the total basket volatility) to analyse the change in diversification.

What were the findings?

The average Sharpe Ratio declined from 0.81 during the back-test to 0.20 in the live environment. This Sharpe ratio is below the 0.31 result found by Suhonen et al. in their 2016 study. While the data sets are different, this is in line with our expectations given the poor performance of most alternative risk premia in 2018.

Some 20% of strategies saw their Sharpe ratio increase in the live environment.

The average Sharpe ratio of the overall sample trended lower over the last decade. Therefore this deterioration began before the majority of the alternative risk premia were available as live indices.

The experience of investors has been that results failed to match those achieved in back-tests. This paper provides a more comprehensive debate on the use of over-fitting.

The principle component analysis produced very similar results in the back-test and live periods. Similarly, when the results are combined into a concentration ratio and compared across the 11 banks, there is no material difference between results in the back-test and live environments.

The Meucci Diversification ratio rose from a level close to the long-term average in 2009 to a record high in June 2015, declining steadily thereafter. At the end of the study, in July 2018, this measure was close to its long-term average.

What are the investment implications?

The deterioration in Sharpe ratio between the back-test environment and the live environment is consistent with previous studies. Therefore this is consistent with the theory that the deterioration can be explained – in part – by the influence of data mining.

The author’s own due diligence of investment bank risk premia strategies finds that the majority use simple, logical rules with few parameters. This reduces the potential influence of over-fitting.

The fact that the deterioration in Sharpe ratios began before many of the strategies went live suggest that data mining is not the only explanation for the decline. This leads us to question the cause of the deterioration. Other factors, such as different economic conditions, may influence the performance of risk premia.

A central premise of factor investing is that the factors must be robust, intuitive, persistent and supported by empirical evidence. Amenc et al. [2015] provide two definitions of robustness: relative robustness – the ability to produce similar performance in similar market conditions; and absolute robustness – the capacity to outperform regardless of market conditions.

We note that the value factor within equities has performed poorly over the last decade. This is a common factor within investment bank risk premia products. This suggests that some of the deterioration in Sharpe ratios may be due to cyclical factors rather than structural. In other words, the value factor may have delivered relative robustness in an adverse environment for the strategy. The cyclical influences on the performance of alternative risk premia is a worthy topic for further research.

The author found that the diversification benefits of investing in a diversified basket of alternative risk premia were robust, being largely unchanged between back-test and live results. The study found that the average Sharpe ratio remained positive in the live environment, albeit significantly lower than suggested by the results in back-tests. This leads the author to conclude that a portfolio of alternative risk premia, diversified across asset classes and strategies, can be expected to produce attractive risk adjusted returns.

Read the full paper

RISK WARNING

The value of investments, and the income from them, can go down as well as up and you may get back less than the amount invested.

The views and conclusions expressed in this communication are for general interest only and should not be taken as investment advice or as an invitation to purchase or sell any specific security.

Any data contained herein which is attributed to a third party ("Third Party Data") is the property of (a) third party supplier(s) (the "Owner") and is licensed for use by Standard Life Aberdeen**. Third Party Data may not be copied or distributed. Third Party Data is provided "as is" and is not warranted to be accurate, complete or timely.

To the extent permitted by applicable law, none of the Owner, Standard Life Aberdeen** or any other third party (including any third party involved in providing and/or compiling Third Party Data) shall have any liability for Third Party Data or for any use made of Third Party Data. Past performance is no guarantee of future results. Neither the Owner nor any other third party sponsors, endorses or promotes the fund or product to which Third Party Data relates.

**Standard Life Aberdeen means the relevant member of Standard Life Aberdeen group, being Standard Life Aberdeen plc together with its subsidiaries, subsidiary undertakings and associated companies (whether direct or indirect) from time to time.