Skip to content

What Backtests Hide

Investors crave certainty about future returns and backtests seem to provide that. Here's what they don't show.

By Adam Collins On 5 min read
Time scale of human history and the small slice where we have quality financial data.

Investors crave certainty about future returns and historical backtests seem to provide that. This post shows some of their limitations and two strategies that haven’t measured up to past performance.

Backtests distort our perception of what’s truly long term. Modern market history starts in the early 1900s. Putting Time In Perspective shows how short of a period that actually is. Everything we know about financial markets is in the light blue rectangle:

How small the past 100 years are compared to recorded human history since 3,500 BC.
Source: Wait But Why

We laugh at investing approaches used 100 years ago…

Old book quote attempting to connect stock market performance to the weather.
Source: The Theory of Stock Exchange Speculation

…and 100 years from now I’m sure people will laugh at some of today’s “evidence-based” strategies too.

But another century of data doesn’t guarantee future investors will be smarter. Financial markets aren’t a field where we can brute force our way to success with data and hard work. In 1969 we put a man on the moon. In 2020 we have much more information and computing power – but still have no clue where the S&P will close next week.

Another issue with backtests is that they can’t capture how popularity impacts performance:

Book quote about how market correlations can change depending on crowd behavior.
Source: The Crisis of Crowding

Capital is rewarded when it’s scarce and competition is low. In 1962 Buffett had $7 million to invest in cheap, overlooked companies. In 2020 there’s $980 billion in value mutual funds and $200 billion in value ETFs. It’s reasonable to assume performance degrades as competition increases and more people chase the same type of investment.

Here are two recent examples where backtests disappointed:

Value Investing

Value strategies tell a compelling story and appeal to the part of us that loves a good sale. The approach was a niche strategy in the 1950s, academics then wrote papers with terrific backtests, and eventually managers raised billions from hopeful investors.

The popularity shifted value away from its contrarian roots:

Tweet on how "the Berkshire meeting is a place to meet 40,000 self-described contrarians."

It’s probably not a coincidence traditional value approaches started underperforming after the seminal Fama-French paper was published:

Simple value investing strategies started underperforming after research was published.
Source: Econompic

Most value funds are still based on the price-to-book ratio. While informative for old economy stocks, it’s much less useful for tech companies and share repurchasers.

“Value investing is dead” is too extreme of a claim. But I wouldn’t hold out hope for the group of strategies that continue to define a company’s value with last century’s definition.

Trend Following

Portfolio hedging was massively in demand after 2008. Some investors turned to trend following, an approach that owns stocks when they’re going up and sells out when they’re going down. Who wouldn’t love this backtest?

Long-only trend following on US stocks helped protect downside in 2000 and 2008.
The trend following strategy shows returns of the Newfound U.S. Trend Equity Index. This index combines multiple trend following models into one and is representative of how the strategy as a whole has historically worked on U.S. stocks. Vanguard’s total market U.S. stock fund VTSMX is used for U.S. stocks. Dividends were reinvested for both strategies. No trading fees or taxes are reflected. The results are hypothetical results, are not an indicator of future results, and do not represent returns any investor actually earned.

Money poured into trend following and its related strategies:

Assets under management for trend funds went from $50 billion in 2006 to $300 billion in 2019.
Source: Financial Times

Recent performance has been lackluster:

Since 2008, long-only trend following on US stocks signaled multiple false positives.
The trend following strategy shows returns of the Newfound U.S. Trend Equity Index. This index combines multiple trend following models into one and is representative of how the strategy as a whole has historically worked on U.S. stocks. Vanguard’s total market U.S. stock fund VTSMX is used for U.S. stocks. Dividends were reinvested for both strategies. No trading fees or taxes are reflected. The results are hypothetical results, are not an indicator of future results, and do not represent returns any investor actually earned.

The main problem with trend strategies is that they’re vulnerable to the speed of a market crash. They work well in slower downturns like 2008, but aren’t fast enough to protect in quick drops like October 1987. This doesn’t mean trend following is broken, but it does mean the strategy can’t always reduce risk like some marketed it after 2008.


Ed Seykota said “everybody gets what they want out of the market.” Regardless of performance, some strategies will always deliver an emotional payoff. What sounds smarter:

  • “I’m in a hedge fund that uses proprietary models to observe 14 million data points per day. They charge 2%, but their backtested alpha has made up for that.”
  • “I pay Vanguard 0.03% for an index fund because I have no idea which stocks will do best.”

People go to Las Vegas and pay money to feel excited. Markets are similar. Complex strategies meet the constant investor desire to do something, regardless of if the activity adds value.

Book quote about how investors have an "inability to do nothing."
Source: Where are the Customers’ Yachts?

Adaptation vs. Perseverance

How can you tell if a strategy’s best days are in the past? I don’t know. But I do know it’s tricky from a manager’s perspective if they need to make a change.

The biggest career risk for a professional investor is being boxed into one type of strategy. Hugh Hendry ran a hedge fund that did great in 2008 and he developed a reputation for being pessimistic. In 2013 he changed his mind and became more bullish on risky assets. History proved him right but Hendry wasn’t rewarded:

Headline on Hugh Hendry closing his hedge fund after changing strategies.

He no longer fit the mold for what his investors wanted, even though a stubborn one-track manager isn’t what they needed.

I respect Hugh because his public reversal was so rare. When confronted with evidence that they’re wrong, most managers get defensive rather than question their assumptions.

Faced with the choice between changing one’s mind and proving that there is no need to do so, almost everyone gets busy on the proof. - John Kenneth Galbraith

For years I managed a strategy focused on downside protection. This spring I finished a research project on a simpler way to achieve that.

I had two choices:

Option #1 was to resist change and option #2 was to evolve as an investor.

Option #1 would have been easy in the short term. Option #2 is what I chose, even though it was tough to say to clients “That strategy I ran for years? There’s a simpler way to do the same thing.” The next few posts will cover what I found.

Summary

  • What we label as long-term market relationships are still relatively new.
  • Strategy popularity can impact performance.
  • Complex strategies sound smart because doing anything feels better than sitting still.