AI Ran 916,000 Backtests — Here’s What It Discovered

During their recent episode, Jake Taylor and Tobias Carlisle discussed AI Ran 916,000 Backtests — Here’s What It Discovered. Here’s an excerpt from the episode:

[Tobias] I think it’s interesting in the context of backtesting. I was saying to Jake before we came on that I have lots of backtest data and I fed it all into Claude and then I asked Claude to run these various backtests and I gave it access to my backtesting software as well. Where I would run dozens maybe over the course of an afternoon, it ran 916,000 backtests testing 1,217 different parameters.

It’s a data mine.

[Jake] Yeah.

[Tobias] Then I said to it, it’s a data mine.

[Jake] You’ve been data mining with a spoon and now you’ve been given an earth mover. Right.

[Tobias] Exactly right. I have some rules for data mining. One of them is that you data mine in the first half of the set and then you test in the second half of the set.

That’s got its own problems as well, but it’s better than nothing. It’s very hard to find anything that persists from one set to the next. I think that’s kind of interesting, but it optimises something in the first set, it doesn’t work in the second set.

I was looking at it in terms of asset allocation because Ray Dalio has that view that you take uncovered later return streams and you get enough of them and then you spread your bets over those and there’s always something that’s working.

[Jake] Lever the bejesus out of it. That’s right.

[Tobias] And then there’s the Harry Brown who wrote fail safe investing and out of that comes the permanent portfolio. And he says that there are these four states of the world where you have prosperity, inflation, stagflation. Is that one of the states?

Yeah.

[Jake] Deflation with economic difficulties.

[Tobias] Monetary tightness?

[Jake] Yeah, maybe that’s it. Something like that.

[Tobias] And so his sort of match up with Dalio’s views of the world where he’s like growth is positive or negative and inflation positive or negative. And so those bad states of the world are negative growth and positive inflation. And if you take that mix and you can see the assets that they say that you need to combat.

So you need gold for inflation, you need small cap stocks, large cap stocks, global stocks. You can have T-bills, you can have 30 year bills, you can have anything in between. You’ve got this huge mix of assets that you can choose from.

And I just said to the machine, look from 1970 to 2000, I think, or whatever halfway through the data is 1998 or something like that. And tell me what the best mix is. And then have a look at that mix in the second half.

And so something goes from one. So there’s like 1,217 different mixes that you can have. Goes from one to like 900 in the second set.

And the ones that’s number one in the second set is like 800 in the first set. And then the one that fits through both, it’s 400 in the first set and 300 in the second set. So you’re a long way from…

And this is optimising for different things too. I can say optimise for sharp, optimise for absolute return, optimise for lowest drawdown. You can do all of these different…

That’s how you end up with 916,000 different tests because each one’s sort of running through all 1,200, optimising for something different every single time through two sets of data and then over the full set. So that’s why it comes up with like so many different things. And then you can ask it to look in different countries as well.

So there’s like 11 developed market countries that have got data going back that far. There’s no good answer.

[Jake] What does it say? Just smoke started coming out of your computer?

[Tobias] I mean, I can give you the answer that is the least worst of all of the options.

[Jake] Right. It’s got to be really unsatisfying.

[Tobias] Yeah. Yeah, it is. But it’s…

Harry Brown’s intuition… Harry Brown’s just put a quarter into bonds, gold, stocks, and something else. T-bills or something.

Cash, I guess.

[Jake] And that did just fine?

[Tobias] I mean, for just a naive, no idea of what’s going to happen, it does fine through the whole thing. It’s positive. It doesn’t have very many losing decades.

And that’s the other problem that you have is like in 1971, gold comes off the gold standard and then has this like explosion where it revalues to market value. Like is that representative of that decade of the seventies is where a lot of the gold bugs rest there. Case for gold.

And then you’ve got a 40 year falling rates from 1980 to 2020. Is that representative of what bonds are going to do? Where we’ve had this incredible bull market in the US for equities.

Is that representative of what global equities are going to look like? So even if you do all of this backtesting and you come up with an answer, is the answer meaningful?

[Jake] Well, and the sad reality is that the actual end of the things that really matter is still very meagre. It’s a low population sample of true catastrophic market drawdowns. Yeah, of course, there’s been lots of corrections, but actual ones that really, really mattered when you start tipping into like Bernstein’s deep risks, there just haven’t been that many of them.

And so to draw too many conclusions from a small sample size is bad stats, right?

[Tobias] Yeah, for sure. So after all that testing, that was sort of what I wanted to do a book on all of that testing. And that’s not this book, but this book is one of the things that I considered while I was doing it, which was, I looked at how to avoid the crashes for the most part.

And there’s basically the answer is there’s no way to do it satisfactorily. Just let me just give everybody the answer from the outset.

[Jake] All right, well, we’re done.

[Tobias] But I’ll tell you everything that I did consider.

[Jake] Yeah.

[Tobias] So I looked at valuation for one, so CAPE as a cyclically adjusted PE as a measure of CAPE’s got problems with this calculation, particularly in the current climate where earnings have raced so far ahead, like the implications of CAPE are that earnings are much, much lower than they really appear. But then it matches up to like Tobin’s Q and Buffett’s gross market capitalisation at GNP. So the answer’s like not that wrong.

Even the market above its own trend gives you roughly the same answer. Like you don’t need to do all of these sort of heroic valuation tricks to get to the answer that the market’s expensive anyway. And the implications of an expensive market, lower returns and a little bit of volatility in the future.

So then I looked at the 10-3 inversion, which is something that I talked about for a long time, from 22 to 24.

[Jake] Yeah, until it stopped working. Then we stopped talking about it. Well, it seemed to have such a good strike rate.

Yeah, it was like nine for nine, wasn’t it, up until this?

[Tobias] It had a great strike rate. And then it just didn’t work this time. Or the NBR didn’t declare leverage.

Or has it worked yet? Well, it’s getting to the point where you’re really stretching the definition to say… Because we un-inverted last 12 months ago, something like that.

And then, well, we went sideways for a little while. I don’t know. The un-inversion doesn’t seem to work.

[Jake] Well, I mean, do you put an asterisk on it when you have what are we doing? $2 trillion of deficit spending per year plus the AA CapEx boom that just maybe kind of kicked the can 12 months down the road? I don’t know.

Will historians go back and look and say like, well, yeah, that one still counts? Maybe. I don’t know.

I think the important maybe, tell me if this is true of what your findings were, is that none of these are useful much for timing.

[Tobias] Nothing’s useful for timing. Valuation is not a useful timing tool at all. Valuation doesn’t work as a timing tool.

There’s no timing tool. The other one I was going to say is the equity risk premium, which we talk about a little bit, which is the surplus of earnings the 10-year government treasury. And where that’s negative, stocks are expensive.

Where it’s positive, then there’s a margin of stocks over risk-free rate. And it’s not predictive at all. But I like it because when you do a valuation, like you’re pulling in, I pull in the 10-year as part of my valuation.

It’s a discount rate. And it’s a threshold. You should be able to earn more than the 10-year over a reasonably foreseeable period of time.

Otherwise, you should be in the 10-year. That would be a much simpler thing to do. But it doesn’t work.

It’s not predictive at all. The only thing that predicts is the absolute PE or the absolute earnings yield. High earnings yield leads to pretty good returns.

Low earnings yield leads to pretty poor returns across a big data set. It obviously doesn’t apply in every single nine.

[Jake] How much time, on average, do you need to arrive at that conclusion? Is it seven years after, 10 years after?

[Tobias] 10 years. It’s a long period of time. It’s not predictive.

[Jake] I have seen the… It was CAPE ratio, and it showed subsequently one year after, it’s just a scatterplot, just total mess cloud. There’s nothing to it.

And then I forget if it was three or five year, it starts to… The dots start to line up into a trend that looks like, okay, yeah, high CAPE, low five-year forward return. And then you get to, I think, 10 years, and it’s still okay.

I think if I remember right. But then 15-year and 20-year, it’s over again. And the line flattens back out because basically it probably has more to do with the business quality than it did the entry valuation or something.

[Tobias] Well, I thought it was most predictive at 20 years, but that point remains that that’s a very At some point, starting valuation matters less after 20 years hence. So I looked at all of those tools for like… But then there’s different types of crashes too.

So I went through all the crashes since 1929, over the last 100 years. So 29, who really knows that maybe that’s an outlier, kind of like a strange… There’s less sophisticated market in a very different way.

But then there’s… 1987 crash is very rapid. 2000s of valuation question, so expensive.

We’ve never seen that as expensive as that before. Then 2008-9, it’s like a financial crisis. Not necessarily valuation, more financial crisis.

So it wasn’t like it was expensive on a cape rate basis, but it was not as expensive as 2000. And then we’re back to 22, which was 2020, which is like a crash, a genuine crash and recovery, but it was over very quickly. 22 was a much slower grinding bear market.

[Jake] Not even a full year’s worth. So none of these sound like they match each other. It’s all like one off.

[Tobias] They’re all totally different.

[Jake] All totally different.