Why diversification doesn't work

You're standing in your canoe, on a beautiful Canadian lake, taking photos of the wildlife, occasionally fishing. Why standing, not sitting? Well, you've read about those disturbing studies that show how sitting is really bad for your long term health; how every hour of television viewing, for example, takes about 20 minutes off your life expectancy, and why the same is probably true for sitting at the computer, sitting reading a book, whatever. So you're standing and that's OK because you're balanced and stable, with your weight distributed uniformly.

Of course, anyone with even a few minutes of experience in a canoe knows this isn't as safe as it seems. What really matters isn't how well-balanced you are when the canoe rests peacefully, but what happens when a few waves come along, kicked up by rednecks passing in a souped-up bass trawler (I lived in rural Virginia for several years, so I know the experience). As you shift your stance to stay upright, and the boat shifts, that balanced distribution vanishes and you can easily tip. Stability demands balance in the midst of the boat's dynamics, not only in the static peace beforehand.

As it turns out, this same lesson applies to investment portfolios -- a new paper in Nature Scientific Reports shows just how important this insight may be.

Famously, of course, Harry Markowitz introduced the idea of diversification into investing back in the 1950s (at least he formalized the idea, which was probably around long before). Using information on the mathematical correlations between the returns of the different stocks in a portfolio, you can choose a weighted portfolio to minimize the overall portfolio of volatility for any expected return. This is maybe the most basic of all results in mathematical finance.

But it doesn't work; it suffers from the same problem as the balanced man in the canoe. This is clear from any number of studies over the past decade which show that the correlations between stocks change when markets move up or down. If the market suddenly plunges downward, you would hope that your well-diversified portfolio, invested as it is in stocks that tend to move unlike one another, would be OK. But when markets move significantly down (or up), it turns out, the correlations are no longer what they were. Trending markets induce strong correlations among stocks that aren't there beforehand, and aren't obvious from long-term averages. So the risks to a portfolio are actually much larger than the simple diversification analysis suggests -- just as the risk of a canoe tipping is much more than it seems to a man standing balanced on a peaceful lake.

The new paper by physicist Tobias Preis and colleagues makes this point with probably the largest data set used so far, looking at the stocks in the DJIA over about 70 years. It's a fairly simple analysis (modulo some nitty gritty details). Roughly, they look at the correlations between different stocks in the DJIA and see how these correlations depend on the recent average return of the DJIA. Are the correlations stable? Or do they go up as the market begins to move? The figure below showing the average correlation coefficient versus the return indicates that the result is clearly the latter: a trending market, in either direction, induces significant correlations among the DJIA stocks.

One of the interesting things here is that this link holds on many different timescales, from 10 days up through two months. The worrying thing for an investor, of course, is that these correlations make the risks of large losses significantly larger than they would appear to be on the basis of long-term correlations alone. As the authors conclude:

... a “diversification breakdown” tends to occur when stable correlations are most needed for portfolio protection. Our findings, which are qualitatively consistent with earlier findings^{42, 44} but quantitatively different, could be used to anticipate changes in mean correlation of portfolios when financial markets are suffering significant losses. This would enable a more accurate assessment of the risk of losses.

 As any canoeist knows, dynamics really matter.