Private information and jumps in the market

Following my second recent post on what moves the markets, two readers posted interesting and noteworthy comments and I'd like to explore them a little. I had presented evidence in the post that many large market movements do not appear to be linked to the sudden arrival of public information in the form of news. Both comments noted that this may leave out of the picture another source of information -- private information brought into the market through the action of traders taking actions:

Anonymous said...

I don't see any mention of what might be called "trading" news, e.g. a large institutional investor or hedge fund reducing significantly its position in a given stock for reasons unrelated to the stock itself - or at least not synchronized with actual news on the underlying. The move can be linked to internal policy, or just a long-term call on the company which timing has little to do with market news, or lags them quite a bit (like an accumulation of bad news leading to a lagged reaction, for instance). These shocks are frequent even on fairly large cap stocks. They also tend to have lingering effect because the exact size of the move is never disclosed by the investor and can spread over long periods of time (i.e. days), which would explain the smaller beta. Yet this would be a case of "quantum correction", both in terms of timing and agent size, rather than a breakdown of the information hypothesis.

 and...

DR said...

Seconding the previous comment, asset price information comes in a lot more forms than simply "news stories about company X." All market actions contains information. Every time a trade occurs there's some finite probability that it's the action of an informed trader. Every time the S&P moves its a piece of information on single stock with non-zero beta. Every time the price of related companies changes it contains new information.

Both of these comments note the possibility that every single trade taking place in the market (or at least many of them) may be revealing some fragment of private information on the part of whoever makes the trade. In principle, it might be such private information hitting the market which causes large movements (the s-jumps described in the work of Joulin and colleagues).  

I think there are several things to note in this regard. The first is that, while this is a sensible and plausible idea, it shouldn't be stretched too far. Obviously, if you simply assume that all trades carry information about fundamentals, then the EMH -- interpreted in the sense that "prices move in response to new information about fundamentals" -- essentially becomes true by definition. After all, everyone agrees that trading drives markets. If all trading is assumed to reveal information, then we've simple assumed the truth of the EMH. It's a tautology.

More useful is to treat the idea as a hypothesis requiring further examination. Certainly some trades do reveal private information, as when a hedge fund suddenly buys X and sells Y, reflecting a belief based on research that Y is temporarily overvalued relative to X. Equally, some trades (as mentioned in the first comment) may reveal no information, simply being carried out for reasons having nothing to do with the value of the underlying stock. As there's no independent way -- that I know of -- to determine if a trade reveals new information or not, we're stuck with a hypothesis we cannot test.

But some research has tried to examine the matter from another angle. Again, consider large price movements -- those in the fat-tailed end of the return distribution. One proposed idea looking to private information as a cause proposes that large price movements are caused primarily by large-volume trades by big players such as hedge funds, mutual funds and the like. Some such trades might reveal new information, and some might not, but let's assume for now that most do. In a paper in Nature in 2003, Xavier Gabaix and colleagues argued that you can explain the precise form of the power law tail for the distribution of market returns -- it has an exponent very close to 3 -- from data showing that the size distribution of mutual funds follows a similar power law with an exponent of 1.05. A key assumption in their analysis is that the price impact Δp generated by a trade of volume V is roughly equal to Δp = kV^1/2.

This point of view seems to support the idea that the arrival of new private information, expressed in large trades, might account for the no-news s jumps noted in the Jouvin study. (It seems less plausible that such revealed information might account for anything as violent as the 1987 crash, or the general meltdown of 2008). But taken at face value, these arguments at least seem to be consistent with the EMH view that even many large market movements reflect changes in fundamentals. But again, this assumes that all or at least most large volume trades are driven by private information on fundamentals, which may not be the case. The authors of this study themselves don't make any claim about whether large volume trades really reflect fundamental information. Rather, they note that...

Such a theory where large individual participants move the market is consistent with the evidence that stock market movements are difficult to explain with changes in fundamental values... 

But more recent research (here and here, for example) suggest that this explanation doesn't quite hang together because the assumed relationship between large returns and large volume trades isn't correct. This analysis is fairly technical, but is based on the study of minute-by-minute NASDAQ trading and shows that, if you consider only extreme returns or extreme volumes, there is no general correlation between returns and volumes. The correlation assumed in the earlier study may be roughly correct on average, but it not true for extreme events. "Large jumps," the authors conclude, "are not induced by large trading volumes."

Indeed, as the authors of these latter studies point out, people who have valuable private information don't want it to be revealed immediately in one large lump because of the adverse market impact this entails (forcing prices to move against them). A well-known paper by Albert Kyle from 1985 showed how an informed trader with valuable private information, trading optimally, can hide his or her trading in the background of noisy, uninformed trading, supposing it exists. That may be rather too much to believe in practice, but large trades do routinely get broken up and executed as many small trades precisely to minimize impact. 

All in all, then, it seems we're left with the conclusion that public or private news does account for some large price movements, but cannot plausibly account for all of them. There are other factors. The important thing, again, is to consider what this means for the most meaningful sense of the EMH, which I take to be the view that market prices reflect fundamental values fairly accurately (because they have absorbed all relevant information and processed it correctly). The evidence suggests that prices often move quite dramatically on the basis of no new information, and that prices may be driven as a result quite far from fundamental values.

The latter papers do propose another mechanism as the driver of routine large market movements. This is a more mechanical process centering on the natural dynamics of orders in the order book. I'll explore this in detail some other time. For now, just a taster from this paper, which describes the key idea:

So what is left to explain the seemingly spontaneous large price jumps? We believe that the explanation comes from the fact that markets, even when they are ‘liquid’, operate in a regime of vanishing liquidity, and therefore are in a self-organized critical state [31]. On electronic markets, the total volume available in the order book is, at any instant of time, a tiny fraction of the stock capitalisation, say 10⁻⁵ −10⁻⁴ (see e.g. [15]). Liquidity providers take the risk of being “picked off”, i.e. selling just before a big upwards move or vice versa, and therefore place limit orders quite cautiously, and tend to cancel these orders as soon as uncertainty signals appear. Such signals may simply be due to natural fluctuations in the order flow, which may lead, in some cases, to a catastrophic decay in liquidity, and therefore price jumps. There is indeed evidence that large price jumps are due to local liquidity dry outs.

Markets are rational even if they're irrational

I promise very soon to stop beating on the dead carcass of the efficient markets hypothesis (EMH). It's a generally discredited and ill-defined idea which has done a great deal, in my opinion, to prevent clear thinking in finance. But I happened recently on a defense of the EMH by a prominent finance theorist that is simply a wonder to behold -- its logic a true empirical testament to the powers of human rationalization. It also illustrates the borderline Orwellian techniques to which diehard EMH-ers will resort to cling to their favourite idea.

The paper was written in 2000 by Mark Rubinstein, a finance professor at University of California, Berkeley, and is entitled "Rational Markets: Yes or No. The Affirmative Case." It is Rubinstein's attempt to explain away all the evidence against the EMH, from excess volatility to anomalous predictable patterns in price movements and the existence of massive crashes such as the crash of 1987. I'm not going to get into too much detail, but will limit myself to three rather remarkable arguments put forth in the paper. They reveal, it seems to me, the mind of the true believer at work:

1. Rubinstein asserts that his thinking follows from what he calls The Prime Directive. This commitment is itself interesting:

When I went to financial economist training school, I was taught The Prime Directive. That is, as a trained financial economist, with the special knowledge about financial markets and statistics that I had learned, enhanced with the new high-tech computers, databases and software, I would have to be careful how I used this power. Whatever else I would do, I should follow The Prime Directive:

Explain asset prices by rational models. Only if all attempts fail, resort to irrational investor behavior.

One has the feeling from the burgeoning behavioralist literature that it has lost all the constraints of this directive – that whatever anomalies are discovered, illusory or not, behavioralists will come up with an explanation grounded in systematic irrational investor behavior.

Rubinstein here is at least being very honest. He's going to jump through intellectual hoops to preserve his prior belief that people are rational, even though (as he readily admits elsewhere in the text) we know that people are not rational. Hence, he's going to approach reality by assuming something that is definitely not true and seeing what its consequences are. Only if all his effort and imagination fails to come up with a suitable scheme will he actually consider paying attention to the messy details of real human behaviour.

What's amazing is that, having made this admission, he then goes on to criticize behavioural economists for having found out that human behaviour is indeed messy and complicated:

The behavioral cure may be worse than the disease. Here is a litany of cures drawn from the burgeoning and clearly undisciplined and unparsimonious behavioral literature:

Reference points and loss aversion (not necessarily inconsistent with rationality):
Endowment effect: what you start with matters
Status quo bias: more to lose than to gain by departing from current situation
House money effect: nouveau riche are not very risk averse

Overconfidence:
Overconfidence about the precision of private information
Biased self-attribution (perhaps leading to overconfidence)
Illusion of knowledge: overconfidence arising from being given partial information
Disposition effect: want to hold losers but sell winners
Illusion of control: unfounded belief of being able to influence events

Statistical errors:
Gambler’s fallacy: need to see patterns when in fact there are none
Very rare events assigned probabilities much too high or too low
Ellsberg Paradox: perceiving differences between risk and uncertainty
Extrapolation bias: failure to correct for regression to the mean and sample size
Excessive weight given to personal or antidotal experiences over large sample statistics
Overreaction: excessive weight placed on recent over historical evidence
Failure to adjust probabilities for hindsight and selection bias

Miscellaneous errors in reasoning:Violations of basic Savage axioms: sure-thing principle, dominance, transitivity
Sunk costs influence decisions
Preferences not independent of elicitation methods
Compartmentalization and mental accounting
“Magical” thinking: believing you can influence the outcome when you can’t
Dynamic inconsistency: negative discount rates, “debt aversion”
Tendency to gamble and take on unnecessary risks
Overpricing long-shots
Selective attention and herding (as evidenced by fads and fashions)
Poor self-control
Selective recall
Anchoring and framing biases
Cognitive dissonance and minimizing regret (“confirmation trap”)
Disjunction effect: wait for information even if not important to decision
Time-diversification
Tendency of experts to overweight the results of models and theories
Conjunction fallacy: probability of two co-occurring more probable than a single one

Many of these errors in human reasoning are no doubt systematic across individuals and time, just as behavioralists argue. But, for many reasons, as I shall argue, they are unlikely to aggregate up to affect market prices. It is too soon to fall back to what should be the last line of defense, market irrationality, to explain asset prices. With patience, the anomalies that appear puzzling today will either be shown to be empirical illusions or explained by further model generalization in the context of rationality.

Now, there's sense in the idea that, for various reasons, individual behavioural patterns might not be reflected at the aggregate level. Rubinstein's further arguments on this point aren't very convincing, but at least it's a fair argument. What I find more remarkable is the a priori decision that an explanation based on rational behaviour is taken to be inherently superior to any other kind of explanation, even though we know that people are not empirically rational. Surely an explanation based on a realistic view of human behaviour is more convincing and more likely to be correct than one based on unrealistic assumptions (Milton Friedman's fantasies notwithstanding). Even if you could somehow show that market outcomes are what you would expect if people acted as if they were rational (a dubious proposition), I fail to see why that would be superior to an explanation which assumes that people act as if they were real human beings with realistic behavioural quirks, which they are.

But that's not how Rubinstein sees it. Explanations based on a commitment to taking real human behaviour into account, in his view, have "too much of a flavor of being concocted to explain ex-post observations – much like the medievalists used to suppose there were a different angel providing the motive power for each planet." The people making a commitment to realism in their theories, in other words, are like the medievalists adding epicycles to epicycles. The comparison would seem more plausibly applied to Rubinstein's own rational approach.

2. Rubinstein also relies on the wisdom of crowds idea, but doesn't at all consider the many paths by which a crowd's average assessment of something can go very much awry because individuals are often strongly influenced in their decisions and views by what they see others doing. We've known this going all the way back to the famous 1950s experiments of Solomon Asch on group conformity. Rubinstein pays no attention to that, and simply asserts that we can trust that the market will aggregate information effectively and get at the truth, because this is what group behaviour does in lots of cases:

The securities market is not the only example for which the aggregation of information across different individuals leads to the truth. At 3:15 p.m. on May 27, 1968, the submarine USS Scorpion was officially declared missing with all 99 men aboard. She was somewhere within a 20-mile-wide circle in the Atlantic, far below implosion depth. Five months later, after extensive search efforts, her location within that circle was still undetermined. John Craven, the Navy’s top deep-water scientist, had all but given up. As a last gasp, he asked a group of submarine and salvage experts to bet on the probabilities of different scenarios that could have occurred. Averaging their responses, he pinpointed the exact location (within 220 yards) where the missing sub was found. 

Now I don't doubt the veracity of this account or that crowds, when people make decisions independently and have no biases in their decisions, can be a source of wisdom. But it's hardly fair to cite one example where the wisdom of the crowd worked out, without acknowledging the at least equally numerous examples where crowd behaviour leads to very poor outcomes. It's highly ironic that Rubinstein wrote this paper just as the dot.com bubble was collapsing. How could the rational markets have made such mistaken valuations of Internet companies? It's clear that many people judge values at least in part by looking to see how others were valuing them, and when that happens you can forget the wisdom of the crowds.

Obviously I can't fault Rubinstein for not citing these experiments  from earlier this year which illustrate just how fragile the conditions are under which crowds make collectively wise decisions, but such experiments only document more carefully what has been obvious for decades. You can't appeal to the wisdom of crowds to proclaim the wisdom of markets without also acknowledging the frequent stupidity of crowds and hence the associated stupidity of markets.

3. Just one further point. I've pointed out before that defenders of the EMH in their arguments often switch between two meanings of the idea. One is that the markets are unpredictable and hard to beat, the other is that markets do a good job of valuing assets and therefore lead to efficient resource allocations. The trick often employed is to present evidence for the first meaning -- markets are hard to predict -- and then take this in support of the second meaning, that markets do a great job valuing assets. Rubinstein follows this pattern as well, although in a slightly modified way. At the outset, he begins making various definitions of the "rational market":

I will say markets are maximally rational if all investors are rational.

This, he readily admits, isn't true:

Although most academic models in finance are based on this assumption, I don’t think financial economists really take it seriously. Indeed, they need only talk to their spouses or to their brokers.

But he then offers a weaker version:

... what is in contention is whether or not markets are simply rational, that is, asset prices are set as if all investors are rational.

In such a market, investors may not be rational, they may trade too much or fail to diversify properly, but still the market overall may reflect fairly rational behaviour:

In these cases, I would like to say that although markets are not perfectly rational, they are at least minimally rational: although prices are not set as if all investors are rational, there are still no abnormal profit opportunities for the investors that are rational.

This is the version of "rational markets" he then tries to defend throughout the paper. Note what has happened: the definition of the rational market has now been weakened to only say that markets move unpredictably and give no easy way to make a profit. This really has nothing whatsoever to do with the market being rational, and the definition would be improved if the word "rational" were removed entirely. But I suppose readers would wonder why he was bothering if he said "I'm going to defend the hypothesis that markets are very hard to predict and hard to beat" -- does anyone not believe that? Indeed, this idea of a "minimally rational"  market is equally consistent with a "maximally irrational" market. If investors simply flipped coins to make their decisions, then there would also be no easy profit opportunities, as you'd have a truly random market.

Why not just say "the markets are hard to predict" hypothesis? The reason, I suspect, is that this idea isn't very surprising and, more importantly, doesn't imply anything about markets being good or accurate or efficient. And that's really what EMH people want to conclude -- leave the markets alone because they are wonderful information processors and allocate resources efficiently. Trouble is, you can't conclude that just from the fact that markets are hard to beat. Trying to do so with various redefinitions of the hypothesis is like trying to prove that 2 = 1. Watching the effort, to quote physicist John Bell in another context, "...is like watching a snake trying to eat itself from the tail. It becomes embarrassing for the spectator long before it becomes painful for the snake."

What moves the markets? Part II

High frequency trading makes for markets that produce enormous volumes of data. Such data make it possible to test some of the old chestnuts of market theory -- the efficient markets hypothesis, in particular -- more carefully than ever before. Studies in the past few years show quite clearly, it seems to me, that the EMH is very seriously misleading and isn't really even a good first approximation.

Let me give a little more detail. In a recent post I began a somewhat leisurely exploration of considerable evidence which contradicts the efficient markets idea. As the efficient markets hypothesis (the "weak" version, at least) claims, market prices fully reflect all publicly available information. When new information becomes available, prices respond. In the absence of new information, prices should remain more or less fixed.

Striking evidence against this view comes from studies (now almost ten or twenty years old) showing that markets often make quite dramatic movements even in the absence of any news. I looked at some older studies along these lines in the last post, but stronger evidence comes from studies using electronic news feeds and high-frequency stock data. Are sudden jumps in prices in high frequency markets linked to the arrival of new information, as the EMH says? In a word -- no!

The idea in these studies is to look for big price movements which, in a sense, "stand out" from what is typical, and then see if such movements might have been caused by some "news". A good example is this study by Armand Joulin and colleagues from 2008. Here's how they proceeded. Suppose R(t) is the minute by minute return for some stock. You might take the absolute value of these returns, average them over a couple hours and use this as a crude measure -- call it σ -- of the "typical size" of one-minute stock movements over this interval. An unusually big jump over any minute-long interval will be one for which the magnitude of R is much bigger than σ. 

To make this more specific, Joulin and colleagues defined "s jumps" as jumps for which the ratio |R/σ| > s. The value of s can be 2 or 10 or anything you like. You can look at the data for different values of s, and the first thing the data shows -- and this isn't surprising -- is a distinctive pattern for the probability of observing jumps of size s. It falls off with increasing s, meaning that larger jumps are less likely, and the mathematical form is very simple -- a power law with P(s) being proportional to s^-4, especially as s becomes large (from 2 up to 10 and beyond). This is shown in the figure below (the upper curve):

This pattern reflects the well known "fat tailed" distribution of market returns, with large returns being much more likely than they would be if the statistics followed a Gaussian curve. Translating the numbers into daily events, s jumps of size s = 4 turn out to happen about 8 times each day, while larger jumps of s = 8 occur about once every day and one-half (this is true for each stock).

Now the question is -- are these jumps linked to the announcement of some new information? To test this idea, Joulin and colleagues looked at various news feeds including feeds from Dow Jones and Reuters covering about 900 stocks. These can be automatically scanned for mention of any specific company, and then compared to price movements for that company. The first thing they found is that, on average, a new piece of news arrives for a company about once every 3 days. Given that a stock on average experiences one jump every day and one-half, this immediately implies an imbalance between the number of stock movements and the number of news items. There's not enough news to cause the jumps observed. Stocks move -- indeed, jump -- too frequently.

Conclusion: News sometimes but not always causes market movements, and significant market movements are sometimes but not always caused by news. The EMH is wrong, unless you want to make further excuses that there could have been news that caused the movement, and we just don't recognize it or haven't yet figured out what it is. But that seems like simply positing the existence of further epicycles.

But another part of the Joulin et al. study is even more interesting. Having found a way to divide price jumps into two categories: A) those caused by news (clearly linked to some item in a news feed) and B) those unrelated to any news, it is then possible to look for any systematic differences in the way the market settled down after such a jump. The data show that the volatility of prices, just after a jump, becomes quite high; it then relaxes over time back to the average volatility before the jump. But the relaxation works differently depending on whether the jump was of type A or B: caused by news or not caused by news. The figure below shows how the volatility relaxes back to the norm first for jumps linked to news, and second to jumps not linked to news. The later shows a much slower relaxation:

As the authors comment on this figure,

In both cases, we find (Figure 5) that the relaxation of the excess-volatility follows a power-law in time σ(t) − σ(∞) ∝ t− β (see also [22, 23]). The exponent of the decay is, however, markedly different in the two cases: for news jumps, we find β ≈ 1, whereas for endogenous jumps one has β ≈ 1/2. Our results are compatible with those of [22], who find β ≈ 0.35.

Of course, β ≈ 1/2 implies a much slower relaxation back to the norm (whatever that is!) than does β ≈ 1. Hence, it seems that the market takes a longer time to get back to normal after a no-news jump, whereas it goes back to normal quite quickly after a news-related jump.

No one knows why this should be, but Joulin and colleagues made the quite sensible speculation that a jump clearly related to news is not really surprising, and certainly not unnerving. It's understandable, and traders and investors can decide what they think it means and get on with their usual business. In contrast, a no-news event -- think of the Flash Crash, for example -- is very different. It is a real shock and presents a lingering unexplained mystery. It is unnerving and makes investors uneasy. The resulting uncertainty registers in high volatility.

What I've written here only scratches the surface of this study. For example, one might object that lots of news isn't just linked to the fate of one company, but pertains to larger macroeconomic factors. It may not even mention a specific company but point to a likely rise in the prices of oil or semiconductors, changes influencing whole sectors of the economy and many stocks all at once. Joulin and colleagues tried to take this into account by looking for correlated jumps in the prices of multiple stocks, and indeed changes driven by this kind of news do show up quite frequently. But even accounting for this more broad-based kind of news, they still found that a large fraction of the price movements of individual stocks do not appear to be linked to anything coming in through news feeds. As they concluded in the paper:

Our main result is indeed that most large jumps... are not related to any broadcasted news, even if we extend the notion of ‘news’ to a (possibly endogenous) collective market or sector jump. We find that the volatility pattern around jumps and around news is quite different, confirming that these are distinct market phenomena [17]. We also provide direct evidence that large transaction volumes are not responsible for large price jumps, as also shown in [30]. We conjecture that most price jumps are in fact due to endogenous liquidity micro-crises [19], induced by order flow fluctuations in a situation close to vanishing outstanding liquidity.

Their suggestion in the final sentence is intriguing and may suggest the roots of a theory going far beyond the EMH. I've touched before on early work developing this theory, but there is much more to be said. In any event, however, data emerging from high-frequency markets backs up everything found before -- markets often make violent movements which have no link to news. Markets do not just respond to new information. Like the weather, they have a rich -- and as yet mostly unstudied -- internal dynamics.

What's Efficient About the Efficient Markets Hypothesis?

The infamous Efficient Markets Hypothesis (EMH) has been the subject of rancorous and unresolved debate for decades. It's often used to assert that markets don't need regulation or oversight because they have a remarkable power to get prices just about right (stocks, bonds and other assets have their correct "fundamental values"), and so never get too much out of balance. Somehow the idea still gets lots of attention even after the recent crisis. Financial Times columnist Tom Harford recently suggested that the EMH gets some things right (markets are "mostly efficient") even if it is also supports unjustified faith in market stability. In a talk, economist George Akerlof took on the question of whether the EMH can be seen to have caused the crisis, and concludes that yes, it could, although there are plenty of other causes as well.

Others have defended the EMH as being unfairly maligned. Jeremy Siegel, for example, argues that the EMH actually doesn't imply anything about prices being right, and insists that, recent dramatic evidence to the contrary, "our economy is inherently more stable" than it was before -- precisely because of modern financial engineering and the wondrous ability of markets to aggregate information into prices. Robert Lucas asserted much the same thing in The Economist, as did Alan Greenspan in the Financial Times. Lucas asserted his view (equivalent to the EMH) that the market really does know best:

The main lesson we should take away from the EMH for policy making purposes is the futility of trying to deal with crises and recessions by finding central bankers and regulators who can identify and puncture bubbles. If these people exist, we will not be able to afford them.

That debate over the EMH persists half century after it was first stated seems to reflect tremendous confusion and disagreement over what the hypothesis actually asserts. As Andrew Lo and Doyne Farmer noted in a paper from a decade ago, it's not actually a well-defined hypothesis that would permit clear and objective testing:

One of the reasons for this state of affairs is the fact that the EMH, by itself, is not a well posed and empirically refutable hypothesis. To make it operational, one must specify additional structure: e.g., investors’ preferences, information structure, etc. But then a test of the EMH becomes a test of several auxiliary hypotheses as well, and a rejection of such a joint hypothesis tells us little about which aspect of the joint hypothesis is inconsistent with the data.

So what does the EMH assert?

In trying to bring some order to the topic, one useful technique is to identify distinct forms of the hypothesis reflecting different shades of meaning frequently in use. This was originally done in 1970 by Eugene Fama, who introduced a "weak" form, a "semi-strong" form and a "strong" form of the hypothesis. Considering these in turn is useful, and helps to expose a rhetorical trick -- a simple bait and switch -- that defenders of the EMH (such as those mentioned above) often use. One version of the EMH makes an interesting claim -- that markets always work very efficiently (and rapidly) in bringing information to bear on prices which therefore take on accurate values. This (as we'll see below) is clearly false. Another version makes the uninteresting and uncontroversial claim that markets are hard to predict. The rhetorical trick is to mix these two in argument and to defend the interesting one by giving evidence for the uninteresting one. In his Economist article, for example, Lucas cites as evidence for information efficiency the fact that markets are hard to predict, when these are very much not the same thing.

Let's look at this in a little more detail. The Weak form of the EMH merely asserts that asset prices fluctuate in a random way so that there's no information in past prices which can be used to predict future prices. As it is, even this weak form appears to be definitively false if it is taken to apply to all asset prices. In their 1999 book A Non-random Walk Down Wall St, Andrew Lo and Craig MacKinley documented a host of predictable patterns in the movements of stocks and other assets. Many of these patterns disappeared after being discovered -- presumably because some market agents began trading on these strategies -- but there existence for a short time proves that markets have some predictability.

Other studies document the same thing in other ways. The simplest argument for the randomness of market movements is that any patterns that exist should be exploited by market participants to make profits. The trading they do should act to remove these patterns. Is this true? Take a look at Figure 1 below, taken from a paper from 2008 by Doyne Farmer and John Geanakoplos. Back in the 1970s, Farmer and others at a financial firm called The Prediction Company identified numerous market signals they could use to try to predict market movements in the future. The figure shows the correlation between one such trading signal and market prices two weeks in advance, calculated from data over a 23 year period. In 1975, this correlation was as high as 15%, and it was still persisting at a level of roughly 5% as of 2008. This signal -- I don't know what it is, as it is a proprietary signal of The Prediction Company -- has long been giving reliable advance information on market movements.

One might try to argue that this data shows that the pattern is indeed gradually being wiped out, but this is hardly anything like the rapid or "nearly instantaneous" action generally supposed by efficient market enthusiasts. Indeed, there's not much reason to think this pattern will be entirely wiped out for another 50 years.

This persisting memory in price movements can also be analyzed more systematically. Physicist Jean-Philippe Bouchaud and colleagues from the hedge fund Capital Fund management have explored the subtle nature of how new market orders arrive in the market and initiate trades. A market order is a request by an investor to either buy or sell a certain volume of an asset. In the view of the EMH, these orders should arrive in markets at random, driven by the randomness of arriving news. If one piece of news is positive for some stock, influencing someone to place a market buy order, there's no reason to expect that the next piece of news is therefore more likely also to be positive and to trigger another. So there shouldn't be any observed correlation in the times when buy or sell orders enter the market. But there is.

What Bouchaud and colleagues found (originally in 2003, but improved on since then) is that the arrivals of these order are correlated and remain so over very long times -- even over months. This means that the sequence of buy or sell market orders isn't at all just a random signal, but is highly predictable. As Bouchaud writes in a recent and beautifully written review: "Conditional on observing a buy trade now, one can predict with a rate of success a few percent above 1/2 that the sign of the 10,000th trade from now (corresponding to a few days of trading) will be again positive."

Hardly the complete unpredictability claimed by EMH enthusiasts. To look at just one more piece of evidence -- from a very long list of possibilities -- we might take an example discussed recently by Gavyn Davies in the Financial Times. He refers to a study by Andrew Haldane of the Bank of England. As Davies writes,

Andy Haldane conducts the following experiment. He estimates the results of an investment strategy in US equities which is based entirely on the past direction of the stockmarket. If the market rises in the period just ended, the strategy buys stocks for the next period, and vice versa. In other words, the strategy simply extrapolates the recent trend in the market. The result? According to Andy, if you had been wise enough to start this procedure with $1 in 1880, you would have consistently shifted in and out of stocks at the right times, and you would now possess over $50,000. Not bad for a strategy which could have been designed in a kindergarten.

Next, Andy tries an alternative strategy based on value. This calculates whether the stockmarket is fundamentally over or undervalued, and buys the market only when value gives a positive signal. The criterion for measuring value is the dividend discount model, first devised by Robert Shiller. If you had been clever enough to devise this measure of value investing in 1880, and had invested $1 at the time, the procedure would have left you with a portfolio now worth the princely sum of 11 cents.

That, according to the weak version of the EMH, shouldn't be possible.

If weakened still further you might salvage some form of the weak hypothesis by saying that "most or many asset prices are difficult to predict," which seems to be true. We might call this the Absurdly Weak form of the EMH, and it seems ridiculous to form such a puffed-up "hypothesis" at all. Does anyone doubt that markets are hard to predict?

But the more serious point with regard to the weak (or absurdly weak) forms of the EMH is that the word "efficient" really has no business being present at all. This word seems to go back to a famous paper by Paul Samuelson, the originator (along with Eugene Fama) of the EMH, who established that prices should fluctuate randomly and be impossible to predict in a market that is "informationally efficient," i.e. in which participants bring all possible information to bear in trying to anticipate the future. If such efficient information processing goes on in the market, then prices will fluctuate randomly. Informational efficiency is what Lucas and others claim the market does, and they take the difficulty of predicting markets as evidence. But it is not, in fact, evidence of anything of the sort.

Think carefully about this. The statement that information efficiency implies random price movements in no way implies the opposite -- that random price movements imply that information is being processed efficiently, although many people seem to want to draw this conclusion. Just suppose (to illustrate the point) that investors in some market make their decisions to buy and sell by flipping coins. Their actions would bring absolutely no information into the market, yet prices would fluctuate randomly and the market would be hard to predict. It would be far better and more honest to call the weak form of the EMH the Random Market Hypothesis or the Market Unpredictability Hypothesis. It is strictly speaking false, as we just noted, although still a useful, crude first approximation. It's about as true as it is to say that water doesn't flow uphill. Yes, mostly, but then, ordinary waves do it at the seaside every day.

So the weak version of the EMH isn't very useful. Perhaps it has some value in dissuading casual investors from thinking it ought to be easy to beat the market, but it's more metaphor than science.

Next up is the "semi-strong" version of the EMH. This asserts that the prices of stocks or other assets (in the market under consideration) reflect all publicly available information, so these assets have the correct values in view of this information.That is, investors quickly pounce on any new information that becomes public, buy or sell accordingly, and the supply and demand in the market works its wonders so prices take their fundamental values (instantaneously, it is often said, or at least very quickly). This version has one big advantage already over the weak form of the EMH -- it actually makes an assertion about information, and so might plausibly say something about the efficiency with which the market absorbs and processes information. However, there are many vague terms here. What do we mean precisely by "public"? How quickly are the prices supposed to reflect the new information? Minutes? Days? Weeks? This isn't specified.

Notice that a hypothesis formulated this way -- as a positive statement that a market always behaves in a certain way -- cannot possibly ever be proven. Evidence that a market works this way today doesn't mean it will tomorrow or did yesterday. Asserting that the hypothesis is true is asserting the truth of an infinite number of propositions -- efficiency for all stocks, for example, and all information at all times. No finite amount of evidence goes any distance whatsoever toward establishing this infinite set of propositions. The only thing that can be tested is whether it is sometimes -- possibly often or even frequently -- demonstrably false that a market is efficient in this sense.

This observation puts into a context an enormous body of studies which purport to give "evidence for" the EMH, going back to Fama's 1970 review. What they all mean is "evidence consistent with" the EMH, but not in any sense "evidence for." In science, you test hypotheses by trying to prove they are wrong, not right, and the most useful hypotheses are those that turn out hardest to find any evidence against. This is very much not the case for the semi-strong EMH.

If markets move quickly to absorb new information, then they should settle down and remain inert in the absence of new information. This seems to be very much not the case. Nearly two decades ago, a classic economic study by Lawrence Summers and others found that of the 50 largest single-day price movements since World War II, most happened on days when there was no significant news, and that news in general seemed to account for only about a third of the overall variance in stock returns. A similar study more recently (2002) found much the same thing: "Many large stock price changes have no events associated with them."

But if we leave aside the most dramatic market events, what about price movements over short times during a single day? Here too the evidence rather strongly contradicts the semi-strong EMH. Bouchaud and his colleagues at Capital Fund Management recently used data for high-frequency trading to test the alleged EMH link between news and price movements far more precisely. Their idea was to study possible links between sudden jumps in the prices of stock prices and possible news items appearing in electronic news feeds, which might, for example, announce new information about a company. Without entering into the technical points, they found that most sudden price jumps took place without any conceivably causal news arriving on the feeds. To be sure, the news entering did cause price movements in many cases, but most large movements happened in the absence of such news.

Finally, we can immediately also dismiss -- with the evidence just cited -- the strong version of the EMH which claims that markets rapidly reflect not only all public information, but all private information as well. In such a market insider trading would be impossible, because insider information gives no one an advantage. If I'm a government regulator about to issue a drilling permit to Exxon for a wildly lucrative new oil field, even my personal knowledge won't permit be to profit by buying Exxon stock in advance of announcing my decision. The market, in effect, can read my mind and tell the future. This is clearly ridiculous.

So it appears that the two stronger versions of the EMH -- which make real claims about how the markets process information -- are demonstrably (or obviously ) false. The weak version is also falsified by masses of data -- there are patterns in the market which can be used to make profits. People are doing it all the time.

The one statement close to the EMH which does have empirical support is that market movements are very difficult to predict because prices do move in a highly erratic, essentially random fashion. Markets sometimes and perhaps even frequently process new information fairly quickly and that information gets reflected in prices. But frequently they do not. And frequently markets move even though there appears to be no new information at all -- as if they simply have rich internal dynamics driven by the expectations, fears and hopes of market participants.

All in all, the EMH then doesn't tell us much. Perhaps Emanuel Dermin, a former physicist who has worked on Wall St. as a "quant" for many years, puts it best: you shouldn't take the thing too seriously, he suggests, but only take it to assert that "it's #$&^ing difficult or well-nigh impossible to systematically predict what's going to happen next." But this, of course, has nothing at all to do with "efficiency." Many economists, lured by the desire to prove some kind of efficiency for markets, have gone a lot further, absurdly so, even trying to make a strength of its own ignorance about markets, indeed enshrining its ignorance as if it were a final infallible theory. Dermin again:

The EMH was a kind of jiu-jitsu response on the part of economists to turn weakness into strength. "I can't figure out how things work, so I'll make that a principle." 

In this sense, on the other hand, I have to admit that the word "efficient" fits here after all. Maybe the word is meant to apply to "hypothesis" rather than "markets." Measured for its ability to wrap up a universe of market complexity and rich dynamic possibilities in a sentence or two, giving the illusion of complete and final understanding on which no improvement can be made, the efficient markets hypothesis is indeed remarkably efficient.