Artificial intelligence can read everything. Markets already have. Why the crowd still outweighs the machine.

 I get a version of the same question almost every week now. If artificial intelligence can read every filing, every earnings call and every news feed faster than any analyst alive, can it finally find the mispriced stock that beats the market? It is a fair question. My answer rests on an idea that earned a Nobel Prize, and that I have built my career and this firm around: the market price is generally the best available estimate of what a security is worth.

The efficient market hypothesis holds that maret prices reflect available information. If that is right, an AI hunting for mispriced securities is not searching a field of hidden bargains. It is searching a field that millions of other participants, many of them now running their own AI, have already picked over. The more powerful the tools become, the faster that field is cleared.

The wisdom of the crowd

The illustration below says it better than I can. On one side of a scale labeled Wisdom sits a thinking machine with its screens, its models and its stack of books. On the other sits a crowd of ordinary people, engineers, teachers, retirees, business owners, each holding a sliver of knowledge the others lack, with a robot sitting among them. The dial tells the story: the crowd is heavier. That is not sentiment. Markets are very large groups of buyers and sellers, and can aggregate dispersed information more effectviely than a single participant. A machine is a single participant.

Figure A. The wisdom of the crowd outweighs the wisdom of artificial intelligence, and AI adds the most when it joins the crowd rather than competing with it. Adam Smith's invisible hand, above, describes how millions of self-interested buyers and sellers arrive at fair prices without anyone directing them. Illustration: Index Fund Advisors, Inc.

The crowd is wise because errors can cancel. One buyer is too optimistic, another too fearful, a third has read a footnote the rest missed. When they trade, their mistakes offset and their knowledge accumulates in the price. An AI, however much it has read, is one opinion. That is why the artist put a second robot on the crowd's side of the scale: AI can join the crowd and make it a little wiser, and its knowledge then ends up in the price like everyone else's. What it cannot do is outweigh the crowd from the other pan.

The first hard evidence for this came from an unlikely place. In 1906, the statistician Sir Francis Galton visited a livestock fair in Plymouth, England, where visitors paid to guess the weight of an ox once it had been slaughtered and dressed. Galton, who had little faith in the judgment of ordinary people, collected 787 of the entry tickets and expected the crowd to be far off. The guessers included butchers and farmers but also plenty of people with no expertise at all, and individually their estimates were all over the map. Yet the median guess was 1,207 pounds, and the ox actually weighed 1,198, an error of less than 1%. Galton published the result in Nature in 1907 under the title "Vox Populi," Latin for "the voice of the people." The individual errors had canceled out, and what remained was the information. In my view, a market price is the same experiment run every trading day, with real money behind every guess.

Figure B. Galton's ox-weighing contest at the West of England Fat Stock and Poultry Exhibition, Plymouth, 1906. The median of 787 estimates was 1,207 pounds; the ox's actual dressed weight was 1,198 pounds. Source: Galton, F. (1907), "Vox Populi," Nature. Illustration: Index Fund Advisors, Inc.

James Surowiecki made this case at book length in The Wisdom of Crowds (2004), with the stock market as one of his central examples. Crowds are wise, he showed, only under four conditions: diversity of opinion, so each person brings private information; independence, so opinions are not copied from neighbors; decentralization, so people can draw on local and specialized knowledge; and aggregation, a mechanism that turns private judgments into one answer. In my view, a market price is the purest aggregation mechanism ever devised.

Surowiecki also described how crowds fail: when people abandon their own information and copy others, which he called an information cascade, or when everyone shares the same biases and the diversity collapses. This is the one place where AI deserves real caution, and it is the opposite of the usual worry. The risk is not that a machine will outsmart the market. The risk is that millions of investors will put the same questions to the same handful of models and act on the same answers, trading independence for a shared opinion. A crowd that all consults one oracle is no longer a crowd. The remedy is the old one: own the whole market, let the price do the aggregating, and resist acting on what everyone else is acting on.

Figure C. A crowd that consults one oracle is no longer a crowd. On the left, a diverse, independent market: every participant brings different knowledge, and the price that emerges reflects all of it. On the right, the same square after everyone has turned to the same model for the same answer, and the price board goes blank, because there is nothing left to aggregate. Illustration: Index Fund Advisors, Inc.

Smith, Hayek and the invisible hand

Adam Smith, looking down from the clouds with his hand extended over the scale, explained the mechanism more than two centuries before anyone built a computer. In The Wealth of Nations he described the invisible hand: each buyer and seller pursues only his own interest, yet the sum of those bargains moves prices to where supply meets demand, with no one planning it and no one in charge. A seller who asks too much finds no buyer and lowers the price; a buyer who bids too little finds no seller and raises the bid. The price that clears the market is the one at which the best-informed participants on each side are willing to transact. That is what I mean by a fair price. Fama's efficient market is Smith's invisible hand, measured.

Rex Sinquefield, who helped build one of the first S&P 500 index funds and co-founded Dimensional Fund Advisors, drew exactly this line in a 1995 debate at a Schwab conference, which I recapped in an earlier IFA article, Active vs. Passive Management. Rex traced the idea from Smith to Friedrich Hayek, who explained that the knowledge needed to price anything "is never possessed by a single individual or a single group" but is dispersed among many participants, and that the price system gathers it. The market, he said, "garners, comprehends and disseminates widely dispersed information better and faster than any system man has deliberately designed." His verdict on fifty years of fund-manager studies was blunt: "In any asset class, the only consistently superior performer is the market itself." His remarks reflect the research available in 1995, and Index Fund Advisors uses Dimensional funds in client portfolios.

Rex's sharpest line was that socialists and active managers are "cut from the same cloth," linked by "a disbelief or skepticism about the efficacy of market prices in gathering and conveying information." Thirty years on, I would add a third member to that club. In my view, an AI deployed to beat the market is a central planner with better hardware: a single mind asserting that it knows more than the price. Hayek's point does not soften because the planner is made of silicon.

Figure D. Cut from the same cloth. A 1950s central planner with paper charts and a red telephone, and today's AI with its screens and predictive models, both trying to set the price of silver for an entire economy. Through the window between them, the marketplace that already knows the answer goes about its business, unnoticed by either. Illustration: Index Fund Advisors, Inc.

A fair price is like a fair coin. Nobody can tilt the coin by studying past flips, and that includes a machine that has memorized every flip in history.

What Fama's hypothesis actually says

Eugene Fama shared the 2013 Nobel Memorial Prize in Economic Sciences for his empirical work on asset prices, including the efficient market hypothesis he formalized in 1970. I have had the privilege of attending numerous lectures given by Gene over the 27 years I have been attending Dimensional Fund Advisors conferences and webcasts, and the thing people most often get wrong about his idea is to caricature it as "prices are always perfect." That is not what it says.

Fama's hypothesis is that prices fully reflect available information. A price is not a prophecy; it is a consensus estimate. New information moves it, but because that information arrives randomly, the price changes are unpredictable. The practical consequence is the one that matters to you as an investor: you cannot expect to earn above-market returns using information the market already has, once you account for the costs and risks of trying.

The companion idea is the random walk theory, which Fama tested in his early empirical work. If today's price already reflects everything known, then tomorrow's change depends only on tomorrow's news, which by definition is not yet known. Price changes therefore behave like a sequence of coin flips: the next one carries no memory of the last.

The random walk is older than Fama, and older than the computer. In 1900 a French mathematician, Louis Bachelier, defended a doctoral thesis, Théorie de la Spéculation, that modeled the prices of French government bonds on the Paris Bourse as a random walk, five years before Einstein used the same mathematics to describe particles jittering in a fluid. My favorite passage is a simple observation about the two sides of a trade: at the moment it occurs, the buyer believes the price will rise and the seller believes it will fall, so the market as a whole believes neither. From that follows the definition of a fair game: the mathematical expectation of the speculator is zero. Every AI that proposes to profit from the next move in a price is, whether it knows it or not, betting against a theorem that has stood since 1900.

Figure E. Bachelier's fair game on the steps of the Paris Bourse. At the moment of every trade, the buyer is sure the coin will land Up and the seller is sure it will land Down, so the market as a whole believes neither. Behind them, the trail of past flips shows no order a robot with a magnifying glass can exploit. Up means a return above the historical median; Down means below it. Illustration: Index Fund Advisors, Inc.

Bachelier's thesis sat unread by economists for half a century. The bridge to the modern era was a 1964 MIT Press anthology, The Random Character of Stock Market Prices, edited by Paul Cootner, and it is the book I point people to when they ask where the efficient market idea came from. Until then, Wall Street's working assumption was that prices moved in trends a careful chartist could map and ride. Cootner gathered the serious statistical studies of the day and let them speak: short-term price movements looked like Brownian motion, not like anything a trend-follower could exploit. The collection also carried the first English translation of Bachelier, along with early work on fat-tailed distributions, Markov processes and option pricing under a random walk. The toolkit today's AI quants use to hunt for patterns was assembled in a book whose central finding was that the patterns are not there.

Paul Samuelson, the first American to win the Nobel Prize in economics and one of the economists who rediscovered Bachelier, turned the insight into a proof. His 1965 paper carried the plain title "Proof That Properly Anticipated Prices Fluctuate Randomly," and its logic is simple. If investors act on everything they know and can anticipate, that information goes into the price at once, and whatever moves the price tomorrow must be the part nobody could foresee. Today's price is therefore the best forecast of tomorrow's, and any pattern an analyst or an algorithm finds in past prices has already been traded away by whoever found it first. Samuelson spent a career on the price system, from how prices ration scarce goods to how trade equalizes the price of labor and capital across borders, and his 1965 proof simply applied Smith's and Hayek's principle to the one market where people are most tempted to believe they know better.

This is what I mean when I say a market is fair. A fair price is like a fair coin. Under the random walk model, the return over the next period has an equally likely chance of landing above or below the historical median return for that level of risk. That median is the risk-appropriate return, the compensation the market offers for the risk being borne. Nobody can tilt the coin by studying past flips, and that includes a machine that has memorized every flip in history.

Figure 1. For a memoryless process, every transition between Up and Down is 50%. Illustrative; no market data shown, so no as-of date applies.

The Nobel committee recognized that this idea reshaped how the world invests. It is the intellectual foundation of index funds, and of my own conviction that today's price is the right price in the only sense that counts: nobody can reliably show it is wrong before the next one arrives.

Notice what the hypothesis does not require. It does not require every investor to be rational, or every price to equal some true value known in hindsight. It requires only that the errors be unpredictable, so that no strategy can consistently exploit them. That is a much lower bar than critics assume, and a much harder one for any trader, human or machine, to clear.

Testing the fair coin

The fair-coin claim is testable, and we built a tool at IFA to test it. Our Markov Chain Calculator applies a simple idea: a Markov chain is a model in which the probability of the next state depends only on the current state, not on the longer history. The calculator classifies each period as Up or Down relative to the median return, then measures the transition probabilities: how often an Up period is followed by another Up, how often a Down is followed by an Up, and so on.

If markets had memory, those probabilities would drift away from 50%. A trending market would show Up followed by Up well above half the time; a mean-reverting market would show the reverse. A fair coin shows every transition near 50%, and the calculator runs a battery of statistical tests, including chi-square, Wald-Wolfowitz runs, higher-order Markov and shuffle tests, to ask whether the observed sequence can be distinguished from chance.

I ran it on a commonly used allocation of 70% stocks and 30% bonds, over the longest period for which live fund data is available, May 1998 through September 2026. I encourage you to change the investment, the dates and the return frequency yourself and watch the same pattern hold: the measured probabilities hover within about a point of 50%, and the statistical tests cannot distinguish the market's sequence from a shuffled version of itself.

Figure 2. Under a fair coin, half of all streaks end after one period and a quarter last two. Long streaks are expected, not evidence of memory. Mathematical illustration; no market data shown.

This is the random walk made visible, and it is the trap waiting for any AI that learns from price history. A model trained on the sequence of past returns is trying to find memory in a process that, by this test, has almost none. It will find patterns, because any finite sample contains them. The Markov chain shows that those patterns do not survive the next flip.

The technology does not sit outside the market looking in. It is the market.

AI is the market, not a visitor to it

The case for AI stock-picking assumes that the machine is a lone genius reading data nobody else reads. In reality, every large asset manager, hedge fund and market maker has access to the same models, the same data vendors and the same computing power. When thousands of AIs parse the same earnings release in the same millisecond, their collective reaction is the new price. The technology does not sit outside the market looking in. It is the market.

This is the paradox that Sanford Grossman and Joseph Stiglitz described in 1980. Prices can only become informative because someone spends money to gather information, and that effort is only worthwhile if prices are not yet fully informative. The equilibrium is a market that is efficient to within the cost of being more efficient. AI drives that cost toward zero. Any edge that a model can find in public data is an edge that a competing model can find at the same cost, and competition among them pushes the reward for finding it toward nothing.

Figure 3. Conceptual illustration of the Grossman-Stiglitz equilibrium. As more participants analyze the same information, the profit available from it falls toward the cost of gathering it. No data shown, so no as-of date applies.

Three further limits apply to any AI hunting for mispricing. First, it learns from the past, and the past is a finite sample full of patterns that appeared by chance. A model that tests enough signals will find some that look profitable in historical data; most disappear when real money follows them. Second, the market adapts. A pattern that is discovered and traded is, by that act, arbitraged away. Third, the returns that remain after competition are compensation for bearing risk, not for being clever. A model that finds stocks with higher expected returns has usually found stocks with higher risk.

None of this says AI is useless in finance. I use it, and we use it at IFA, for cutting costs, executing trades, managing taxes and keeping portfolios on target. What it cannot do is step outside the equilibrium it helped create. The sharper the tools, the tighter the consensus, and the fewer the mistakes left for anyone to profit from.

The experiment has already been run

Quantitative funds have used machine learning, natural-language processing and alternative data for years, long before the current wave of generative AI.

Fama and Kenneth French examined this directly in their 2010 Journal of Finance paper, "Luck versus Skill in the Cross-Section of Mutual Fund Returns." Their finding was that, in aggregate, active managers' returns before costs were roughly what one would expect from the market, and after costs the typical fund fell short. The few funds with returns high enough to suggest skill were about as numerous as chance alone would produce.

Figure F. Many investors are certain they can pick the winners, but the results often look like a dartboard. Illustration: Index Fund Advisors, Inc.

AI changes the speed of the search, not the arithmetic of the contest. It changes how quickly the losers lose.

AI changes the speed of the search, not the arithmetic of the contest. Every dollar one investor earns above the market is a dollar another investor earned below it, and the costs of playing come off the top. A faster search in a game that is zero-sum before costs and negative-sum after them does not change who wins on average. It changes how quickly the losers lose.

So the question I would ask you is not whether AI is impressive. It plainly is. The question is whether paying for an AI-driven attempt at mispricing is likely to pay off. The academic record and the fund-performance record point the same way, and neither offers much comfort to the stock picker, silicon or otherwise.

Which risks does the market pay you to bear?

Accepting that prices are right does not leave you with nothing to decide. It changes the question from "which stock is wrong?" to "which risks does the market pay me to bear?" That is the question the Fama-French five-factor model answers, and it is why I regard it as the most reliable framework for building a portfolio.

Fama and French published the five-factor model in 2015, extending the three-factor model they introduced in 1993. It explains differences in expected returns across stocks using five dimensions: the market premium for holding stocks rather than Treasury bills; the size premium for holding smaller companies rather than larger ones; the value premium for holding stocks with high book value relative to price; the profitability premium for holding firms with robust operating profitability; and the investment premium for holding firms that invest conservatively rather than expanding their assets aggressively.

Figure 4. The five dimensions of expected return in the Fama-French five-factor model (2015). Diagram only; no performance data shown. Factor premiums are not guaranteed.

The model does not claim these stocks are mispriced. It claims the opposite. Their higher expected returns are the market's compensation for risks or characteristics that investors, in aggregate, require to be paid for. That is exactly what an efficient market should produce, and it is why the factors have been documented across decades and across international markets rather than vanishing once published, as a true mispricing would.

This gives you a principled way to construct a portfolio without pretending to know something the market does not. Decide how much market risk you can carry; our Risk Capacity Survey exists for exactly that purpose. Tilt toward small, value, profitable and conservatively investing companies to the degree your time horizon and your tolerance for tracking a different path than the broad index allow. Diversify across thousands of securities so no single company's news dominates. Then hold, rebalance and let the premiums work over time.

The expected reward for those risks is a feature of an efficient market, not a bet against one.

In my view, that is a sturdier foundation than an algorithm searching for errors in prices set by other algorithms. I want to be direct about the trade-off, because I have lived through it with clients: factor premiums are not guaranteed, each has gone through long stretches of underperformance, and investors who tilt toward them must accept that their portfolios will differ from the market, sometimes painfully. But the expected reward for those risks is a feature of an efficient market, not a bet against one. If you would prefer to minimize tracking error relative to the broad market and keep costs low, we also advise clients on alternative portfolios geared to the total U.S., international and emerging markets, for both stocks and bonds, without a factor tilt, such as a mix of Vanguard large-blend stock ETFs and Vanguard Total Bond Market funds. You can see the range of fund options we use to implement our portfolios at IFA.com/fund-portfolios, and an IFA Wealth Advisor can explain which one fits your situation.

Should AI change what you do?

Which brings me to the question underneath all the others. If you already hold a passively managed, low-cost, globally diversified portfolio, should the arrival of AI change what you do? No. If anything, it is a reason to keep it.

Consider what AI changes and what it leaves alone. It may make markets more efficient, which means the price you pay for an index fund is a better estimate of value than it was a decade ago. It can lower the cost of trading and administration, and some of those savings may flow through to fund expenses. It makes it easier for your adviser to rebalance, harvest tax losses and keep your allocation on target. Every one of those effects improves the passive investor's position.

What AI does not change is the arithmetic that made passive investing sensible in the first place. The market return is still the average of all investors' returns before costs. Index funds still seek to capture that return at low cost, and like the market itself, they can lose value. And the premiums that a factor-tilted portfolio is designed to earn still come from bearing risk, which no amount of computing power removes. If anything, a world in which every active manager has the same AI is a world in which their collective edge over the index shrinks further, while their fees do not.

The temptation will be real. Over the next few years you will see funds, newsletters and apps promising that their model has cracked the market. Some will have good recent numbers, because with enough entrants some always do. The efficient market hypothesis predicts both the promises and the disappointment that follows them.

My advice has not changed in more than twenty-five years, and AI has only strengthened it. Stay diversified, stay low cost, stay tilted toward the dimensions of return the evidence supports, and let the machines compete with one another to make your prices more accurate. Invest and relax.

Sources

1. Bachelier, L. (1900). Théorie de la Spéculation. Doctoral thesis, University of Paris; English translation in Davis, M., and Etheridge, A. (2006), Louis Bachelier's Theory of Speculation: The Origins of Modern Finance, Princeton University Press.
2. Cootner, P. H., ed. (1964). The Random Character of Stock Market Prices. MIT Press.
3. Fama, E. F. (1970). "Efficient Capital Markets: A Review of Theory and Empirical Work." Journal of Finance.
4. Galton, F. (1907). "Vox Populi." Nature, 75, 450–451. galton.org/essays/1900-1911/galton-1907-vox-populi.pdf
5. Samuelson, P. A. (1965). "Proof That Properly Anticipated Prices Fluctuate Randomly." Industrial Management Review, 6(2).
6. Grossman, S. J., and Stiglitz, J. E. (1980). "On the Impossibility of Informationally Efficient Markets." American Economic Review.
7. Fama, E. F., and French, K. R. (2010). "Luck versus Skill in the Cross-Section of Mutual Fund Returns." Journal of Finance.
8. Fama, E. F., and French, K. R. (2012). "Size, Value, and Momentum in International Stock Returns." Journal of Financial Economics.
9. Sharpe, W. F. (1991). "The Arithmetic of Active Management." Financial Analysts Journal.
10. Fama, E. F., and French, K. R. (2015). "A Five-Factor Asset Pricing Model." Journal of Financial Economics.
11. The Royal Swedish Academy of Sciences (2013). Prize in Economic Sciences, awarded to Eugene F. Fama, Lars Peter Hansen and Robert J. Shiller.
12. Index Fund Advisors, Inc., Markov Chain Calculator, com/calculators/markov-chain.
13. Surowiecki, J. (2004). The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations. Doubleday.
14. Sinquefield, R. (1995). Opening address, Schwab Institutional conference, San Francisco, Oct. 12, 1995; recapped in Hebner, M., "Active vs. Passive Management," com/articles/active-vs-passive-management.

Important disclosures

This article is for informational purposes only and reflects the opinions of Mark Hebner and Index Fund Advisors, Inc. It is not investment, tax or legal advice. All investing involves risk, including the possible loss of principal; the value of investments may increase or decrease. Past performance is not a guarantee of future results. Factor premiums are not guaranteed and may underperform for extended periods. Individual investor accounts (performance, allocation, etc.) may vary. Investors cannot invest directly in any index. Data presented in IFA's Markov Chain Calculator is hypothetical back-tested performance and is not actual client performance; see www.ifabt.com for back-tested performance disclosures. Charts in this article are illustrative or mathematical and do not display market or portfolio returns. Index Fund Advisors, Inc. is registered with the SEC as an investment adviser; however, such registration does not imply a certain level of skill or training and no inference to the contrary should be made.