The Wisdom of Crowds

A reader's summary of the claim that the average judgment of a large group can beat its experts: where the evidence comes from, the conditions it needs, and why a comment section meets almost none of them.

The term at a glance

The wisdom of crowds is the finding that when many people each estimate the same quantity, the average of their answers is often closer to the truth than most of the individual answers, and sometimes closer than any expert's. The phrase became common after journalist James Surowiecki used it as the title of a 2004 book. It is routinely invoked in arguments about up-votes, review scores, prediction markets and open encyclopedias, usually as a reason to trust whatever a large number of strangers agree on. The research behind it is narrower than that, and the exceptions matter as much as the rule. It is the optimistic partner of the groupthink and availability cascade stories told elsewhere on this site.

Origin

The founding anecdote comes from the statistician Francis Galton. At a livestock fair in Plymouth in 1906, visitors paid a few pence to guess the weight of an ox once it had been slaughtered and dressed. Galton obtained the roughly 800 tickets, set aside those that were illegible or incomplete, and analyzed the 787 that remained. In a short note titled “Vox Populi” in Nature in March 1907, he reported that the middle estimate was 1,207 pounds against an actual weight of 1,198 pounds, an error of under one percent. He later gave the arithmetic mean as 1,197 pounds. Galton, who had no great faith in democracy, wrote that the result was more creditable to the trustworthiness of a democratic judgment than he would have expected.

History and context

The mathematical idea is older than the ox. In 1785 the Marquis de Condorcet argued, in his essay on the application of analysis to majority decisions, that if each voter is more likely than not to choose the correct option, and the voters decide independently, the chance that the majority is right grows toward certainty as the group gets larger. The same theorem has a dark side that is often left out: if each voter is more likely to be wrong than right, a larger group makes the wrong answer more certain.

Surowiecki's book gave the idea a checklist. A crowd is likely to be wise, he argued, when it has diversity of opinion, when its members judge independently, when knowledge is decentralized and drawn on locally, and when there is a way to aggregate the judgments into one answer. Take away any of these and the crowd becomes ordinary. His examples ranged from the search for the lost submarine Scorpion in 1968 to stock markets and horse-race odds.

Experiments on the conditions arrived later and are the least flattering part of the story. In a 2006 study in Science, Matthew Salganik, Peter Dodds and Duncan Watts let thousands of participants download unknown songs. Those who could see what others had downloaded produced more unequal outcomes, and the same songs rose or fell differently in parallel worlds that started identically. In 2011 Jan Lorenz and colleagues, writing in PNAS, asked people factual questions and then showed them each other's answers. The spread of the guesses narrowed, but the group did not become more accurate: it simply became more confident in a smaller range. In a 2013 Science experiment on a social news site, Lev Muchnik, Sinan Aral and Sean Taylor found that one randomly assigned early up-vote raised a comment's final rating by about a quarter, on average.

Main ideas

Critique

• The anecdote is one data point. A single contest, collected by a man looking for evidence, does not show how often crowds beat experts. Later repeats with jelly-bean jars and similar tasks support the idea but they are puzzles with a known answer, not a general proof.
• Condorcet assumes what the web lacks. The theorem needs independent voters who are each better than chance. A thread where people read earlier replies before writing their own breaks independence, and a topic on which most people start misinformed breaks the second condition.
• Averages are not what platforms show. Galton computed a central estimate from every slip. A feed ranks by early reactions and engagement, which rewards whatever got a head start, as the 2006 and 2013 experiments suggest.
• It is stretched to questions it cannot answer. A crowd can estimate a weight, but whether a policy is fair or a song is good has no ground truth to approach. Citing the “wisdom of crowds” for such questions confuses a measurement result with a popularity result.
• Crowds have known failures. Housing bubbles, rumor cascades and mass panics are also crowds at work. The honest summary is that groups are good at some problems under some conditions, which is a less quotable claim than the title.

Impact

The idea shaped how much of the internet is built. Rating systems, prediction markets, community-edited reference works and crowd-sourced fact-checking such as the notes systems on social platforms all rest on the hope that many small judgments add up to a good one. It also gave a vocabulary to the opposing worry that a crowd with a single source of information is only an echo. For a reader, the practical test is the one Galton's fair passes: ask whether the people behind a number judged separately, whether there is an answer they could be wrong about, and whether the number is an average of them all or a ranking of whoever spoke first.