David H. Freedman, Wrong: Why Experts* Keep Failing Us—And How to Know When Not to Trust Them (New York: Little, Brown and Co., 2010), 304 pp., $25.99.

Kathryn Schulz, Being Wrong: Adventures in the Margin of Error (New York: Ecco, 2010), 416 pp., $26.99.

Charles Seife, Proofiness: The Dark Arts of Mathematical Deception (New York: Viking Adult, 2010), 304 pp., $25.95.

[amazon 0316023787 full]CHAFE THOUGH we might at our helplessness, even the most abrasive cynics cannot avoid relying on various species of experts—economists or oncologists or climatologists or . . .—in forming opinions. The world is just too complex to decode it on our own. And as the world has grown evermore complicated, there has been a corresponding surge in demand for metaexpertise—for those who promise to guide us through the intellectual labyrinth, distinguishing the snake-oil salesmen from the fact profferers. Of course, just because we want something does not mean that it exists. But it is a good bet that people will pop up claiming they can deliver the answers gift wrapped tomorrow morning.

 

THANK GOODNESS, it would seem then, that there are three new books by accomplished journalists David Freedman, Kathryn Schulz and Charles Seife to help us laymen figure out who is engaging in trickery-by-data-distortion and who not, so we can all better decode whom and what to trust.

I am often pigeonholed as an “expert on experts” because I have a long-standing interest in the (rather large) gaps between the confidence of political pundits and their forecasting accuracy. In reviewing these books by experts on experts, one could say I have morphed into an even-higher life form: an expert on “experts on experts.”

Before ascending further on the Great Ladder of Being (all it would take is for a blogger to review this review—and for me to reply—to move up to Rung Number Five), I had better stop the infinite regress. Why not start by asking: How much traction can we get from heeding these authors’ advice in coping with that recurring challenge of modern existence—separating the wheat from the chaff?

Each author offers what appear to be sensible, sometimes deeply insightful, guidelines. Seife warns us in Proofiness about how easily we can be seduced by an assortment of specious statistical arguments—and about the hazards of Potemkin numbers that give weak arguments an aura of credibility. He notes especially that we are never more vulnerable to this sort of “proofiness” than when false claims of precision cut along convenient ideological lines. In essence, when experts’ sureties affirm our own beliefs (Democratic, Republican or otherwise), we are quick to take them on faith.

But we should be wary of almost all “facts” delivered to us (whether they jibe with our vision of reality or not), for as David Freedman documents in Wrong, experts know a lot less than they claim—and that this is, as Marxists were fond of saying, no accident. There are such powerful and perverse institutional incentives for experts to overclaim the validity of their data and their conclusions, we should not be shocked that many ambitious scientists succumb to the I-have-figured-out-all-the-answers temptation (indeed, the surprising thing is perhaps that so many resist the siren calls of media acclaim).

[amazon 0670022160 full]And Schulz explores in her own subtly seductive way the experience of “Being Wrong”; why it is so hard to admit error but how, with the help of brilliant philosophers and clever perceptual illusions, we might learn to stop taking ourselves so damned seriously and embrace William James’s liberating insight: “Our errors are surely not such awfully solemn things. In a world where we are so certain to incur them in spite of all our caution, a certain lightness of heart seems healthier than this excessive nervousness on their behalf.”

It is hard to argue with writers as attractively open-minded as this threesome—and I caught only a few errors, none consequential. And certainly if we all internalized the authors’ collective wisdom we would be quite a bit better off. If we mastered the rudiments of statistical reasoning, recognized that experts are all too human and cultivated the fine art of self-interrogation, we’d reduce our exposure to some expensive mistakes.

On the personal side of the ledger, we would pay less money for dubious financial-services advice (and shift into low-transaction-cost, diversified mutual funds); we would be less easily swept up in panics over food additives or vaccinations; and, we would be less likely to gamble our lives on alternative medical treatments. Richer and healthier are not bad returns on an investment of less than $100 in books.

On the public-policy side of the ledger, we would be quicker to sense when we are being tricked by politicians who make their arguments in nominal dollars rather than inflation-adjusted ones, who imply that correlation means causation, and who insinuate that they have policy formulas that allow us to escape the tedious trade-offs that genuine domestic- and foreign-policy experts know all too well—those between consumption and investment, equality and efficiency, deterrence and reassurance.

[amazon 0061176044 full]And if all that were not reward enough, we could dodge a lot of silly watercooler arguments. For instance, Seife’s discussion of “electile dysfunction” shows how meaningless it was to get hot and bothered over who really won the Bush-Gore—or the Franken-Coleman—election. Both were, despite the media brouhaha, to put it simply, ties. Many have a hard time grasping this truth because we have not deeply internalized the elementary statistical concept of measurement error. A result can be off a little here or off a little there; a definitive result is an impossibility, no matter how many chads we count. Indeed, there are some situations in which we can do no better than tossing a coin—an option that our highest courts could have safely considered if most citizens were as statistically savvy as Charles Seife.

So, there are tangible senses in which we would be better off if we were all “smarter”—and “smarter” about what it means to be smart (it is not just processing speed and crunching power—it is a capacity for constructive self-criticism). But how feasible is it for the statistical layman? Can we truly become functioning educated skeptics armed with the sage wisdom of these authors? I see no better way to get at the answer than to ground ourselves in a real-world experiment. Can we decipher the validity of a given expert “finding”?

 

IT IS essential to choose tough test cases for this process—for judging metaexperts—cases in which, most importantly, we don’t already know the outcome and thus don’t know which experts proved to be on the right and wrong sides of history. We know in principle but in practice frequently forget how easy it is to tut-tut about expert stupidity from the lofty perch of hindsight. Many in my faculty-club bar still cannot fathom how those fools in the Bush administration managed to underconnect the dots prior to 9/11 and then overconnect the dots for Iraq’s WMD. Obviously, the barfly psychologists tell us, the politicians must have fallen prey to “inattentional blindness” prior to 9/11 (sometimes we are so focused on one task, we miss the eight-hundred-pound gorilla strolling through the crowd) and then fallen prey to “randumbness” prior to March 2003 (sometimes we see patterns that we wish or expect to see, not what is actually there).

Any fool can find fault ex post. But who can spot the faults—and fix them at acceptable cost—ex ante?

Tough test cases for metaexperts should have other properties as well. They should carry high policy stakes that bring into conflict heavyweight thinkers and researchers who make bold claims and hold prestigious posts in research universities and think tanks. At the risk of sounding intolerably elitist, it is too easy to discredit dubiously credentialed proponents of conspiracy theories. We need test cases with experts who, even if you’re ideologically inclined to blow them off, are hard to ignore. Here are four examples:

• Are Paul Krugman and Joseph Stiglitz right (circa summer 2010) that we are teetering on the edge of the third great depression of modern economic times—and that it would be a monumental error to start reining in deficit spending? Or are the conservative and libertarian critics of Keynesian stimulus spending, such as Gary Becker, correct about negative multipliers and the need to focus on reincentivizing the private sector?

• Who is closer to being right (circa summer 2010) about the future of Sino-American relations: James Fallows (there are lots of marital stressors but divorce is far from inevitable) or Niall Ferguson (get ready for a nasty divorce, probably sooner rather than later)?

• On which sect of climate scientists should we place our multitrillion-dollar global-warming bets: The pessimists who see looming catastrophe or the optimists who see weak trends and much uncertainty about causation? Those willing to consider geoengineering solutions or those who see the risks of unintended consequences as too great?

• Which psychologists and social scientists have staked out the more accurate position in the great American racism debate: The optimists who believe the survey evidence that prejudice has hit all-time lows or the pessimists who declare that surveys fail to measure unconscious forms of racism that make it extremely difficult for African Americans to be treated fairly in job markets?

Image: Essay Types: Book Review