unSpun (15 page)

We're not saying that you should track every single fact back to its origin. That's clearly not possible. You don't have to go to the National Archives in Washington to read the original copy of the U.S. Constitution; you can read the text any number of places, including the National Archives website. You don't have to subscribe to
The New England Journal of Medicine
to get the gist of the latest medical research if you can read an accurate summary in a good newspaper, which is usually easier for nonmedical readers to understand anyway. We
are
saying that primary sources are more reliable than secondary sources. And when the inferences you will draw from the information are very important to you, it is best to check the primary source. Track your information upstream. Be wary of secondhand accounts, and even more wary of thirdhand stories.

RULE #5:
Know What Counts

W
HEN YOU SEE NUMBERS BEING USED, BE SURE YOU KNOW WHAT'S
being counted, and what's not. Definitions matter. We wrote earlier about numerical flimflams—“cuts” that are really slower increases, and “average” tax breaks that are much bigger than most people will ever see. Often this sort of confusion and deception can be avoided with a clear understanding of exactly what the numbers are supposed to represent.

Even the simple act of counting becomes complicated in real life, because we have to make choices about what to count. George W. Bush is counted as our forty-third president, even though he's only the forty-second person to hold the office. Grover Cleveland is counted twice—as the twenty-second president and also as the twenty-fourth—because he served one term, was defeated, then was elected four years later to a second term. And if just counting to forty-three can be a problem, imagine what can happen when numbers get into the billions and the opportunities for massaging the count multiply accordingly.

For example, in his 2006 State of the Union address, President Bush said, “Every year of my presidency, we've reduced the growth of nonsecurity discretionary spending.” But he sure hadn't reduced the growth of federal spending, which had shot up 42 percent since he first took office. Look carefully at what he was counting. The word “discretionary” excludes so-called entitlement programs, including Medicare, which is experiencing the largest expansion in its history because of Bush's addition of a prescription drug benefit. Notice also the term “nonsecurity,” which excludes the entire Pentagon budget and also huge extra outlays for the wars in Iraq and Afghanistan and everything the Bush bean counters choose to sweep under the heading of “homeland security.” Bush was counting only some relatively small categories of spending that hadn't risen so quickly.

Democrats play the same counting games. The Democratic National Committee ran a TV ad in advance of the speech we just mentioned, faulting Bush for “2.8 Million Manufacturing Jobs Lost,” among other things. Notice the word “manufacturing.” In fact, the U.S. economy had gained a net total of more than 2 million jobs since Bush's first day in office (first losing 3 million, then gaining 5 million). Most people probably aren't aware that manufacturing these days accounts for only a little more than one job out of every ten in the United States. And manufacturing had been almost the only major sector to decline under Bush's entire tenure. The DNC wasn't counting jobs in fast-growing sectors such as construction, health care, and finance.

Numerical flimflams are probably as old as numbers themselves, and we've already discussed a several of them. Still worth reading is the classic
How to Lie with Statistics,
a little book that Darrell Huff wrote more than half a century ago. Some of his examples sound quaint today. For example, he questions a claim that Yale graduates of the class of 1924 were earning an average of $25,111 a year, because that seemed far too grand a sum. Huff wrote that in 1954; the equivalent in 2006 dollars would be somewhere close to $180,000. Nevertheless, the numerical and statistical tricks Huff exposes are ageless and still in everyday use, and his style is so readable that we've tried to match it in this book.

RULE #6:
Know Who's Talking

I
F YOUR DOCTOR RECOMMENDS IT, PROTECTING YOURSELF AGAINST
a heart attack or stroke by taking aspirin can cost as little as a penny a day. The aspirin reduces the likelihood that blood clots will form. However, studies have been appearing in medical journals suggesting that large numbers of patients need to be tested for “aspirin resistance” in case aspirin no longer protects them. These same studies suggest that many patients may need an aspirin substitute costing $4 a day. But who's paying for these studies? As
The Wall Street Journal
pointed out in a front-page story in April 2006, many of those raising the “aspirin resistance” alarm have financial ties to the companies that stand to profit from selling the tests and drugs. The newspaper reported that one study, which declared that “resistance” affects perhaps 30 percent of those taking aspirin, was funded by the test's maker, Accumetrics, Inc., and by Schering-Plough Corp., which sells a drug being tested for its potential benefit to patients resistant to aspirin. This funding wasn't disclosed by the medical journal that published the article.

Such financial ties should make us skeptical of the research findings. This is a clear conflict of interest. The author's private interest conflicts with his responsibility to provide unbiased, trustworthy research—the public interest. Did the study's author, deliberately or otherwise, skew the research toward a finding that would create a profit for the sponsors, and make them more inclined to pay him to conduct future research? What other research did these companies fund, and did it come up with contrary findings that the companies suppressed? We're not saying that drug companies shouldn't finance research, or that paying for a study automatically produces the result they want. But knowing who's behind a statement is important in considering how much weight to give it.

It's not always obvious who's behind a study or a group. Until it disbanded in 2002, the “Global Climate Coalition” had a name that sounded neutral and a website showing happy children and green fields; but it was a lobby group made up of trade associations for industries including oil, chemicals, logging, agribusiness, and utilities, all of them financially motivated to avoid taxes or defeat regulation of their emissions of greenhouse gases. The groups named “Americans for Good Government,” “Americans United in Support of Democracy,” and “Maryland Association for Concerned Citizens” funnel donations to candidates who support Israel, according to the Center for Responsive Politics. The Alliance for Justice and the Committee for Justice sound alike, but they work for exactly opposite ends. The Alliance is a liberal group that opposed some of President George W. Bush's judicial nominees, while the Committee is a conservative group that ran TV ads supporting his Supreme Court picks. The latter was founded by C. Boyden Gray, who was White House counsel (and tennis partner) to Bush's father. Groups whose names seem to indicate support of a noble policy position (“good government” or “justice”) may be committed to a specific party or industry. Beware.

Self-interest doesn't make a statement false. What an electric utility says about what's coming out of its smokestacks can be accurate, even if the company quite naturally would like to avoid paying for expensive scrubbers. And an environmental group's statements might not exaggerate the dangers and extent of pollution, even if the group does get more money by telling donors the skies are being poisoned than it would if it said the air is getting cleaner, as happens to be the case. (As we mentioned earlier, the EPA's tracking of the six major air pollutants shows a 12 percent decline in the five years ending in 2005, which amounts to a decrease of more than 19 million tons per year of volatile organic compounds, sulfur dioxide, carbon monoxide, nitrogen oxides, soot, and lead.) Both sides in the environmental debate have a clear motive to tilt the facts in one direction or the other, and so we shouldn't accept either at face value. Which brings us to:

RULE #7:
Seeing Shouldn't Necessarily Be Believing

W
HEN WE GAVE AS OUR FIRST RULE THAT YOU CAN'T BE 100 PERCENT
certain of anything, you might have said to yourself, “I'm certain of what I see with my own eyes.” Don't be. Researchers have found, for example, that people can rather easily be talked into seeing things that aren't there (or saying they do). In one of the most famous experiments in the history of social science, the late Solomon Asch showed lines of different lengths to groups of students and asked each to say which was longer or shorter. But only one student in each group was a test subject; the others were Asch's confederates, whom he had instructed. When the test subjects heard a majority of others say that the longer line was the shorter, they often said the same even though the opposite was obviously true. In fact, 37 percent of the subjects expressed the bogus view of the majority. In an even earlier experiment, from 1935, the social psychology pioneer Muzafer Sherif showed people in a dark room a light that was not moving. They reported that the light was moving—and, more important, they gave reports of the amount of movement that were consistent with what they had heard others report.

Personal experience can mislead us. Eyewitness testimony is notoriously unreliable even when given in court, under oath. Years of research show that witnesses regularly pick innocent “foils” from police line-ups. As we write this, 189 persons have been exonerated after DNA tests showed they had been wrongly convicted, according to the Innocence Project. And more than 70 percent of those were convicted on the basis of mistaken eyewitness testimony!

Scholars also tell us that people tend to overestimate how well they remember their own experiences. If you have siblings, you can test this by picking some major event in your family's past that you and they shared: each of you write an account, then see how well they agree. Scholars have found that an apparently distinct memory of something that occurred long ago may be a reconstruction, often a self-serving one. As distance from the event increases, memory decays.

Even very smart people misremember things. Alan Greenspan, the former chairman of the Federal Reserve Board, is by most accounts a very smart man, yet even he was surprised to find how much his own memory had shifted with respect to a crucial conference call on April 12, 1991. Greenspan had proposed lowering interest rates, but other members of the Federal Open Market Committee strongly objected and blocked the move. Bob Woodward reports in his 1994 book
The Agenda
that when Greenspan looked at a verbatim transcript of the meeting, he discovered “that his own memory was faulty and imbued with recollections that were self-interested but not true.”

That somebody feels quite certain is no guarantee that his or her memory is accurate. Scholars who study the relationship between confidence and accuracy come up with mixed results. Sometimes witnesses who say they are positive are right, and sometimes they are not, and so far psychologists have only a dim understanding of why.

RULE #8:
Cross-check Everything That Matters

B
Y NOW IT SHOULD BE OBVIOUS THAT RELYING ON A SINGLE SOURCE
of information is a good way to be steered wrong. However, we can be more confident about a conclusion when different sources using different methods end up agreeing on it.

Different newspapers sometimes convey quite different impressions of the same event. When Governor Arnold Schwarzenegger of California appeared at a breakfast honoring Martin Luther King, Jr., in 2006, the
Oakland Tribune
reported that he got a “chilly reception” and the
San Jose Mercury News
described it as “hostile,” but the
San Francisco Chronicle
's headline cited a “surprisingly warm welcome.” So which was right? The
Mercury News
story reported as follows: “After a few scattered boos, the audience listened politely to his speech and laughed at his Terminator jokes.” Some of the labor union people who attended had talked of walking out to embarrass the Republican governor, but didn't. Whether that was “hostile” and “chilly” or “surprisingly warm” depends on whether you focused on the boos or the laughs. As for the word “surprisingly,” “surprise” depends on who is surprised and what they were expecting. In this case, reading two or three newspapers produced a much more balanced picture of the event than reading any one headline or report.

Weighing Evidence

The rules that apply to evidence in trials give us a good starting point for thinking about how we should weigh facts in our everyday lives. We've already mentioned the weakness of hearsay and secondhand accounts and considered why we should give them much less weight than firsthand accounts or physical evidence. And we've mentioned our preference for primary sources; we like to check the full transcript of an interview rather than rely on a paraphrase or partial quote. Here are some of the other factors we consider at FactCheck.org.

The Man in the Hole

How did the United States know it had the right man when soldiers pulled a disheveled wretch from a six-foot-deep “spider hole” in Ad-Dwar, Iraq, on December 13, 2003? How did they prove to Iraqi skeptics that Saddam Hussein had really been captured? By cross-checking and using multiple methods of verification.

The man announced, “I am Saddam Hussein, president of Iraq, and I am willing to negotiate,” but maybe he was one of Saddam's doubles. First, U.S. officials had to convince themselves. Former ambassador L. Paul Bremer recalls that, after a shave and haircut, the captive looked like the right man: “There could be no doubt: the face in this photo was Saddam Hussein,” Bremer writes in his book My Year in Iraq. Then four captured officials who had worked closely with Saddam Hussein were brought in. “Each prisoner had verified that the wizened man slouched on an army cot in that windowless room was in fact Saddam Hussein.” But maybe they were protecting the real Saddam Hussein. So U.S. officials used digital voice analysis to confirm that their prisoner's voice was the same as the voice in archived recordings of the Iraqi leader, and they compared saliva swabs from the prisoner with DNA from “Saddam family samples.”

To convince skeptical Iraqis, Bremer brought in a delegation from the U.S.-appointed Governing Council. “They could then confirm publicly they had seen the prisoner.” That did it: delegation members betrayed no hint of doubt that it was the former tyrant who was now in U.S. custody; they even berated him for his actions as Iraq's president. “Why did you have Sayyid Muhammad al-Sadr murdered in 1999?” asked one. “Saddam Hussein, you are cursed by God,” said another. “What do you say now about the mass graves?” challenged a third. The prisoner didn't bother denying his identity; instead, he justified his acts by calling his victims “criminals…thieves, traitors…Iranians.” Arab news media carried compelling narratives of the meeting.

The process provides a classic instance of what scholars call “converging on certainty,” when different methods all arrive at the same conclusion.

S
WORN
T
ESTIMONY

People don't generally go to jail for lying to a reporter, but they can be imprisoned for perjury if they are shown to have lied under oath. For that reason, we can give greater weight to sworn testimony than to unsworn statements such as news releases, news conferences, or TV interviews. Lying to a congressional investigating committee or an FBI agent or a bank examiner can also be a criminal offense, so unsworn statements made to official inquiries also deserve weight, though not quite as much as statements made under oath.

S
ELF-INTEREST

If somebody stands to profit (or to avoid loss), we naturally give his or her statements less weight than those of a neutral observer. We don't dismiss a statement just because it's made by a big corporation, or by a trial lawyer who stands to gain a multimillion-dollar fee by suing that corporation, but we don't accept such a statement at face value, either. We just assume that each party is probably giving us only one side of the story, and we look for additional evidence. We have to be alert to the possibility that a partisan is twisting the facts, either deliberately or because he or she is honestly blind to facts on the other side.

C
ONFESSIONS

A statement does deserve special weight, however, when the person speaking makes a confession, or states facts contrary to his or her own interest. An example occurred when President Bush answered a question on December 12, 2005, by saying, “How many Iraqi citizens have died in this war? I would say thirty thousand, more or less, have died as a result of the initial incursion and the ongoing violence against Iraqis.” His administration had previously refused to estimate civilian casualties, but now the president was in effect endorsing figures of the Iraq Body Count project, which compiles them from public, online media reports and eyewitness accounts. Bush's statement is evidence that the U.S. government can't refute the Iraq Body Count tabulation (which had risen to a minimum of more than 52,800 by January 2007). There are some higher estimates, disputed figures from the British medical journal
The Lancet,
which we'll have more to say about later. Our point here is that Bush's embrace of the Iraq Body Count figure is what lawyers call an “admission against interest”—a truth that it hurt him to say. We should count that as evidence that at least that many Iraqis had died.

R
EPUTATION OF THE
A
UTHORITY

When a medical study appears in
The New England Journal of Medicine,
we know it has gone through a systematic screening process called peer review, in which other knowledgeable scientists are asked to comment or point out possible flaws. The original author may then respond with clarifications or additional data. By contrast, we should always be skeptical of “scientific breakthroughs” that are announced at a news conference without any independent review by other experts. For example, when a news conference in 2002 proclaimed the birth of the first cloned human being (supposedly named “Eve”), it created a brief sensation. But good reporters were quick to point out that the man behind the announcement, a French former journalist named Claude Vorilhon, had renamed himself Rael, claimed to be a direct descendant of extraterrestrials who created human life on earth, and founded a cult. Neither “Eve” nor the mother of the supposedly cloned baby ever appeared publicly. Reasonable people gave the unsupported announcement zero weight and quickly dismissed it as a silly fraud.

T
RANSPARENCY

Look for transparency whenever a claim is made. Is the publisher of a poll telling you the statistical margin of error and exactly how the poll takers asked the question? If not, don't give much weight to the result. Political candidates who are challenging entrenched incumbents like to release polls showing that they are “closing the gap” or even have a lead, in order to convince potential donors they can win. But such polls can be tailored to produce a positive result by including loaded questions. The challenger might ask, “Did you know the incumbent is a wife-beater?” These so-called push questions nudge the respondent toward the desired answer, and a poll containing them is called a push poll. Questions can also be worded in ways that bias the result. One survey conducted by the Annenberg Public Policy Center found a dramatic difference in support for school vouchers depending on whether such phrases as “taxpayers' money” or “private schools” were included in the question. And polls asking about support for public financing of political campaigns come out one way if the poll taker asks about “banning special-interest contributions from elections” and another if they ask about “giving tax money to politicians” as a substitute.

When reading a news story or article, ask whether the reporter or author is telling you where the information came from. We supply footnotes at FactCheck.org, with links to the sources we are using if they are available free on the Internet, so that readers may find more information or check that we're getting it right. When you see somebody claim that “a study” has backed up their claims, ask how it was conducted, how many people participated and under what conditions, and whether it really supports what's being said.

P
RECISION

Sometimes evidence isn't nearly as precise as portrayed. A good example is a pair of studies that produced shocking headlines about deaths in Iraq, studies that have since been widely questioned and disparaged. Both studies were published in the British medical journal
The Lancet,
and both were produced by a team from Johns Hopkins University in Baltimore. The first was released five days before the 2004 presidential election, and estimated that 98,000 Iraqis had died as a result of the invasion ordered by President George W. Bush in March 2003. The second was released less than a month before the 2006 midterm House and Senate elections, and estimated that the Iraqi death toll had reached 654,965 from the invasion and the violent aftermath. Both were several times higher than other generally accepted estimates.

However, neither estimate was an exact count, just the midpoint of an exceptionally broad range of possibilities. For the first estimate, the authors calculated that their “confidence interval” was somewhere between 8,000 deaths and 194,000 deaths. In the language of statistics, that means a 95 percent probability that the actual figure fell somewhere within that huge range. Put another way, there was 1 chance in 40 that the actual number was less than 8,000 and an equal chance that it was greater than 194,000. As the critic Fred Kaplan put it in an article for the online magazine
Slate,
“This isn't an estimate. It's a dart board.” For the second estimate the dart board was larger, between 393,000 and 943,000 deaths. Such wide ranges of uncertainty are much larger than the plus or minus 2 or 3 percent we are used to seeing in U.S. public opinion polls, and should tell us to beware.

The exceptionally imprecise estimates of the
Lancet
studies stem from the relatively small sample used to produce them. The estimates came from interviews in 33 clusters for the first study, 47 for the second. Using such randomly chosen “clusters” is a statistical method commonly used when it isn't practical to draw a random sample of individuals from an entire population. But other experts criticized the
Lancet
authors for using too few. “I wouldn't survey a junior high school, no less an entire country, using only 47 cluster points,” said Steven Moore, a Republican consultant who had conducted polling in Iraq for Coalition forces. One of the
Lancet
authors, Gilbert Burnham, replied that “surveying more clusters would have also meant more risk to the survey team.” He said, “Had we used 470 clusters, our range of plausible values would have been about 3 times narrower.” It is also possible that the results would have been far different.

Indeed, a survey of Iraq by the United Nations Development Programme used 2,200 cluster points, compared to only 33 used by the first
Lancet
study conducted four months later. And the study—
Iraq Living Conditions Survey 2004
—estimated only 24,000 deaths, roughly one quarter as many as
The Lancet
estimated at the time.

C
ONVERGENCE

In Chapter 6 we mentioned the notion of convergent evidence, and said that when different methods arrive at similar estimates, those estimates are more credible. The reverse is also true: when results diverge, we should be more cautious. To be sure, the
Lancet
studies seem to support each other, but both produced results that are far higher than those of others. The Iraq Body Count project, for example, tabulated in November 2006 that between 47,016 and 52,142 deaths had been reported in Iraqi and international news media as a result of the 2003 invasion and the continuing violence. That's just 7 to 8 percent of
The Lancet
's 654,965 figure published the previous month. It's true that the IBC estimates almost certainly missed some deaths that weren't reported, but we judge it unlikely that they could miss so many.

The 2004
Lancet
study was inconsistent both with the Iraq Body Count tabulations and with the United Nations survey. Shortly after
The Lancet
had estimated 98,000 war deaths, the Iraq Body Count put the count between 14,619 and 16,804 as of December 7, 2004. The United Nations survey estimates war deaths at between 18,000 and 29,000, with the midpoint of that range at 24,000.

After the second
Lancet
study, Iraq Body Count officials issued a “reality check,” disputing it and pointing out inconsistencies with other data. Delving into the details, they said that if the
Lancet
study was valid it would mean, among other improbabilities, that an average of 1,000 Iraqis had been killed by violence every single day in the first half of 2006, but that only one of those killings in ten had been noticed by any public surveillance mechanism. They said it also would mean 800,000 Iraqis suffered blast wounds or other serious conflict-related injuries over the preceding two years and that 90 percent of them went unnoticed by hospitals.

We can't say the
Lancet
studies are wrong. Unlike Mitch Snyder's “meaningless” estimate of 3 million homeless persons, which we discussed in Chapter 6, the
Lancet
estimates both were derived using scientifically accepted methods and were published in a reputable, peer-reviewed journal. The findings also are stoutly defended not only by the authors but by some independent experts as well. Nevertheless, given both the extraordinary imprecision of the figures and their failure to square with other observations, we can't accept them as accurate until and unless validated by other researchers using a much larger sample.