Super Crunchers (11 page)

“It's what I was taught and it's what patients have been taught to expect,” he said.

Worse yet, there is an extensive literature on “medical myths” that persist in practice long after they've been refuted by strong statistical evidence. On the ground, many practicing doctors still believe:

Vitamin B
₁₂
deficiencies must be treated with shots because vitamin pills are ineffective.

Patching the eye improves comfort and healing in patients with corneal abrasions.

It is wrong to give opiate analgesics to patients with acute abdomen pain because narcotics can mask the signs and symptoms of peritonitis.

However, there is evidence from carefully controlled randomized trials that each of these beliefs is false. It's not surprising that the untrained general public clings to “folk wisdom” and unproven alternative medicines. But how do medical myths persist among practicing physicians?

Part of the persistence comes from the idea that new studies aren't really needed. There's that old Aristotelian pull. The whole idea of empirical testing goes against the Aristotelian approach that has been a guiding principle of research. Under this approach, researchers should first try to understand the nature of the disease. Once you understand the problem, a solution will become self-evident. Semmelweis had trouble publishing his results because he couldn't give an Aristotelian explanation for why hand-washing saved lives. Instead of focusing on the front-end knowledge about the true nature of disease, EBM shows the power of asking the back-end question of whether specific treatments work. This doesn't mean that medical research should say goodbye to basic research or the Aristotelian approach. But for every Jonas Salk, we need a Semmelweis and, yes, a Berwick to make sure that the medical profession's hands are clean.

The Aristotelian approach can go seriously wrong if doctors embrace a mistaken conception or model for how an ailment operates. Many doctors still cling to mistaken treatments because they have adopted a mistaken model. As Montana physician and medical myth-buster Robert Flaherty puts it, “It makes pathophysiologic sense, so it must be true.”

Moreover, once a consensus has developed about how to treat a particular disease, there is a huge urge in medicine to follow the herd. Yet blindly playing follow the leader can doom us to going the wrong way if the leaders are poorly informed. Here's how it happens: we now know that B
₁₂
pills are just as effective as shots—they both work about 80 percent of the time. But imagine what happens if, just by chance, the first few patients who are given B
₁₂
pills don't respond to the treatment positively (while the first few that receive B
₁₂
shots do). The pioneering clinician will respond to this anecdotal evidence by telling her students that B
₁₂
pills aren't effective supplements. Subsequent doctors just keep giving the shots, which are in fact effective 80 percent of the time but which are also a lot more expensive and painful.
Voilà,
we have an informational cascade, where an initial mistaken inference based on an extremely small sample is propagated in generation after generation of education.

Other physicians reject evidence-based medicine, saying that there often aren't any good statistical studies to guide their decision. Even today, many medical procedures do not have systematic data to support them. Lisa Sanders, the “Diagnosis” columnist for the
New York Times Sunday Magazine
and an internist at Waterbury Hospital in Connecticut, puts it this way: “Even though evidence-based medicine has made tremendous strides,” she says, “still only a tiny fraction of what we do in the course of a day is actually evidence-based.”

Lisa is the perfect person to step back and think about the rise of evidence-based medicine, because she has lived through the sea change in emphasis from the early days of her training in the 1990s. “Statistics really wasn't taught until a few years ago,” she says. “I was on the beginning of the wave.”

Born and raised in South Carolina, Lisa is a true renaissance woman. In a former life before becoming a doctor, she was an Emmy-winning producer for CBS news. Her current “Diagnosis” column is the inspiration for the hit TV show
House,
where she is the lead medical consultant. She cares passionately about evidence-based medicine, but sees its limitations.

“Right now I am teaching all of these residents all the stuff of the physical exam,” she tells me with characteristic candor, “but I don't tell them how sensitive it is or how specific it is or what its positive predictive value is because mostly we don't know. I try to—to whatever extent I know about it. But it is still a huge gray area, and when you think that one of your fundamental tools—the physical exam and taking the history—is not at all evidence-based at this point, it's just kind of crazy. So that is where we are. We are just beginning.”

In some ways, it shouldn't be surprising that there aren't always evidence-based answers. Medical research is damned expensive. The cost not only cuts down the quantity done, it also biases what kinds of questions get studied and reported. Pharmaceutical companies fund efforts to prove that their drugs are effective, but are not about to ask whether non-proprietary treatments are equally as good.

And even when rigorous empiricism exists, it may not provide information that is relevant to the patient in question. The protocols for clinical trials routinely disqualify patients with many comorbid diseases so that the study can exclusively assess the impact of a treatment on a particular disease. And there has been an unfortunate tradition of excluding women and minorities from these trials.

It's also reasonable for doctors to resist statistical studies that are badly done. Some studies don't ask the right questions or control for sufficient variables. A few years back, a study concluded that excessive caffeine consumption increased the risk of heart disease—without controlling for whether the patients in the study smoked. Turns out that smokers were more likely to drink a lot of coffee and that smoking, not caffeine, was the real cause for the heart failure. This kind of resistance is completely consonant with evidence-based medicine. EBM only asks doctors to assess the quality of different types of evidence and to give high-quality, systematic research the appropriate weight when making their treatment decisions. The problem is that many doctors fail to make any assessment of relevant EBM results because they never learn that a study exists.

“You Could Look It Up”

A physician's decisions can't be driven by statistical results if the physician doesn't know what the statistical result is. For statistical analysis to have impact, there needs to be some kind of a transmission mechanism that disseminates the analysis to the decision makers. The rise of Super Crunching is often accompanied and nurtured by improvements in the technology of transmission so that decision makers can more quickly access and respond to data in real time. We've even seen automation of this transmission link in randomized Internet testing applications. Google's AdWords not only reports your results in real time, but it will automatically shift your web page image toward the version that produces the best results. The faster Super Crunching results are available, the faster they can change a decision maker's choices.

In sharp contrast, the practice of medicine before the EBM movement was shackled with an extremely slow and inefficient mechanism for disseminating medical results. The Institute of Medicine estimates that it took “an average of 17 years for new knowledge generated by randomized controlled trials to be incorporated into practice, and even then application [was] highly uneven.” Progress in medical science occurred one funeral at a time. If doctors didn't learn about something in medical school or in their residency, there was a good chance they never would.

As in other Super Crunching contexts, the EBM movement has tried to shorten the time it takes to disseminate important results. The central, but probably most deeply resisted, demand of EBM is the call for physicians to research the problems of their particular patients. Just as Semmelweis angered doctors by calling on them to wash their hands several times a day, the evidence-based medicine movement has had the impertinence to ask doctors to change what they do with their time.

Of course, doctors shouldn't do patient-specific research for all their patients. It would be a huge waste of time to hit the books when a patient presents with the classic symptoms of a common cold. And in the emergency room, there simply isn't time to do research. But academics who have “shadowed” practicing physicians find that about one in three new patients presents questions that would benefit from a review of current research. And this proportion rises for patients newly admitted to a hospital. Yet very few physicians in these studies actually take the time to look up the answers.

The critics of evidence-based medicine often focus on the lack of information. They claim that in many instances high-quality statistical studies just don't exist to provide guidance for the myriad of questions that arise in day-to-day clinical decision making. A deeper reason for resistance is just the opposite problem: there is too much evidence-based information for any individual practitioner to reasonably absorb. On coronary heart disease alone, there are more than 3,600 statistical articles published each year. A specialist who wants to keep up in the field would have to read more than ten articles every day (including weekends). At fifteen minutes an article, that's two and a half hours a day dedicated to reading about just one class of disease. “And it would be a waste of time,” Lisa says. “Most of those articles aren't any good.”

Clearly, asking physicians to devote such a large amount of time to sifting through mountains of statistical studies was never going to be feasible. From the very beginning, the advocates of evidence-based medicine understood that
information retrieval
technology was critical to allowing practicing physicians to pull the relevant and high-quality information from the massive and ever-changing database of medical research. Indeed, in their original 1992 article, Guyatt and Hackett looked into their crystal ball and imagined how a junior medical resident might respond to “a 43-year-old previously well man who experienced a witnessed grand mal seizure.” Instead of just asking a senior resident or attending physician what to do, the evidence-based practitioner would proceed “to the library and, using the ‘Grateful Med program,' conduct
…
a computerized literature search…. The search coststhe resident $2.68, and the entire process (including the trip to the library and the time to make a photocopy of the article) [would take] half an hour.” The idea that a resident could go to the library, locate, and photocopy an article in half an hour in retrospect was ludicrously optimistic, but something has happened since 1992 to make the idea of half-hour research much less of a pipe-dream: Al Gore's friend, the Internet. It's sometimes hard to remember, but the first web browser wasn't released until 1993.

“It was technology—the computer, the Internet—that finally made it possible for the doctors in the trenches, doctors taking care of patients, to systematically practice evidence-based medicine,” Lisa says. “We all now have access to the newest and best in medical research right at our own desks. We can find out quickly, for example, that arthroscopic knee surgery for osteoarthritis and postmenopausal hormone replacement for the prevention of heart disease have lost their standing as effective therapies.” Because of the web, every examination room now can have a virtual library.

The retrieval technology of the web has made it a lot easier for physicians to find the results that relate to the specific problems of specific patients. Even though there are more high-quality statistical articles than ever, it is simultaneously faster than ever for physicians to find the haystack's needle. A host of computer-assisted search engines now exist that are dedicated to putting doctors in touch with relevant statistical studies. Infotriever, DynaMed, and FIRSTConsult are all there to help doctors find the summaries of cutting-edge research with just a few clicks of their mouse.

These summaries of results usually have a web link so that the physician can click through and see both the underlying study and all subsequent studies that have cited it. But even without clicking, a doctor can tell a lot from the initial search result just by looking at the “Level of Evidence” designation. Today, every study is given a grade (on a fifteen-category scale developed by the Oxford Center for Evidence-Based Medicine) as a shorthand way to inform the reader of the quality of evidence. The highest possible grade (“1 a”) is only awarded when there are multiple randomized trials with similar results, while the lowest grade goes to suggested treatments that are based solely on expert opinion.

This change toward succinctly characterizing the quality of the evidence may well be one of the most important impacts of the evidence-based medicine movement. Now a practitioner in evaluating a study recommendation can have a much better idea of how much he or she can trust the advice. One of the coolest things about Super Crunching regressions is that they not only make predictions but are able to simultaneously tell you how precise the prediction is. The same kind of thing is now happening with these levels of evidence disclosures. EBM not only makes treatment recommendations but simultaneously tells the physician the quality of the data backing up the recommendation.

This grading of evidence is a powerful response to the naysayers who claim that an evidence-based approach won't fly because there simply aren't enough studies to answer all the questions that doctors need to answer. Grading still lets experts answer pressing questions of the day even in the absence of authoritative statistical evidence. It just requires them to reveal the limits of the field's current knowledge. The level of evidence-grading scale is also a simple but real advance in information retrieval. The harried physician can now cruise through dozens of Internet search results and distinguish anecdotes from robust multi-study outcomes.