Vaccination News Home Page

http://www.nytimes.com/2002/06/28/health/28MAMM.html

Mammogram Team Learns From Its Errors

By MICHAEL MOSS

ENVER — Seven years ago, Dr. Kim A. Adcock started a revolution in mammography: He decided to keep score.

Dr. Adcock had just become radiology chief at Kaiser Permanente Colorado, and he was already hearing whispers of problems with his staff. So he pored over the doctors' records, counted the cancers they had missed and printed their batting averages in bar charts and graphs.

Advertisement

This was deeply controversial territory. To many doctors, keeping score was yet another assault on their autonomy and prestige. It could also, they warned, be dangerous: The statistics were tricky and easily twisted. The malpractice lawyers would pounce. Worse still, if women knew how many cancers their doctors had missed, they might avoid mammograms altogether.

When Dr. Adcock looked at the numbers, though, he saw a promise of revelation, a fair and rigorous way to hold his mammography doctors — and perhaps doctors in other specialties — accountable for their work. That was what Americans were demanding from the health care system, wasn't it?

So he pushed on. When he discovered that one doctor had missed 10 cancers in the space of 18 months, he fired him. Over the next two years, he fired two others who were missing more than their share of tumors. He then reassigned eight doctors who were not reading enough films to stay sharp — or for the data to show how sharp they were. "I had to assume they might be dangerous," he explains.

The immediate result was sensational headlines and much in-house angst. But today, Dr. Adcock's team is missing one-third fewer cancers and has achieved what experts say is nearly as high a level of accuracy as mammography can offer.

"Every mammography program in the country should be doing something like this," says Dr. Robert A. Smith, the American Cancer Society's screening chief.

Very few do. In fact, what Dr. Adcock has created is a mirror image of American mammography as usual — an industry that remains deeply troubled 10 years after Congress set out to clean it up through its own experiment in medical regulation.

At the heart of Dr. Adcock's experiment was his willingness to confront his doctors and focus on their skill in spotting tumors in the swirls and shadows of X-ray film, what experts call the hardest job in radiology. For the government, facing withering resistance from physicians, regulating doctors proved too politically risky. At the very moment in 1995 that Dr. Adcock was beginning to hold his doctors to the statistical fire, regulators were settling for a system that concentrated on the X-ray machines and the images they produced.

In that breach, a yearlong examination by The New York Times has found, the government has fallen far short of its pledge to ensure high-quality mammography for all. Here in Denver, Dr. Adcock has winnowed his team down to a few specialists. By contrast, most of the 20,000 doctors in the United States reading breast X-rays are generalists with limited training and practice in mammography. Many lack the skill needed to do so effectively, yet neither they nor their patients have the tools to find out who is good and who is not.

Keeping score, though, is not simply a matter of identifying and weeding out the worst practitioners. For Dr. Adcock and his admirers, the statistics offer a way to approach a more pervasive, and more elusive, problem that increasingly preoccupies the entire medical profession: the mistakes that, to varying degrees, all doctors make.

At Kaiser in Denver, the statistics anchor a regimen of continuous education that far outstrips the few hours a year the government requires. The Denver doctors are constantly analyzing their errors, searching for those meaningful patterns of shadow that they have missed, perhaps again and again.

The Denver group is not the first to use statistics to track doctor performance. A small but growing number of other mammography programs are beginning to keep score. Several states now publish individual doctors' death rates for open-heart surgery. But seven years in, the Kaiser mammography group has perhaps gone as far as anyone in creating a statistical system for holding doctors accountable for their work.

Still, even many of Dr. Adcock's admirers point out that he has achieved his success in the closed and relatively manageable confines of a health maintenance organization. They wonder if it can be replicated broadly, especially since the government has not fulfilled its promise of a national registry of cancer cases. At Kaiser, if a woman receives a breast cancer diagnosis, a doctor can find past mammograms and see if her case was missed. In the world at large, a doctor will often do a mammogram and never see or hear of the patient again.

In the end, though, the most delicate obstacle may be the doctors themselves. Doctors have been pushed a good distance from their traditional pedestals. But few have done so especially happily, and rare is the physician eager for the psychic roughing up that comes when the Denver doctors are forced to confront their mistakes.

Dr. Ken Heilbrunn, a Seattle radiologist who says he admires what Dr. Adcock has done, calls this the "shame" factor, and manipulating it is the stealth ingredient of the Kaiser method. "To really improve your skills," he explains, "you have to repeat this shameful moment over and over."

It's a tricky business, this question of the doctor's image. Even today, Kaiser is reluctant to advertise its turnaround, and it would share only some data with The Times. Too many people still believe doctors walk on water, one official explained, so how can we brag about making fewer mistakes?

In wielding those mistakes, Dr. Adcock says he pledged from the first to avoid emotions and hold everyone accountable, including himself. After all, he says, "it is easy for us to delude ourselves about the quality of our work."

Setting a New Standard

The revolution began with a shot in the dark.

When several physicians complained about an apparent missed case in September 1994, Kaiser dealt with it in a typically ad-hoc way: the radiologist in question was encouraged to have another doctor double-read his films for a while. There was no reason to go further, Kaiser reasoned, since even experts make mistakes.

But soon after, the H.M.O. named a new radiology chief, Dr. Adcock, with a different approach, drawn from his personality and personal experience.

Advertisement

Five years before, Kaiser had hired Dr. Adcock for a variety of X-ray work. He had little grounding in mammography and a cautious, statistical turn of mind not entirely common in a doctor. In fact, he had thought about becoming a lawyer, and in medicine had sought out a specialty about as far removed from patients — and especially, he says, their blood — as possible.

Starting out at Kaiser, he had dreaded missing too many tumors. "A good deal of what we do in radiology does not have the same sort of health implications," he says. "`With mammography, you're looking for the opportunity to save a life."

He devised a personal oversight system, using what are known as medical-outcome data, in which a doctor's action is tracked to see how the patient fared.

Mammography, he felt, was well-suited to a statistical approach. Unlike, say, hip surgery, with its many gradations of success — is it the ability to walk, or run, with or without a limp? — the equation in mammography is fairly straightforward. The radiologist concludes that a woman appears to have cancer or not, and over time that judgment is proved right or wrong.

If mammography was the ideal medium, Kaiser was the ideal laboratory, since it already tracked its members. So when Dr. Adcock began his new job, he quickly homed in on his suspect employee. The doctor, it turned out, had not missed just one case; he had apparently missed a lot.

In taking the matter to his bosses, Dr. Adcock says now, he realized he was stepping into a running debate. He remembered the furor, and the mixed lessons, of the heart-surgery initiative in New York.

After the surgeons' scores began appearing, the heart-surgery death rate had fallen by about 40 percent. Dr. Mark Chassin, a former New York State health commissioner, says hospitals were pressed to fix underlying problems. New Jersey and Pennsylvania have since begun their own listings.

Some researchers suggested, however, that other factors might have driven down the death rate. They questioned the soundness of the data. They warned that surgeons might be increasing their scores by avoiding higher-risk patients, a criticism that prompted the state to refine its system.

But where others saw controversy, Dr. Adcock saw opportunity.

"For me," he recalls, "it was a feeling of exhilaration that here at last was some aspect of medicine that could be measured and managed."

He rechecked his numbers, then sat down with Kaiser officials, lawyers and public-relations people. They were worried about many things — negative publicity, malpractice claims, women turning away in skepticism. How many might die because they stopped getting tested?

But there was another danger they could not ignore. The radiologist had read 3,000 mammograms, and if Dr. Adcock was right, a dozen or so women he had said were fine in fact had breast cancer.

Making Tough Decisions

Finding those women was a huge job. The radiologist's films had to be culled from the files and reread by several doctors.

They concluded that 259 women needed follow-up X-rays. Kaiser brought these women back in, gave biopsies to 30, and in the end, 10 women were found to have cancer, the H.M.O. says.

Word eventually reached The Rocky Mountain News, a local newspaper, which reported it as a front-page medical scandal.

In its defense, Kaiser said it had uncovered the situation through its own detective work. Steve Krizman, who edited the Rocky Mountain News' coverage and later joined Kaiser as a spokesman, said he was skeptical enough about Kaiser's assertion to mention it only briefly in the pieces.

"I thought, `That's how they are trying to spin it,' " Mr. Krizman recalls.

But if the news media missed the broader story, some of the women involved did not.

At first, Ann Veenstra felt spun when she got a phone call asking if she would mind getting another mammogram. "I felt something was wrong," says Ms. Veenstra, an administrative assistant.

It felt especially wrong in her case. The mammogram had been her first, a baseline test at 40; she had not planned another for five years. When she turned up with cancer, she says, "I was so very angry."

But Kaiser explained how it had found her cancer, and she realized that after potentially killing her off, the H.M.O. may have saved her. "After I got over my initial shock and anger, I appreciated that someone was checking and double-checking," she says. "It's unbelievable to me this is not nationwide."

Missed breast cancer is a leading malpractice complaint. But Kaiser was sued by just one woman, who eventually settled.

The radiologist, Dr. James A. Walsh, was crushed when Kaiser asked him to leave, his former colleagues say. He was 60, with two children in graduate school. "It was a painful moment," he said recently.

Dr. Walsh said he felt singled out for undue scrutiny and had been treated unfairly and unprofessionally. He said that an expert had found that only three of the missed cancers could be legitimately blamed on him, and that such an error rate fell within acceptable bounds.

"I think I was right and they were wrong," he says.

The Kaiser official who headed the Denver affiliate's quality-control program, Dr. Andrew M. Wiesenthal, says the treatment of Dr. Walsh was "exceedingly fair."

"We didn't take any action until it was patently clear that he didn't do this very well," Dr. Wiesenthal says.

The Colorado medical board placed Dr. Walsh on probation, and he eventually moved to North Carolina. He says he attended numerous training programs and is now reading mammograms as a fill-in radiologist in four or five states.

"All the medical staffs I work for have no problems at all with my work," he says.

Starting Fresh

Even with all the hubbub, Dr. Adcock's bosses gave him a free hand to dig deeper.

"Jim Walsh was a lovely, lovely guy," says Dr. Deborah S. Shaw, one of the radiologists on the team. "But we knew this was the right thing to do."

Which is not to say that the team did not feel wrenched by the firing, and by all the publicity. The doctors could not help but wonder who would be next.

Over the next few years, several more radiologists were fired or resigned in the face of concerns about their interpretive skill. Then Dr. Adcock spotted an even trickier problem. Nearly half the original 20 radiologists were reading far fewer mammograms than the others. They met the federal minimum of 480 a year, but with the others reading as many as 14,000, Dr. Adcock agreed with experts who say the government minimum is far too low.

Advertisement

Moreover, the low-volume doctors were not accumulating enough data to show if they were good. So he simply assumed they were not, and restricted them to other radiology tasks, like CAT scans.

How did they feel? Rather relieved, it turns out. Dr. John A. Siebert, for one, says mammography was monotonous, particularly since he might screen 200 healthy women before finding one cancer. An instructor once told him to pretend that each X-ray was his mother's, but that trick, he says, went only so far. "It's sort of tedious," he says. "You have to sort of slap yourself to look at them."

Others say they had trouble mustering — and holding onto — the intense yet relaxed concentration needed to find the more subtle tumors, what some of the Denver doctors call "the Zen zone."

"It was hard for me to get in the groove," Dr. John W. Grudis says.

Improving Accuracy

In a dark basement room, Dr. Shaw takes a deep breath, clears her mind and begins the hunt for breast cancer. It takes her just minutes to stumble.

She sits facing a large machine shaped like a player piano that holds a reel of mammograms, and when she spins the films of a 55-year-old woman into view, she is riveted to a whitish spot in the shadows.

"This one cluster has my attention," she says. "I can't tell you why, but it looks funny."

She dictates instructions for the woman to return for further testing. But the biopsy finds only normal cells.

The doctors do not take these "false positives" lightly, given the physical and psychic pain they can inflict on a patient. But there is a weightier side of the coin, the moment when a doctor finds a tumor that looks as if it has been around awhile. Then the question becomes, was the cancer visible on earlier films? If so, who read them?

"It's a horrendous experience, just an explosion of emotions at once," says Dr. Gerald L. Lourie, another team member. "You know you are either going to be free as the judge says, `Not guilty,' or you look and you know you just missed this one cold."

What distinguishes the Denver team from most others is its systematic embrace of frequently occurring shame.

The regimen begins with competency tests, in which the doctors run through a stack of mammograms. Many mammogram doctors never take even one such quiz. In Denver they do so at least three times a year. Not only do these tests allow the doctors to study their errors; they also build confidence for those who do well.

Once a year, Dr. Adcock also sends out lists of actual cancers missed, known as false negatives, so the doctors can pull the files and commit their mistakes to memory.

"That's your boss telling you, `These are the ones that weren't so hot,' " says one of the doctors, Richard A. Propper.

Last comes the toughest scrutiny of all. The doctors' hits and misses and other statistical variables are displayed in brightly colored charts for all to see.

Mammography everywhere is a constant balancing of possible harm: between missing too many cancers and ordering too many needless biopsies. But the Denver doctors say their continuous scrutiny enables them to spot weaknesses in their work before they do inordinate harm.

Over time, they say, they have made an important discovery about why they miss some tumors. Breast cancer has many different shapes on an X-ray: a line of little white dots, perhaps, or a star-shaped blob known as architectural distortion. By testing and keeping score, the Denver doctors found that they sometimes obsessed over one type and neglected the others.

Today, the team's accuracy is close to what experts say is the best mammography can offer.

Women have been told that mammograms can find 90 percent of breast cancer. But that figure stems from ideal conditions in research, and recent real-world samplings in two states show that doctors are finding just over 70 percent of the cancers in women who get regular exams.

Some clinics are doing much worse. Four of the six busiest centers in a study of screening in North Carolina are averaging about 65 percent. That is, they miss one cancer for every two they find. (Not all missed cancer can be blamed on the doctor; the X-rays might be poorly taken, and many tumors are simply too hard to see.)

The Denver team, stuck near 70 percent before it began its makeover, is now scoring 80 percent. By another measure, it is finding cancers at an earlier stage, allowing for earlier treatment. It did this without increasing the number of women it sends to biopsy. Just as critically, the group says, its team is consistently good, doctor to doctor. Women need not worry about having their X-rays read by a weak member of an otherwise strong team.

What that means, in the simplest terms, is that the Denver doctors are finding about 15 more cancers a year than they would have at their previous accuracy level. (Kaiser says it does not know if that improvement has affected its breast-cancer death rate.) In a country where 192,000 breast-cancer cases are diagnosed each year, that same increase in accuracy could mean finding upwards of 10,000 more annually.

Seeking a Better Way

Dr. Adcock is branching out. He has begun looking at outcome data for other radiology procedures, like breast biopsies, in which doctors can cause bleeding or miss the targeted cells.

And the news from Denver is starting to get around Kaiser's loose nationwide confederation of H.M.O.'s.

"Kim Adcock is at the cutting edge of everything in radiology," says Dr. William E. Drobnes of Kaiser's Maryland affiliate, "and I'm shamelessly trying to steal this."

A similar effort is under way in British Columbia, and about 120 clinics in the United States, mostly in North Carolina and New Hampshire, are volunteers in a study designed to help doctors improve their skills.

Still, this is a revolution of small steps. Even at the nation's leading cancer centers, doctors say they cannot do all Dr. Adcock has done.

Advertisement

Partly, it is that vast and inevitable well of psychic resistance. Equally important, few medical organizations can control information the way Kaiser can, as an H.M.O. that provides all of its patients' care.

"Everybody would like to do this if they could. It's a wonderful learning experience," says Dr. David Dershaw, the mammography chief at Memorial Sloan-Kettering Cancer Center in New York. "But the search for false negatives is difficult, cumbersome and expensive."

Sloan-Kettering does track false positives, and Dr. Dershaw says he is confident that his doctors, all trained by him, are highly skilled.

Still, he has never calculated their skill by tracking missed cancers. That would require contacting all the women who got negative mammograms — tens of thousands each year — to see if they later received diagnoses of breast cancer.

"I've been trying to reach one woman for three days," Dr. Dershaw says. "And I'm trying to give her the results of her biopsy. Just imagine what it would take to reach every woman who comes in."

In pursuit of a better way, Congress a decade ago ordered the creation of a national cancer registry that radiologists could search for patient records. But the system remains a cumbersome and piecemeal hodgepodge of state archives.

The data are also difficult to interpret, especially for the many doctors reading just a few hundred films a year. Several years' worth would be needed to be meaningful.

Some clinics are trying other approaches. A few have two radiologists read every X-ray independently; others are using novel computer programs that show promise in seeing some hard-to-find cancers.

Even so, when experts talk about doctors' skills, the discussion almost always circles back to the conundrum federal officials wrestled with when they wrote the mammography rules a decade ago: How to improve quality without diminishing access to care. If doctors start dropping out of mammography because they score badly in tests or performance audits, where will women go?

The balancing act gets trickier and trickier. New research is stoking concern about doctors' competency. At the same time comes anguished talk about doctors driven away by skyrocketing malpractice rates and shrinking reimbursement.

To some experts, the solution lies in a radical-sounding reorganization: centralized facilities across the country where large numbers of mammograms would be read by small teams of highly skilled, and presumably enthusiastic, experts. In the future, digital mammography, a recent and still-experimental innovation, could make sending films as easy as e-mail.

Which is remarkably similar to what they do in Denver. Call it the two-step mammogram. Women are still X-rayed at satellite offices, but the films are shipped to the central complex where Dr. Adcock's six-member team works. This has also meant lower costs for a procedure that many radiologists see as a money-losing obligation.

For now, though, many people are banking on the federal government, hoping it will pay more attention to the doctors.

Legislation moving through Congress to extend the federal mammography rules would have the Institute of Medicine, an independent research group, study several matters, from doctor training to interpretive skill. Breast-cancer screening advocates are quietly pushing Congress to take strong steps, sooner. The American College of Radiology, which accredits the nation's mammography doctors, says it would support a federal requirement for periodic competency drills.

Many experts, like Dr. Robert A. Schmidt of the University of Chicago, say that only a complete government overhaul can do the job, starting with financial incentives and ending with tools to assess doctor skill.

He is not holding his breath. "There are lots of arguments you can make in deciding to do nothing," he says. "Even with the way mammography is now, you could still say you're still doing more good than harm."

Working as a Team

Even Dr. Adcock is wary of having the government police doctors' performance. "I could see that being counterproductive," he says.

On the other hand, left to themselves, it is not clear how many doctors would do what Dr. Adcock did when his data turned on him.

The Denver doctors all have their own reading styles. Dr. Shaw likes to press her red-nailed fingers against the X-rays when she zeroes in on a problem spot. Dr. Geoffrey D. Friefeld burns through films at a torrid two-minute pace.

Dr. Adcock is a fretter. "Oh boy, I hate it when that happens," he said one afternoon last summer when he couldn't make up his mind. "This one is very hard to let go."

Then his latest scores came in, and he really started to worry. He was dumping more X-rays into an ambiguous pile, having failed to decide if they showed cancer or not. Holding his charts, he said, "I look at that and think, my goodness, have I forgotten how to read mammograms?"

He labored over the tougher cases, and even his body language — big exhales and slouching — seemed to show his concern. He thought about the radiologists he had exiled, including Dr. Walsh.

Then his volume began to slip as he spent more time on management duties, and he wondered: If his accuracy slipped, too, would he see it in his data? "That was the hardest thing," he says, "knowing that I might not be able to tell."

Late last year, his volume slipped below 200 a month, and as his colleagues watched his numbers drop, they feared the worst. If he did not stop himself, Dr. Shaw says, "I would have had to tell him to."

On Jan. 1, Dr. Adcock decided to stop reading mammograms. He did not want to burden the team with his workload, since the original group of 20 was down to 6. But he says he had a bigger obligation in bailing out of a task he had come to love: "I'm protecting the patients against myself."

Vaccination News Home Page

ALL INFORMATION, DATA, AND MATERIAL CONTAINED, PRESENTED, OR PROVIDED HERE IS FOR GENERAL INFORMATION PURPOSES ONLY AND IS NOT TO BE CONSTRUED AS REFLECTING THE KNOWLEDGE OR OPINIONS OF THE PUBLISHER, AND IS NOT TO BE CONSTRUED OR INTENDED AS PROVIDING MEDICAL OR LEGAL ADVICE.  THE DECISION WHETHER OR NOT TO VACCINATE IS AN IMPORTANT AND COMPLEX ISSUE AND SHOULD BE MADE BY YOU, AND YOU ALONE, IN CONSULTATION WITH YOUR HEALTH CARE PROVIDER.