Swing and a Miss: Do Quality Measures Have Dubious Value?
In 2006, Tampa Bay Rays executive James Click wrote an essay in the book Baseball Between the Numbers about how light-hitting backup catcher Mike Redmond fared so well against Hall of Fame pitcher Tom Glavine. During his 13-year career, Redmond had a respectable batting average of .287, but he didn't play regularly because of his mediocre defense and lack of power.
Glavine, on the other hand, spent the majority of his 22-year career as one of the best pitchers in Major League Baseball, twice winning the Cy Young Award as the best pitcher in the National League.
Redmond faced Glavine 51 times before retiring after the 2010 season, and he got a hit nearly 44 percent of the time. Not only that, he displayed more power against Glavine than he typically did against any other pitcher. In other words, when this marginal batter faced this all-time great hurler, Redmond seemed to turn into a modern-day Babe Ruth.
So what did Redmond know about Glavine that other hitters didn't? Nothing. This was a statistical anomaly, a function of small sample size. Redmond didn't face Glavine enough times to accurately say that he had any true advantage other than luck. It's possible that something gave Redmond some benefit against Glavine, but they didn't face each other enough for us to say that with certainty.
What does this anecdote have to do with health care? We have to be careful about what we assertively call "quality." For example, in my practice, I have a panel of roughly 600 patients. This is smaller than a typical family physician's panel because a significant part of my time is spent teaching medical students and residents, and I also have various administrative duties. Of these 600 patients, a little more than 50 have type 2 diabetes. With the various quality metrics that assess the care I provide to individuals with diabetes (blood pressure management, hemoglobin A1c testing and results, eye exams, foot exams and kidney disease screening -- all per the recent core quality measures from CMS), can we be certain that the sample size of my patient panel is large enough to rule out a statistical oddity in my "quality?"
Do we even know what the appropriate sample size is for reliable measurement of my quality? Not that I can find.
Spencer Nam, an analyst at the Clayton Christensen Institute for Disruptive Innovation, has pointed out that the measurement of quality in other industries focuses on process, but this is not how we measure it in health care.
"Relying on metrics when there is no agreement on a standardized process causes some of the patient care decisions to be made based on reimbursability, while adding administrative responsibilities to track measures that are irrelevant to patient outcomes," Nam said. "Instead of improving efficiency and effectiveness," not to mention the true quality of care provided, "these metrics become extremely burdensome to the system."
So if the sample size is not standardized and the metrics themselves may not be true measures of quality, what do we do? The examples of disruptive models that Nam discusses, such as Geisinger Health System and Intermountain Healthcare (as well as, theoretically, accountable care organizations), focus their quality measures on a broader scale. Although these organizations still take some account of quality as measured for individual physicians and other health care professionals, it is the quality measures of health care teams and care processes that actually make the difference.
If we rely on teams to care for patients and populations, then it makes no sense to measure quality for a single physician. There are others in my clinic who care for patients with diabetes, but we all use the same support teams to deliver the essential services to these patients. Shouldn't the clinic's overall team be measured instead of the individual clinician? I think so, but we're not there yet.
So how do we know if I'm like Mike Redmond, the mediocre player who was lucky against top-notch talent, or Tom Glavine, the ace who was dominated in a small sample? The best way to accurately answer that question is to see which team actually won the most.
Kyle Jones, M.D., is a faculty member at the University of Utah Family Medicine Residency Program in Salt Lake City. He is the director of primary care at the Neurobehavior HOME Program, a patient-centered medical home for those with developmental disabilities. You can follow him on Twitter @kbjones11.
Subscribe to receive e-mail notifications when the blog is updated.
- Putting Patients at the Center of My 'Clinic of the Future'
- Crackdown on Undocumented Immigrants Poses Broad Health Risks
- Stay Kind, Stay Connected, Stay Well: Advice for New Residents
- I Referred My Patients to Subspecialists; So Why Did They See NPs?
- It's Complicated: What We Wish Legislators Knew About Health Care
Our other AAFP News blog
Leader Voices Blog - A Forum for AAFP Leaders and Members