In the last few posts, I’ve been focusing on the role of data science in addressing the challenges of covid-19. While I will continue to address this topic in this newsletter, today I’d like to describe recent work by a team from MIT evaluating whether hospital quality measures accurately measure quality of care (email me for a copy if you can’t get behind the paywall).
Quality measures are metrics used by the Center for Medicare and Medicaid Services (CMS) to compare the quality of care at different hospitals. Important examples are “timely and effective care” measures (e.g. the administration of aspirin to patients with heart attack) and scores on the HCAHPS patient experience survey. These measures play a central role in value-based reimbursement models, whereby insurers pay for quality rather than quantity of care.
A challenge with quality measures, however, is that different hospitals serve different patient populations. A hospital with sicker patients may have higher readmission rates than a hospital serving a healthier population even if they offer the same quality of care. In fact, patients in poor health may be referred to higher quality hospitals (such as large academic medical centers), leading to seemingly poor performance for these hospitals.
To account for this, quality metrics are reported on a risk-adjusted basis, attempting to control for the variation in patient population. But, these risk adjustment techniques frequently face major (and justified) criticisms over their ability to accurately capture the health of a population based on a handful of comorbidities and demographic variables. In essence, many hospitals would like to claim that their patient population is sicker than their risk-adjustment scores would suggest.
Ultimately, this is a question of correlation and causation: does the provision of timely and effective care as defined by CMS measures actually lead to better outcomes? Or are the two merely correlated? Establishing causal relationships without randomized trials is very difficult even with modern machine learning techniques, requiring vast amounts of highly accurate data. So in the MIT team’s work, the researchers came up with a clever trick.
When an ambulance transports a patient to the emergency room, its crew decides to which hospital in the area it will take the patient. It turns out that different ambulance companies prefer different hospitals. Moreover, ambulances are assigned to patients arbitrarily, based on which ambulances happen to be available nearby. This means that ambulance allocation can be used as an “instrumental variable”, a statistical technique that adjusts the data to make it look as if the patients were, in fact, participating in a randomized trial.
The researchers used ambulance allocation to explore two main outcomes: mortality and hospital readmissions. Using the instrumental variable technique, they demonstrate that better quality measures do, in fact, lead to reduced readmission and mortality rates. For example, an increase of 10 points in timely and effective care measures leads to a 10% reduction in one-year mortality, while a 10 point increase in patient satisfaction leads to a 14% reduction in the rate of readmission.
This approach does have certain limitations. For example, it is most appropriate for emergency care and says less about the quality of care for chronic conditions. But overall, it seems that better quality measures do lead to better patient outcomes, even after risk-adjustment.
What can you learn from this? First, many provider- and payer-facing healthcare organizations utilize CMS quality measures in their daily operations, and it is reassuring to have increased confidence in their validity. Second, the important insight in this case relied on deep domain knowledge -- patterns of ambulance allocation -- and not on particularly sophisticated techniques (instrumental variable analysis is actually fairly simple.) Domain knowledge will remain crucial for all applications of data science in healthcare for the foreseeable future, and effectively integrating it into the work of data science teams will be critical for their success.