Rapid experiments to improve health-care delivery

Death from colorectal cancer can be prevented by early detection. Controlling high blood pressure would prolong the lives of the nearly 500,000 Americans who die from this disease each year. Vaccinations prevent tetanus, which could otherwise be lethal.

Clearly, preventive medicine can make a big difference to health.

And yet most people don’t get the preventive care that could save their lives. Indeed, as of 2015, only 8 percent of US adults 35 and older had received all immunizations, cancer screenings and other high-priority services recommended for them.

Researchers seeking to change that are borrowing a page from Facebook, Google and other tech companies. By rapidly comparing small differences in how they communicate with patients — a process known in the tech world as A/B testing — health-care workers can quickly learn what works and what doesn’t. The approach has already delivered several actionable improvements, though not everyone is convinced of its value.

Tech-oriented companies use A/B testing to make decisions about marketing slogans, web page colors and lots of other options. The key is randomization, meaning that people are randomly assigned to see different versions of whatever is being tested. Does a bigger “subscribe” button on a website generate more clicks than a small one? Does one headline over a story capture more readers than another?

Leora Horwitz, an internist and health services researcher at NYU Langone Health, and her colleagues adopted this technique — which they call rapid randomized controlled trials — to learn how to improve the delivery of health-care services. Randomized controlled trials, or RCTs, are widely used in medicine, typically to test new drugs or other disease treatments. For example, patients may be randomly assigned to receive a new drug or the current standard treatment, then followed for months or years to assess whether the new drug works better. But those trials are slow and expensive, in part because researchers have to recruit people willing to be in a medical experiment.

Rapid RCTs, by contrast, are not used to study new treatments, so nobody has to be recruited to participate. Rather, Horwitz’s goal is to improve health-care delivery through quick trials in which one could repeatedly test and fine-tune changes to health-care delivery based on what researchers learn from each test.

“We are randomizing what we’re doing so that we can quickly and accurately assess whether what we are doing is working,” says Horwitz, who wrote about the approach in the 2023 Annual Review of Public Health.

For example, Horwitz and her colleagues wanted to figure out how to get patients to book appointments for care gaps — preventive services that are overdue. Because of the huge number of patients, physicians’ offices can’t contact everyone by telephone or through the online patient portal that NYU Langone uses to communicate with patients. So the health system needed to understand what type of reminders were most effective.

In the A/B test, patients with care gaps were divided into two sets: those who had signed up for an online portal account and those who had not. Patients in each set were then sorted into different groups based on their health-care history. Patients who, based on past behavior, were unlikely to initiate appointments on their own were put in higher-risk groups; those who in the past had eventually booked their own appointments were assigned to lower-risk groups.

In one part of the test, several thousand patients who had no portal account were randomized so that some received a telephone call reminder and others did not. Among those who received a phone call, patients booked appointments to address 6.2 percent of the care gaps, compared with just 0.05 percent among those who did not.

In another part of the test, some patients with patient portal accounts received a reminder message through that channel, while others did not. Of those who received the message, 13 percent scheduled the needed services, compared with 1.1 percent of those who were not contacted.

Importantly, the experiments revealed that for the subgroups of patients who were high-risk and the least likely to get their preventive services without a nudge, a phone call reminder was the most effective way to reach them. Shortly after the test results were known, NYU Langone prioritized all its highest-risk patients to receive telephone reminders and greatly expanded its use of patient portal messages for the rest.

“When we learn something, we apply that to all of our messaging quickly,” Horwitz says. That immediately extends what they’ve learned to tens of thousands of people. “That’s gratifying,” she says.

Langone’s A/B testing is why thousands of NYU Langone’s female patients are now receiving short messages to remind them to schedule their mammograms. The researchers used rapid RCTs to test the wording on reminders sent through the online portal: Would shorter messages get better results? Indeed, nearly twice as many patients scheduled mammograms after getting a 76-word reminder as those who received the old 155-word message.

In another investigation, to find out how to boost vaccination rates among very young children, Horwitz and her team turned to rapid randomized tests that compared one-text and two-text reminders to parents, against parents who received no text reminder at all. Only the two-text reminder — one sent at 6 p.m., the other sent at noon two days later — made a difference, tripling the number of appointments scheduled. Most appointments were made after the second text, suggesting that this booster reminder was what triggered the parents to act.

Though it’s still new to the health-care sector, the idea of rapid RCTs is catching on. One research team — an economist, a physician and a public policy expert, none of whom was affiliated with Horwitz’s group — used the technique to learn how to increase the use of preventive care services by Black men, the US demographic group with the lowest life expectancy.

They recruited more than 1,300 Black men from Oakland-area barbershops and flea markets, asked them to fill out a health questionnaire, and gave them a coupon for a free health screening. A pop-up clinic, staffed with 14 Black and non-Black male doctors, was set up to provide the screenings, and the participating men were randomly assigned to a Black or a non-Black doctor. The result: Black men randomly assigned to Black physicians were much more likely to get diabetes screenings, flu vaccinations and other preventive services than those assigned to non-Black doctors.

Some experts doubt that rapid A/B testing will ever become commonplace in health care. Darren DeWalt, a physician who directs the Institute for Healthcare Quality Improvement at the University of North Carolina, likes the concept but he thinks most health-care organizations will avoid it for ethical reasons, because people often disapprove of randomization, even in the context of something as innocuous as appointment reminders. “People in this country don’t like the idea that they are randomly allocated to something, even something as simple as that,” DeWalt says. “There’s a lot of suspicion around researchers in health care.”

Others criticize A/B testing as tinkering at the margins. Pierre Barker, chief scientific officer for the nonprofit Institute for Healthcare Improvement in Boston, believes that significant improvements in health-care delivery require an in-depth analysis of the problem to be solved, which may require many changes to the system. By contrast, rapid randomized controlled trials focus on a single, discrete change — say, the words used in a telephone script — rather than a broader effort to understand why patients don’t get preventive services and what can be done to change that.

“The attractiveness is how fast it can move, more than the size of the impact,” he says. “I remain to be convinced that you can get more than a small incremental change” from rapid randomized controlled trials.

It is true that the vast majority of NYU Langone’s care gaps were not resolved by the new reminders, says Horwitz, but the tests did provide information that led to thousands of potentially life-saving services being performed. That is what convinces her that the health-care industry should embrace rapid randomized trials.

“If you were working for a web company or an airline or any other industry, you would randomize as a matter of course — this is the standard practice,” she says. “But it is still very foreign in health care, and it shouldn’t be.”