Stone Inequality & Social Policy Seminar: Katy Milkman

The Power and Pitfalls of Megastudies for Advancing Applied Behavioral Science

Katy Milkman, James G. Dinan Professor at The Wharton School of the University of Pennsylvania

Abstract: Increasingly, policymakers are turning to behavioral science for insights about how to improve citizens’ decisions and outcomes. However, these insights can only inform policy insofar as they are comparable—and unfortunately, different intervention ideas are typically tested across different samples on different outcomes over different time intervals. In my talk, I will introduce the “megastudy,” a massive field experiment in which the effects of many different interventions are compared in the same population on the same objectively measured outcome for the same duration.  I will then share results from four megastudies that my team has conducted over the last several years to illustrate the power and pitfalls of this methodology. I will describe a megastudy targeting physical exercise among 61,293 members of an American fitness chain in which 30 scientists worked in small, independent teams to design a total of 54 different four-week digital programs encouraging exercise. Next, I will discuss a megastudy encouraging flu vaccination at doctor’s appointments among 47,306 patients of two large U.S. health systems in which 42 scientists worked in teams to design a total of 19 different text reminders using a variety of different behavioral science principles.  I will then describe a megastudy encouraging in-pharmacy flu vaccinations among 689,693 Walmart pharmacy patients in which 44 scientists worked in teams to design a total of 22 different text reminders using a variety of different behavioral science principles.  Finally, I will discuss a megastudy encouraging COVID-19 bivalent booster vaccination among 3.5 million patients of CVS Pharmacy in which 13 scientists worked together to design a total of 8 different interventions including text reminders and free round-trip rides to CVS.  I will discuss the accuracy of experts’ and laypeople’s forecasts of the performance of different interventions tested in these studies, share best practices for running megastudies, and describe some key limits of this approach to applied behavioral science research.

Katy Milkman is the James G. Dinan Professor at The Wharton School of the University of Pennsylvania, host of Charles Schwab’s popular behavioral economics podcast Choiceology, and the former president of the international Society for Judgment and Decision Making. She is also the co-founder and co-director of the Behavior Change for Good Initiative, a research center with the mission of advancing the science of lasting behavior change. Over the course of her career, Katy has worked with or advised dozens of organizations on how to spur positive change, including Google, Walmart, CVS, the White House, the United Nations and the U.S. Department of Defense. She has published over 60 research papers in leading academic journals such as Nature, The Journal of Finance, and The Proceedings of the National Academy of Sciences and is the author of the international bestselling book How to Change: The Science of Getting From Where You Are to Where You Want to Be, which was named one of the eight best books for healthy living in 2021 by the New York Times. Katy is particularly proud that in 2022, she won her university’s highest teaching award, the Provost’s Lindback Award for Excellence in Teaching. She writes frequently about behavioral science for major media outlets such as The Washington Post, The New York Times, CNN and The Economist. Katy earned her undergraduate degree from Princeton and her PhD from Harvard.