p-Hacking and False Discovery in A/B Testing
(with Ron Berman, Leonid Pekelis, and Christophe Van den Bulte)
Abstract: We investigate whether online A/B experimenters "p-hack" by stopping their experiments based on the p-value of the treatment effect. Our data contains 2,101 commercial experiments in which experimenters can track the magnitude and significance level of the effect every day of the experiment. We use a regression discontinuity design to detect p-hacking, i.e., the causal effect of reaching a particular p-value on stopping behavior.
Experimenters indeed p-hack, especially for positive effects. Specifically, about 57% of experimenters p-hack when the experiment reaches 90% confidence. Furthermore, approximately 70% of the effects are truly null, and p-hacking increases the false discovery rate (FDR) from 33% to 42% among experiments p-hacked at 90% confidence. Assuming that false discoveries cause experimenters to stop exploring for more effective treatments, we estimate the expected cost of a false discovery to be a loss of 1.95% in lift, which corresponds to the 76th percentile of observed lifts.
Lotteries and Savings In Mexico
(with Paul Gertler, Sean Higgins, and Enrique Seira)
Abstract: Despite the benefits of saving in formal financial institutions, take-up of no-fee formal savings accounts is low among the poor. Surprisingly, even after opening a savings account, use of the account is often low. In a large randomized experiment across 110 bank branches throughout Mexico, we provide a temporary incentive to both open and use a savings account: we offer prize-linked savings accounts with cash-prize lotteries, where lottery tickets are awarded as a function of savings balances. We find that 41% more accounts are opened in treatment branches than in control branches on average, and the number of accounts opened in treatment branches increases steadily over time while the lotteries were being offered. Although the incentive to save is temporary as lotteries are only offered for two months, the new accounts continue to be used over time. After five years, clients who opened accounts in response to the lottery continue saving and making transactions at the same rates as those who opened accounts in control branches during the same months.
Presentations: UC-Berkeley Economics Development Lunch Seminar (slides)
Trust, Information, and Society: An Experiment
Abstract: Trust is an essential ingredient for unlocking economic surplus. However, consider the prisoner's dilemma---all parties gain from cooperation, yet each party has an incentive to deviate. How can we organize society to unlock the possible gains from trust in such situations? We've all had experiences that indicate it is possible. Studies have shown prosocial individuals are more trustworthy. We can take advantage of this fact and suggest pairing prosocial individuals with less prosocial individuals who will trust them if their type is known. In this case, it takes information, timing, and only one pro-social individual to unlock the trust surplus. I find information actually decreases overall trust and does not impact less prosocial individuals view of the world. Consequently, too much information might negatively influence cooperation and trust by changing our biases.
Presentations: Bay Area Experimental Economics Workshop (slides)
I was a research assistant to Paul Gertler (UC Berkeley) from 2013 to 2015 on his interdisciplinary entrepreneurship training study in Uganda. I designed a labor market survey for Ugandan Youth. In 2015, I went to the field to pilot test the survey and help train enumerators to implement it. This study will measure the long term outcomes of an entrepreneurship training intervention. The intervention was specifically designed to tease out the importance of hard skills versus soft skills-- a classic debate in business schools among psychologists and economists.
In undergrad, I was a research assistant for Latika Chaduhary. I worked on a paper discovering the ROI to for-profit universities as well as collecting data on Indian Railroad stock prices.