Leveraging Lotteries for School Value-Added: Testing and Estimation

Conventional value-added models (VAMs) compare average test scores across schools after regression-adjusting for students’ demographic characteristics and previous scores. The resulting VAM estimates are biased if the available control variables fail to capture all cross-school differences in student ability. This paper introduces a new test for VAM bias that asks whether VAM estimates accurately predict the achievement consequences of random assignment to specific schools. Test results from admissions lotteries in Boston suggest conventional VAM estimates may be misleading. This finding motivates the development of a hierarchical model describing the joint distribution of school value-added, VAM bias, and lottery compliance. We use this model to assess the substantive importance of bias in conventional VAM estimates and to construct hybrid value-added estimates that optimally combine ordinary least squares and instrumental variables  estimates of VAM parameters. Simulations calibrated to the Boston data show that, bias notwithstanding, policy decisions based on conventional VAMs are likely to generate substantial achievement gains. Estimates incorporating lotteries are less biased, however, and yield further gains.