Economic Policy Group co-organizers, John Cochrane and Valerie Ramey, hosted a talk on “Quantifying Non-Sampling Variation: College Quality and the Garden of Forking Paths.”
Presenter: Jeffrey Smith, the Paul T. Heyne Distinguished Chair in Economics and the Richard Meese Chair in Applied Econometrics at the University of Wisconsin–Madison.
Moderator: John Cochrane, the Rose-Marie and Jack Anderson Senior Fellow at the Hoover Institution
SUMMARY
Empirical economics papers report standard errors that quantify the uncertainty associated with sampling variation but rarely consider non-sampling variation in a systematic way. Nonsampling variation arises from researcher study design choices related to measurement of key variables, functional form, tuning parameters, model selection procedures, and so on. This paper documents the current state of play regarding non-sampling variation and describes approaches from inside and outside of economics to quantify such variation more systematically. We provide a worked example in the form of an analysis of the labor market and educational effects of college quality on degree completion and earnings using data from the National Longitudinal Survey of Youth 1997 cohort (NLSY-97). In our analysis, we consider multiple ways to create our college quality index, multiple ways to deal with item non-response in our conditioning variables, multiple ways to code our earnings outcome measures, and multiple ways to choose a specification for our conditioning variables. We find that in this context, in which sampling variation matters a lot due to the relatively small sample size of the NLSY-97 and relatively high residual variance of our outcomes, these dimensions of non-sampling variability imply uncertainty in our parameter estimates on a par with the sampling variation.
To read the slides, click here.
WATCH THE SEMINAR
Topic: Quantifying Non-Sampling Variation: College Quality and the Garden of Forking Paths
Start Time: October 29, 2025, 12:15 PM PT
PARTICIPANTS
Jeffrey Smith, Valerie Ramey, John Cochrane, John Taylor, Annelise Anderson, Richard Coillot, Camille DeJarnett, Nick Gebbia, Siddarth Gundapaneni, Eric Hanushek, Ken Judd, Chris Karbownik, Evan Koenig, David Laidler, Charles Leung, Jacob Light, Megan Liu, Elena Pastorino, Armando Perez-Gea, Rocío Sánchez Mangas, Paola Sapienza, Richard Sousa, Bebel Vieira, Mike Wu
