WHAT NEXT:
The Key to Research Influence

By Michael J. Petrilli

Quality data and sound analysis matter, after all

It is notoriously difficult to determine the impact of research and scholarship on public policy. While we all might like to imagine that the perfect study, a powerful book, a trenchant article, even a well-timed op-ed might turn the tide of history, this rarely happens. And when it does, it takes eons. Consider Charles Murray’s definitive work on social policy, Losing Ground, published 12 long years before Congress enacted welfare reform. Or George L. Kelling and James Q. Wilson’s 1982 Atlantic Monthly article “Broken Windows,” which took just as long to gain traction in the nation’s police departments. Or, to pick an example from education, the late Milton Friedman’s voucher proposal (circa 1955), which wasn’t implemented in any form until Milwaukee’s program began in 1990.

Nevertheless, those of us at the Thomas B. Fordham Foundation, unabashed influence peddlers for sure, were curious about the impact of education studies published over the past decade or so. (We’re turning ten this year.) Among other things, we wanted to know whether the most influential studies were also the best studies, or if there was something else that explained their prominence.

So we contracted with the Editorial Projects in Education Research Center, publishers of Education Week, to find out. They surveyed a host of education insiders, analyzed citations in academic journals, and tallied media hits. They computed scores across those three categories and identified 13 studies that stood head and shoulders above the rest (see Figure 1).

The list is eclectic. The studies range from large-scale assessments (National Assessment of Educational Progress [NAEP] and Trends in International Mathematics and Science Study [TIMSS]), to evaluations of specific interventions (class-size reduction and vouchers), to commission reports (National Reading Panel, National Commission on Teaching and America’s Future), to data analyses (Education Trust on teacher quality, Jay Greene on graduation rates). What to make of them? Is a study’s influence related to its rigor?

At first blush, the most rigorous methodology, randomized field trials (RFT), appears not to fare well, since few such studies made the list. But solid RFT studies exist in only a handful of areas in education—class-size reduction, early reading, vouchers—and all appear here. Indeed, it’s hard to think of any RFT work more than a few years old that didn’t make the cut.

Policy analysts using other methodologies can also take heart. Smart, provocative data analyses or syntheses pack a punch. Look at the Education Trust on teacher quality, for example. Its “studies” were short policy briefs that combined solid research, unimpeachable data, and a powerful and popular ideology (“close the achievement gap”) in an accessible format.

Of course, influence is inevitably linked to impact, to discernible changes in policy or practice (for better or for worse). On this count, all sorts of studies have enjoyed success. The National Reading Panel led directly to the federal Reading First program. The Tennessee Student/Teacher Achievement Ratio Experiment (STAR) class-size study inspired action at the federal and state levels. Jay Greene’s work on graduation rates prompted a commitment by the governors to report these data in a consistent fashion. Ed Trust’s work led to the “highly qualified teachers requirement in NCLB.” Paul Peterson’s voucher studies helped shape the D.C. voucher experiment authorized by Congress. And the American Diploma Project report motivated at least 25 states to vow to align their K–12 outcome standards with their higher-education entrance expectations. None of these actions was automatic—each was pushed by advocates with agendas—but it’s fair to say that policy or practice would be different had the studies never been published.

As for NAEP and its international cousin, TIMSS, though they surely differ in kind from the others, we dare not underestimate their influence. NAEP isn’t called the “Nation’s Report Card” for nothing, and the man on the street (and the reporter in the field) cares most of all about whether test scores are rising, which states are falling behind, and whether we can compete in world markets. These “horse race” questions are likely to drive the education influence industry for years to come.  

THE WINNERS: THIRTEEN INFLUENTIAL EDUCATION STUDIES National Assessment of Educational Progress (NAEP) The National Assessment of Educational Progress (NAEP), broadly known as the “Nation’s Report Card,” is the only nationally representative assessment that enables comparisons of results across states and jurisdictions as well as changes in results over time. Trends in International Mathematics and Science Study (TIMSS) TIMSS assesses student achievement in mathematics and science at the 4th- and 8th-grade levels in the United States and other participating countries. In 2003, some 46 countries took part. National Reading Panel (NRP), “Teaching Children to Read: An Evidence-Based Assessment of the Scientific Research Literature on Reading and Its Implications for Reading Instruction,” 2000 Congress convened the National Reading Panel in 1997 and charged its members with reviewing research-based findings on reading instruction. The panel reviewed only studies that appeared in English in a refereed journal, that focused on children’s reading development from preschool through grade 12, and that were experimental or quasi-experimental in design. Tennessee Student/Teacher Achievement Ratio (STAR) Experiment In this four-year longitudinal class-size study, more than 7,000 students in 79 schools were randomly assigned to one of three classroom situations: small class (13 to 17 students per teacher), regular class (22 to 25 students per teacher), or regular class with a full-time teacher’s aide. An analysis of academic achievement found that smaller class sizes resulted in higher achievement than either of the regular class–size situations. The National Academies’ Commission on Behavioral and Social Sciences and Education (CBASSE), “Preventing Reading Difficulties in Young Children,” 1998 This commission reviewed research on topics related to the development of reading and reading outcomes, such as normal reading development and instruction, factors identifying groups and individuals at risk for reading difficulties, and prevention, intervention, and instructional approaches related to positive reading outcomes. The commission report found that many factors can promote learning to read, such as exposure to experiences in early childhood, promoting motivation related to reading, and attendance at schools that provide effective reading instruction. William L. Sanders on Value-Added Methodology and the Tennessee Value-Added Accountability System This statistical methodology introduced a new paradigm for predicting student academic progress and comparing the prediction to the contribution of individual teachers (or value added) as measured by student gain scores. Sanders’s methodology can provide an indication of an individual teacher’s effectiveness based on his or her students’ performance. The Education Trust on Teacher Quality The Education Trust has produced a number of noted reports on teacher quality. “Good Teaching Matters: How Well-Qualified Teachers Can Close the Gap” (1998) makes the case that the capability of the teacher, rather than influences from outside the classroom, has the strongest effect on student learning. In addition, the report highlights data showing that poor and minority students are more likely to be taught by less-qualified teachers. National Commission on Teaching & America’s Future (NCTAF), “What Matters Most: Teaching for America’s Future,” 1996 This commission report maps out a plan for providing every child with high-quality teaching by attracting, developing, and supporting excellent teachers. It contends that the capability of the teacher has the strongest effect on student learning, and that “recruiting, preparing, and retaining” quality teachers is the most important way to improve education. The National Academies’ Commission on Behavioral and Social Sciences and Education (CBASSE), “How People Learn: Brain, Mind, Experience, and School,” 1999 This commission report brings together findings from the fields of neuroscience, cognitive and social psychology, human development, and emerging technologies in examining the processes of effective learning and the environments in which learning best takes place. Key findings include the idea that students come to the classroom with preconceived notions of how the world works. If their schooling does not engage these notions, students may not grasp new concepts that they are taught. Richard F. Elmore on School Reform Drawing on his personal interactions with Community District #2 in New York City, Elmore promotes the idea that school reform cannot be imposed through artificial constructs developed by outside policymakers, but must begin from the inside with a commitment by educators to develop the knowledge, structures, and practices at the heart of instruction. His work examines the promise of distributed leadership, the potential for improved incentive systems to promote effective practices and bring them to scale, and the necessity to maintain a tight focus on instruction within an internalized accountability system. Jay P. Greene on High-School Graduation Rates Greene’s “High School Graduation Rates in the United States” (revised in 2002) employs a unique method for calculating graduation rates and presents his own rates for each of the 50 states, for distinct racial and ethnic group breakdowns, and for the 50 largest school districts. Greene’s results suggest that high-school graduation rates in the United States are lower than the public previously believed. American Diploma Project (ADP), “Ready or Not: Creating a High School Diploma That Counts,” 2004 Achieve, Inc., the Education Trust, and the Thomas B. Fordham Foundation established the American Diploma Project, whose report contends that the high-school diploma too often fails to signify that students are well prepared for postsecondary education or the workplace. As a remedy, it provides “college and workplace readiness benchmarks” designed to help states align their high-school assessments and graduation requirements with the demands of credit-bearing college courses and quality jobs. Paul E. Peterson on School Choice and Vouchers Peterson’s studies of voucher programs, including publicly funded programs in Milwaukee and Cleveland, find that participating African American students make greater academic gains than similar students who are not enrolled in the program. Peterson’s research also addresses the equality issue in the vouchers debate, rejecting the notion that voucher programs “skim” the best and brightest students away from public schools.