Students in private schools learn more and score higher on standardized tests than
their counterparts in public schools. Some say this does not prove that private schools
are better but only shows that children from more motivated families (who are willing and
able to pay the tuition) attend private schools. As former Wisconsin state school
superintendent Herbert Grover, an arch-critic of school choice, has argued, "Do
private school children outperform children in public schools? It’s hard to imagine
that they wouldn’t, given the initial advantages they enjoy from their parents."
To see whether students actually learn more as a result
of attending a private school, my colleagues and I are currently evaluating a
school-choice pilot program in New York City funded by the School Choice Scholarships
Foundation (SCSF).
In February 1997 the SCSF offered public-school students from
low-income families who were entering grades one through five a chance to win a $1,400
annual scholarship, good for at least three years, to help defray the cost of attending a
private school, either religious or secular. Over 20,000 applications were received.
Ninety percent of the applicants were either Latino or African American. Scholarships were
awarded by means of a lottery. Some 1,200 SCSF scholarships were used to attend some 225
participating private schools. Students began school in the fall of 1997.
Because SCSF awarded scholarships by means of a lottery, it was
possible to evaluate the pilot program using a scientific method regularly employed in
medical research, the randomized field trial (RFT). In a medical RFT, one group is given a
pill, the other a placebo. Individuals are assigned to one or another group by lot, or, in
scientific parlance, at random. This method is preferred over all others, because the test
and control groups, on average, can be assumed to be similar, save for the medical
intervention under investigation. Positive results from RFTs are required in order to win
approval of a medication from the Food and Drug Administration.
In my view, education innovations should be subjected to similar
testing before being introduced on a wide scale. Unfortunately, this seldom happens, in
part because public schools typically resist rigorous, independent evaluations, but also
because the Department of Education, unlike the FDA, has not provided strong research
leadership. Fortunately, SCSF was willing to permit a rigorous, independent evaluation of
its pilot program, and my colleagues and I were able to obtain funds for the evaluation
from a broad network of private foundations.
The lottery was held in mid-May 1997. The firm responsible for the
evaluation, Mathematica Policy Research, administered the lottery in order to leave no
doubt about its integrity; SCSF announced the winners.
To estimate the effects of attending a private school, the mathematics
and reading components of the Iowa Test of Basic Skills were administered in the spring of
1997 to scholarship applicants. Each component of the test took approximately one hour to
complete. Students participating in the evaluation were tested again in the spring of
1998. Both the scholarship students and students in the control group were tested in
locations other than the school they were currently attending. To guarantee similar
testing conditions, both for scholarship students and students in the control group, the
tests were administered under the supervision of the evaluation team.
Each student’s performance was given a national percentile ranking
between one and one hundred. The national average is 50. The data indicate that these
students are educationally disadvantaged: overall, average test scores were below the 30th
percentile. Results were collected from approximately 85 percent of the participants in
the evaluation, an unusually high response rate from a low-income, inner-city population.
Our evaluation focused on students entering grades two through five,
because only from them were we able to obtain baseline test-score data. Baseline
test-scores were unavailable for those entering first grade, because those children were
still in kindergarten at the time of application.
After one year, the national percentile ranking of students attending
private schools was, on average, two points higher in reading and mathematics than the
ranking of the comparison group that remained in public schools. Differences were uneven
in grades two and three, but choice students in grades four and five achieved
substantially higher scores, six percentile points more in math, and four points more in
reading.
Long-term Rewards
When reporting these effects of school choice, Education Week
headlined them as "modest" the New York Times found them
"slight." Whether or not these gains after one year are slight or substantial
depends in part on what happens in later years. Nonetheless, there is reason to conclude
that the effects of choice on the performance of students in their middle years is already
sizable enough to merit careful consideration.
Scholars typically calculate effect sizes in standard deviations. As an
indication of both the average score and the degree of variation from it, standard
deviation allows us to compare results across different data sets. One can grasp its
essential quality by keeping in mind that one standard deviation is approximately the
current difference between the average test scores of blacks and whites nationwide. The
effects of school choice on students in fourth and fifth grade are roughly one fifth of a
standard deviation. If similar effects occur in subsequent years, these are large enough
to bring the scores of minority students up to the levels currently attained by whites.
This would be taking a large step toward achieving equal educational opportunity across
ethnic groups, something most people would regard as a major accomplishment.
More to the point, these test scores are not a triviality, or a
hobgoblin only of interest to academic researchers. Students who score higher on
standardized tests are more likely to remain in school, more likely to achieve a college
degree, more likely to remain married and avoid welfare dependency, and more likely to
enjoy a higher family income. According to the best available estimates, a gain of one
standard deviation in test scores will later in life increase that person’s family
income by over 20 percent. If students in the choice program in New York City simply hold
the gains they have already achieved, one could expect their family income, on average, to
be 4 percent higher than it otherwise would have been. Assuming a modest annual income of
$30,000, that’s an increase of $1,200 a year. If these estimates are reasonably
accurate, the philanthropists in New York will realize an ample return on their charity
dollar, once these students enter the labor force.
Another way to consider the effects of the SCSF program is to compare
them to the results of a different intervention. Very few education innovations of
interest have been subjected to a random field trial, but one. Class size reduction from
25 to 15 students has been rigorously evaluated by this method. It is worth comparing the
results of a school choice field trial with the results from a class-size reduction
experiment, because both innovations can be introduced rather straight-forwardly by
legislative action. (Other reforms, such as requiring students to do more homework, are
much more difficult to mandate by legislative fiat.)
The class size RFT was conducted in Tennessee, where students were
randomly assigned to classes of different sizes. No incremental effects on student
learning were observed for students after the first grade. Among first graders, effect
sizes varied between .15 and .30 standard deviations. Fred Mosteller, one of those
involved in the experiment, observed, "although effect sizes of the magnitude of 0.1,
0.2, or 0.3 may not seem to be impressive gains for a single individual, for a population
they can be quite substantial."
Congress was apparently persuaded by such reasoning and by the results
from the effect sizes observed in Tennessee. After extensive policy deliberations in which
the Tennessee evaluation was frequently mentioned, in 1998 Congress enacted legislation
appropriating $1.1 billion for the purpose of reducing the size of elementary school
classes.
The effect sizes observed in our evaluation of the New York scholarship
program in grades four and five do not differ materially from those observed in Tennessee
in grade one. The effects among fourth and fifth graders of attendance at a private school
were, on average, .23 and .18, not much different from the .15 to .3 effects observed in
the first grade of the Tennessee Study—the only grade where incremental class size
effects were detected. Following Mosteller’s guidelines, these effect sizes, observed
after just one year in the program, can be said to be "quite substantial."
From a cost-benefit perspective, school choice seems a better
intervention than reductions in class size. To get effects of about .2 standard
deviations, class sizes in the Tennessee study were reduced from approximately 25 to
approximately 15 students. If such reductions were introduced as a school reform more
generally, it would increase the size of both the teaching staff and classroom space by 40
percent. Per pupil costs could be expected to increase by approximately 20 percent (if it
is assumed that classroom costs constitute about half the cost of public schooling). By
comparison, the per pupil cost of school choice is minimal; the taxpayer may in fact enjoy
some savings, if eventually competition among schools leads to more effective education at
lower cost.
Moreover, the incremental benefits of class size reduction disappear
after first-grade. If larger differences between the test scores of scholarship students
and those in the control group appear in subsequent years in New York City, the benefits
of school choice will clearly outstrip those obtained through large reductions in class
size.
When we initially announced our findings, Sandra Feldman, president of
the American Federation of Teachers, offered the interesting hypothesis that class size
and school size probably accounts for the results that we observed. "I see it as a
validation of the need for small class sizes, and for smaller schools that are orderly and
disciplined," she said. To test this hypothesis, we conducted a statistical test in
order to ascertain whether any of the following characteristics could account for the
higher test scores in the private schools: (1) class size (2) school size (3) discipline
problems (4) school-parent communications (5) school resources.
Data on these potential explanatory factors were available from
information contained in the questionnaires administered to parents when the students were
tested. Although parents reported that private schools were superior in all five respects,
only discipline problems had a large and consistently positive effect on both the math and
reading scores of the two older grades. Class size had no significant effect. Students in
larger (not smaller) schools did slightly better in math (but not reading). Improved
parental communications had a positive effect on math (but not reading) scores, and
additional resources had a positive effect on reading (but not math) scores. Most
importantly, none of these factors, nor all of them, reduced the size of the effects of
receiving a scholarship to attend a private school in fourth and fifth grade.
These new pilot programs provide
new opportunities to find out whether students learn more when families are given a choice
of schools.
Perhaps the program’s impact comes from the sheer fact of choice:
the opportunity to better match older students with an appropriate school. But, more
likely, it is some constellation of many factors that affect scores in ways not easily
captured by a statistical model. In any case, the advantage of attending a private school
is not readily reduced to any one or single set of factors.
As we have pointed out, the advantages of attending a private school in
New York City are not clearly evident until a student enters fourth and fifth grade. This
finding is consistent with other indications that in American education problems begin
during the middle years of schooling. According to the National Assessment of Educational
Progress (NAEP), students in fourth grade are performing at higher levels than their
counterparts a generation ago. Gains over the past two decades have been particularly
large for students from minority groups. But NAEP data also show that, after fourth grade,
initial gains disappear. In fact, students nationwide learned less between fourth and
eighth grade in the 1990s than they did in the ‘70s. The slippage seems even greater
in high school. Similarly, international comparisons reveal that U.S. fourth-grade
students keep up in science and math with most of their peers abroad (though not with the
Japanese and Koreans). But by eighth grade U.S. students trail those in all other leading
industrial nations, and by twelfth grade they fall to near the bottom of all participating
countries. If the problems in American education develop in the middle years of schooling,
perhaps it is at this point that the advantages that come with school choice are
particularly evident.
Of course, the findings from New York City are simply first-year
results. Our evaluation is scheduled to continue for two more years, and only time will
tell whether the initial gains are maintained in the future. It remains to be seen whether
school choice, if generalized to a larger population, will yield comparable gains. But it
does seem time to begin larger-scale experiments.
An Historical Perspective
The unique quality of the SCSF pilot program can be appreciated by
situating its evaluation within the long-running controversy over research on public and
private schools. In the early 1980s two nationwide studies, one conducted by a team headed
by sociologist James Coleman, the other conducted by John Chubb and Terry Moe, reported
that high school students learned more in private than in public schools. School choice
critics questioned the findings from both studies on the grounds that the students in
private schools came from families more committed to their children’s education.
Both studies had anticipated this argument by taking into account
family background characteristics, such as education and income. But critics say that no
amount of statistical tinkering can ever fully correct for the selection effect: families
who pay to send their child to private school are almost certainly more involved in and
concerned about their child’s education, even after adjusting for demographic
characteristics. Even the Coleman research team admitted, the "difference between
parents, by its very nature, is not something on which students in public and private
schools can be equated" in a statistical analysis.
Beginning in the late 1980s, a number of publicly and privately-funded
school choice pilot programs began providing researchers with opportunities to consider
the question anew. Educational outcome information is currently available from programs in
San Antonio, Indianapolis, Milwaukee, and Cleveland. In the next few years, still more
information will become available not only from New York City but also from other pilot
programs that are getting underway in many other cities, including Washington, D.C.,
Dayton, and San Antonio.
These new pilot programs provide new opportunities to find out whether
students learn more when families are given a choice of school. For one thing, differences
in family background have been reduced, compared to the national surveys mentioned above,
because most of these programs are limited to inner-city children from low-income
families. More importantly, from a research perspective, these scholarships are often
awarded by lottery whenever the number of applicants exceeds the number of scholarships
available. Because a lottery is used to award the scholarships, these programs can be
evaluated by means of an RFT.
Unfortunately, many of the school-choice pilot programs conducted thus
far do not permit an RFT. Privately funded programs in Indianapolis and San Antonio
admitted students on a first-come, first-serve basis. Such admission procedures have a
fairness of their own, and they are easy to administer, but any findings from these
programs may be contaminated by the selection effect. After all, those families who are
quick, clever and well-connected enough to get a first-come, first-serve scholarship are
likely to have other attributes that favorably affect their child’s educational
attainment. Nonetheless, test score results from these experiments are mainly positive.
For example, the scores of students participating in the school choice program in San
Antonio increased between 1991-92 and 1993-94, while those of the public-school comparison
group fell. In Indianapolis, students in private schools did better than students in
public schools, particularly in grades six through eight.
Inconclusive Cleveland
Much the same can be said for the disparate findings that have emerged
from research on the state-funded pilot program that began in Cleveland in the fall of
1996. Although the state required that the scholarships be awarded by lot, various legal,
political, and administrative problems made it impossible to gather data necessary to
conduct an RFT. As a result, both the research team that I headed and other researchers
were forced to rely upon less precise research techniques.
In 1997 my colleagues and I found that students attending the Hope
schools, two newly established choice schools serving 25 percent of the students
previously attending public schools, gained 9 national percentile rank points in math and
6 percentile points in reading. But because no control group was available for comparison
purposes, we cannot be sure that a comparable group of students would not have achieved
similar gains in Cleveland’s public schools.
All in all the evidence that
school choice enhances achievement of low-income students has now become quite
substantial.
Another evaluation by Indiana University’s School of Education
found no programmatic effects on the test scores of 94 third grade choice students. The
Indiana University evaluation suffers from a number of limitations:
1. The study analyzed only third-grade test scores; no information is
available for students in kindergarten, first or second grades.
2. To control for student achievement prior to the beginning of the
scholarship program, the evaluation used implausible second-grade scores collected by the
Cleveland Public Schools before the beginning of the choice experiment when students were
still in public school. These dubious second-grade scores tell us that students from
central-city, low-income, largely one-parent families were performing in second grade, on
average, at approximately the national average. Yet in an independently proctored test
administered one year later, the same students scored, on average, 40 percentile points in
reading. Clearly, the previous second-grade test scores were inflated.
3. The evaluation excluded Hope school students from the evaluation,
despite the availability of comparable test-score data.
In the end, firm conclusions cannot be drawn from the studies of the
scholarship program in Cleveland. In neither our research nor that of the Indiana
evaluation team was it possible to compare similar groups of students by means of an RFT.
The state-funded program begun in Milwaukee in 1990 also required that
scholarships be awarded by means of a lottery, if applicants exceeded places available. In
this case, the lottery was successfully conducted; as a result, data are available from an
RFT for the first four years after the program was started (school years 1990-91 to
1994-95). Unfortunately, no data are available after 1995.
The original evaluation of the Milwaukee choice program did not
carefully analyze the data from the randomized field trial but instead compared students
from low-income families with a cross section of public school students whose parents were
motivated enough to return a mailed questionnaire. Although this research reported no
systematic achievement effects of enrollment in a private school, its findings are
problematic because the study compared choice students with public-school students
enjoying much more advantaged families.
When these data were released to the general public, my colleagues and
I analyzed the data from the RFT. Although the data collection was less complete in
Milwaukee than in New York City, making the findings less definitive, they are nonetheless
of interest. We found that enrollment in the program had about the same modest effects for
all students (regardless of grade) during the first year of the program, just as was
observed in New York City. But we also found that choice students scored much higher in
years three and four. The differences in these years were as much as one quarter of a
standard deviation in reading and one third of a standard deviation in mathematics. Once
again, these gains are large enough that, if similar gains are made in the remaining years
of education, they have the potential of bringing minority students up to the level
currently achieved by white students.
That choice students did not demonstrate improved performance until the
third and fourth years is quite consistent with a common-sense understanding of the
educational process. Choice schools are not magic bullets that transform children
overnight. It takes time to adjust to a new teaching and learning environment. The
disruption of switching schools and adjusting to new routines and expectations may hinder
improvement in test scores in the first year or two of being in a choice school.
Educational benefits accumulate and multiply with the passage of time. As Indianapolis
choice parent Barbara Lewis explains the process: "I must admit there was a period of
transition, culture shock you might call it. He had to get used to the discipline and the
homework. . . . But Alphonso began to learn about learning, to respect the kids around him
and be respected, to learn about citizenship, discipline, and doing your lessons. . . . My
son has blossomed into an honor roll student."
Note to Government: More Choice
School choice programs are too recent to provide information on their
effects on college attendance, though the private school choice program in Milwaukee
(PAVE) reports that 75 percent of those who have graduated from high school have gone on
to college. More systematic information on the effects of attendance at a Catholic high
school are contained in a recent University of Chicago analysis of the National
Longitudinal Survey of Youth, conducted by the Department of Education, a survey of over
12,000 young people. Students from all racial and ethnic groups are more likely to go to
college if they attend a Catholic school, but the effects are the greatest for urban
minorities. The probability of graduating from college rises from 11 to 27 percent, if
such a student attends a Catholic high school.
The University of Chicago study confirms results from two other
analyses that show positive effects for low-income and minority students of attendance at
Catholic schools on high school completion and college enrollment. As one researcher
summarized one of these studies, it "indicates a substantial private school advantage
in terms of completing high school and enrolling in college, both very important events in
predicting future income and well-being. Moreover . . . the effects were most pronounced
for students with achievement test scores in the bottom half of the distribution."
All in all the evidence that school choice enhances the achievement of
low-income students has now become quite substantial. Although additional RFTs are
desirable, the results from the first year of the New York City evaluation suggest that,
at least for children in grades four and five, there are clear benefits for low-income
minority students that come from attendance in private schools.
The results from New York tend to confirm findings from a wide variety
of previous studies that used less definitive research methods. Only time will tell if the
choice students in this program score much higher in later years as they did in Milwaukee.
If Congress regards the research evidence sufficient to justify the
$1.1 billion federal intervention to reduce class size appropriated in 1998, then the
evidence is equally sufficient to justify comparable state and federal expenditures on
school-choice experiments.
