The evidence on school choice: A real 'reality check'

[Guest post by Greg Forster]

A blog called Oklahoma Education Reality Check has posted an intemperate and inaccurate attack on my report summarizing the empirical research on school choice. My review finds a very strong consensus in favor of school choice among the empirical studies. For those interested, here’s a review of OERC’s misrepresentations and miscellaneous fallacies.

For the record, my report surveyed the research on school choice and found:

• Twelve empirical studies have examined academic outcomes for school choice participants using random assignment, the “gold standard” of social science. Of these, 11 find that choice improves student outcomes—six that all students benefit and five that some benefit and some are not affected. One study finds no visible impact. No empirical study has found a negative impact.
• Twenty-three empirical studies (including all methods) have examined school choice’s impact on academic outcomes in public schools. Of these, 22 find that choice improves public schools and one finds no visible impact. No empirical study has found that choice harms public schools.
• Six empirical studies have examined school choice’s fiscal impact on taxpayers. All six find that school choice saves money for taxpayers. No empirical study has found a negative fiscal impact.
• Eight empirical studies have examined school choice and racial segregation in schools. Of these, seven find that school choice moves students from more segregated schools into less segregated schools. One finds no net effect on segregation from school choice. No empirical study has found that choice increases racial segregation.
• Seven empirical studies have examined school choice’s impact on civic values and practices such as respect for the rights of others and civic knowledge. Of these, five find that school choice improves civic values and practices. Two find no visible impact from school choice. No empirical study has found that school choice has a negative impact on civic values and practices.

The most damning charge in their post is that I “cherry pick” the evidence, leaving out studies I don’t like. That would indeed be grievous, if it were true. They back up this charge by saying that I did not include in my review the official study of the D.C. voucher program, headed by Pat Wolf.

The accusation is clear, emphatic, and misspelled: “Patrick Wolfe’s official (and dismal) report (Wolfe et al 2010) on the program in Washington D.C. is not cited, a serious omission.”

Unfortunately, the charge is also false. I cite Wolf’s 2010 report on pages 8, 29, and 31. Oops!

OERC claims Wolf’s report (excuse me, “Wolfe’s” report) is “dismal” for the voucher program, crowing that “Wolfe” found negatively for the program even though he supports vouchers. But while Wolf found no change in test scores, he also found the following:

The Program significantly improved students' chances of graduating from high school, according to parent reports. Overall, 82 percent of students offered scholarships received a high school diploma, compared to 70 percent of those who applied but were not offered scholarships. This graduation rate improvement also held for the subgroup of OSP students who came from "schools in need of improvement."

So the study they characterize as “dismal” actually found that the program dramatically reduces high school dropout rates without any corresponding reduction in standards of academic achievement.

Oops again!

Here’s a point-by-point review of other problems in the post:

“Empirical evidence by definition would indicate that an idea has gained acceptance in the scientific community. There is no such broad acceptance.”

Not at all—empirical evidence is empirical evidence regardless of whether the scientific community agrees or disagrees about how it is interpreted. The evidence is empirical regardless of whether the studies run 95/5 one way or split 50/50. However, in this case the studies do produce a clear pattern in favor of school choice, as my report showed. Of course, you can find plenty of scientists who don’t support school choice, but—as the OERC post itself emphasizes in its comments about “Wolfe”—the personal opinions of the researcher are irrelevant to what the data show.

“The Friedman Foundation, while named after and founded by a Nobel Laureate economist, has become an apologist for school choice.”

Who wants to tell them?

“As such, the Friedman Foundation is no longer a valid source for unbiased information on school choice.”

So when “Wolfe” endorses school choice and produces a study that is (supposedly) negative, his support for vouchers is irrelevant to his scholarship. It proves he is, in OERC’s own words, “objective.” But when the Friedman Foundation endorses school choice, that somehow undermines my scholarly credentials. 

This game is fun! Can anyone play? The author of the OERC post is against school choice, so that person is clearly not a reliable source of information about the research on school choice, and no one should believe anything they say.

By this standard, studies that find smoking causes cancer are illegitimate because they’re all conducted by people who think smoking causes cancer.

That having been said, it is also true that most of the research on school choice is not funded or conducted by the kind of advocates who are (selectively) deplored by OERC. The post even quotes a passage from my report in which I point out that fact, but does not address the point. I wonder why.

The post lavishes a great deal of loving attention on a single 2010 study, of inferior methodological quality, that found against school choice, and then says “There is no empirical evidence. Please stop.”

One study versus the whole body of other evidence leads you to say “there is no empirical evidence”? Please stop, indeed.

“This is important, because the data in the meta study was mostly older than 2010.”

Yes, we regret to inform you that most of human history did occur before 2010.

However, if my report were to exclude all studies from before 2010, guess what it would find? An even stronger consensus of the research in favor of school choice. Just look at the charts in the report, which list all the studies by date and indicate their findings.

“Forster includes an analysis of the financial impact on ‘taxpayers.’ The issue is not the financial impact on taxpayers. The issue is the draining of financial resources from public schools. One could argue for or against this idea, but a careful reading of Forster reveals that his study has nothing to say about this.”

Wrong; “a careful reading” would have revealed that I discuss this issue on p. 3-4 and again on p. 15-16. Among other things, I write:

When a student leaves a public school using a choice program, the school loses all the costs associated with educating that student but not all the funding. As has been noted already, almost all federal and local education spending does not vary with enrollment, so those funds stay when students leave. This means public schools are left with more money to serve the students who remain. This is one possible explanation for the positive impact choice has on public school outcomes.

A recent empirical study on schools nationwide supports these findings. Benjamin Scafidi examined school finances in every state and found that out of a total of $12,450 spent per student in 2008-09, 64 percent ($7,967) was made up of variable costs that change with the number of students enrolled. This means school choice programs would produce significant financial windfalls for local schools as long as they redirected less than that amount per student.

An extra $7,967 per student is a lot of money, but it can’t buy OERC a decent argument.