Charter schools, and whether or not we should have them in Louisville, is a hot topic, and bound to get even hotter. At Forward Kentucky, we’re beginning a series of articles exploring whether charter schools are as miraculous as their promoters state, or as evil as their detractors claim.
As we do, though, there is one request we’d like to make of all parties involved: Let’s be honest about charter schools, and about the data.
A charter school is a publicly-funded school that has a charter that grants it greater flexibility in its operations. It might have extra instructional time, or develop a stricter behavior code, than an ordinary public school. Charter schools are quite popular across the country: 43 states have them, with Kentucky among the 7 states that do not.
In recent weeks, Kentucky newspapers have devoted a lot of ink to discussing the merits of charter schools. Conservatives typically laud charter schools as the key to Kentucky students’ academic outcomes. A recent example is an opinion piece that appeared in the Courier-Journal, written by Jeanne Allen of the Center for Education Reform. She writes: “A meta-analysis of four research studies shows that black students in charter schools score better than their district peers in both English and math.”
There are several important problems with this passage. The first problem is that many studies have been done on how charter schools affect student achievement, and there is a wide variety of outcomes in such studies. For instance, two recent CREDO studies of charter schools in Ohio and Texas found that charter school students scored significantly worse in mathematics than their public school peers.
When there is as wide a variation in study results as there is in charter school research due to geographic and cultural variability, it is possible to cherry pick studies that support one’s ideological preferences. Indeed, the problem of result variability is embedded in the statistical testing procedures themselves: the familiar false positive problem. If one has 100 such studies and one uses the conventional statistical standards for testing the hypothesis, 5 such studies should ordinarily come out being “statistically significant” simply due to chance alone. From a policy perspective, the best studies are ones that have results that sample from a broad population and achieve very high levels of statistical significance.
One study that clearly meets these standards is the 2013 Credo study by Stanford University. This study compared academic outcomes in reading and math in 27 states. In addition, the study matched each of 1.5 million charter students in 27 states with a comparable student from a traditional public school in the same state. In other words, each student was matched with a “virtual twin” on ability and demographic variables that are known to affect student performance. Moreover, the study covered 86% of all charter school students from the 27 states covered. In other words, the 2013 CREDO study effectively negates the problem of cherry picking the data because of its breadth and because of its enormous sample size.
What did the 2013 Credo study find?
- In math, there were no differences in achievement between charter schools and traditional public schools.
- In reading, the students in the charter schools performed better. This difference was statistically significant; in other words, there was very little chance that this finding was a false positive.
- However, the reading difference, while definitely true, was also relatively small: .01 standard deviations. For comparison, most research studies consider a difference of .20 SDs to be a small effect size. To use another example, if we compare the heights of two males, a difference of .01 of a standard deviation equates to a difference of about .03 inches: (i.e., three sheets of paper). In other words, the performance advantage of the charter schools in reading scores was much ado about very little: not enough to get excited about, much less drive ambitious school reform.
As you can see, anyone claiming that this study proves that charter schools are obviously better either doesn’t know how to read research … or is just being dishonest. And there is a lot of dishonest grandstanding on this issue – on BOTH sides of the discussion.
At Forward Kentucky, one of our goals is to always be honest with the data, and we hope that others will follow that rule as well. We will have more to share about charter schools, and more questions to ask. In the meantime, let’s remember the immortal words of Senator Daniel Patrick Moynihan:
“Everyone is entitled to their own opinions; they are not, however, entitled to their own facts.”
– 30 –
Comments
