The Education Exchange: Charters’ “Lost Decade” Found Again?

PAUL PETERSON, HOST:

This is the Education Exchange with Paul Peterson. I am the Director of the Program on Education Policy and Governance at Harvard University. Thank you for joining us. A decade ago, urban charter schools were celebrated for their no excuses model. The largest and most influential of the charters was KIPP Academy, and its motto spoke for the whole movement: Work hard. Be nice. There were no excuses for failure. Every child can go to college. If teachers pour energy into their instruction, everyone at the school encourages one another and students pay attention and do their homework. But 10 or 15 years ago, the urban charter atmosphere changed. Low achievement was blamed on institutional racism, not on the failure to work hard. KIPP dropped its slogan. White administrators were treated with suspicion. Steve Wilson, a white founder of Ascend Learning, another successful charter network, was asked by his board to surrender his leadership position. And in a book about his experience, Wilson speaks of the “Lost Decade” when urban charters lost their way. Well, are these just rumors and reports? Do we have any solid evidence that charters are not pursuing their mission as successfully as they did in the past? To investigate this question, Geoffrey Kocks, an economist at MIT, has taken a careful look at charter school effectiveness in Massachusetts, a state that pioneered the urban charter movement. Kocks’s paper is entitled “Two Decades of Massachusetts Charter School Effects.” Well, I’m very pleased to have Geoffrey Kocks with me on the Education Exchange today. Thank you, Geoffrey, for joining me.

GEOFFREY KOCKS: Thank you for having me on the program.

PETERSON: Well, Geoffrey, we’ll get to your methodology in a minute, but can you first share with our listeners your bottom line? Has charter school effectiveness fallen since 2015 in the schools in Boston?

KOCKS: Yes, I’m certainly happy to. So, I started this project with two key research questions. The first is that we already know from a large body of literature that Massachusetts charter schools have historically boosted student test scores. But these papers only extended through cohorts of students who applied to the schools through 2014. And of course, a lot has happened both to the charter sector and the educational landscape more broadly since that time period. So, the first question was how have the effects of the state’s charter schools evolved since then? And the second question is after seeing potential changes in charter school effects, I was curious what explanations were consistent with both the declines in charter school effects and the subsequent rebounds. So, my headline finding for the first question is that while charter schools did boost student test scores on average over time, these effects varied substantially across the study period. So, during the earliest period, going up to about 2014, I find consistent with the previous literature that charter schools indeed continued to boost student test scores. So, for example, on the MCAS, which is the state test that students take in Massachusetts, each year of charter enrollment increased math scores by about 0.25 standard deviations.

PETERSON: What’s a 0.25 standard deviation? I know you know what it means. I think I know what it means, but can you just put it in layman’s terms?

KOCKS: Yeah, so that’s about equivalent to a 10 percentile point increase. If you have a student that’s starting at about the 50th percentile in math, for example, that would suggest that after about one year of charter enrollment, they’d be in the 60th percentile. on the state test.

PETERSON: Can I say it is something like a half years’ worth of learning or a full year’s worth of learning or anything like that? Is that a little over-interpretation of what you can say?

KOCKS: We don’t have it in those terms in the paper.

PETERSON: So, you’re not sure exactly how to put it in terms of something like, so shift of 10 percentile points or percentage points on the distribution, which is a little tricky.  But it sounds like a big effect.

KOCKS: Yeah, so we don’t map that into years of learning, for example, but we map it in terms of percentile points on the state exams.

PETERSON: All right. So, do you look at any other educational outcomes besides test scores?

KOCKS: So, the main outcomes that I look at are test scores, so both the MCAS scores as well as SAT scores. I find in the first half of the 2010s, these large increases in MCAS scores. But in the second half of the decade, these indeed dropped to about zero. So, for both math and English, there was no effect of the charter schools compared to the traditional public schools.

PETERSON: So, it goes from this 10 percent thing to zero, you’re saying. So, in the recent period of finding no advantage, you might as well go to your neighborhood school as to go to one of these charter schools on average.

KOCKS: Yeah, so in the second half of the decade, there was no differential effect compared to traditional public schools. And then what I find interesting is that these effects seem to reemerge after the Covid pandemic. So, when we look at students who enter in 2020 or later, there’s some rebound in the effects.

PETERSON: I think it’s important to say that you found this decline before Covid, right? It isn’t like this was caused by Covid because the scores were— The difference between charters and the regular public schools in Boston; that difference was narrowing in the period before Covid. Not…It wasn’t caused by Covid.

KOCKS: Yes, that’s exactly right. So, the decline in the charter school effects is a pre-Covid phenomenon. And in fact, they seem to be doing relatively better compared to the public schools since the pandemic.

PETERSON: So, the pandemic actually put an end to this decline, you’re saying, and there’s been a bit of a rebound. Have they gone back to where they were in the early period?

KOCKS: So, in terms of levels of student achievement, we still see that there’s room for both of the sectors to grow compared to how they were doing pre-pandemic. But if we look at the causal effect of a student attending a charter school versus a traditional public school, it seems like that charter advantage has reemerged since the pandemic.

PETERSON: Now, one of the things you just mentioned here is that the charter school advantage depends on two things. One is how well the charter school is doing, but also it depends on how well your traditional public school is doing. And in Boston, one worries a lot about the traditional public schools. There’s a lot of feeling out there that that’s an institution that’s broken. Now, do you see any signs as to whether or not the Boston Public Schools were actually getting quite a bit better in the latter part of the second decade? And maybe that’s the explanation. It has nothing to do with charters, but it’s just that the public schools were getting better.

KOCKS: Yeah, so that’s one explanation that I was interested in. So, when I look at just the levels of the test scores in the traditional public schools, I don’t find any substantial increases in their performance throughout the decade. Now there are some signs of some school practices that may have been improving in organizational changes during that period. So, for example, one characteristic that I look at is whether teachers were qualified in the subject area that they were teaching, which is something that I measure based on whether they had an undergraduate major in that subject area or whether they were licensed to teach in that area. And I find that those shares were declining in the charter sector while they were increasing in the traditional Boston Public Schools so that might reflect changes in how they were recruiting teachers, for example.

PETERSON: Well, did they have more money? I mean, I know that a lot of money became available during Covid, especially to your traditional public sector. But I don’t know about the period just before that. So, did you look at, you know, just the resources available, financial resources available to the two sectors?

KOCKS: So that’s not something that I look at directly in this paper. I think there’s been some anecdotal evidence that the traditional public schools may have been increasing their pay scales during that period, which may have made it more attractive for teachers to teach in that sector, for example.

PETERSON: And did the charter school…how well are they funded? Are their resources as adequate as they are in the traditional sector?

KOCKS: That’s not something that I look at directly in this paper. But of course, they are given public funding for the schools.

PETERSON: But is it at the same level per pupil?

KOCKS: I’m not sure off the top of my head.

PETERSON: OK, well, so that might be a possible explanation is that the charter schools didn’t have as much resources as they had previously. That there was a starving because, you know, there was a lot of political opposition to charters in 2019. The voters were asked to, you know —

KOCKS: Yeah, definitely, so there’s been a lot of attention for example on like the referenda for the charter schools and things like that so what I do in the paper is I try to look at what are the potential explanations for the decline in the 2010s and I pinpoint two potential explanations that coincided with the decline in charter effects. So, one of those which I alluded to earlier is the decline in teacher qualifications, so I find that fewer teachers in the charter schools during that period majored in the subject area that they were teaching or were licensed in that area, for example.

PETERSON: How important is that? You know, when I’ve looked at that, I did study that question. Admittedly, it was in Florida, not Massachusetts, but I couldn’t see any effect of whether you were trained in a particular area or not, because, you know, a lot of this is happening at the elementary and middle school level. You’re teaching a lot of basic material so how important is the…that you have training in the field? Do we have much evidence that that’s important in the literature?

KOCKS: Yeah. That’s a great question. Well, first of all I look at the middle schools and high schools so not the elementary schools in this case but second of all the literature is a bit mixed on those effects. But it might reflect something more fundamental just about how easy it is to attract well-qualified teachers that have sterling academic credentials to fill these positions. So, it might not be the credentials themselves, but rather that’s reflecting how easy it is to get qualified teachers to fill the positions.

PETERSON: I just noticed you mentioned that you’re looking at middle school and high school students. You did not include elementary school students in your analysis.

KOCKS: That’s right. I don’t look at elementary school in this study. It’s just middle school and high school.  I also want to go back quickly to the potential explanation. One is teacher qualifications. The other is leadership turnover, which is another important part of the organizational structure. So, another trend that I find during that same period is that there was a large turnover of school principals in the charter schools during the years immediately preceding this decline. It seems like a lot of the leaders, and especially a lot of the founding leaders of these schools, may have left the organizations around that time that could have created instability in the school practices and also might have moved away a bit from some of the original visions of the schools there.

PETERSON: But wasn’t that happening in the district sector as well?

KOCKS: So not at the same rate. So, what I do in the study is I compare this for the charter sector relative to the traditional public schools.

PETERSON: A lot of people want to know, well, exactly who is going to the charter schools? I know you’re comparing charter schools or district schools using the lottery method. So that method gets you a pretty accurate comparison. But what is the composition of the charter school in your study? What’s the race and ethnic composition, for example?

KOCKS: Yeah, so the charter sector in Massachusetts has more racial minority students and more low-income students than the traditional public schools. The nice thing about using this lottery method is that we’re comparing only the students who applied to the charter schools. So, we’re abstracting from any of those differences in the underlying student characteristics there. By law in Massachusetts, if charter schools have more applicants than seats, they’re required to run an admissions lottery to randomly admit the students. So, it creates a nice random variation in the opportunity to actually attend the charters.

PETERSON: Right. Did that change over time? I mean, is that ethnic composition pretty stable over the time period?

KOCKS: Overall, it’s relatively stable. There have been slight increases in the share of Hispanic and Latino students, slight decreases in the share of black students, but it’s remained a larger share of racial minority students than the traditional publics.

PETERSON: Now, how about the family income, family education? Do you have any information on that?

KOCKS: So I look at that as well. I find that that’s been relatively stable over time. And that rules out that as an explanation for why the effects are changing as well. So it doesn’t seem like it’s due to large changes in the composition of students who are applying or their characteristics.

PETERSON: So how about the comparison between the Boston schools and the charter schools elsewhere in Massachusetts? We’ve been focusing in on Boston. Maybe that’s all my fault. But can you go into a little bit on whether you’re finding the same thing in the charter schools elsewhere in the state?

KOCKS: Yeah, so a lot of the changes that I find are driven by these Boston area charter schools. The effects for the schools outside of Boston have been relatively stable over time. So, it seems like a lot of the action is taking place in Boston. So those were the schools that were performing best in 2014 and earlier. Those are the schools that experienced the biggest decline throughout the 2010s. And they’re also the schools that have experienced the biggest rebound since the pandemic. So, for example, Boston area charter schools, if we look at their effects on state English exams, are doing even better now than they were prior to their decline. So, in the first half of the 2010s.

PETERSON: So, this is sort of a Boston middle school and high school story. It’s not necessarily a story of all charter schools in Massachusetts. So why do you think this is not happening in the other part of Massachusetts, out in, say, Springfield, or … I’m not sure exactly where all the non-Boston charter schools are located, but they’re in a variety of places, rural and suburban and some smaller cities, but you don’t see it out there. So, what do you think is the reason you see it more in one place than another?

KOCKS: Yeah, that’s a great question. You’re right; the non-Boston charter schools are sort of spread geographically across the state, some in more rural areas, some in sort of suburban areas. Some of it could be just accidents of timing. So, it could be when the Boston charter schools had their leaders leave. And some of it could be Boston-specific problems in recruiting teachers. So, as we mentioned, the pay scales may have changed in the Boston public schools, for example, in ways that made it differentially more attractive. There are also issues like higher cost of living that might be more specific to the Boston area, and they’ve made it more difficult to have high staffing in these schools.

PETERSON: Do you have any thoughts as to why the comeback actually occurs since Covid? This is sort of the most encouraging part of your study, is that maybe this lost decade is behind the charter movement and they can move forward with confidence at this point. How confident are you in that finding? Do you have enough observations to know for sure this is happening? And what do you think are the reasons why that might be?

KOCKS: So, the evidence looks pretty strong that the Boston area charter schools in particular and the charter sector as a whole has been performing relatively well since the pandemic. And I was really interested in understanding why. I conducted a survey of Massachusetts charter school administrators to ask about the practices that they’ve been doing since the pandemic and what they’ve actually changed since the pandemic. And this has been motivated by the idea that there’s a rhetoric of charter schools being areas of innovation where there’s more flexibility to experiment with different learning practices and that might have resulted in an ability for them to pivot more readily than traditional public schools did. So, I got responses from 30 schools in that survey and what I do is I correlate the school specific effects since the pandemic with what they report doing since the pandemic and I find some evidence of particular practices that have been the most effective. So, this is all correlations for this part of the study. But for example, I find that schools did better that were adopting more frequent assessments for students. So, they might have been more readily able to gauge exactly how students were doing in the classrooms and respond to learning challenges after identifying those.

PETERSON: Do you find any evidence that the schools that had fallen the most rebounded the most? Was it just sort of like, oh, we’ve got to change our practices, and so we’re going to turn around? Or that wasn’t the factor?

KOCKS: There’s definitely some evidence of that. So, for example, the Boston area charter schools are the ones that had the biggest decline in the second half of the 2010s, and those are also the same schools that experienced the biggest rebound since the pandemic.

PETERSON: So, can you generalize beyond Massachusetts? Is there any research elsewhere?

KOCKS: So to my knowledge, this is the first lottery-based evidence that’s occurred since the pandemic. There have been some studies using observational methods in other areas like Tennessee, for example, and those have found somewhat similar findings of the charter schools potentially doing better at promoting learning recovery since the pandemic. But of course, those are using different methods and might have some of the selection bias that would occur without a lottery study.

PETERSON: So, on the lottery study, are these schools still as popular as they once were? I mean, if there’s signs that they aren’t as effective, you might think that people no longer are pouring into these charter schools as they were. How about the…are all of them oversubscribed today or are there a lot more that are not oversubscribed, and you can’t do a lottery study?

KOCKS: It’s always been the case that not every charter school in the state was oversubscribed. So, I’m always limited to the set of schools that have that oversubscription. There’s still a large number that have been oversubscribed since the pandemic, for example. But there aren’t quite as many students applying as there were during the peak in the early 2010s, for example.

PETERSON: How much of a change has there been? I mean, could that be a possible explanation is that you’re looking at a different set of schools now than in the earlier period?

KOCKS: Yeah, so that’s one explanation that I ruled out as well. So, in some of the analyses, I restrict the sample just to the schools that were oversubscribed and in the early period and can trace the effects of those same schools over time. So that holds a set of schools constant in the analysis and ignores the fact that some schools might be going in and out, for example. And I find very similar results when looking at that subset of schools.

PETERSON: Now, what do you see as the future of the charter school movement? Is it under a lot of pressure, partly because of maybe it not performing as well? You’ve got a big homeschooling movement today. The private sector is popular with these Education Savings Accounts. The Trump administration is supporting a private sector, not really against charters, but not as much for them as it is for the private schools. You see the landscape changing.

KOCKS: I think one interesting thing that I got out of this study is that it’s important for the charter schools to constantly be reevaluating the effectiveness of the methods that they’re using and being able to pivot quickly in response to new challenges. So, I saw how quickly these school effects could change over time, both for better and for worse, and that these charter effects aren’t something static that will always be high regardless of what else is going on. So, I think constant evaluation and constant accountability for the sector is very important for them to maintain this advantage in test scores.

PETERSON: Well, how about moving beyond test scores and sort of academic outcomes? How about social and emotional learning? Many people think this is just as important, maybe more important than academic learning. Were you able to get any window on that?

KOCKS: I don’t look at that directly in the study. All of my outcomes are looking at… test scores for now. There are plans in the future to look at things like chronic absenteeism to continue evaluating how charter schools have affected those sorts of outcomes.

PETERSON: So, at this point, we don’t really know whether or not the charter schools have re… I mean, this absenteeism topic and chronic absenteeism has just totally escalated in the post-Covid era. And maybe the charter schools’ rebound is due in part to their… at getting students to come to school. Is there any signs of that?

KOCKS: Yeah, that’s definitely something that I’m interested in exploring, but I don’t have evidence on that point right now.

PETERSON: Well, we look forward to your next study helping us out in that dimension, because that could be part of what’s going on here.

KOCKS: Thank you. I’m excited for that ongoing work.

PETERSON: Thank you very much, Geoffrey. It’s a great study, but oh, I know what, there is one question that I didn’t ask that I think I really need to ask, and that is Steve Wilson’s book. I started with that at the beginning, and I wasn’t sure whether or not you thought he was onto something there, or was he just talking about his own experience and not something you could generalize to other urban charter schools? Do you think his analysis has something to … are you inclined to agree with him?

KOCKS: Oh yeah, I’m glad that you came back to that. So, one thing that I looked at as a potential explanation is whether these changes in school disciplinary practices could explain what was going on. For example, in Massachusetts, there was a law passed, chapter 222 that went into effect in 2015 that made it more difficult for schools to suspend students. And so, there’s been some rhetoric that that might have been responsible for some declines in charter school effects if it was more difficult for schools to pursue practices that were commonly called “no excuses”. And I find that that doesn’t seem to be a big factor in what was going on. The decline in charter effects actually preceded what was taking place with that law. And I find declines in suspensions in both the traditional public schools and the charter schools within Boston. The charter schools were starting from higher levels, but it seems like it affected both of the sectors.

PETERSON: So you think perhaps there are other factors that are driving this that are more important than the ones that Steve Wilson settled on?

KOCKS: Yeah, so my analysis suggests that the organizational and the staffing factors seem to be more important as a part of that story.

PETERSON: Well, thank you, Geoffrey, for your fascinating analysis. I agree with you. This is the first lottery study, the first random assignment study, the first definitive study that looks at how charter schools work over a long time period. And it greatly enhances what we know about this sector. So thank you very much for joining me on the Education Exchange.

KOCKS: Thank you for having me.

PETERSON: I’ve been speaking with Geoffrey Kocks. He’s an economist at MIT who has just released a new paper entitled “Two Decades of Massachusetts Charter School Effects.” I am Paul Peterson. This is the Education Exchange. Please join me every Monday at noon when our weekly podcast is released on the Education Next website.