The Exceptional Results Of Sensible Teacher Salary Reforms

>> John Taylor: Very honored to have Eric Hanushek speak to us today, couldn't be more honored, quite frankly. Education, what more could we want, what could we hope for?

>> John Taylor: We're all in the same business. It's called The Exceptional Results of Sensible Teacher Salary Reforms, take it away.

>> Eric Hanushek: Thanks very much.

I was going to bring in a prop, I have a prop that suggests that the reforms that we're talking about has a copyright of 1962. It's just that we don't get those reforms. What I wanted to do today is talk mostly about work that we've done in Dallas, Texas.

Dallas has gotten very little attention for the fact that it has completely changed the way it hires and pays and evaluates teachers. Dallas has for now close to ten years, been working on a set of school reforms that turn out, as I will demonstrate, to be pretty powerful in terms of results.

And then you say, well, which of the other 13,500 school districts in the country have adopted what Dallas has done? And you find that not very many. And so that's one of the puzzles at the end. But what I want to talk about is what Dallas has been able to do.

I have to warn you that occasionally I may break into spontaneous sneezing and coughing, it comes with two negative COVID tests. So that's just an old cold that I might be having here.

>> John Taylor: False positive?

>> Eric Hanushek: False negative. I'm going to take a little bit of a moment that is tangentially related to this discussion.

And I take the moment because I've been unable to convince very many people that the current situation in education is really desperate in many ways. So I'm going to talk a little bit about the pandemic and what's going on in schools today and then turn to Dallas and then come back a little bit to what that implies for the pandemic.

The Dallas situation is a complete change in how they pay and hire teachers. And then they applied some of the things they learned from which teachers were good and which weren't, to induce some of the better teachers to go into some of the worst schools in Dallas. And it turns out that that works.

And so that's the full story. Let me talk a little bit, sort of an aside, just to motivate this. I don't have to tell people here that skills and income go together in the United States. If I compare what is the return to skills as measured by just standard math achievement tests or something like that, and think of a mentor earnings equation with measures of what people know as opposed to how many years of schooling they have, this shows us returns across the world.

It comes from a OECD survey that gave survey instruments to a random selection of the adult population in 30-some countries and also gave them math and reading tests at the same time. So from that, you can look around the world, and what you see is that the United States rewards skills more than every other country in the initial survey.

There have been a couple recent additions to the survey that suggests that maybe Singapore has higher rewards than the US. But what this says is that the us labor market really pays attention to the quality of workers. Read backwards because of the pandemic, it says that the US punishes the lack of skills more than any other country in the world.

And so if you take what we know about the pandemic. Well, let me first go. The part that is more controversial when I speak to all my macroeconomic colleagues is that skills of the population also seem to be very closely related to long run growth rates around the world.

And the pandemic means that the US will have a poor labor force sometime in the future as poor-achieving students go out into the labor market and that by historical standards that we see would imply lower growth rates. And of course, lower growth rates are really important in terms of economic outcomes.

So what does this mean for the US? Well, this is the picture behind what I just said. If you array test scores on a horizontal axis against long-run growth rates, 1960 to 2000 growth rates, average growth rates of GDP per capita, it's conditional upon what 1960 GDP was.

But other than that, it's simply the relationship between test scores internationally and growth rates. You can explain three quarters of the variation in long run growth rates with just test scores across countries, yeah.

>> Speaker 3: So the economists might object to that.

>> Eric Hanushek: Yeah, I know.

>> Speaker 3: You said conditionally in the 60s- It is important, especially in our larger discussions, that skill is a level effect.

And so every bit of common sense is this is gonna have a level effect on GDP. Now, you grow to get up to the higher levels, but, you know, okay, so grow forever at another 1%.

>> Eric Hanushek: Why not? So I mean, I'll concede this point that whether you look at levels or growth rates, right, this is sort of consistent with endogenous growth kinds of models.

And it sort of says that people with a higher-skilled, more knowledgeable population invent more things. And so as opposed to you can't keep buying more years of schooling because you'd run out of age.

>> Speaker 3: Or adopt more things if you're not in-

>> Speaker 3: Let's just say it's not Obvious that growth is right.

There are level effects or growth rate effects.

>> Speaker 4: You can really go astray. So some of the leading really crazy climate change estimates are because they did exactly this. And all you have to do is do the time series model right, and it disappears.

>> Eric Hanushek: So if you believe in the level version of this, take two-thirds of the estimates I'll give you in the future.

That's what the difference is if you believe in level of differences.

>> John Taylor: For the next 30 years, it doesn't really matter but how far you extrapolate.

>> Speaker 5: I think the situation is worse than that. If you put it into a standard growth model, not endogenous growth model, that we, with the Chad Jones research astrology-

 

>> Eric Hanushek: The people on that side of the table, yes. Okay, I agree with that.

>> Speaker 5: Something happens favorable or unfavorable in your case, then it's gradually erased as time goes by.

>> Eric Hanushek: Yeah, I understand that.

>> Speaker 5: So it's not even a level effect.

>> Eric Hanushek: Well, that's why the conditional has that in it.

 

>> John Taylor: Conditional has a lot, yeah.

>> Eric Hanushek: The conditional is exactly that point that there.

>> Speaker 3: You only need to educate the top 10% and let the wage slaves be. If it's about levels, you need to educate.

>> Eric Hanushek: Okay, I knew I was gonna make a mistake here. This is not what I really want to talk about and I'll defend it separately.

This is an introductory mistake that I was trying to motivate everything. To write this off as Rick made an error here of introducing something that he didn't wanna talk about.

>> Speaker 6: How to expect a tough audit.

>> Eric Hanushek: So if I'm going to quickly run through what would happen if you took the observed learning losses in the United States, where we now have data on what were the scores of kids before the pandemic and what were the scores of kids after the pandemic or mostly after the pandemic, you find that there were huge losses for the US.

The estimates I gave you that, luckily we were the labor economics part of this, which people also would object to, says that this is a five to 6% average loss of lifetime earnings for anybody was in school during the pandemic, basically.

>> John Taylor: Why would you object to that?

 

>> Eric Hanushek: I'm not Endogenous discuss this, John. I'm carrying through to just trying to finish out my mistake, because I have slides on my mistake, okay? And here's the number of that you would get from the growth rates of the previous estimates. You can use the whole adjustment of taking two thirds of that number and you get a present value that is ten or more times the total cost of the pandemic on business losses and unemployment and so forth in present value terms.

So it's-

>> Speaker 7: What interest rate did you use for that?

>> Eric Hanushek: Three, 3%, and the important thing from my standpoint is that these losses are permanent unless the schools get better than they were in March of 2020. If we just go back to March of 2020, you're going to have these permanent losses of this cohort that aren't gonna be made up for.

And so that gets into what I really wanna talk about today.

>> Speaker 5: Just on that score, do we know anything from other tragic national experiments, like wars or something? What happened to school kids that were? How much was made up, if any?

>> Eric Hanushek: The best, yes, there are a series of studies of long-term shutdowns from union activities in different countries, Argentina, Colombia, Belgium.

The best evidence, I think, is from Germany. Germany, after World War II, some of the lander in Germany had a school year that began in January instead of September, and they decided that they wanted to get on the international calendar of moving everybody to a September school year.

And they did this by having short school years. And so you can pick out a cohort that had the short school years in Germany when they made this adjustment. And you can follow it through the Social Security data that they are worse than the group right before them and worse than the group right after them in terms of what happened because of these short school years.

 

>> Speaker 6: Does it make up for three quarters of a year or?

>> Eric Hanushek: Half a year.

>> Speaker 6: That's two-thirds of the-

>> Eric Hanushek: Them.

>> Speaker 6: Okay, great,

>> Speaker 9: two-thirds of this number is very large.

>> Eric Hanushek: The two-thirds is to translate growth models into level terms as opposed to endogenous growth terms.

 

>> Speaker 9: Two-thirds of that, it's still a huge number.

>> Eric Hanushek: Yeah, yes, agreed. So just a little quick lead in. The way that we've been dealing with the pandemic is I don't think we're quite back to where schools were in January of 2020 by all measures. There's all disruption in lots of schools.

But the most frequently sort of talked about remedial effort is to have more time in the current schools, either lengthen the school day or lengthen the school year. If it's done in a voluntary way, you know that it just expands the variations in outcomes because the higher achieving kids take advantage of more time and more activities and the lower achieving kids don't.

And we have all kinds of problems. The Los Angeles school system was going to make up for this by adding four different days to the school year. They did it on a voluntary basis by adding it the first two days of the last Christmas break. At the end of last year, about a quarter of the kids actually came to class.

And you can't imagine what they could possibly do with a random 25% of the population showing up for some extra schooling other than perhaps improve their soccer games. I mean, there's no other way to think of doing that. And we've also seen out of the pandemic, a series of negative moves.

There's been a movement to stop having any testing and accountability in the schools. That people have tried to link to the pandemic by saying things like, well, we know kids are worse off. There's no sense testing them to find that out. There have been lots of distractions. You will probably notice that nobody in Washington DC has discussed K to 12 policy.

We're only interested in loan forgiveness and maybe early childhood education, but there's no discussion at all of K to 12 policies. And then we have things like union actions in Oakland. We had an eight day strike last May in Oakland that they had already settled all the pay and benefits issues.

And the questions were whether we have reparations in Oakland, whether we have sustainable plantings around all the schools and whether there are water coolers in all the schools, and housing homeless kids in the schools. And so, in the midst of having these learning losses, the unions thought that it was important to stop schooling for eight days.

 

>> Speaker 10: I'm aware California gutting the curriculum. We can't do algebra anymore. Is that just California's nattiness?

>> Eric Hanushek: Yeah, that's more California, even though California is in the bottom third of the nation in terms of performance, people pay attention to California. At least textbook manufacturers do, cuz it's got an 8th of the us student population in California schools.

Okay, that's all? I'm going to say that schools have to get better and that the only way that we really know how to do this is by trying to affect teacher quality. That's the only persuasive policy instrument that we can think of. People who've talked about teacher policies have generally put it in terms of, well, let's have just more teachers, reduce class size and so forth.

 

>> Speaker 5: On the teacher quality thing, I've read about things like directed instruction, take teachers who aren't very good and say, here's exactly what you're doing and that's supposed to make things better. And also, great teachers and a bad curriculum can't be wonderful.

>> Eric Hanushek: So I'm not a big curriculum kind of person.

I think it does have an influence in it. And at least when you do stupid things the way California does, it has to have some negative influence. There is evidence on direct instruction, which is people revolt against because it doesn't allow teachers to do what they do and so forth.

It's worked dramatically in developing countries where there have been dramatic improvements with direct instruction, where you can get actually the right level of instruction into the classroom. But that's not what I'm going to talk about. I'm going to assert for the most part, which is something I pretty closely believe, that some people are good at this teaching and some people aren't.

And that we have to have a selection process that allows us to have the people that are good at it do more teaching and the people that aren't good at it do less teaching. And that's what this Dallas story is about. My simple solution is along these lines for the pandemic, and I'm gonna turn to Dallas in one click here but use the more effective teachers more intensively.

Just provide incentives to have effective teachers teach larger classes, and then use the ineffective teachers less intensively either as a second-best solution, give them smaller classes as a first best solution, get rid of them. And this is an idea that has not caught on relative to lengthening the school day in the school year.

But I'm going to give you some evidence that potentially these are policies that pay off.

>> Speaker 4: You have to be careful about the incentives there. I was just joking about referees. If you do a good referee report, you get more requests. So you wanna be careful about the incentive system for quality.

 

>> Eric Hanushek: Yeah, no, no, but I pay them more. I pay them more. I would give them less effective teachers to do all the grading and administrative things. I would do a variety of other things to induce good teachers to do more. And most of the evidence suggests that any reasonable changes in class size aren't going to offset the difference in quality of teaching.

So Dallas, let me talk about Dallas. Dallas had a set of reforms that began in 2013. There's a couple underlying papers, this one of them which was distributed, the first one was distributed. I think there are a couple bureau papers, and I'm gonna talk about both of them today and give you the overview of them.

What happened was that Dallas introduced really comprehensive reforms in the way they evaluated teachers and paid teachers, and ultimately, and how they employed teachers around the district. There was something called the Principal Excellence Initiative that was put together in 2012,13, and then subsequently the Teacher Excellent Initiative that did a similar thing in 2014,15.

What they did was to put in an entirely new set of evaluation rules, structure for the system where they had supervisory ratings of all the people. If there were test scores and achievement data, they used those data to get value added measures of teachers and combine that. They had survey information from the kids and the parents that entered into all of this.

And there was some use of aggregate school performance. So how did the whole school do as a collection that went into it. What they did was pay based pay on the evaluation. So historically, pay for teachers has been how much experience do you have, and do you have in a graduate degree?

And it's a lookup matrix, and you look up what the pay is for anybody. Dallas did away with that entire system. There were no remnants of it. There's a little phase in that went, On, but they went entirely to pay according to the evaluations that people had. And later, the second paper that we'll talk about here links the provides incentives for better teachers to work in worse schools.

And so they use the whole thing. Now what happened here was that you take the teachers

>> Speaker 11: 1 second, you want to say a word about how they were able to get this done. This is kinda stunning relative to most of the basic idea of unions prevent any change.

I'm just radical.

>> Eric Hanushek: Sure.

>> Speaker 11: So I don't wanna do the track about the results, but if you could just spend a minute or two on how that was done.

>> Eric Hanushek: Yeah, I was gonna come back to that at the end, but I'll do it now. There was a superintendent who had done a similar thing in a small district in Colorado.

His name was Mike Miles, and he came and worked with the school board. He, while he was there, had a favorable draw on the school board that a majority would listen to him. Texas has no collective bargaining. That's not to say that they don't have unions. They have unions that are very active.

But he spent a couple years trying to convince the Dallas school board that they should adopt these programs, and a majority of the school board voted with them. There were some who didn't want it. He got this all in place after this long, elaborate process, and just as the teacher program was going into effect, he quit.

He'd had enough of dealing with the school board, which in Dallas, the school board is sort of on a knife edge of whether it's something reform oriented or not and so forth. And he got a good draw and managed to get this through. Having said that, when he got it through, it stuck even as he left, because, among other things, it was hard to go back to the old system.

It wasn't just an add on to the old system and a bonus that you get, but in fact, it was an entirely different system. And he had this clever idea that he would have a teacher advisory council that was clearly, it made sense to make it up with the most effective teachers.

But if you think for a minute the most effective teachers are the ones that are least interested in going back to the old system, and it's managed to stay pretty much in effect, all of these things get watered down. The punchline at the end, I'll tell you now, is that last May, the Texas Education Agency, the state Department of Education, took over the Houston school district after a long, involved legal battle.

And Mike Miles was appointed to be the superintendent in Houston, where he is now. And there is some hope, because he. I'll come back to that at the end. But he's trying to put this similar system in place very rapidly in Houston. So there might be two districts in the country.

 

>> Speaker 3: Why did the teachers unions, dominated by people with lots of experience, lots of phony baloney degrees and high salaries, not immediately go out on strike over this?

>> Eric Hanushek: Well, they wanted to get paid. The contract changed. The contract changed. There was a phase in that meant that nobody actually lost money from the old system, and it went into place, and then there was nothing to go back to.

 

>> Speaker 3: So there's no collected bargain? Somehow there's a union, but not collective bargaining?

>> Eric Hanushek: Well, anybody can. There's a private tennis club here, too. I mean, they can join a union. It has an influence because it's louder and noisier than other places, but it does not enter into directly into collective bargaining in Texas at all.

 

>> Speaker 3: The contract is written by the school board, and you as a teacher can take it or go work at McDonald's.

>> Eric Hanushek: Yeah, but I mean, obviously this is, there's a political activities behind the scene, but yes.

>> Speaker 12: So you said this was not budget neutral, I take it, because everybody got as much in the new system as in the old system.

 

>> Eric Hanushek: It was not budget neutral, and I don't remember how they, maybe the school board voted some extra money for it.

>> Speaker 12: Well, it'd be nice to know how much.

>> Eric Hanushek: Well, we'll come back to that. Let me come back to that at the end, because Texas is, as a state, trying to promote ideas like this by providing state funds to support these systems.

So everybody is divided into seven categories, from unsatisfactory to broad note. Progressing means that you actually need some help. It's proficient and exemplary. I mean, there are rules that you can't move more than one category in a year or something, and you don't get dropped down unless you have two years of poor performance and so forth.

But this determines salaries where you are. These were in your budget neutral point, Ken. These were fixed size categories. One of the things that you find in US school evaluations is that 94.3% of the people are good and there's a very small number that aren't. Well, this has fixed proportions in each of these categories.

They haven't quite worked out the dynamics of that, right? If you start getting rid of poor teachers and get more, then I don't think that they completely understand that yet.

>> Speaker 13: What's the distribution across these categories?

>> Eric Hanushek: Do I have that? I'll show it to you in the schools we're looking at.

I don't think I have it for the whole state. There are very few in the exemplary category. There's lots in the various proficient categories and some in progressive.

>> Speaker 14: You could address this later, but I'm curious. What is the impact of unsatisfactory or low quality teacher leaving the district?

Well, this is what's happening in badly run district. What you see is people leaving. And going to better district. And I wonder whether here you have also an inflow or better quality teacher. And some of the effect is driven by that.

>> Eric Hanushek: So I think there is a movement to get better quality in partly because you can get higher pay.

It's Houston that I'll come back to at the end that Mike Miles is trying to institute. This system is one of the few school districts in the country today that has no vacancies as it went into this school year. It had 30 vacancies in mid August for a school district that has 200,000 kids and I don't know how many teachers.

With more money and with the promise of being rewarded for your performance, you get people that are willing to come in. Now, they don't explicitly fire unsatisfactory teachers, which is different than the only other big system that has something like this is Washington DC, which is somewhat different.

It's built on big bonuses or big changes in base pay for the best teachers in Washington DC and firing the worst teachers. And they fired 500 or 600 teachers in Washington DC based on performance. That was not part of this system. Although if you're unsatisfactory, you can't expect a lot of income and increases in the future, so you get more people to leave.

 

>> Speaker 15: Are you gonna show us the compensation effects of these different categories?

>> Eric Hanushek: Of course, I am. Of course, I am. That's the whole purpose of this. I'm gonna show you the impact on students.

>> Speaker 16: This is a design question. What's the ratio of-

>> Eric Hanushek: No, I'm not gonna show you that.

I don't have that.

>> Speaker 16: Give us any sense. Is it like,

>> John Taylor: we have a question over here.

>> Speaker 17: Rick, if you're unsatisfactory for two years in a row, you're still allowed to teach?

>> Eric Hanushek: I think so. I think so.

>> Speaker 17: The poor kids that get lumbered with those, why don't they have a sharpened pencil or something?

That's ridiculous that they're still teaching kids.

>> Eric Hanushek: They're still teaching kids in every other school in the country.

>> Speaker 18: This is the United States, Nick, come on. This is the United States. You can't fire people just because they're incompetent.

>> Eric Hanushek: And that happens all over the place, right?

I mean, it really does. There are very few dismissals for poor performance. There are some, but not a huge number.

>> Speaker 19: Is this really how it works in the sense if you're three years, you're 28 years old and you get exemplary, you're the highest paid teacher around. You've been there for 30 years and you get a proficient rating, you're gonna get paid, so there's no age.

Every other business is kind of an age earnings profile. But is it really just or does the ratings kind of?

>> Eric Hanushek: No, in the phase in there is an age, you'll see an age component, but in the end system there is not.

>> Speaker 19: Because the rating does depend on, it's a supervisor's assessment, so.

 

>> Eric Hanushek: Sure, and there are rules about a brand new teacher enters as a progressing something. And there are rules on how quickly you can move from up in categories and to be an exemplary teacher. There are a bunch of different hurdles, but you can move up over time.

 

>> Speaker 19: Okay, got it.

>> Speaker 20: And what are the weights on the different, like the teachers are evaluated by a supervisor and their students write about them, parents write about them, there's test scores. Are they teaching to the test or?

>> Eric Hanushek: So I think that not everybody has test scores that can be used to measure their performance.

For those that do have test scores, which in Washington DC is about a quarter of the teachers in Washington DC can be evaluated by test scores. I think it's more in Dallas because they developed some new tests that were available. I think if you have both observational ratings and test scores that are about evenly split.

If you don't, the observational ratings are the most. They include things like ten to 12 random observations of teachers for at least 15 minutes over the year and one or two longer observational meetings. And they're highly structured. The observational ratings are a bit structured on what's going on in the classroom and what people are doing.

And I think that there's pretty good evidence that those are quite closely related to student performance, the evaluations. So it's not just I walk in and say, Bob, you're not doing a good job and walk out. It's that I have to record that over multiple times.

>> Speaker 21: It isn't just about the weights because if some of the inputs, for example, have much more variability than others, then because they're forced into buckets, the source of that variability is going to be the source of the ranking.

 

>> Eric Hanushek: Yeah, so I haven't actually looked at that and I don't think anybody has, but you're absolutely right about that.

>> Speaker 17: Sorry to interrupt again. I'm still stuck on this. Is this transparent? I mean, are these rankings known so the parents know that they've got the person who was ranked unsatisfactory?

 

>> Eric Hanushek: No, this is public information. It is public information. I'll come back to that in Texas. The commissioner in Texas was actually on the school board in Dallas before he was commissioner, and likes this system, and has put in place a variant of this from the state level of incentives for it.

And his idea is exactly trying to get a better market for teachers and effective teachers by this information becoming public, and people perhaps bidding for good teachers and maybe even parents revolting about bad teachers, Evans. Okay, we've got that. So let me talk about evaluating the overall impacts, first on Dallas, and then I'm gonna talk about the specific applications to supporting difficult schools.

So the problem, of course, is that the reforms fly to the whole district, and it's a little bit hard to know exactly how to evaluate that. We're going to rely upon the magic of synthetic control groups to try to develop a comparison group for Dallas and to see what happened.

What we're going to do is try to match all of the schools in Dallas to schools in the 20 other high poverty, big districts in the state of Texas. And we're going to try to have a control group for Dallas that is made up of a composite of schools in other large urban districts as seen in 2013.

2013 is kind of important because the principal system comes into effect in 2014 and the teachers in 2015 and so we're going to try to get a control group before this really comes into effect. And then I'm going to try to convince you that the control group is reasonable by looking at the transition period between 2013 and 2015.

 

>> Speaker 22: What's the principal system?

>> Speaker 23: Principal system also?

>> Eric Hanushek: Yeah, there's a principal system it's almost exactly the same is that there are rating categories for principals and they get paid different amounts.

>> Speaker 23: That went first.

>> Eric Hanushek: As far as we can tell, that went into effect a year before, which was a good thing because you got a presumably better teacher principals there but they can't do much if they can't control who's in their schools and what their schools look like.

 

>> John Taylor: I have a question over here.

>> Speaker 24: I assume Dallas is like other big cities and it's a low performing district overall.

>> Eric Hanushek: Yes, and I'll show you that in just a second, so everything I do is gonna use the state testing system, which changed a little bit over this time period, but I don't think that affects our results.

But on average, Dallas is about 0.3 standard deviations below the state average beforehand and our synthetic control group is 0.3 standard deviations. Below 0.3 standard deviations is a pretty big number for the average of the school district.

>> Speaker 25: What does that translate towards like, years of skill attainment?

Because I don't know what.

>> Eric Hanushek: The rule of thumb is that one standard deviation of performance is three to four years of schooling. So this is like a year of schooling difference for on average, of every kid in its district compared to the rest of the state. So take kids at the same age, schooling level, they're a year behind in terms of learning in Dallas compared to the rest of the state.

So what we find is that before things come into effect in 2015, basically when the teacher program kicks in, the pre-trend that we have in 2015, 2013 and 2014, which wasn't used in creating this synthetic control group looks the same, and there's statistical tests. I'm not going to provide any of the gory statistical details behind this, I'm going to give you the answers and we can go into them if you want but here's the picture, yes?

 

>> Speaker 26: There any experimentation with charters or anything like that happening at this time period?

>> Eric Hanushek: With charters, yeah, charters are coming into effect throughout Texas over this entire period. There are quite a few charters in Dallas and in Texas, texas is a fairly large charter state. We don't have anything that would indicate that there's any real interactions between the charters and this but we haven't looked in very much detail at that.

 

>> Speaker 27: I have one quick design question, I was curious whether you considered using regression discontinuity based on the bins of ratings and teachers proximate the thresholds that created incentive or salary differences?

>> Eric Hanushek: I don't think we have their performance data before Tom, we haven't done that, which is a possibility, I guess, I don't think there's anything on that.

 

>> Speaker 27: I don't think it quite gives you the same full average treatment effect of interest, but it speaks to the theory of change because it leverages the contrast.

>> Eric Hanushek: Yeah, and so I don't know if we can follow teachers all before the rating system and into it, i's a possibility.

And so here's the answers, this is math scores of math achievement and what you see is years beforehand there must be some difference here. The black dots are Dallas Independent School District, Dallas ISD, and the red dots and lines are the synthetic control group. And what you see is that before 2015, we've got a control group that tracks this, and then we let the system rip after 2015.

And what you see is that the control group stays at roughly 0.3 standard deviations below the state average. A synthet is what's on the vertical axis and the Dallas system goes to about 0.08 standard deviations below the state average, so that within four years, one of the poorer districts in the state is starting to approach the state average by this change in the pay system.

 

>> Speaker 28: This looks almost too good to be true in the pre-trend. Why, since it's separate school districts, 2007, did we just have a year worth of dummies or why in the world would schools across the state? It means it's a signal, not noise, that it's happening to both.

 

>> Eric Hanushek: So what schools are changing in all kinds of ways. There was something called a 2008 recession, too. I don't know if that doesn't quite work out, there are policies that are. Changing across the state and within districts all the time. So the synthetic control group is trying to find a group of schools that mimic the performance in our schools in Dallas, but that would pick up any gross changes, like changes in the testing system, changes in the accountability system, changes in the spending and so forth.

So that we have sort of artificially, through 2012, created a control group that is matching the.

>> Speaker 28: What would happen?

>> Eric Hanushek: So if you look at test data for the entire US over time, you see some jumpiness. Even though there are 100,000 kids taking the test, there's some-

 

>> Speaker 28: Is the comparison point a fixed object, or is it also changing year to year?

>> Eric Hanushek: The comparison point-

>> Speaker 28: When you say they're 0.3 standard deviations below, is that below something that's fixed over time?

>> Eric Hanushek: It's below, no, it's below the state average in that year.

>> Speaker 28: That year, okay.

 

>> Eric Hanushek: So we've normalized all the tests to the state average for the particular year.

>> Speaker 14: And so there is a lot, I mean, at least in the data they used for Florida, there is a lot of change across years because they change the nature of these tests.

>> Eric Hanushek: Yeah, sure, but-

 

>> Speaker 14: So part of it could be explained by that.

>> Eric Hanushek: That's going on in the background, yeah, yeah.

>> Speaker 4: Okay, what fraction of all of the children in Texas are in Dallas? And the reason I ask is, to the extent that the Dallas scores go up, that will affect the state average.

So in some ways, you're weighting it against Dallas, this is my adding-up idea from last week.

>> Speaker 4: Yeah.

>> Eric Hanushek: Bring this out, there are 130,000 kids in Dallas, and I am remembering, and this is memory of 2 or 3 million kids in the state. So Dallas is a big state and it has lots of big districts and so forth, so I don't think that that's driving much.

 

>> Speaker 29: Fred wouldn't be affected by that anyway.

>> Eric Hanushek: Right.

>> Speaker 14: Eric, I'm curious, and maybe you're gonna go there, this is related to the question before. So in principle, this effect could be driven by two mechanisms. The first one is teachers somehow start working harder. So if you think about the developing world, in the past, the teacher wouldn't show up.

 

>> Eric Hanushek: Yeah, yeah.

>> Speaker 14: The fact that they show up, their rating are gonna go higher, or teacher move in and out of different qualities. So you could separate the two if you have the teacher fix effect, which I think is-

>> Eric Hanushek: We tried to do that to some extent, and what we find is about a third of this difference looks like it's the composition of teachers and two-thirds is something else.

 

>> Speaker 3: Pell has a really good point, just the rating isn't doing this, it means something was done with that rating that-

>> Eric Hanushek: Yeah, yeah, sure.

>> Speaker 3: Where I guess we're getting to.

>> Eric Hanushek: Yeah, well, we're at, this is it in the sense that here's the outcome. I'm gonna talk about using this rating system to provide incentives and assign teachers to bad schools in a minute.

 

>> Speaker 3: But they must have done something with it. If you just rate the teachers and-

>> Eric Hanushek: Well, I mean, what they

>> Eric Hanushek: If you talk to Mike Miles, he would say it's a whole combination of the system of providing support for individual teachers, better leadership of the principals, some use of more standardized curriculum, and so forth.

Mike Miles speaks in larger pictures, and it's hard to actually separate this out into individual components. Let me come back at the end and talk about Houston because that has some interesting parts.

>> Speaker 30: Are you going to break this down by type of school or type of student?

Do you have the data that allow you to do that?

>> Eric Hanushek: No, I'm not gonna do that.

>> Speaker 30: What about type of performance?

>> Eric Hanushek: Hm?

>> Speaker 30: Math's only one measure.

>> Eric Hanushek: Not true. Here's reading. And reading is in, in every evaluation of anything in schools, reading performance is less affected by schools than math performance.

And that's what you find here, you find the same pattern, but it's muted that the same analysis suggests that this system had an effect on reading, too.

>> Speaker 30: But is there evidence that moving a good teacher from a good school to a bad school has a positive net effect?

That seems to be the motivation behind the incentive program, right?

>> Eric Hanushek: He's not a shill.

>> Eric Hanushek: I didn't give him this question.

>> Eric Hanushek: Let me talk about accelerating campus excellence or the ACE program, which is designed to do exactly what your question says with this system. There were two concerns that the leadership of the district had.

One was it might be harder to get a good evaluation in a crummy school. And secondly, we were worried about crummy schools and wanted to fix them. So what they did was develop a program that starts a year after the overall teacher evaluation system went into effect. That took, well, four elementary schools.

In ACE I, there's going to be two different ACE programs that are there. One of the schools was identified as cheating on test scores in an earlier period. So we dropped that from the analysis. It turns out not to make much difference if we included it or not, but it's not good to have cheating schools in your analysis.

And what the ACE program, there was a second version of this later on that they were concerned about, the ACE I program was the very bottom ranked schools in the district. The ACE II was essentially the second from the bottom schools in the district in a subsequent program that came into effect later.

What they did was the following. First, they assigned an effective teacher to each of these crummy schools. I should say poor-performing schools we're broadcasting here. Each of these. Low-performing schools got a new principal. None of the existing teachers in these bad schools could automatically stay in these schools.

They had to reapply if they were interested in staying.

>> Eric Hanushek: They were paid bonuses based upon their prior rating, effectiveness rating on the system. And there was a little bit of extra money for other things, but most of the money went for personnel. And this is back to Ken's point about whether it's cost neutral or not.

 

>> Speaker 14: Can you give us a sense of what the percentage of how big the incentive had to be to move to a harder school?

>> Eric Hanushek: Yes. Thank you. I didn't set this question up either. Thanks to my colleagues for asking the right questions at the right time. $2,000 signing bonus to anybody who would go there.

The principal got an annual stipend of $13,000. The teachers, and this is the part that's unique to Dallas, the teachers got different amounts of annual stipends depending upon their prior rating so that the best got $10,000 annual stipend, middle got 8,000, and lower got 6,000. Now, what this does is contrast very dramatically with the way that schools have dealt with underperforming schools in the past.

In the past they have said, we have trouble getting people into our schools. We will give a $5,000 bonus to anybody who wants to teach in this poor school. My general economic principle that's applied here is that bad teachers like more money as much as good teachers. And so you end up with, on average, the same mix of teachers going into these bad schools as was there before or within the district.

And historically, anytime they put on combat pay or extra pay for the poor schools, the results have indicated nothing happens. But not completely is the case, not completely the case.

>> Speaker 31: Good teachers like to teach good students. So actually, I would guess there would be a negative selection in that direction

 

>> Speaker 4: and in safe environments.

>> Eric Hanushek: Sure. So, I mean, historically, what's happened is that in many school districts, you have a right to select your school based upon your seniority. And the more senior teachers find it's easier to teach in high-performing, schools and they migrate to high-performing schools.

The low performing schools get all the new teachers because they have no choice. And then historically, nothing happens other than you maintain the historic differences in performance of kids.

>> Speaker 32: Eric, what are ballpark average salaries, 50, 60, 70,000?

>> Eric Hanushek: There were about 60,000 at this time,

>> Speaker 33: depending on your objective.

It's not obvious this is right. So suppose you want to raise the average test score in the district. You might want to match the good teachers with the really good students.

>> Eric Hanushek: Yes.

>> Speaker 33: In New York, putting the worst possible teachers in Stuyvesant High School might not be the smartest thing.

 

>> Eric Hanushek: That's why I said that the goals of this school district leadership were twofold. One is trying to make sure that the system was fair, and secondly was to being able to deal with the worst schools. I mean, that is a design issue that I think we could have a good open discussion about at the end, because if nobody leaves or comes into the system, this is really zero sum.

And what you've done is dealt with the poorest performing, but at the cost of elsewhere in the system.

>> Speaker 34: Do you know anything about value added by good teachers by type of student? I mean, do they just get out of the way of good students?

>> Eric Hanushek: No, I don't think-

 

>> Speaker 34: Or is this proportional or what?

>> Eric Hanushek: I don't think that there's much. I mean, the idea of value adders is try to get estimates that are independent of the background of students. And I don't think that the normal measures of estimates of these are very closely related to the backgrounds of kids.

 

>> Speaker 34: So we don't know much about the point John's getting at. Is this-

>> Eric Hanushek: No, no, I think if you believe what I just said, we know a lot about what John just said. If you want to maximize the impact on the schools, and, in fact, good kids will do better with a good teacher, then you'd reassign your best teachers to the highest-performing kids.

 

>> Speaker 14: But maybe the good kids don't need such a good teacher because-

>> Eric Hanushek: Well-

>> Speaker 14: I mean, I think-

>> Eric Hanushek: The US they do when we were 31st in the world in math performance.

>> Speaker 35: The teacher who knows calculus, sending that teacher down to-

>> Speaker 14: Yeah, no, no, I agree.

 

>> Speaker 35: Be a waste, and then even I needed a teacher in-

>> Speaker 14: In the Florida public data, it was forbidden, I mean, by the Florida Department of Education to calculate value added. It was really, I mean, we could get the data, but we couldn't calculate it. So I think the answer is we don't know.

 

>> Eric Hanushek: Well, I mean, it's worse than that.

>> Speaker 14: No, by background of students.

>> Eric Hanushek: This is a policy decision in California, where it is illegal to relate any teacher to any student, student performance.

>> Speaker 36: There's also the selection question. To the extent that you ruin your top-end high schools, those kids leave the public schools.

 

>> Eric Hanushek: Yeah, yeah.

>> Speaker 14: Yeah, they have to-

>> Eric Hanushek: No, no, I'm not saying that this is necessarily the right objective function we ought to be following, or we might follow different ones. What I'm trying to say is that first Dallas objective function was to try to deal with their poorest performing schools.

And so this was a system designed to try to deal with the very rock bottom of their schools. And what I'm gonna show you is that it did, okay?

>> Speaker 37: Now, people have mentioned the word background. Now, we do know backgrounds of students. We know that you have data in the background.

We know that those factors are extremely important in performance. And so when you talk about a school being high performing, perhaps it's just that the kids are all from backgrounds where you expect them to be high performing.

>> Eric Hanushek: That's-

>> Speaker 37: That's kinda like a value added thing, but all of this discussion assumes that That this.

That what you're seeing here is value added or not value added as opposed to.

>> Eric Hanushek: No, no, no, this is. At this point, choosing the schools is not value added in any sense.

>> Speaker 37: No, but in terms of your analysis of, you're talking about a poor performing school and a high performing school, but you're not in your evaluation.

 

>> Eric Hanushek: Right now, I'm not saying anything about value adder. I'm saying that we chose schools on their lowest performance and historically doing the worst and looked at, see whether we could change that by an incentive structure.

>> Speaker 14: And the assumption is that the student population didn't move, so you're pretty safe there.

 

>> Eric Hanushek: Yeah, well, and we verified that actually that this wouldn't. The results we got are nothing a function of families moving in and out of these schools. We know that.

>> Speaker 38: Are we gonna see the results of it? So sorry.

>> Eric Hanushek: Yeah, I'm trying to get there, but I have to deal with my inquisitive colleagues.

So here's the story of on the left hand side are the ACE 1 schools, the ones where these incentives were provided. The blue lines of are the distribution of teachers across categories in 2015, right before ACE went into effect. And the red lines are what happens after the incentives are provided.

And so what you see is that the red and blue districts are largely non overlapping. And it says teachers respond to incentives, which is something that has been debated in the education world of whether teachers respond to incentives, but they do. The right-hand side is our control group.

The control group, by the way, for this study, is the 15% next highest achieving schools in terms of levels. We take the lowest for ACE program, and then the next 15%, their performance is the control group here. And in the control group, you see that before ACE and after ACE, nothing happened.

They don't have very many on the right side of the distribution of the exemplary and high proficiency categories, and they don't change.

>> Speaker 39: That's really surprising. I would have expected him to leave the next highest.

>> Speaker 14: Where do they come from?

>> Speaker 39: They go to semi bad school.

Those ones are gonna go to advanced.

>> Eric Hanushek: No, no, we're talking about 2% of the total population here, of the total school teacher population. And so it's not enough to swing the remaining district.

>> Speaker 39: There's an adding up constraint, those guys in the right-hand side.

>> Eric Hanushek: There's an adding up constraint, that is true.

That what I said before, and I believe, is that it's zero sum for the whole district. But you don't see it in specific schools because it's a small part of specific schools. Okay, so this is what happens. You give-

>> Speaker 40: What is the change on ACE 1, it seems to be changed from-

 

>> Eric Hanushek: What happens in ACE 1 is you go from having all unsatisfactory and progressing teachers in the lowest categories of proficient, to having higher proficiency teachers, and in the highest groups and exemplary. This is just a frequency distribution of the teachers in the before and after ACE. And so nothing happens in non ACE schools that you can detect, but that in ACE schools, they move, okay?

And so what happens? This is ACE 2, which is the same thing, same sort of structure of having controls that are next group higher. We left out ACE 2 schools that were going potentially be a control group for the ACE 1 schools. We didn't put them in. It doesn't make any difference.

And what you see is ACE 2 schools that came into effect two years later had exactly the same result. The teachers responded, highly effective teachers, by their prior ratings, responded. They moved to these bad schools. Okay, and so what happened? Here's what happened. This is a diagram that my colleagues made, and I should have broken apart, but I'm gonna try to walk you through this.

This has got both ACE 1 and ACE 2 schools on it. The circles there are ACE 1 schools. The dark circles are the ACE 1 schools. The empty circles are the control group for ACE 1. And similarly, the darkened diamonds are the ACE 2 schools. And open diamonds are the control group.

What you'll see is, if you follow the control group across all the years from 2012 to 2019, both the control groups for the ACE 1 and ACE 2 schools are basically flat. Nothing happens in those schools over this time. And what you see here is that in the first year of the ACE program, scores jump from sort of roughly four-tenths of a standard deviation below the city mean here.

This is the city mean to the city mean, and they remain high. They immediately jump when you change the composition as I showed you in the last slide. You get the same thing in ACE 2 schools. In the first year of ACE 2, the scores jump. But then, partly because of a Judd comment question and partly because of the not thinking it through.

In 2019, the ACE 1 schools have been doing very well, so they were no longer in the category of needing help, and they removed most of the incentives for teachers there. It turns out that a large number of the better teachers left, some going to ACE 2 schools, by the way.

And what you see is that scores plummet. When they take off the incentives, teachers leave, and they start reverting back to To their old level of performance. So I think this little picture, I think, which you can reproduce in various regression forms and so forth. This little picture, I think, identifies the fact that differential incentives matter, that are based upon the effectiveness of teachers, and that you have the power to take the worst schools based upon family background and everything else.

And move them up to the to the city average if you provide them with good teachers or good principals.

>> Speaker 41: There's two parts of this, and one part of it is less compelling than the other part. The part that's less compelling, and maybe you'll persuade me otherwise is the part about how teachers respond to incentives.

What you really show is that some teachers, a small number of teachers relative to the system as a whole, are responding to the incentives. This is the point about these schools only account for. So is it the case that there was a large number of teachers who wanted to transfer into these systems, good teachers, but they weren't able to?

 

>> Eric Hanushek: I don't know that story, I don't know what happened there.

>> Speaker 41: Okay.

>> Eric Hanushek: I'll come back to that a little bit in terms of Houston, I know a little bit about Houston.

>> Speaker 14: I have a related question, which is, I think for teachers to go into a bad school, I think a great incentive would be that the principal is good.

So maybe was the crucial aspect of this thing is that they moved at the time in which, because if you go to a place, which is crazy you're not gonna be as effective, right? I guess you cannot separate the two, but.

>> Eric Hanushek: That is in the background, you're absolutely correct that the working conditions depend upon the quality of the of the boss and what the structure set, so, yes.

You had a second, second.

>> Speaker 41: No, I said the other part of it that is compelling is just the effect of having good teachers there, that's not subject to the concern I just raised.

>> Eric Hanushek: Yeah, yeah, so we'll come back. We don't know the answer to that of can can you change the entire 130 schools in Dallas as opposed to just a subset of 7 elementary schools that I'm looking at here?

There's some middle schools, by the way, that we didn't analyze in the same way, or we did, but it's harder because of testing. So you see the same thing, by the way, in reading, answering John Taylor. It's not just math, you get the same increase in 1st year ace, you get the same increase in 2nd year ace, and you get the same decline when you take off the incentives, everything is just more muted with reading.

Which is not a surprise when you see any of the performance of schools.

>> Speaker 41: These effects are actually huge if I understand them right. Because what they say is like, go back to the, well, stay here the first year, effective aces, erasing a half standard deviation. And that's a year and a half of schooling.

 

>> Eric Hanushek: So let me go back to the others. It's not that much, let me go back to the other one where it is close. It goes close to a half standard deviation.

>> Speaker 41: That's a racing. And that's a year and a half of schooling. According, you said it earlier, a standard deviation is about.

So in just one year, you were racing a schooling gap of a year and a half, bro, that's an enormous effect.

>> Eric Hanushek: It is enormous, it's absolutely enormous. And it brings these schools up to the district average, the worst schools, and it brings the entire district of Dallas up to the state average with the whole system.

 

>> Speaker 42: The upper schools did not lose.

>> Eric Hanushek: They did not according to the

>> Speaker 42: previous measure,

>> Eric Hanushek: yeah, yeah.

>> John Taylor: Now, awfully quick, yes.

>> Speaker 43: You're always talking about teachers as if it was a homogeneous group of people, but math versus reading, we're talking about different group of teachers.

 

>> Eric Hanushek: No, these are elementary schools where they're teaching everything.

>> Speaker 43: Okay, okay, sorry.

>> Eric Hanushek: These are elementary school element, should have made that clearer. These are not specialists. So the more compelling story, I think, is that we then looked at, we were looking at performance in grades 1 through 5, actually, we test them in.

It's grade 3 and 5, in the previous things where we test kids, they go to middle school in grade 6. And the question is, do you see these performance gains holding for grade 6 after they've left the ace schools? And so the left hand side is the full distribution of performance of ace kids versus control kids, controls are the red and ace are the blue.

And then we look in 2019 and you see that this distribution has been systematically across the board move, slip to the right, and that this is the longer term effects that you see now. If you look at just kids that have had one year of ace from the and then left, they don't get much of an impact.

So what you, what you see with the one year that our interpretation at least, is that it's easier to change the short run test scores than it is the underlying deeper learning. But after two or three years, you see that kids have a lasting impact in subsequent performance.

Suggesting that this is not just teaching to the test or anything, but that they are, in fact, learning things that carry through.

>> Speaker 44: Or that teaching to the test is actually useful.

>> Eric Hanushek: Some of us have argued for that test. Some of us have argued for that yes in terms of general accountability, that if you have a reasonable test you want.

 

>> Speaker 4: Part of what they're learning is how to learn.

>> Eric Hanushek: Yeah, okay, so let me give you some conclusions here, and then we can talk more widely. So I take this as supporting my general unrecognized policy proposal to improve teacher effectiveness to deal with the pandemic problem that teachers respond to incentives.

And having effective teachers makes a difference, because what we're seeing here is. Change the composition of the teaching force, nothing about class size or anything else. And if you get better teachers in, you get better performance of the kids.

>> Eric Hanushek: I think this looks like it's scalable because we have several full schools that are doing this.

We have prior evidence. I think that the Washington, DC, whoops, Thomas left, Washington, DC, system has lasting impacts that has improved schooling in Washington, DC. It's still not a very good system, but the kids have done better. And I've given you two districts out of 13,500 where we do something sensible, what I would call sensible, and get some results when I wake up optimistic.

I think that, well, these are institutions that we could maybe move someplace else. When I wake up pessimistic, I wonder, are these individuals, Michelle Ree fighting in Washington, DC, and Mike Miles fighting in Dallas or in prior incarnation, jeb bush fighting in Florida, where individuals are able to get some changes that when they leave, they sort of stay but get whittled down over time.

And we have an, I don't know if it's the right experiment. There's Houston, because as I already told you, Mike Miles stepped out of the Texas schools for a while and then was brought back as superintendent in Houston. And this addresses a little bit of Steve's question.

>> Eric Hanushek: There's 200 schools in Texas.

In Houston, 27 of them were identified as failing. And that gave the possibility to the state commissioner to replace the school board and the superintendent and bring in administration. So Mike Miles came in as superintendent and basically took a variant, it's not exactly the same system, but a variant of Dallas, and sort of said, here's what we're gonna do in these 27 schools.

It involved spending more money and did that. And then over the summer, he was doing training for the principals in Houston, and he had all the principals together, and he explained to all the principals what was being done in the 27 schools. And a bunch of other principals said, well, why don't we do that in our school, too?

And he said this is pretty serious stuff. You gotta make sure that your teachers and you really wanna do this because it's pretty serious. And we're not going to give you all the money that the 27 get. And something, I think it's 53 schools went back to their teachers, got agreement that they should change things in these dramatic ways in evaluation, everything.

And they joined the system in a sort of second category of being in this evaluation system. We don't know any of the answers yet. There's lots of town meetings where people yell at Mike Miles, in part because he eliminated 23 librarian positions to arguing in a sort of Judd like way that, well, money isn't available for everything.

We have to make some choices

>> Speaker 46: about other states, other areas.

>> Eric Hanushek: Let me, let me say a little bit more, one more thing about Texas. Texas three years ago, as I said, the current commissioner had been from Dallas, and three years ago, the Texas legislature passed something called HB3, house bill three.

Which, among other things, had a pot of money in it that would give incentives to any school district in Texas that put in place a reasonable accountability system and use that to help bring good teachers into poorer schools. This is actually my idea of how you can do this.

He didn't say institute the Dallas system or the Houston system. He said, if you can come up with a reasonable system that's based upon your own capacity and your own interests, we'll provide the support from outside to deal with the Judd budget constraint and we'll help to pay for this.

But in part to try to get a market for these teachers to be identified in various school districts as good. The money followed the good teachers. Essentially, the state would provide more money for the good teachers. And presumably, I don't have any actual data on this. If a good teacher leaves San Antonio and goes to some other district, the money presumably follows the teacher.

So that you have starting to have some sort of pseudo market for teachers and effective teachers. It's gonna work. I don't know. It looks to me like there's a lot of evidence here that if you do sensible things, it pays off.

>> Speaker 47: The Dallas accountability and compensation system, as you described it, I would think would greatly undercut the demand for advanced education degrees for teachers.

 

>> Eric Hanushek: Hopefully.

>> Speaker 47: Hopefully. Right, right. So part of the institution, the institutional barriers aren't just the unions, they're the whole credentialing.

>> Eric Hanushek: Yeah.

>> Speaker 47: That education schools are a part of.

>> Eric Hanushek: I mean, so look at this Dallas. I mean, Texas has 1000 school districts and you have this one Dallas breaking away.

They have no collective bargaining, and yet the system has a lot of inertia in it. It keeps doing what was done before. And

>> Speaker 47: is there any evidence on what happened to teacher enrollment in Dallas?

>> Eric Hanushek: Yeah, there's a little bit of evidence that, and there's, I think, more evidence from Washington, DC.

In Washington, DC, you at least got the poorer teachers to leave more readily and had the best teachers stay around a little bit longer. It's a little bit hard to get comparisons of who is moving in and who's moving out because you don't have measures of their effectiveness before.

Further in the district. So it's a little bit hard to do that. I think, well, I can do this in two minutes, and I won't be attacked for it.

>> Eric Hanushek: I think it's really hard to ever guess who's gonna be a good teacher before they're in the classroom.

And what these systems do is emphasize performance after they're in the classroom. I don't know, okay. Well,

>> Speaker 48: What you're finding is that there's sort of a generalized good teacher.

>> I'm actually surprised that somebody who's a good teacher in a fancy school can also be a good teacher.

 

>> Eric Hanushek: That's the evidence that we see. And we see that evidence that people who are good are good.

>> Speaker 26: You've mentioned just public schools. But I'm vaguely aware of the near miraculous things that some charter schools, like the success academies.

>> Eric Hanushek: Yeah.

>> Speaker 26: We've in New York where they go into the absolutely.

 

>> Eric Hanushek: Yeah.

>> Speaker 26: Most horrible schools and they turn out kids who are beating state averages, which seems to sort of go further to prove your point. Good teachers can do wonderful things with a difficult input level.

>> Eric Hanushek: Yeah, and. There's no magic system and that's part of the problem here.

Success academies has a very scripted curriculum and program and they monitor it. And teachers that aren't performing well according to that system don't stay around. But it's hard to put that in place in lots of other places.

>> Speaker 50: What really you're showing us is you can get some way in that direction even in a public school system, at least so long as it doesn't have a big.

 

>> Eric Hanushek: Yep.

>> Speaker 50: Blue city union problem.

>> Eric Hanushek: Yeah, exactly, that's the message. The discouraging message is I'm talking about two school districts or a handful of school districts out of 13,000.

>> Speaker 51: Rick, we have a question online.

>> Eric Hanushek: Yes, Paul.

>> Paul: Hello Rick.

>> Eric Hanushek: Yes, hi, Paul.

>> Paul: Yeah, so I have a short question.

And then depending on the answer, I'll have a longer question. So the short question is, are these the average scores at these schools that you're talking about? Or are they the individual level scores of the each student at the school?

>> Eric Hanushek: They're the average of the individuals.

>> Paul: The average of the individuals at the school level.

 

>> Eric Hanushek: Yeah.

>> Paul: Well, because when you say standard deviation of 0.4, that's a very big number. I mean, in the study that Mathematica did of Washington DC, the total impact was about a tenth of a standard deviation. And that was over the whole impact. It wasn't just in one year or anything like that.

So you're getting much bigger impacts here. But I'm wondering if you can really use that same standard deviation unless you've got population of individuals that you're looking.

>> Eric Hanushek: That's an individual. That's a standard deviation of individual scores.

>> Paul: It's not school average.

>> Eric Hanushek: It's not the standard deviation of the school average, it's the standard deviation of the individual scores.

 

>> Paul: All right, well, that's more impressive. I mean, that's truly impressive. So that invites my other question, which is. It's very hard to keep that going to scale. I mean, if you go to DC, you've got a whole innovation at scale for the system as a whole. You've got it in a specific set of schools almost any innovation in a pilot kind of setting.

Once you go to scale, you lose a lot of the impact. So do you have?

>> Eric Hanushek: That's been the historical comment, but here we went from AC one to AC two and got exactly the same results. We started with our evaluation was three. There were four elementary schools, and then it went up to eight elementary schools in AC two and it stayed.

You could reproduce this as you went up. Now, there may be limits to it. But this is more scalable than almost any other experiment that I know of.

>> Paul: Did I hear you say you went from three to eight?

>> Eric Hanushek: Yeah, yeah, we went.

>> Paul: Okay, yeah, I don't really see that as failing.

I mean, you're a long ways from being at scale in Dallas.

>> Speaker 53: John Taylor had to leave, so I'm gonna take his prerogative and call time. And thank you very much.

>> Eric Hanushek: Okay.

>> Speaker 4: Thank you.

>> Speaker 54: That doesn't wanna go. Go.