Behind the Headlines of Teacher Quality Research
Michael Diedrich, Minnesota 2020, November 12, 2013 – “It works!”
That’s what we all want to hear. In education reform, the stakes are high, the equity gaps are increasingly well-known, and the thirst for solutions is strong. We know the outcomes are a problem, and we want to fix it. This has driven research (and investments) into a variety of areas, one of the most visible of which is teacher quality. Within the last month, two different studies have come out on the topic, often accompanied by triumphant headlines that over-represent the findings. The answer to, “Does it work?” is usually, “Yes, sometimes.” For these two studies, the conditions on “sometimes” diminish much of the “yes.”
Teacher Transfer “Works”
The more recent of the two is a study by the federal Department of Education’s Institute on Education Sciences [PDF]. It looked at the effects of the Talent Transfer Initiative, a program that created incentives for teachers to transfer to schools in greater need. It aimed to examine how well effective teacher could transfer across schools, a question which has significant ramifications for district transfer policies.
Many of the headlines about this summary are some variation on, “It worked!” They suggest that there are benefits to be gained from transferring teachers, but the evidence from the report only supports a very narrow set of conditions and benefits. We cannot conclude from this report, for example, that districts should adopt more aggressive teacher transfer policies.
A more accurate summary of the research findings (though definitely not headline-sized) might go something like this:
Of the eligible elementary and middle school teachers with high “value-added” test score calculations, roughly 5% transferred schools in exchange for a $20,000 incentive. Of that group, the elementary teachers appear to have improved their students’ test scores by 4 to 10 percentile points relative to similar students, and the middle school teachers did not appear to have a significant impact on their students’ scores.
The research did not consider high school teachers, did not look at the effects of involuntary or non-incentivized teacher transfer, and did not offer significant policy recommendations to districts. It did not consider other means of identifying effective teachers, and did not look at student outcomes beyond math and reading test scores. That would be a lot to put on one study, which is why any one study shouldn’t be relied on too much as the basis for policy. The research is still valuable; it is clearly an interesting outcome that could guide future thinking about teacher transfers and quality. It just should not be used to justify major policy changes right now.
DC’s IMPACT Evaluations “Work”
The other major piece of research that got attention was a National Bureau of Economic Research working paper by Thomas Dee and James Wyckoff. It examined the effects of the DC school system’s IMPACT teacher evaluation system. Many of the headlines discussing this paper suggest that IMPACT improves teacher performance. Again, this is only true for a limited range of outcomes and under specific conditions.
One of the major caveats for this paper is the definition of “teacher performance.” For many in the accountability section of education reform, “teacher performance” should be defined by effects on student performance. Those people will be disappointed by this study, which did not give serious consideration to student performance. Instead, it looked at how teachers improved their score on IMPACT’s rubric.
A more accurate summary of this research might read:
Teachers identified at the low end of the IMPACT score range who were threatened with dismissal, given particular areas within their control to improve, and offered instructional coaching to improve in those areas, were more likely to improve their IMPACT score. Similarly, teachers near the high end who were offered a significant bonus to “cross the finish line” into the highly effective category by improving in particular areas under the control, were also more likely to improve their IMPACT score.
So, in addition to not seriously considering the effects on student performance, this research does not say much about IMPACT’s ability to help the majority of teachers in the middle improve. That’s an ongoing concern about teacher evaluation systems, and part of why we probably shouldn’t rely too much on them as drivers of improvement.
I raise these points because the kind of headlines and one-sentence blurbs that inevitably wind up accompanying these pieces of research often end up being used to justify major policy changes that the research does not support, based on claims the researchers did not make, with effects on students, schools, and teachers that the research did not claim to address.
Education research generates many interesting findings, and it is crucial to our ability to keep moving our schools forward. It is also too often misrepresented and abused in the service of particular agendas, and we should be willing to challenge that behavior.