About six months ago, I saw several stories saying that "Having just one black teacher can keep black kids in school," to quote NPR's summary. They all noted the magnitude of the effect: almost 40% reduction in dropout rates for low-income black boys. I located the paper on which the stories were based and thought about posting on it, but it was a long paper by the time I got around to reading it, the attention seemed to have passed. However, last week's NY Times magazine had a list of statistics on education, and one of them was "exposure to at least one black teacher in Grades 3 to 5 reduced the probability of low-income black male students dropping out of school by almost 40%." So that led me back to the paper.
The thing that originally attracted my attention was not the general idea that having a black teacher would help to keep black children in school, which seemed plausible, but that it could reduce dropouts by 40% for any group. There is a lot of data on basic educational outcomes like finishing school, and by the standards of social science it's high quality data. Moreover, there are a lot of people who have studied the issue, so it seems that any simple and straightforward way to dramatically reduce dropout rates would have been discovered long ago.
The paper reports that the estimated effect on dropout rates is -.04 for all black students, -.06 for persistently low-income black students, and -.12 for persistently low income black male students. Since about half of students are boys, that suggests that the estimated effect on persistently low income black female students would be about zero, and indeed they report an estimate of 0.00 for that group. So the issue was treating only the big estimate as worthy of interest. If you believe that there are differences in the effects on boys and girls (and the difference appears to be statistically significant), both of the estimates are equally important; if you don't, you should just report the estimate for boys and girls combined. The differences between persistently low income students and other students don't appear to be statistically significant (it's hard to tell from the tables), so maybe you should just report the estimate for all students.
There's also a more complex issue which relates to the way that they got the estimate. The simple approach would be to do a regression with dropping out as the dependent variable, and having a black teacher plus some other variables as independent variables. But the authors say that those estimates "are likely biased by unobserved student characteristics that jointly predict classroom assignments and long-run outcomes, even after conditioning on the basic socio-demographic controls in X and school FE (Rothstein 2010). For example, students with lower achievement (Clotfelter, Ladd & Vigdor, 2006) and greater exposure to school discipline (Lindsay & Hart, 2017) are more likely to be matched to black teachers, and these factors likely affect long-run outcomes as well." That is, black teachers tend to be given the kind of students who are at higher risk of dropping out. The authors had an idea on how to eliminate this potential bias. They had multiple students from each school, which means that they could include a dummy variable for each school. That's a reasonable thing to do, since it's generally agreed that some schools are more effective than others. They also had five different classes of students: those who started third grade in 2000, 2001, 2002, 2003, and 2004. Because of new hires, departures, and leaves, the percent of the teaching staff that was black could change from year to year. Those personnel changes would depend on idiosyncratic individual factors--getting pregnant, reaching retirement age, having a spouse get a job offer in another state--so they would be random from the point of view of the students. So you can use within-school variation in the racial composition of the teaching staff over time as a substitute ("instrument") for the original variable (having a black teacher or not) and get unbiased estimates.
This approach strikes me as clever but not very convincing. Teachers' decisions to stay or go will depend partly on how rewarding it is to work in a school. That could depend on student performance (teachers like it when their students do well) or on things that might affect student performance, like discipline problems, or how well teachers get along with the administration. Things get more complicated because what matters is differential effects on black and white teachers, but I can think of possibilities here too: for example, black teachers may be particularly interested in how the black students are doing. I think I might trust the simple results more than the results from their method--at any rate, I'd like to see them, but they aren't reported in the paper.
This isn't a straightforward mistake, but the sort of difference of judgment that often comes up with research, and the authors could probably say more in defense of their approach. But I will stick with my original feeling that a 40% reduction in dropout rates for anyone is too big to be believed.
No comments:
Post a Comment