An Interview with Daniel Koretz: On the “Leo” Problem

This is precisely the kind of talk that would get a teacher branded for "making excuses" despite its veracity.

This from Michael F. Shaughnessy at EducationNews.org

Daniel Koretz is the author of the book "Measuring Up: What Educational Testing Really Tells Us" published by Harvard University Press. He has been interviewed twice previously about some of the major concerns relative to standardized testing and in this interview, he discusses yet another problem in the testing arena.

1) In your book, in the chapter on error and reliability, you discuss the LEO Effect. Could you briefly tell us about this "Leo Problem."

The term came from a conversation among teachers in a Maryland school who were puzzling about a cohort of children who, grade after grade, performed more poorly than was typical for that school.One of the teachers said that he understood the problem: "It's Leo."Leo was a very disruptive student whose behavior impeded the work of every class he was placed in.

Leo might have been unusual, but the problem they were discussing is not.The performance of small groups of students—certainly, one class, but even the within-grade enrollment for entire schools if they are not very large—is highly unstable from year to year because of essentially random differences between successive cohorts of kids.This variation will be found even if the quality of teaching is constant.In one of my classes, "the Leo effect" has become the shorthand term for this year-to-year inconsistency.

2) You define the LEO effect " as short term fluctuations in aggregate scores, caused by sampling error". That is a very nice way of saying that one major discipline problem can cause a major skewing of the data. Yes or No?

The Leo effect is broader than this. Even if no children in the class are disruptive, there will be "good crops and bad crops," as one teacher put it.Some cohorts perform better than others, even in the absence of disruptive students.You might say that disruptive kids are a special case—and a particularly vexing one—of the Leo effect.

3) Almost all teachers will tell you about a student who is disruptive on an almost daily basis, and who disrupts the classroom environment, hence the teacher's ability to teach. How significant is the problem (which you have alluded to as the LEO problem?

We know that the inconsistency in performance from one year to the next is typically very large, but to my knowledge, we don't have firm data indicating how much of this inconsistency is attributable to disruptive students, as opposed to other factors.However, I think there is no doubt that the problem of disruption is sometimes large. Disruption has a multiplier effect that some other differences among groups need not have.

For example, suppose that one year, you happen to draw two students who are not at all disruptive but have severe problems learning math.Unless you find some way to offset their difficulties, their performance will lower your average test score, but it may not greatly affect the performance of other students in the class.In contrast, a disruptive student can interfere with the learning of the entire class.In other words, disruption has a ripple effect, and it can be large.

4) I have been to Finland, and have been told that when a student is SO disruptive, so problematic, so much of a discipline problem that they are simply removed from the school and either institutionalized or sent out into the work force to work. Is this a sane, ethical, moral approach, or what should be done with the various students who, despite all good efforts, remain problematic?

I started my career as a classroom teacher of students with behavioral and emotional problems—I provided the special placement for students who were too disruptive for regular classes—so I have given this question a great deal of thought.However, this is more a question of values and policy than a technical issue.My personal view is that schools must make serious efforts to control disruption in the context of regular placements, but if the disruption is sufficiently severe and difficult enough to control, it is to everyone's benefit to find an alternative placement.From the perspective of measurement, the question is simpler.

For many reasons—disruptive kids are just one—a classroom's scores for a single year year have to be taken with a grain of salt...