Two things all sides of the US public education debate will agree on: too many students are not doing well, and we need a way to measure which students and which schools need the extra help.

Beyond these expressions of general concern, opinions diverge in all directions. So most will agree that we need to assess students’ and school’s progress, but there’s no consensus about how to set about it.

A recent study in The American Review of Public Administration shows why measurement can be so controversial. Different measures paint very different pictures. The authors, William Miller of the University of Illinois and Brinck Kerr and Gary Ritter of the University of Arkansas, describe three approaches to school performance:

The Adequate Yearly Progress (AYP) method simply averages the scores of all students in a single grade within a school on a standardized test. This average score is then compared to a goal or standard. The controversial No Child Left Behind policy, implemented by the Bush administration, uses this method.

Rather than comparing student test score averages to a set standard, the Value-Added Model, compares them to earlier test scores for the same group. Thus the value-added model focuses on progress rather than proximity to a goal.

The Adjustment Performance measure is similar to AYP except for one notable difference: it factors in a handicap. It determines an expected performance level for a school given how poor its students are, how many students it serves, and other factors that could make educating students more challenging. Then it compares students’ actual scores to their expected scores.

Miller and company measured the math and literacy performance of Arkansas schools serving second and fourth grade students in 2002 and 2004 using each of the three methods. And their results are surprising.

Schools serving mostly low-income African American students performed quite poorly compared to other schools when the AYP method was used. However, when the researchers considered students’ average progress using the value-added method, the progress of the majority of African American schools was quite similar to that of other schools. Moreover, when the researchers applied a handicap using the adjusted performance measure, the African American schools significantly outperformed other schools.

No Child Left Behind sanctions schools that don’t meet standards. Yet, as the authors of this article point out, “the sanctioning will be the result of their students’ lower baseline status rather than the school’s ability to move students forward”.

• Summary of “School Performance Measurement: Politics and Equity” by William Miller, Brinck Kerr, and Gary Ritter in The American Review of Public Administration, March 2008, Volume 38, Issue 1, pp100-117.