After recognizing the excellent analysis by Michael Vasquez and David Smiley of Florida’s school grades being strongly correlated with out-of-school factors associated with the students*, I have received several important related points from Vasquez about the unintended lessons coming from one of the most often cited reform states in the U.S.
First, Vasquez pointed me to Matthew DiCarlo’s What Florida’s School Grades Measure, And What They Don’t, in which DiCarlo explains:
A while back, I argued that Florida’s school grading system, due mostly to its choice of measures, does a poor job of gauging school performance per se. The short version is that the ratings are, to a degree unsurpassed by most other states’ systems, driven by absolute performance measures (how highly students score), rather than growth (whether students make progress). Since more advantaged students tend to score more highly on tests when they enter the school system, schools are largely being judged not on the quality of instruction they provide, but rather on the characteristics of the students they serve.
New results were released a couple of weeks ago. This was highly anticipated, as the state had made controversial changes to the system, most notably the inclusion of non-native English speakers and special education students, which officials claimed they did to increase standards and expectations. In a limited sense, that’s true – grades were, on average, lower this year. The problem is that the system uses the same measures as before (including a growth component that is largely redundant with proficiency). All that has changed is the students that are included in them. Thus, to whatever degree the system now reflects higher expectations, it is still for outcomes that schools mostly cannot control.
The really important aspect of DiCarlo’s analysis is that Florida’s accountability system has likely caused harmed instead of attaining the lofty goals often associated with accountability policies, as DiCarlo concludes:
In other words, there are many Florida schools with lower-performing students that are actually very effective in accelerating student performance (at least insofar as tests can measure it). This particular ratings system, however, is so heavily driven by absolute performance – how highly students score, rather than how much progress they have made – that it really cannot detect much of this variation.
Closing or reconstituting these schools is misguided policy; their replacements are unlikely to do better and are very likely to do worse. Yet this is what will happen if such decisions are made based on the state’s ratings.
Florida will have to do a lot more than make tweaks to truly improve the high-stakes utility of this system. In the meantime, one can only hope that state and district officials exercise discretion in how it is applied.
Both Vasquez and Smiley’s 2013 analysis and DiCarlo’s 2012 analysis, however, are even more troubling in light of a St. Petersburg Times (Florida) March 21, 1999, article, “A lesson in grading schools,” by Kent Fischer and Geoff Dougherty:
Bush believes the grades, A through F, will make it easier for parents to compare schools and assess how they are doing. A noble goal.
But a St. Petersburg Times analysis indicates his school-grading system may be fundamentally flawed.
It takes no account of the impact poverty has on student achievement, though many studies have proven that children from wealthy families generally outscore children whose parents are poor. So Bush’s grades are more apt to reflect the relative wealth of a school’s student body rather than the competency of its teachers, the newspaper’s analysis shows.
Fourteen years ago, then, Fischer and Dougherty accurately identified the flawed Florida school grading plan, but also acknowledged the key ignored hurdle facing education, poverty:
If there’s one thing that has been firmly established by research, it is the impact social factors have on student achievement. This does not mean poor kids can’t achieve. Many do. But poor children often lead transient lives, may suffer from malnutrition and endure higher rates of abuse and neglect than other children. They also tend not to be exposed to books, music and other cultural influences that help ready young minds for school.
Research – and the Times’ analysis – shows that when large numbers of students are considered, poverty reliably predicts test scores. The Times’ analysis found that depending on which test is given, from 69 to 79 percent of the difference in test scores among schools is explained by poverty.
That seems to ignore several significant demographic factors, like single-parent homes and student mobility. But many of those factors have an extremely high correlation to poverty and thus are effectively included in the analysis.
The bottom line, according to many experts: Any grading system that fails to take poverty into account is flawed.
Ultimately, Fischer and Dougherty offered Florida parents a message still relevant today: “There’s more to a school than good test scores. When trying to gauge school quality, educators suggest parents do some investigating.”
* See below a graphic (click to open and then click again to enlarge) related to Vasquez and Smiley’s article they were unable to include in the online article: