Education, education reform, reading, Teaching, Testing

Mississippi Miracle, Mirage, or Political Lie?: 2019 NAEP Reading Scores Prompt Questions, Not Answers [Update September 2023]

6 Comments

Update September 2023

Grade Retention Harms Children, Corrupts Test Data, But Not a Miracle: Mississippi Edition

UPDATE

CRUMBLING SCHOOLS, DISMAL OUTCOMES: Alexander v. Holmes County Board of Education was supposed to change everything for Southern black children, Bracey Harris

UPDATE 15 February 2022

Opinion: Reeves’ Education Mirage

Key points:

To make his case, Reeves — much like the Mississippi Department of Education itself — is chronically selective in his statistics, telling only part of the story and leaving out facts that would show that many of these gains are either illusory or only seem to be impressive because the state started so far behind most of the rest of the nation….

Even the state’s impressive improvement on the National Assessment of Educational Progress may not be quite all that it seems.

From 2013 to 2019, the latest year for which results are available, Mississippi students rose faster in fourth grade reading than anyone on the national test. They improved their ranking from 49th to 29th. The gains in math were even more impressive, jumping from 50th to 23rd during that same time frame.

Reeves attributes the progress to “third grade gate,” the reform pushed through by Republicans in 2013 that requires third graders to demonstrate they are at least minimally proficient in reading before they advance to fourth grade.

The Republican belief is that the threat of having to repeat a grade has prompted students, their families and teachers to work harder to be sure that doesn’t happen.

Another interpretation has been offered, though. It’s that because of third grade gate, Mississippi’s lowest performing students get an extra year of instruction before they take the fourth grade test. With the state failing more than twice as many students in their early years as the national average, that could create a significant advantage, though probably a short-lived one.

The research remains inconclusive on this point. It’s not on the others.

REEVES’ EDUCATION MIRAGE

UPDATE 7 December 2022

Note the trends in Mississippi’s NAEP Reading scores from 1992 through 2022 and the contrast between grade 4 and grade 8:

Note that Mississippi’s grade 4 reading scores on NAEP show:

  1. MS has steadily improved scores over thirty years despite adopting different reading programs, implementing different state standards, and reforming with multiple reading policies.
  2. MS has had two large gains—one from 2005 to 2009, 7 points, and one from 2013 to 2019, 10 points, dropping to 8 with 2022—raising a question about the role of SOR legislation since that first large gain is pre-SOR.
  3. MS remains well below proficient on average, similar to other high-poverty states.

Note that Mississippi’s grade 8 reading scores on NAEP show:

  1. Apparent “gains” made in grade 4 have disappeared by grade 8 (consistent with research on systematic phonics instruction for reading).
  2. MS grade 8 scores on reading have remained relatively flat for 24 years.
  3. MS grade 8 larger increases happened in 2002-2003 and 2017-2019, again one well before and another after SOR legislation.

UPDATE 4 July 2023

Column: How Mississippi gamed its national reading test scores to produce ‘miracle’ gains (see HERE also)

MISSISSIPPI’S MIRACLE: Do we really have a “Mississippi mirage?”

MISSISSIPPI’S MIRACLE: Has the revolution reached the eighth grade?

UPDATE: Mississippi reading isn’t so miraculous after all

Mississippi Miracle, Mirage, or Political Lie?: 2019 NAEP Reading Scores Prompt Questions, Not Answers

There is a disturbing contradiction in the predicted jubilant response to Mississippi’s outlier 4th-grade results from the 2019 NAEP reading test. That contradiction can be found in a new article by Emily Hanford, using Mississippi to recycle her brand, a call for the “science of reading.”

This is a great deal to ask of the average reader, but Hanford’s argument is grounded in a claim that most students in the U.S. are being taught reading through methods that are not supported by scientific research (code for narrow types of quantitative research that can identify causal relationships and thus can be generalized to all students).

However, the contradiction lies in Hanford’s own concession about the 2019 NAEP reading data from Mississippi:

The state’s performance in reading was especially notable. Mississippi was the only state in the nation to post significant gains on the fourth-grade reading test. Fourth graders in Mississippi are now on par with the national average, reading as well or better than pupils in California, Texas, Michigan and 18 other states.

What’s up in Mississippi? There’s no way to know for sure what causes increases in test scores [emphasis added], but Mississippi has been doing something notable: making sure all of its teachers understand the science of reading.

To be fair, there is a way to know, and that would be conducting scientific research that teases out the factors that can be identified as causing the test score changes in the state.

In her missionary zeal for the “science of reading,” Hanford contradicts herself by taking most of the article to imply without any scientific evidence, without any research, that Mississippi’s gains are by her fervent implication a result of the state’s embracing the “science of reading”: “In 2013, legislators in Mississippi provided funding to start training the state’s teachers in the science of reading.”

Let me stress here a couple points.

First, scientific research connecting classroom practices to NAEP test scores is rare, but in the 1990s, comparative data were released on 1992 scores in 1997. That research showed a possible link between whole language practices and higher NAEP scores—something that Hanford and her “science of reading” followers may find shocking since they routinely claim that whole language and balanced literacy are not scientifically supported.

Therefore, it is simply far too soon after the release of the 2019 NAEP scores to suggest any relationship between classroom practices (as if they are uniform across an entire state) and NAEP scores. Any implications about Mississippi are premature and irresponsible to make for journalists, politicians, or advocates for education.

Premature and irresponsible.

Second, data from Mississippi are more than 4th-grade 2019 reading—if we genuinely want to know something of value about teaching children to read.

Mississippi’s outlier 4th-grade reading scores are way more complicated once we frame them against longitudinal NAEP scores as well as 8th-grade reading scores. These, then, are more data we should using to ask questions about Mississippi instead of making rash and unscientific claims:

MS reading grade 4 trend

4th grade reading trends

MS score gaps grade 4

4th grade score gaps

MS reading grade 8 trend

8th grade reading trends

MS score gaps grade 8

8th grade score gaps

Here are some complicated takeaways from this larger picture:

  • If the “science of reading” is the cause of recent gains in 4th-grade reading in MS, how do we explain that MS has seen a trend of increased scores since 1998 and pretty significant jumps between 2005 and 2009[1], well before the shift identified by Hanford in 2013?
  • Why does MS still show about the same gaps between Black and white students as well as between socioeconomic classes of students since 1998 if how we teach reading is the key factor in achievement?
  • And a really powerful question concerns 8th grade: Are any 4th-grade gains by MS (or any state) merely mirages since many states with 4th-grade gains see a drop by 8th grade and since longitudinal 8th-grade scores are mostly flat since 1998?
  • UPDATE: Todd Collins has raised another important caveat to the 4th-grade reading gains in Mississippi because the state has the highest 3rd-grade retention percentages in the country:

But Mississippi has taken the concept further than others, with a retention rate higher than any other state. In 2018–19, according to state department of education reports, 8 percent of all Mississippi K–3 students were held back (up from 6.6 percent the prior year). This implies that over the four grades, as many as 32 percent of all Mississippi students are held back; a more reasonable estimate is closer to 20 to 25 percent, allowing for some to be held back twice. (Mississippi’s Department of Education does not report how many students are retained more than once.)

This last concern means that significant numbers of students in states with 3rd-grade retention based on reading achievement and test scores are biologically 5th-graders being held to 4th-grade proficiency levels. Grade retention is not only correlated with many negative outcomes (dropping out, for example), but also likely associated with “false positives” on testing; as well, most states seeing bumps in 4th-grade test scores also show that those gains disappear by middle and high school.

UPDATE

(USDOE/Office of Civil Rights) – Data 2017-2018

Sources

Ultimately, if anyone wants to argue that how we teach reading in the U.S. must be grounded only in a narrow view of “scientific” (and that is a terrible argument, by the way), then any claims we make about the effectiveness of those practices must also be supported by scientific research.

Despite efforts to make Mississippi a shining example of how all states should address reading policy, we should be using Mississippi (and the 29 states scoring higher) to examine all the factors contributing to why students achieve at the levels they do on NAEP reading.

Unless of course we have real political courage and are willing to admit that NAEP and any form of standardized testing are the wrong way to make these decisions.

Here’s something to think about in that regard: As long as we use this sort of testing, we will always have some states above the average, several at the average, and some below the average—resulting in the same nonsensical hand wringing we see today that is no different than any decade over the last 100 years.

I recommend instead of all the scientific research needed to make any fair claim, we stop the testing, make teaching and learning conditions better, make the lives of children and their families in the U.S. better, and do the complicated daily work it requires to serve the needs of all students.

NOTE

[1] Hanford contradicts herself again and open the door to another question:

For years, everyone assumed Mississippi was at the bottom in reading because it was the poorest state in the nation. Mississippi is still the poorest state, but fourth graders there now read at the national average. While every other state’s fourth graders made no significant progress in reading on this year’s test, or lost ground, Mississippi’s fourth-grade reading scores are up by 10 points since 2013, when the state began the effort to train its teachers in the science of reading. Correlation isn’t causation* [emphasis added], but Mississippi has made a huge investment in helping teachers learn the science behind reading.

There is an 8-point jump in 4th-grade reading in MS from 2002 to 2009—well before the 2013 shift to the “science of reading”—thus how is that explained? [UPDATED]

* For the record, causation is a key component of “scientific,” which Hanford espouses for reading, yet she stoops to correlation (not scientific) to make her argument.