PISA Brainwashing: Measure, Rank, Repeat

When Mary Catherine Bradshaw, a teacher since 1984 in Nashville, TN, announced her retirement from public schools, Bradshaw pointed her finger at one major reason, standardized testing:

[S]he says standardized testing is the reason….

Testing, she said, has taken away from instructional time and taken the joy out of learning.

Much has changed, she said, since she took her first job as a teacher at Hillsboro in 1984 when she said she was attracted to its diversity and commitment to academic reputation.

“There was more of a focus on the whole student, the joy of learning, building a community and finding one’s own passion in the midst of the K-12 experience,” she said.

“Now, with the focus on testing, data collection and closing a too narrowly defined gap among learners, I have found myself ready to retire from public education.”

Bradshaw’s concern about the loss of joy due to the central place of testing in education is echoed in a recent statement about PISA rankings [1], as Peter Wilby details in Academics warn international school league tables are killing ‘joy of learning’:

Now nearly 100 leading educational figures from around the world have issued an unprecedented challenge to Pisa – and what they call “the negative consequences” of its rankings – in a letter to its director, Andreas Schleicher….

“Education policy across the world is being driven by the single aim of pushing up national performance levels on Pisa,” says one signatory, Stephen Ball, professor at London university’s Institute of Education. “It’s having a tremendously distorting effect, right down to the level of classroom teaching.” Another signatory, Sally Tomlinson, research fellow at Oxford university’s education department, says that, though the Pisa league tables appear to be scientifically based, “you really can’t compare a country the size of Liechtenstein with one the size of China and nor can you compare education systems that developed over the years in different political, social and cultural contexts”.

The signatories are particularly concerned about the UK, the US and other countries imitating schools in Asian countries that come high in the Pisa rankings. They are suspicious of Shanghai’s success. “Shanghai’s approach is an incredibly strategic one,” says Ball. “Their students practise the tests. It’s difficult to see what their maths teachers can say to ours except ‘teach to the test’.”

While international rankings based on test scores have influenced public perception of U.S. public education for at least 60+ years (see Hyman Rickover’s books lamenting U.S. rankings, for example), state rankings based on NAEP and SAT/ACT scores have also been central to perception as well as policy, especially since the early 1980s.

While the open letter to Schleicher is a powerful and important challenge to the misleading influence of PISA, the essential problem is high-stakes testing coupled with ranking as well as a persistent misinterpretation of test data (see this excellent examination of how test scores are misunderstood and misused).

As I have addressed often about the SAT (see HERE and HERE), even when a comparison of states appears fair and accurate—South Carolina with Mississippi, for example, since the states share a similar high-poverty demographics of students—the reality is far more complex: MS has a higher SAT average score than SC because the test-taking populations of students are significantly different despite the overall student populations being similar:

Two Southern states, Mississippi and South Carolina, share both a long history of high poverty rates (Mississippi at over 30% and SC at over 25%) and reputations for poor schools systems. Yet, when we compare the SAT scores (pdf) from Mississippi in 2010 (CR 566, M 548, W 552 for a 1,666 total) to SAT scores in SC (CR 484, 495, 468 for a 1,447 total), we may be compelled to charge that Mississippi has overcome a higher poverty rate than South Carolina to achieve, on average, a score 219 points higher.

This conclusion, based on a “few data points”, is factually accurate, but ultimately misleading once we add just one more data point: the percentage of students taking the exam. Just 3% of Mississippi seniors took the exam, compared to 66% in South Carolina. A fact of statistics tells us that SC’s larger percentage taking the exam is much closer to the normal distribution of the all seniors in that state, thus the average must be lower than a uniquely elite population, such as in Mississippi. Here, the statistics determined by the populations taking the exam trump the raw data of test averages, even when placed in the context of poverty. (The truth about failure in US schools)

Even if the open letter about PISA prompts reform by the OECD, we have evidence that the problem will persist. For example, The College Board struggles with both the statistical complexity of SAT data (see here about the recentering) and the misleading use of SAT data to rank states:

Educators, the media and others should:

8.1 Not rank or rate teachers, educational institutions, districts or states solely on aggregate scores derived from tests
that are intended primarily as a measure of individual students. Do not use aggregate scores as the single measure to
rank or rate teachers, educational institutions, districts, or states.

And yet, each year when SAT data are released, the media, political leaders, and public school critics rank states and pronounce schools a failure.

The open letter about PISA implores, “Slow down the testing juggernaut,” adding:

OECD’s narrow focus on standardised testing risks turning learning into drudgery and killing the joy of learning. As Pisa has led many governments into an international competition for higher test scores, OECD has assumed the power to shape education policy around the world, with no debate about the necessity or limitations of OECD’s goals. We are deeply concerned that measuring a great diversity of educational traditions and cultures using a single, narrow, biased yardstick could, in the end, do irreparable harm to our schools and our students.

Once we apply the brakes, we must then take a close look at the fundamental policy errors—high-stakes standardized testing, labeling, sorting, and ranking—and then abandon those practices for alternatives that address inequity both outside and inside schools and that honor the essential dignity and humanity of students and their teachers.

For Further Reading

Among the Many Things Wrong With International Achievement Comparisons, Gene V. Glass

More Things Wrong with International Assessments Like PISA, Gene V. Glass

[1] As full disclosure, I am a signatory on the letter.


  1. Pingback: PISA Brainwashing: Measure, Rank, Repeat – @ THE CHALK FACE
  2. Pingback: PISA Brainwashing: Measure, Rank, Repeat | the becoming radical | MI ED News Clips
  3. Pingback: Once Again, NAEP? Nope: “states and schools have lied about the rigor of their courses” | the becoming radical
  4. Pingback: Schneider’s Ten Reform Claims: A Reader | the becoming radical

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s