Edujournalism and Eduresearch Too Often Lack Merit

What do Marta W. Aldrich’s Teacher merit pay has merit when it comes to student scores, analysis shows and Matthew G. Springer’s Teacher Merit Pay and Student Test Scores: A Meta-Analysis have in common?

Irony, in that they both lack merit.

Let’s be brief but focus on the nonsense.

Well, as Aldrich reports about Springer’s research, a meta-analysis (this is research-speak that is supposed to strike fear into everyone since it is an analysis of much if not all of the existing research on a topic; thus, research about research), we now have discovered that merit pay in fact works! You see, it causes [insert throat clearing] “academic increase … roughly equivalent to adding three weeks of learning to the school year, based on studies conducted in U.S. schools, and four weeks based on studies across the globe.”

Wow! Three to four weeks of learning. That is … nonsense.

So here are the problems with our obsession with the hokum that is merit pay.

First, to make the process of giving teachers merit pay in order to create greater student learning, we have to have a metric for student learning that is quantifiable and thus manageable. Herein is the foundational problem since all of these studies use high-stakes test scores as proof of student learning.

This is a problem since standardized testing is at best reductive—asking very little of students and far more efficient than credible.

Next, very few people ever question this whole “weeks (or months) of learning” hokum—which is a cult-of-proficiency cousin of the reading grade level charade.

Researchers should explain to everyone that “weeks of learning” can often be a question or two difference on any test. In short, it is something that can be done statistically, but means almost nothing in reality. Three to four weeks out of a 36-week academic year.

Finally, and this is hugely important, merit pay linked to standardized test scores codified as proof of student learning necessarily reduces all teaching and learning to test prep and fails due to Campbell’s Law:

The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.

Notice here “corruption” and “corrupt.” Merit pay is guaranteed to corrupt the evidence and the entire teaching/learning process.

Similar to the obsession with choice and competition, the media and research fetish for merit pay is mostly about ideology—some believe outcomes are mostly about effort (thus, teachers are lazy) and are committed to merit pay regardless of the evidence or the unintended consequences.

As Mark Weber Tweeted about the claims of the study:

“Absurd” seems here to be an understatement, but, yes, this reporting and meta-analysis are themselves without merit and yet another example of the folly that is edujournalism and edureform in the U.S.

Why No Accountability for the Accountability Hawks?

It is a disturbingly easy list to make: police officers shooting/killing defenseless black males and females, Rush Limbaugh, OJ Simpson, Karl Rove, Hillary Clinton, Michelle Rhee, Bill Gates, ad infinitum.

Those shielded by privilege (wealth, race, celebrity, status) from the consequences of being held accountable for their actions.

And what is most disturbing is that among accountability hawks, those are the people least likely to be held accountable.

In the accountability era of education reform, the accountability hawks have been left unscathed even as they work to create school choice (public funds sent to private schools outside the accountability paradigm), more charter schools (relieved of accountability), and uncertified teachers (Teach For America).

Those accountability hawks, the politicians and the billionaire education hobbyists, are never held accountable as each policy and reform-of-the-day fails before s/he moves on to the next Great Reform.

Complicit in this failure to hold accountability hawks accountable have been spineless edujournalism and edupresses that have abdicated their role to press release journalism in the service of the edureformers.

And thus, as Audrey Amrein-Beardsley details:

Just this week, in Education Week — the field’s leading national newspaper covering K–12 education — a blogger by the name of Matthew Lynch published a piece explaining his “Five Indisputable [emphasis added] Reasons Why You Should Be Implementing Value-Added Assessment.”

I’m going to try to stay aboveboard with my critique of this piece, as best I can, as by the title alone you all can infer there are certainly pieces (mainly five) to be seriously criticized about the author’s indisputable take on value-added (and by default value-added models (VAMs)). I examine each of these assertions below, but I will say overall and before we begin, that pretty much everything that is included in this piece is hardly palatable, and tolerable considering that Education Week published it, and by publishing it they quasi-endorsed it, even if in an independent blog post that they likely at minimum reviewed, then made public.

Shame on Lynch, shame on EdWeek, but this is hardly anything out of the ordinary.

This is edujournalism as we have known it for decades now.

All hail the accountability hawks, and let neither evidence nor accountability deter their march!

Media Fail, 10,000 hours, and Grit: The Great Media-Disciplines Divide, pt. 2

In his The Danger of Delegating Education to Journalists: Why the APS Observer Needs Peer Review When Summarizing New Scientific Developments [1], K. Anders Ericsson makes several key points about how the mainstream media present disciplinary knowledge to the public, focusing on Malcolm Gladwell’s misleading but popular 10,000 hour rule.

Ericsson’s key point includes:

Although I accept that the process of writing an engaging popular article requires considerable simplification, I think it is essential that the article does not contain incorrect statements and misinformation. My primary goal with this review is to describe several claims in Jaffe’s article that were simply false or clearly misleading and then discuss how APS might successfully develop successful methods for providing research summaries for non-specialists that are informative and accurately presents the major views of APS members and Fellows. At the very least they should not contain factually incorrect statements and avoid reinforcing existing misconceptions in the popular media.

Through the Gladwell/10,000 hour rule example, Ericsson provides an important argument relevant to the current (and historical) public debate about school quality, teaching and learning, and education reform.

Much in the same way Gladwell has misrepresented research (which is typical within the media), and how that has been uncritically embraced by the media and public (as well as many if not most practitioners), a wide array of issues have received the same fate: learning styles, “grit,” collaborative learning, progressive education, charter schools, school choice, language gap, and so on.

Even when a claim or practice has a kernel of research at its source, popular oversimplification (often by journalists, but practitioners as well) and then commercialization/politicizing (creating programs and policies through publishers, “star” advocates, and legislation) significantly distort that research.

Education Has Failed Research, Historically

John Dewey represents an odd paradox in that he is possibly the most mentioned educator in the U.S. (either as the source of all that is wrong in education or idealistically cited as all that is right about how school could be), despite the reality that Dewey is mostly misunderstood and misrepresented; and thus his philosophy, progressivism, remains mostly absent in U.S. public schools.

Dewey can be blamed, in part, for this reality because he refused on principle to allow his experiments in education to be carefully catalogued because he believed no educational practice should be come a template for others.

Throughout much of the twentieth century, Lou LaBrant, a vigilant progressive educator, spent much of her career practicing and advocating for progressive literacy instruction, but LaBrant also confronted the many instances of how progressivism was misrepresented.

Broadly, and early, LaBrant recognized the public confusion about progressivism:

Two adults speak of “progressive education.” One means a school where responsibility, critical thinking, and honest expression are emphasized; the other thinks of license, lack of plans, irresponsibility. They argue fruitlessly about being “for” or “against” progressive education. (LaBrant, 1944, pp. 477-478)

But she also confronted how progressivism was mostly distorted in its application. LaBrant’s criticisms still reflect why education has failed research, and why research has not failed education.

Credible educational research-based philosophy, theory, and pedagogy are often corrupted by oversimplification.

In 1931, LaBrant published a scathing criticism of the popularity of the project method, an oversimplification of Dewey that resulted in students doing crafts in English class instead of reading or writing:

The cause for my wrath is not new or single. It is of slow growth and has many characteristics. It is known to many as a variation of the project method; to me, as the soap performance. With the project, neatly defined by theorizing educators as “a purposeful activity carried to a successful conclusion,” I know better than to be at war. With what passes for purposeful activity and is unfortunately carried to a conclusion because it will kill time, I have much to complain. To be, for a moment, coherent: I am disturbed by the practice, much more common than our publications would indicate, of using the carving of little toy boats and castles, the dressing of quaint dolls, the pasting of advertising pictures, and the manipulation of clay and soap as the teaching of English literature. (p. 245)

Credible educational research is often corrupted by commercialization/politicizing, reducing that research to misguided programs/legislation.

“[L]anguage behavior can not be reduced to formula,” LaBrant (1947) argued (p. 20)—emphasizing that literacy growth was complicated but flourished when it was child-centered and practical (for example, in the ways many privileged children experience in their homes because one or more of the parents are afforded the conditions within which to foster their children’s literacy).

By mid-twentieth century, LaBrant (1949) had identified the central failure of teaching reading: “Our language programs have been set up as costume parties and not anything more basic than that” (p. 16).

For at least 80-plus years since LaBrant fought this fight, the same patterns of media, political, public, and practitioners failing educational research have continued

Oversimplification, Commercialization/Politicizing: Recovering the Evidence

The list is incredibly long, too long to be exhaustive here, but consider the following: sloganism (“Work hard. Be nice.”), silver-bullet ideologies (“grit,” 10,000-hour rule), miracle schools (KIPP), evidence-based programs (Dibbles, 4-block, 6-traits), common sense claims and policy absent evidence (Common Core), and trendy legislation (3rd-grade retention policies as reading policy, merit pay) as well as politicized government reports (National Reading Panel).

Each of these can be traced to some kernel of research (sometimes robust bodies of research, and sometimes cherry-picked research), but all of these represent a current and historical fact: Education has failed research, but research has not failed education.

When educational research is reduced to scripts or programs/legislation, that knowledge base is invariably distorted, corrupted—as Ericsson details well above.

Journalists, politicians, and commercial education entities have all played a fundamental and crippling role in this reality; thus, as Ericsson argues, educators, scholars and researchers must not allow the fate of educational research to remain primarily in the wrong hands.

We have a public and professional obligation to confront these oversimplifications as well as the commercialization/politicizing of educational research. And we must do this through our public work that speaks to those failures and the public simultaneously.

As LaBrant and Ericsson reveal, unless we take that call seriously, we too are part of the reason education continues to fail research.


LaBrant, L. (1949). A genetic approach to language. Unpublished manuscript, Institute of General Semantics, Lakeville, CT.

LaBrant, L. (1947). Um-brel-la has syllables three. The Packet, 2(1), 20-25.

LaBrant, L. (1944, November). The words they know. The English Journal, 33(9), 475-480.

LaBrant, L. (1931, March). MasqueradingThe English Journal, 20(3), pp. 244-246.

For Further Reading

U.S. and Education Reform Need a Critical Free Press

My Open Letter to Journalists: A Critical Free Press, pt. 2

NPR Whitewashes “Grit” Narrative

Shiny Happy People: NPR, “Grit,” and “Myths that Deform” pt. 2

How I Learned to Distrust the Media (about Education)

My (Often Painful) Online Education

[1] See original and downloadable link to the paper here.

What We Know Now (and How It Doesn’t Matter)

Randy Olson’s Flock of Dodos (2006) explores the evolution and Intelligent Design (ID) debate that represents the newest attack on teaching evolution in U.S. public schools. The documentary is engaging, enlightening, and nearly too fair considering Olson admits upfront that he stands with scientists who support evolution as credible science and reject ID as something outside the realm of science.

Olson’s film, however, offers a powerful message that rises above the evolution debate. Particularly in the scenes depicting scientists discussing (during a poker game) why evolution remains a target of political and public interests, the documentary shows that evidence-based expertise often fails against clear and compelling messages (such as “teach the controversy”)—even when those clear and compelling messages are inaccurate.

In other words, ID advocacy has often won in the courts of political and public opinion despite having no credibility within the discipline it claims to inform—evolutionary biology.

With that sobering reality in mind, please identify what XYZ represents in the following statement about “What We Know Now”:

Is there a bottom line to all of this? If there is one, it would appear to be this: Despite media coverage, which has been exceedingly selective and misrepresentative, and despite the anecdotal meanderings of politicians, community members, educators, board members, parents, and students, XYZ have not been effective in achieving the outcomes they were assumed to aid….

This analysis is addressing school uniform policies, conducted by sociologist David L. Brunsma who examined evidence on school uniform effectiveness (did school uniform policies achieve stated goals of those policies) “from a variety of data gathered during eight years of rigorous research into this issue.”

This comprehensive analysis of research from Brunsma replicates the message in Flock of Dodos—political, public, and media messaging continues to trump evidence in the education reform debate. Making that reality more troubling is that a central element of No Child Left Behind was a call to usher in an era of scientifically based education research. As Sasha Zucker notes in a 2004 policy report for Pearson, “A significant aspect of the No Child Left Behind Act of 2001 (NCLB) is the use of the phrase ‘scientifically based research’ well over 100 times throughout the text of the law.”

Brunsma’s conclusion about school uniform policies, I regret to note, is not an outlier in education reform but a typical representation of education reform policy. Let’s consider what we know now about the major education reform agendas currently impacting out schools:

Well into the second decade of the twenty-first century, then, education reform continues a failed tradition of honoring messaging over evidence. Neither the claims made about educational failures, nor the solutions for education reform policy today are supported by large bodies of compelling research.

As the fate of NCLB continues to be debated, the evidence shows not only that NCLB has failed its stated goals, but also that politicians, the media, and the public have failed to embrace the one element of the legislation that held the most promise—scientifically based research—suggesting that dodos may in fact not be extinct.

* Santelices, M. V., & Wilson, M. (2010, Spring). Unfair treatment? The case of Freedle, the SAT, and the standardization approach to differential item functioning. Harvard Educational Review, 80(1), 106-133.; Spelke, E. S. (2005, December). Sex differences in intrinsic aptitude for mathematics and science? American Psychologist, 60(9), 950-958; See page 4 for 2012 SAT data:

Where Is Our “Sense of Decency”?

Before teaching The Crucible in my American literature courses during my two decades as a high school English teacher in rural Upstate South Carolina, I played the students R.E.M.s “Exhuming McCarthy,” which “makes an explicit parallel between the red-baiting of Joe McCarthy‘s time and the strengthening of the sense of American exceptionalism during the Reagan era, especially the Iran-Contra affair” (Wikipedia).

The song includes an audio from the McCarthy hearings, including this soundbite of Joseph Welch confronting Joe McCarthy:  “Let us not assassinate this lad further, Senator….You’ve done enough. Have you no sense of decency, sir, at long last? Have you left no sense of decency?”

Part of The Crucible unit asked students to examine how societies continue to repeat the basic flaws of abusing power and oppressing powerless groups of people. Despite the lessons of the Witch Trials and the Red Scare/McCarthy Era (with the Japanese Internment in between), Americans seem hell-bent on doubling down on policies and practices that are authoritarian, hypocritical, and simply mean—especially if those policies can be implemented by people with power onto the powerless.

Current education reform needs a McCarthy hearing, and we need to confront those driving those reforms with “You’ve done enough. Have you no sense of decency, sir, at long last? Have you left no sense of decency?”

For example, consider the following:

History is replete with evidence that the ends do not justify the means.

While there remains great political and public support for grade retention, for example, a huge body of evidence shows that retention negatively impacts students retained, taxpayers, and peers not retained—all for mixed results of short-term test scores.

The only justification for grade retention is giving the appearance of being tough (raising a key question about how tough any adult is for lording him/herself over a child).

Americans’ puritanical roots are some of our worst qualities, and especially where children and other marginalized groups are concerned, Americans need to regain our sense of decency.

We would be well advised to begin with how we reform our schools.

Welcome, Doctors, to the Brave New World of Corporate Reform!

What are the problems?

What is the evidence the problems exist?

What is the quality of that evidence?

Who are the stakeholders in the problems and solutions?

What are the perspectives of those stakeholders?

What are the perspectives of the stakeholders with experience and expertise in the problems and solutions?

Who stands to gain personally, professionally, and financially from the problems and solutions?

In the pursuit of any sort of reform, the right questions are essential—as is credible evidence—before solutions can be identified as valid, useful, and potentially effective. The great failure of democracy is that it appears those elected to power have neither the ability to ask the right questions nor the propensity to seek credible solutions. Those leaders are, however, eager to claim problems and support solutions that benefit them.

“In a bold experiment in performance pay, complaints from patients at New York City’s public hospitals and other measures of their care — like how long before they are discharged and how they fare afterward — will be reflected in doctors’ paychecks under a plan being negotiated by the physicians and their hospitals,” announces the lede to “New York City Ties Doctors’ Income to Quality of Care.”

“Bold” apparently means “making decisions based on ideology and not a shred of evidence.”

The article makes no case that doctor pay currently poses any sort of genuine problem—just that doctor pay is “traditional.”  Further, the article does acknowledge two important facts:

“Still, doctors are hesitant, saying they could be penalized for conditions they cannot control, including how clean the hospital floors are, the attentiveness of nurses and the availability of beds.

“And it is unclear whether performance incentives work in the medical world; studies of similar programs in other countries indicate that doctors learn to manipulate the system.”

For those of us struggling against a similar baseless current of teacher evaluation and pay reform, these details are all too familiar: (1) Concerns about accountability being linked to conditions over which a worker has no control (or autonomy), and (2) A complete disregard for the mountain of evidence that merit pay of all kinds proves to be ineffective and triggers for many negative unintended consequences:

“‘The consequences in a complex system like a hospital for giving an incentive for one little piece of behavior are virtually impossible to foresee,’ said Dr. David U. Himmelstein, professor of public health at the City University of New York and a visiting professor at Harvard Medical School, who has reviewed the literature on performance incentives. ‘There are ways of gaming it without even outright lying that distort the meaning of the measure.’ …

“Dr. Himmelstein also said doctors could try to avoid the sickest and poorest patients, who tend to have the worst outcomes and be the least satisfied. But physicians within the public hospital system have little ability to choose their patients, Mr. Aviles said. He added that he did not expect the doctors to act so cynically because, ‘in the main, physicians are here because they are attracted to that very mission of serving everybody equally.'”

The medical profession is poised to experience the complete failure of democracy that has been the fate of educators for at least three decades now. Democracy has spawned a legion of people with power but no expertise, and the result is a template for reform that ignores clearly identifying problems, fails to gather credible evidence, bypasses a wealth of experience and expertise, and imposes the mechanisms of inequity that brought those in power to that power.

As a result, buried late in this article on doctor pay reform is a cautionary tale:

“But Dr. Himmelstein said there were still hazards in the city’s plan. He said that when primary-care doctors in England were offered bonuses based on quality measures, they met virtually all of them in the first year, suggesting either that quality improved or — the more likely explanation, in his view — ‘they learned very quickly to teach to the test.'”

Educators, sound familiar?