Category: Educational Research

“A new study shows,” Education, and the Media

As I continue to document, the mainstream media believe everyone is an expert on education (except educators, of course).

In today’s two-experts-collide, know-nothing David Brooks comes out against GPA while latching onto Angela Duckworth’s “grit” sequel that is poised to maintain her racism/classism train to fame and fortune.

As John Oliver has now confronted (see below), the mainstream media love “a new study shows,” but almost always gets everything wrong.

Educational research continues to suffer this fate in the mainstream media, where, for example, the elites maintain our focus on students struggling just need more “grit,” and the self-serving counter to that: high achieving, successful people are so because of, primarily, their “grit”! (Ahem, and not their enormous privilege.)

Don’t hold your breath, but let’s imagine a world in which Brooks and Duckworth hold forth on this truth:

If you are black/brown and/or poor, your “grit” will still get you less than those gifted white privilege at birth.

Or how about:

Instead of the fatalism of saying that life is going to be hard for black/brown and/or poor people, and thus we need to make them extra “gritty” through abusive “no excuses” schools, why don’t we eradicate the social forces making their lives suck? [1]

Nope. We’ll just keep getting the sort of breezy hokum John Oliver brilliantly unmasks here:

[1] Also, imagine a world in which we discover lead in paint is dangerous for children so we conduct a study on children who survive exposure to lead pain in order to equip all students with that quality—instead of eradicating lead in paint. That’s the “grit” research in a nutshell.

Today in “Don’t Believe It”

More often than not, mainstream media and think tanks produce claims about education that are without credibility.

Sometimes the source is also lacking credibility, but many times, the source has good intentions.

Today in “Don’t Believe It,” let’s consider both types.

First, NCTQ—a think tank entirely lacking in credibilityissued a report claiming that teacher education is lousy, basing their claims on a fumbled review of textbooks assigned and course syllabi.

Don’t believe it because NCTQ bases the claims on one weak study about what every teacher should know, and then did a review of textbooks and syllabi that wouldn’t be allowed in undergraduate research courses.

See the full review here.

Next, despite genuinely good intentions, Kecio Greenho, regional executive director of Reading Partners Charleston, claims in an Op-Ed for The Post and Courier (Charleston, SC) that South Carolina’s Read to Succeed, which includes provision for third-grade retention based on high-stakes test scores, “is a strong piece of legislation that gives support to struggling readers by identifying them as early as possible.”

Don’t believe it because Read to Succeed is a copy-cat of similar policies across the U.S. that remain trapped in high-stakes testing and grade retention, although decades of research have shown retention to be very harmful to children.

See this analysis of Read to Succeed, the research base on grade retention, and the National Council of Teachers of English’s resolution on grade retention and high-stakes testing.

When you are confronted with claims about education, too often the source and the claim are without merit, but you have to be aware that those with good intentions can make false claims as well.

Edu-Journalists Know Only Two Stories (And They Are Both Wrong)

This post is mainly a public service because it has become stunningly clear that trying to engage journalists and the media covering education in order to prompt change is falling on willfully deaf ears.

My public service is to save you, dear reader, time. If you see or view a story in the media covering education, you can expect only two frames: (1) If the coverage is about public schools, the message is CRISIS!, but (2) if the coverage is about someone (anyone) without expertise or experience in education, the message will be breathless awe at their courage to finally be dragging that miracle to life that the horrible public school system has been unable to do lo these many years.

Sal Khan? Wow:

As we’ve reported, students anywhere now can get free SAT test prep both online and in person at some Boys & Girls Clubs of America. The move may help level the playing field by improving test prep for less-affluent students to get them ready for the newly revamped SAT, which remains a pillar of college admissions despite the growth in 2015 in “test optional” schools.

It’s part of what Khan Academy calls its core duty to help provide “a free, world-class education for anyone, anywhere.”

Teach For America? No way!:

In Teach For America lingo, that would be called a hook, a compelling way to think about a concept so that it’ll stick in students’ minds….

Those concerns are reflective of the new tack toward preparation taken by TFA’s Dallas-Fort Worth region, which is trying to move away from teacher-directed instruction in favor of techniques that focus more on students drawing conclusions on their own.

And rather than giving its new corps members a crash course on lesson planning, typically the first step in the TFA’s summer training, the Dallas region supplied its recruits with 700 ready-made lessons, focusing instead on giving candidates feedback on the finer points of carrying off a lesson well.

But what about that pesky skill reading that no one has really figured out—or possibly has never even tried to figure out?

Don’t fret! Buy psychologist Daniel Willingham’s new book!

Or (thank goodness another book to buy!):

About five years ago, the chief executive officer of the Uncommon Schools charter network offered up a lofty charge during a routine staff meeting: “Figure out” reading instruction.

OK, since I said at the outset I was planning to save you time, I’ll stop here, but rest assured, I could do this for hours because the list of (wow!) edu-saviors with almost no or often no experience or expertise in education who are heralded by the media as if the field doesn’t already exist but (again, thank goodness!) these innovative types are here to save the day is almost endless itself: Bill Gates, Michelle Rhee, KIPP founders Mike Feinberg and Dave Levin [1], and those already noted above.

Search those names and the pattern is the same: Fair-and-balanced edu-journalists open with breathless amazement at brave Edu-Genuis X and then later notes the person or the program has “received some criticism” (without nary a nod toward whether or not Edu-Genius X is credible, without nary a nod toward whether or not the criticism is credible—because, hey, that’s not a journalist’s job, right?).

We are left then with two related but wrong edu-journalist approaches to education reporting: public education is (and always has been) in crisis; therefore, the only people we can count on to save us is someone (anyone) outside of education.

However, the real story is much more complicated.

Much needs to be reformed in public schooling in the U.S.—much that has been historical failures.

But those problems are more about the structural wall of bureaucracy that has always existed and is even thicker today between the rich and powerful research and expertise in education as a field and practice, and the ability of teachers to implement their profession.

“A brief consideration,” LaBrant wrote in 1947, “will indicate reasons for the considerable gap between the research currently available and the utilization of that research in school programs and methods.”

LaBrant’s claim remains today, and has been documented specifically about how middle and high school students are bing taught writing. Applebee and Langer discovered:

Overall, in comparison to the 1979–80 study, students in our study were writing more in all subjects, but that writing tended to be short and often did not provide students with opportunities to use composing as a way to think through the issues, to show the depth or breadth of their knowledge, or to make new connections or raise new issues…. The responses make it clear that relatively little writing was required even in English….[W]riting on average mattered less than multiple-choice or short-answer questions in assessing performance in English…. Some teachers and administrators, in fact, were quite explicit about aligning their own testing with the high-stakes exams their students would face. (pp. 15-17)

Applebee and Langer emphasize that the negative consequences of high-stakes testing distinguish this study from their earlier work and that accountability has essentially stymied the influence of writing research, professional organizations, and teacher professionalism.

In other words, teachers today know more about teaching writing and have a more robust research base on what works in teaching writing but are unable to implement that knowledge base because of the accountability bureaucracy that has supplanted teacher professionalism.

Also damning is that this negative dynamic is even more pronounced for our most vulnerable students:

By far the greatest difference between the high poverty and lower poverty schools we studied stemmed from the importance that teachers placed and administrators placed on high-stakes tests that students faced. In the higher poverty schools, fully 83% of teachers across subject areas reported state exams were important in shaping curriculum and instruction, compared with 64% of their colleagues in lower poverty schools. (Applebee & Langer, 2013, p. 149)

And their is a truly ugly irony to this mess: Political leadership and edu-preneurs (remember, buy the book and program!) are chanting “college and career ready” while pushing mainstream education that guarantees students lack the rich and complex behaviors that would serve them well in higher education, their careers, and (god forbid) their lives. As noted above (and writing as someone who teaches first-year writing at the university level), students are being denied writing instruction they need and deserve because of accountability (except for privileged students at private schools that are not shackled by the mandates of politicians who send their children to those private schools).

So, I know this isn’t the stuff of breathless awe and fair-and-balanced journalism, but the reality is that we are failing the profession of education, the research of the field of education, public education as a great democratic experiment, our children, and our country. But the solution to that huge set of problems is to tear down the bureaucratic wall of accountability and rebuild public schools with the leadership of educators.

[1] See this excellent examination of KIPP and the clueless media coverage of education “hot shot” experiments by know-nothing edu-preneurs:

For Russo to ignore the uneven outcomes of KIPP schools as a possible reason for their recent lack of attention is odd, but not surprising as reform cheerleaders are often blind to any objective evidence that does not support their narrative.


Beware the Technocrats: More on the Reading Wars

Since it is Academy Awards season, let me start with film as context.

Whiplash has received a great deal of Oscars buzz with five significant nominations. But that film praise is interesting to frame against a review that considers how the film’s topic, jazz, is portrayed:

The mediocre jazz in Damien Chazelle’s new film, “Whiplash,” the story (set in the present day) of a young drummer (Miles Teller) under the brutal tutelage of a conservatory professor (J. K. Simmons), isn’t itself a problem. The problem is with the underlying idea. The movie’s very idea of jazz is a grotesque and ludicrous caricature.

“Mediocre,” “grotesque,” and “ludicrous caricature” are certainly not the stuff of Oscars, one would think, but this contrast of responses to the film represents well my problem with the Mt. Rushmore of technocrats who are cited with missionary zeal whenever you spend much time in the reading wars (see the comments here): Daniel Willingham, John Hattie, E.D. Hirsch, and Grant Wiggins.

With some qualifications for Wiggins (who taught high school and coached for 14 years, but has focused primarily on assessment since then), these often cited men are primarily quantitative researchers who are not within the field of literacy (Hirsch’s background is literature, not literacy) and have created cottage industries out of their names/work: Willingham as a psychologist, Hattie as a researcher/consultant, Hirsch as a core knowledge advocate, and Wiggins as proponent of understanding by design and consultant.

As I have noted before, most of my concern here is how certain advocates for phonics and direct instruction in literacy use the Mt. Rushmore of technocrats to close the door on the reading wars—not with any of these men or their work specifically (except Hattie [1]).

Therefore, I must offer, Beware the technocrats, because of the following:

  • Beware the seductive allure of statistics, numbers, and “scientific” research. As I have detailed more often than I would have liked, a perfect example of this concern is the prevalence of the Hart and Risley research on the “word gap,” which persists despite many concerns being raised about not only the research itself, but also the deficit ideology that drives the conclusions. Of course, high-quality experimental and quasi-experimental research matters, but many aspects of teaching and learning require and lend themselves to other research paradigms—notably qualitative action research conducted by classroom teachers with the real populations they teach.
  • Beware the momentum of cottage industry gurus. Hattie, Hirsch, and Wiggins have created entire careers for themselves—books, workshops, consultations. I remain deeply skeptical of such ventures (see also Nancie Atwell and a whole host of gurus on the “softer” side of research and within literacy as well). Even the best people with the best intentions can find themselves victims of “‘filthy lucre,'” but just as the higher the quality of scientific research, the more likely it means less to real-world teaching, the urge to reduce an evidence base or best practice to a program means that evidence and practice are mostly ruined.

In the reading wars, then, I witness time and again that the advocates for intensive phonics, phonics programs, and direct instruction grounded in prescribed content are either not within the field of literacy [2] or themselves invested in programs that benefit from those positions (the Common Core debate represents the same issue since most advocates stand to benefit from Common Core being implemented, some politically and some financially).

Which brings me back to Whiplash. If you know little or nothing about jazz, the film likely appears more wonderful than if you do.

I have a thirty-plus year career in literacy, including teaching literacy (mostly writing) and scholarship addressing literacy. That context for me renders the Mt. Rushmore of technocrats not insignificant, but certainly less credible than a century of research and practice by literacy practitioners and researchers that informs my practice.

There is a tyranny to certainty among those who wield the work of Willingham, Hattie, Hirsch, and Wiggins in ways that end the conversation, that shut the door on a broader basis of evidence to inform, not mandate, practice. There is a greater tyranny of commerce lurking here also, using “scientific” as a mask for commercialization.

Both serve to further de-professionalize teachers, and both often result in classroom practices that may raise test scores but create nonreaders.

And thus, when Hattie is cited (yet again) during the reading wars, for example:

I posted a question in Pamela and Alison’s article last week, but didn’t get a response from anyone. My question is: if the “effect size” of synthetic phonics (according to Hattie’s research) is 0.54, and that of whole language learning is 0.06, does that mean:

  1. That whole language actually does have an effect; and
  2. Should we therefore use the two approaches in the ratio of 1:9 (i.e. the difference in their effect sizes)? (scroll to the comment from John Perry, who, I must add, is being reasonable here)

I share the exasperation Richard Brody expresses at the end of his review of a jazz film that uses Buddy Rich as the icon for the film’s protagonist: John Hattie. John Visible Learning Hattie.

In terms of evidence, that has the opposite effect intended.

See Also

Education ‘experts’ may lack expertise, study finds

Taming the Wild West of Educational Research, Simon P. Walker

[1] Those who rush to use Hattie are proof of solid research fail to note that his work has been challenged for quality, even within the quantitative paradigm; see:

[2] Since many people continue to refer to the National Reading Panel report, please examine Joanne Yatvin’s minority view, starting about page 444, including:

In the end, the work of the NRP is not of poor quality; it is just unbalanced and, to some extent, irrelevant. But because of these deficiencies, bad things will happen. Summaries of, and sound bites about, the Panel’s findings will be used to make policy decisions at the national, state, and local levels. Topics that were never investigated will be misconstrued as failed practices. Unanswered questions will be assumed to have been answered negatively. Unfortunately, most policymakers and ordinary citizens will not read the full reviews. They will not see the Panel’s explanations about why so few topics were investigated or its judgments that the results of research on some of the topics are inconclusive. They will not hear the Panel’s calls for more and more fine-tuned research. Ironically, the report that Congress intended to be a boon to the teaching of reading will turn out to be a further detriment.

As an educator with more than 40 years of experience and as the only member of the NRP who has lived a career in elementary schools [emphasis added], I call upon Congress to recognize that the Panel’s majority report does not respond to its charge nor meet the needs of America’s schools.

More on Evidence-Based Practice: The Tyranny of Technocrats

Depending on your historical and literary preferences, spend a bit of time with Franz Kafka or Dilbert and you should understand the great failure of the standards movement in both how we teach and how we certify teachers—bureaucracy.

Bureaucracy tends to be inadequate because bureaucrats themselves are often lacking professional or disciplinary credibility or experience, depending, however, on the status of their authority to impose mandates. For education, Arne Duncan serves well as the face of the bureaucrat, an appointee who has only the bully pulpit of his appointment to hold forth on policy.

However, as corrosive to education—and ultimately to evidence-based practice—is the technocrat.

Technocrats, unlike bureaucrats, are themselves credible, although narrowly so. For technocrats, “evidence” is only that which can be measured, and data serve to draw generalizations from randomized samples.

In short, technocrats have no interest in the real world, but in the powerful narcotic of the bell-shaped curve.

As a result, a technocrat’s view often fails human decency (think Charles Murray and Richard Herrnstein) and certainly erases the very human reality of individual outliers.

The face of the technocrat—in fact, the technocrat’s technocrat—is Daniel Willingham, whose work is often invoked as if handed down by the hand on God, chiseled on tablets. [1] [Note: If you sense snark here, I am not suggesting Willingham’s work is flawed or unimportant (I would say important but narrow), but am being snarky about how others wield the technocratic hammer in his name.]

And it is here I want to return to a few points I have made recently:

  • Even the gold standard of experimental research fails the teacher in her day-to-day work because her classroom is not a random sampling of students, because her work is mostly with outliers.
  • And in the teaching moment, what counts as evidence becomes that teacher’s experience couched in that teacher’s content and teaching knowledge as all of that happens against the on-going evidence of the act of teaching.

Stewart Riddle, offering yet another effort in the reading war, is essentially speaking for evidence-based practice while raising a red flag against the tyranny of the technocrat, embodied by the systematic phonics crowd (those who wave the Willingham flag, for example).

On Twitter, in response to my piece on evidence-based policy and practice, Nick Kilstein raised a great point:

My ultimate response (prompting this blog):

My thoughts here, building on the bullet points above, are that having our practice informed by a wide range of evidence (including important evidence from technocrats, but also from other types of evidence, especially qualitative research [2] that can account for outliers, nuance, and the unexpected) is much different than having our practice mandated by evidence (think intensive, systematic phonics for all children regardless of needs or fluency because that is the program the school has adopted).

For day-to-day teaching, the tensions of the disciplines remain important: what we can measure against what measuring cannot address.

When Willingham proclaims that a certain type of research does not support the existence of learning styles, for example, teachers should use that to be very skeptical of the huge amount of oversimplified and misguided “teacher guides” and programs that espouse learning styles templates, practices, and models. [3]

But day-to-day teaching certainly reveals that each of our students is different, demanding from us some recognition of those differences in both what and how we teach them.

It is in the face of a single child that technocrats fail us—as Simon P. Walker notes:

Some educational researchers retreat to empiricist methods. Quantitative studies are commissioned on huge sample sizes. Claims are made, but how valid are those claims to the real-life of the classroom? For example, what if one study examines 5,000 students to see if they turn right rather than left after being shown more red left signs. Yes, we now with confidence know students turn left when shown red signs. But so what?  What can we extrapolate from that?  How much weight can that finding bear when predicting human behaviour in complex real world situations where students make hundreds of decisions to turn left and right moment by moment? The finding is valid but is it useful?

If that child needs direct phonics or grammar instruction, then I must offer them. If that child is beyond direct phonics and grammar instruction or if that direct instruction inhibits her/his learning to read and write, then I must know other strategies (again, this is essentially what whole language supports).

The tyranny of the bureaucrats is easy to refute, but the tyranny of the technocrat is much more complicated since that evidence is important, it does matter—but again, evidence of all sorts must inform the daily work of teaching, not mandate it.

Professional and scholarly teachers are obligated to resist the mandates by being fully informed; neither compliance nor ignorance serves us well as a profession.

[1] For more on worshipping technocrats, explore this, notably the cult of John Hattie and that those who cite his work never acknowledge the serious concerns raised about that work (see the bottom of the post).

[2] Full disclosure, I wrote a biography for my EdD dissertation (published here), and also have written a critical consideration of quantitative data.

[3] See, for example, how evidence (Hart and Risley) functions to limit and distort practice in the context of the “word gap.” The incessant drumbeat of the “Hart and Risley” refrain is the poster child of the tyranny of technocrats.

The “Word Gap”: A Reader

The AMC series, based on the iconic graphic series The Walking Dead, has finally included Rick admitting, “We are the walking dead” (Season 5, Episode 10).

Viewers witness the inevitable lethargy of living always under the threat of zombies, a reduced existence in which even stabbing a zombie in the brain is executed with a resignation that borders on macabre camp:

Maggie is confronted with death—and the walking dead—throughout the episode. We open to her weeping, as a walker shambles up behind her. She casually stands, and knifes the zombie in the skull. Later she finds a walker tied up and gagged in the trunk of a car. She must have been tied that way when she was alive, and starved to death before turning. It’s a horrible thought. Glenn kills that one for her. At the barn, she finds a third walker, this one apparently camped out there before she died.

I have explored the power of zombie narratives to examine the weight of living in poverty and the paralysis of anxiety, but here I want to add that one study and the term “word gap” are also yet more proof of the zombie apocalypse.

The “Word Gap” That Will Not Die

Like Maggie, I am nearly numb, having spent over thirty years in education mostly having to refute constantly misguided policy and misinformed media.

The most resilient and disturbing among those experiences is the term “word gap” and the single study that will not die—this time from Elizabeth Gilbert:

The term “word gap” was first coined in the 1995 Hart/Risley study that found low-income children are exposed to 30 million fewer words than their higher-income peers before age 3. This study and others have linked poor early literacy skills to lifelong academic, social and income disparities. Word gap initiatives primarily target low-income parents to help them understand the effect they have on their children’s cognitive development. Unfortunately, this misses another important part of the problem.

The deficit view perpetuated by Hart and Risley (not the credibility of the study or its claims) is as contagious as the zombie virus infecting everyone in The Walking Dead universe.

And while it would be easier just to lie down, give in, I remain steadfast against the “word gap” throng; thus, please take the time to consider the following reader:

In this article, we argue that strong claims about language deficiencies in poor children and their families based on the Hart and Risley study are unwarranted. Further, we argue that the uncritical acceptance of Hart and Risley’s findings is emblematic of a trend among some educators, educational policy makers, and educational researchers to readily embrace a deficit stance that pathologizes the language and culture of poor students and their families (Dudley-Marling, 2007; Foley, 1997). We hope that this critique will help teachers resist “research-based” policies that aim to fi x the language and culture of poor and minority students with whom they work.

Currently, the most cited study detailing the deficiencies of low-income children is that of Hart and Risley (1995). This study has been criticized by language scholars, including in a past issue of Language Arts (Dudley-Marling & Lucas, 2009), although the criticism seems to have had little impact….

Of course, there are many problems with the Hart and Risley study (Dudley-Marlin & Lucas, 2009; Michaels, 2013; Miller & Sperry, 2012). Among the criticisms are the reduction of language to vocabulary knowledge, the extrapolation from six families in one state to all poor children, whatever their familial, cultural, linguistic, and community circumstances, and the lack of a correlation between children’s vocabulary as mea- sured at age 3 and their school performance in third grade (noted by Hart & Risley [1995] themselves, on p. 161). Nonetheless, it is widely assumed that this study illuminates the “deficiencies” of poor children and that these account for school difficulties. This assumption is simply a continuation of societal ideologies linking income, race, and lan- guage deficiency that have been part of this coun- try since colonial times (e.g., the views of slaves’ capacities).

And thus the worries Labov (1972) articulated many years ago have come to pass: educators (and researchers and policy makers, I add) view chil- dren through the taken-for-granted lens of language deficiency and, thus, attempt to “repair the child, rather than the school.” One consequence of this is the indiscriminate erasure of children’s language strengths. (pp. 200-201)

Hart and Risley, Meaningful Differences in the Everyday Experience of Young American Children—Google it and in .15 seconds you get over 100,000 hits. Hart and Risley’s book Meaningful Differences (1995) is the most-cited piece of academic work that attempts to explain what goes wrong with poor kids, with grand extrapolations and claims (which you’ll see that I argue are totally unsubstantiated) about how poor children will fare in school and later life—based on their early home experiences with language. The book purports to demonstrate (with what I will call pseudo-scientific elegance) that poor children (in their study six families, all black, all on welfare) are doomed before they enter school because 1) their parents don’t talk to them as much as upper middle class parents (13 upper SES, “professional” families—where the parents were predominantly professors, all white except one); and 2) poor children don’t experience as many “quality” features in the talk with their parents.

Because of the severe methodological flaws in the study, these conclusions are unwarranted. To truly investigate the relationship between quantity of interaction and vocabulary growth, we need at least two completely independent measures — (1) a measure of quantity of interaction such as that used by Hart and Risley, and (2) a measure of vocabulary size such as a vocabulary size test.

The differences are striking….

Neither the approach of concerted cultivation or the accomplishment of natural growth is without flaws. Both have strengths and weaknesses [emphasis added]. Middle-class children, for example, are often exhausted, have vicious fights with siblings, and do not have as much contact with their extended families as working-class and poor children. But when children enter institutions such as schools and health care settings, the strategy of middle-class child rearing of concerted cultivation is far more in compliance with the current standards of professionals than is the approach of the accomplishment of natural growth. There are signs that middle-class children gain advantages, including potentially in the world of work, from the experience of concerted cultivation. Working-class and poor children do not gain this benefit.

Unlike fatalism in The Walking Dead, World War Z is a zombie narrative offered after the apocalypse, a tale told with a dark optimism since humans have survived the rise of the living dead.

There are lessons in this version as well, particularly about the possibility of an antidote—about choosing to see the world differently in order to make a different world.

Let us put the term “word gap” to rest, permanently, along with the nearly compulsive urge to cite Hart and Risley.

Related and Recommended

Why we need to smash up the concept of the achievement gap in tiny little pieces, Andre Perry

What Are Evidence-Based Practices and Policies in Education?

The best laid schemes o’ Mice an’ Men
Gang aft agley,
An’ lea’e us nought but grief an’ pain,
For promis’d joy!

“To a Mouse,” Robert Burns

If John Steinbeck’s Of Mice and Men offers a fictional dramatization of Scottish poet Robert Burn’s dire warning in poetry, above, then the mangled federal education policy popularly known as No Child Left Behind (NCLB) may be a top candidate for real-world proof of Burns continuing to be right 230 years later.

Heralded as bi-partisan and bold, NCLB has been unable to manage its central directive: scientifically based policy and practice in education.

The wider accountability paradigm of education reform driven by standards and high-stakes tests and NCLB have proven to be failures, but with the reauthorization of ESEA (NCLB) now on the table, those failures provide ample evidence for how to move forward with education reform and policy.

A policy memo from NEPC now stresses the importance of making evidence-based decisions during reauthorization:

Kevin Welner and William Mathis discuss the broad research consensus that standardized tests are ineffective and even counterproductive when used to drive educational reform. Yet the debates in Washington over the reauthorization of the Elementary and Secondary Education Act largely ignore the harm and misdirection of these test-focused reforms. As a result, the proposals now on the table simply gild a demonstrably ineffective strategy, while crowding out policies with proven effectiveness. Deep-rooted trends of ever-increasing social and educational needs, as well as fewer or stagnant resources, will inevitably lead to larger opportunity gaps and achievement gaps. Testing will document this, but it will do nothing to change it. Instead, the gaps will only close with sustained investment and improvement based on proven strategies that directly increase children’s opportunities to learn.

First, to heal the damage done, we must admit those clear failures. Next, we must change course away from accountability, standards, and high-stakes tests. Finally, we must clearly identify the reasons for educational struggles and failures in order to embrace the best policies and practices to prompt genuine and effective reform.

That reauthorization process, the new reform agenda, and then the daily practice of running schools and teaching students must re-embrace evidence-based policies and practices, but not without clarifying exactly what that means.

Two powerful lessons about creating evidence-based policy and practice can be drawn from the NCLB era: (1) simply codifying and mandating policy and practice must be evidence-based do not make that occur, and (2) the National Reading Panel’s procedures and outcomes highlight that “evidenced-based” when politicized is just as subject to human whims and corruption as anything else.

The inevitable train wreck of the Common Core (doing the same thing over and over while expecting different results) can be avoided now if we learn from NCLB; otherwise, we continue a long history identified by Lou LaBrant, former president of NCTE, in 1947: “A brief consideration will indicate reasons for the considerable gap between the research currently available and the utilization of that research in school programs and methods.”

What Are Evidence-Based Practices and Policies in Education?

What counts as evidence has been the basis of stringent debate within the disciplines throughout the history of organized disciplines. In fact, that tension is how disciplines continue to seek knowledge, define themselves, and thus crawl closer and closer to the ultimate goal of Truth.

The two lessons noted above, however, show that once partisan politics are the process for mandating what counts as evidence, the credibility of that evidence is essentially destroyed.

Along with the National Reading Panel (NRP), A Nation at Risk demonstrated as a process how partisan political goals corrupt evidenced-based conclusions. The panel creating A Nation at Risk and the NRP had conclusions dictated first and then selected the evidence to reach those conclusions. Codifying what counts as evidence through a political process corrupts knowledge (and thus policy and practice) while forcing disciplinary debate about what counts as evidence to remain on the sidelines (and thus ineffective).

Briefly (and at the risk of oversimplification), the debates within fields over what counts as evidence tend to be between those embracing quantification and generalizability and those who who embrace qualitative data in order to raise and pursue essential questions.

In most disciplines, experimental and quasi-experimental (and thus quantitative) research has historically (and still currently) dominated those debates. The rise of qualitative research, however, has both expanded the disciplines and forced quantitative analysis of the world to address that reducing phenomena to numbers is both limited and limiting.

So the first problem with codifying what counts as evidence through the political process is that mandates (through legislation and funding) narrow and render static a process that must remain vibrant and organic in order to be effective.

For education as a discipline, then, the added political layer (again, think A Nation at Risk, NRP, or the USDOE) is the problem that mutes the already existing and rich professional organizations that have the needed knowledge base and guidelines for both how to teach students the disciplines and what to teach students within the disciplines.

Since it is difficult to clearly separate policy and practice in education, let me end with some concrete examples in order to give the bones of the question what counts as evidence flesh: corporal punishment, grade retention, and teaching reading.

Corporal punishment remains relatively common in parenting in the U.S., but it also lingers in schools in about a third of states. The American Psychological Association (APA), however, has a clear stand against corporal punishment based on over 60 years of evidence.

This represents one level of the needed relationship between government mandates and the disciplines as that informs education: When a professional field has a clear understanding of an issue, policy should reflect that stance. In other words, corporal punishment should be banned in public schools—not because some partisan political committee has studied the issue, but because the APA has done so and over a long period of time, while taking into consideration a wide and varied body of evidence.

A second level of the needed relationship between government mandates and the disciplines as that informs education is grade retention. Like corporal punishment research, grade retention research is robust and suggests that grade retention is mostly harmful. But at this second level, how the political process fails is strongly highlighted. Currently, states are mandating the opposite of what the research reveals (embracing high-stakes testing as a trigger for grade retention).

At this second level, based on the disciplinary evidence over a long period of time and built on a wide and varied body of evidence, grade retention must not be mandated, but not banned either. In cases such as grade retention, policy should caution against the practice, but allow that, as the research shows, some children may benefit from the practice, but professionals closest to those case are best for making that decision.

A third level involves daily classroom teaching of the disciplines; I’ll focus here on reading by highlighting whole language and balanced literacy.

Reading policy and practice are possibly the most debated areas of partisan political agendas. Everyone believes children need to learn to read—and almost everyone thinks s/he knows how that should happen.

Whole language has been codified (see California) and banned (see NRP), and balanced literacy has also been codified (see New York).

Complicating this third level is that politicians, the media, parents and the public, and even practitioners often misunderstand practices such as whole language and balanced literacy. In the cases of California and New York, whole language and balanced literacy did not fail; the political and implementation processes failed them.

Whole language commercialized (textbooks, programs), tested in high-stakes contexts, and prescribed (standards, curriculum and pacing guides) ultimately is not whole language (or best practice).

The lesson at this third level is that political policy always corrupts classroom practice because classroom practice is never as simplistic as policy. Even the gold standard of experimental research fails the teacher in her day-to-day work because her classroom is not a random sampling of students, because her work is mostly with outliers.

At this third level, the political mandate must address only that teachers are provided the opportunity to decide for each student and during each teaching moment what counts as evidence. And in the teaching moment, what counts as evidence becomes that teacher’s experience couched in that teacher’s content and teaching knowledge as all of that happens against the on-going evidence of the act of teaching—which is exactly what balanced literacy is:

Spiegel 3

The third level, then, is the arena of professionalism. A reading teacher must come to class equipped with the knowledge of her field of literacy (one powerfully informed by whole language and balanced literacy, for example, but also a field in constant tension due to the debates about what literacy is and how to teach it), and then, capable of providing different students different approaches in order to provide the learning needed in the pursuit of each child’s literacy.

The evidenced-based real world of teaching reading is messy, chaotic, and cumulative over a period of time that cannot be predicted for any individual child.

Simply put, no politician or political committee in DC or any state house has any business or ability to mandate the daily teaching of children. The political job is to ensure professionals have the opportunity to be professionals—and to create a process of transparency so that tax payers know that professionals are provided for children and doing their work as experts in the disciplines.

Federal and state policies are misguided when they are prescriptions that supersede the complex and on-going knowledge of the disciplines.

As the NEPC memo above notes, what counts as evidence now in education reform is the political disaster that is NCLB. The grand lesson of that evidence is that political mandates in education—detailed in three levels above—are creating problems, not solving them—and at tremendous expense.

In education reform, we need political humility and a new era of recognizing both the existing power of the disciplines and the the professional possibilities of teaching.

We do not need a commission of a wide range of stakeholders to fumble badly again what, for example, the field of literacy has been carefully examining for decades, what the field of literacy could easily and quickly provide every teacher in the U.S.—the wide range of strategies for ensuring each child becomes an eager and empowered reader.

If our goal is truly evidence-based practices and policies in education, the evidence suggests we must first have partisan politics step aside.