Will Free-Pass Mainstream Media Clean Up Their Chetty Mess?

The New York Times couldn’t hold back: Big Study Links Good Teachers to Lasting Gain.

And for two years now, the so-called Chetty study that claims teacher quality directly causes higher lifetime earnings for their students has been re-released by Chetty and colleagues as well as rehashed by the free-pass mainstream media with a fervor that seems at least over-the-top.

When the Chetty numbers (has anyone noticed that the lifetime earning numbers appear to change?) are put into perspective, all the air should deflate from that misleading balloon: $50,000 gained over a lifetime (40 years) is only about 1.5-2 tanks of gas a month. But inflating that balloon to $1.4 million for a class of 28! Now you have something … (Hint: An overinflated balloon, possibly filled with poop.)

Initial scholarly responses to the Chetty balloon were cautious and critical (see notably Bruce Baker and Matthew Di Carlo here), but the free-pass mainstream press kept on keeping on.

Thus, since I have made a case for our needing a critical free press (in other words, “free press” and not “free pass” found among press-release journalists), we are at a key moment with the release of a thorough review of the Chetty study, a review that discredits the claims, Review of Measuring the Impacts of Teachers by Moshe Adler:

Can the quality of teachers be measured the way that a person’s weight or height is measured? Some economists have tried, but the “value-added” they have attempted to measure has proven elusive. The results have not been consistent over tests or over time. Nevertheless, a two-part report by Raj Chetty and his colleagues claims that higher value-added scores for teachers lead to greater economic success for their students later in life. This review of the methods of Chetty et al. focuses on their most important result: that teacher value-added affects income in adulthood. Five key problems with the research emerge. First, their own results show that the calculation of teacher value-added is unreliable. Second, their own research also generated a result that contradicts their main claim—but the report pushed that inconvenient result aside. Third, the trumpeted result is based on an erroneous calculation. Fourth, the report incorrectly assumes that the (miscalculated) result holds across students’ lifetimes despite the authors’ own research indicating otherwise. Fifth, the report cites studies as support for the authors’ methodology, even though they don’t provide that support. [emphasis added] Despite widespread references to this study in policy circles, the shortcomings and shaky extrapolations make this report misleading and unreliable for determining educational policy.

So my question now is: Will the free-pass mainstream media clean up their Chetty mess?

I suspect we will not have a NYT scorching headline, we will not even have a NYT article, we probably will not see interviews on NPR with Adler, and I am skeptical about Education Week‘s coverage (beyond some bloggers).

The fair and balanced mainstream media, alas, (those journalists who cannot judge the credibility of the research they cover) will not fall all over themselves to cover and then repeat for two years to come the popping of the Chetty balloon because that would mean admitting their own incompetence, which is the sweet allure of fairness that leaves us all misinformed.

Debating the Gates Moratorium, Or Life among the Roadbuilders

As a lifelong fan of science fiction (SF), I want to start by invoking a SF convention that never gets old: time travel (but I’ll spare you the whirlwind prestidigitation mastered by Kurt Vonnegut, who married time travel and non-linear narrative in Slaughterhouse-Five).

Once upon a time (well, I must confess, I enjoy a bit of genre-splicing also), Bill Gates, billionaire and burgeoning education hobbyist, began tossing his considerable expertise (read: money) at small schools projects. Low and behold, Gates eventually looked at the research his own foundation gathered, declared the project a failure, and scuttled away—only to decide that the greatest scourge on the planet was the enormous number of “bad” teachers failing our children everyday!

Since we are now back in time (and you may notice a pattern here about history repeating itself, and such), let’s look at that Gates/small school dynamic as I blogged about this in April and May 2011:

And, Gates’s small schools experiment? While Gates himself declared the experiment a failure, Marshak explains that Gates’ small school experiment actually exposes Gates’ own inability to understand the education dynamics he claims to reform. But ample evidence reveals both that Gates is inexpert and remains unsuccessful as an education reformer. See here, here and here – including his failure to understand statistics and the charts he enjoys using to make his points here. (Accountability? Start at the Top, OpEdNews 11 April 2011)

Let’s do some truly basic math.

First, consider that Bill Gates, a billionaire whose wealth and success have been built on computer innovation and entrepreneurship, has been an education reformer for many years now–stretching back to a small schools focus:

“Bill Gates used to believe that one of the solutions to failing schools was to create smaller ones with 500 students or fewer. His foundation spent $1 billion toward this; seeing the opportunity to bring in private dollars, districts started shifting to smaller schools. Small schools became the big new trend. But then the foundation conducted a study that found that, by itself, school size had little if any effect on achievement. The foundation dropped the project and moved on to teacher reform, but by then some urban districts throughout the nation had changed to small–and more expensive to operate–schools.”

So the first formula is:

Gates initiative + Gates funding = abandoned schools in the wake of failure (with no consequences for Gates)

As the Los Angeles Times reports above, Gates is now focusing on teacher quality–including calls for teacher evaluations tied to test scores measuring student achievement against the common core standards.

This suggests a new formula:

Gates money + common core standards + testing industry = profit for Gates and testing industry at the expense of students, learning, and public education. (If There Remains Any Question, Daily Kos 1 May 2011)

Let’s return now to the present where the Gates machine has called for a moratorium on all that Common Core and VAM stuff because his own people’s research appears to refute what Gates has been pontificating about in the compliant media gaze that only Gates seems worthy of receiving (I hate to beat this to death, but do any of you see a pattern here?).

I want to offer now that this call for a moratorium is another teachable moment—those who fail to learn from history are doomed to repeat it, and all that—and therein we have at least two important lessons:

First, Gates/small schools, Gates/VAM, and Gates/Common Core are all the same flawed dynamic in which political leaders, the media, and the public confuse wealth with expertise. Remove Gates’s billions and he would have zero credibility in any of these ventures—and I still maintain that the small school debacle is the most telling part of this story [1] because Gates misread his own research, declared erroneously the commitment a failure, and simply walked away leaving others to hold the bag (roadbuilder he has always been, roadbuilder he will always be).

And this leads to the debate and discussion that has blossomed from the Gates moratorium announcement.

So our second lesson is somewhat positive: If we could ever set aside confusing wealth with expertise, we may come to recognize that among educators, scholars, and researchers we already have a wealth of expertise that could better serve our goals of education reform.

And thus, I maintain my stance that the Gates moratorium is a sham, but I want to highlight here that within the credible responses to that moratorium call, we see how education reform should be debated: by those who know the field:

And while I enjoy coming back to and finding new SF again and again, in education reform I am eager to step off the “Déjà vu all over again” merry-go-round.

Bill Gates, about that moratorium? No thanks. And while we are at it, no thanks to all the rest either.

If you’d stopped building your roads we never asked for right through the middle of our villages, we would have time to take care of business here, instead of constantly staying one step behind underneath the rubble of the disasters you create.

[1] Please read carefully Good Doubt and Bad Doubt from 2007 and Why Did the Gates Small-High-Schools Program Fail?: Well, Actually It Didn’t from 2010 for some really powerful time travel. And this is just must-read: Bill Gates should hire a statistical advisor.

My Open Letter to Journalists: A Critical Free Press, pt. 2

Dear Journalists (especially those who write about education):

After posting my U.S. and Education Reform Need a Critical Free Press, which represents a recurring effort in my public work to address the problems with journalism about education and education research/reports, I continued to interact with Juana Summers (NPR) and Stephen Sawchuk (Education Week) on Twitter. Those exchanges have suggested to me that I need to examine more fully what my concerns raised specifically about mainstream coverage of NCTQ’s reports mean to my wider call for a critical free press.

First, I think I need to establish the foundational context of my complaints about journalism/education journalism.

I believe journalists and teachers should be (must be) comrades in arms because a free press and universal public education are essential foundational institutions for a free people.

I am not, however, suggesting that this camaraderie is some sort of wink-wink, nod-nod collusion between the two professions in which we “cover” for each other, but that we are comrades joined by the same mission to build the free society that many claim the U.S. seeks. In fact, as comrades I expect we should be each other’s most vigilant and accurate critics to insure that we both stay the course.

And that builds on my second larger context for my concerns about journalism and my call for a critical free press. My use of the term “critical” is the source of my calls for reform of both education and journalism—two fields that reach their potential when critical, but fail when they are bound by traditional expectations of impartiality, calls that teachers and journalists avoid being “political.” Critical teachers and critical journalists are activists; they use their professions as mechanisms for change. Apolitical teachers and journalists are essentially defenders of the status quo (thus, the calls for impartiality are always loudest from those with power and wealth).

So I want to return briefly to my criticisms of mainstream media coverage of NCTQ’s latest report.

While Summers, Sawchuk, and I exchanged Tweets related to my post, Adam Bessie, whose important public work refuting the “bad” teacher meme is central to my points here, offered a Tweet that simply identified a fact about Gates funding among NPR, Education Week, and NCTQ:

This Tweet represents the central issue to my concerns: Among the New Media (blogs, Twitter, Facebook, etc.), many critical educators have been confronting the disproportionate and inappropriate (because Gates has no credibility in the field of education, but nearly endless funds) influence of Gates on education reform—an influence that I have confronted often with a question: If Bill Gates had no money, who would listen to him about education reform? No one.

Also, Bessie’s Tweet about the ubiquity of Gates funding has finally begun to gain some traction in the mainstream press. But bloggers still carry the greatest weight for being critical about the influence of Gates on education reform. (The most common places now to find critical journalism is in the New Media, such as blogs at Education Week [see Anthony Cody and Nancy Flanagan] and The Washington Post [see Valerie Strauss] or alternative press such as Truthout and AlterNet.)

Despite Bessie’s Tweet only stating facts, Sawchuk immediately responded with this:

It is at “offensive” that I think we should all pause and consider carefully.

I do not in any way think Sawchuk is a careless, “bought,” or shoddy journalist. In fact, as I stated to him, I have interacted with Sawchuk because I respect him, his work, and his profession (I do not interact with others about whom I cannot say the same).

I must add that my concern with Sawchuk’s coverage of NCTQ (as well as the other coverage I identified) is that the work fails because it conforms to the flawed traditional convention of fairness that Sawchuk mentions directly.

The traditional view of fairness in journalism has been brilliantly skewered recently by John Oliver on his HBO show: Oliver exposes that being “fair” in the climate change debate—having one person for “both sides” debate the issue—actually greatly misrepresents the current understanding within the field of science for the lay public. Mainstream journalists committed to this sort of “fair and balanced” are doomed to fail the much more important goal of accuracy. As Oliver demonstrated, the ratio of for and against climate change within the sciences is not 1 to 1 as a “fair” debate implies, but about 97 for and 3 against (and while Oliver didn’t explore this, a careful look at the against shows that even those 3% are less credible within the field).

If we extrapolate the Oliver Rule, then, to education reform, we do not find an equal 1 to 1 ratio of research on using value-added methods (VAM) to evaluate and dismiss teachers because the field overwhelmingly refutes using VAM in highs-stakes situations (even pro-VAM researchers call for “modest” uses of VAM) and mostly ideological advocates and political leaders (without expertise) endorse VAM for high-stakes education policy. However, VAM advocacy garners primary coverage in the mainstream press with little attention paid to the more credible research refuting its high-stakes and disproportionate use.

So let me be very clear here about both Sawchuk being offended and the difference between critical journalists and fair journalists.

I’m sorry, but journalists and journalism will always look bad when money speaks louder than expertise (see again Gates).

Now, imagine, journalists, if every day your field was repeatedly and inaccurately trashed for all the public: U.S. public education is a failure because our tests scores rank poorly internationally (misleading), schools with impoverished students have low test scores because of low expectations by the teachers (untrue), U.S. public education is failing because of corrupt teachers’ unions (untrue and basically opposite of the evidence), to improve public schools we need to identify “bad” teachers and fire them (untrue), public school teachers are “bad” mostly because they have tenure (untrue) [1], and the list goes on.

I genuinely regret Sawchuk being offended because he doesn’t deserve it, but I must emphasize that Sawchuk is among the media who are complicit in offending teachers, teaching, and schools everyday because journalists are quick to assume the misguided pose of “fair” and unwilling to assume the needed position of critical.

Many issues simply do not have “sides” (rape, genocide), and to be honest, most issues do not have equally credibly sides.

Does teacher education/certification need to be reformed (full disclosure: I am a teacher educator)? Absolutely.

But NCTQ has no credibility and garners its influence through the impact of money and media endorsement, and not validity (just as Gates has done).

Louann Reid, Chair of NCTE’s Conference on English Education, has identified this problem perfectly in her rebuttal of NCTQ’s recent report:

The recommendations are, however, backed by considerable funding, which helps extend NCTQ’s reach. CEE doesn’t have that kind of funding, but we do have reliable researchers and educators who can mobilize to tell the true stories of effective English teacher education. And I believe we must do so.

And herein lies the problem. While I also spurred some offense by my use of “press-release journalism,” the inordinate and uncritical coverage of NCTQ by the mainstream press proves my point that mainstream journalists respond to press releases (funding) while the experts (NCTE/CEE) remain mostly ignored.

And that’s the problem with “fairness” as the journalist’s guide instead of “critical.”

As researcher and scholar Bruce Baker added to the Twitter discussion, critical journalism would have responded to NCTQ quite differently:

It may appear “fair” to respond to NCTQ as one perspective in the education reform debate, but it isn’t beneficial to afford an organization and a report without merit more credence (or even the same) as the contributions of those who have credibility.

And choosing to cover a topic is a political choice; coverage is never unbiased. And framing a topic is also a political choice (what perspective to present first, how to frame in the headline and lede, etc.). In truth, assuming a dispassionate pose is always dishonest since as humans we are always being political. I suggest we are all better off being openly and purposefully political instead of conforming to misleading norms of “neutrality.”

Yet, this is how the mainstream media carry on day after day—especially in the misguided assault on teachers, public schools, and now higher education.

Legions of hardworking and dedicated classroom teachers are offended daily by the mainstream media being complicit in a false story being told by those with money and an agenda—while that same mainstream media either offer secondary equal time [2] or ignore a powerful group of educators, researchers, and scholars who have the experience and expertise to reform education as it needs to be reformed.

Journalists, if you are ever offended, I would add that coincidentally you are now educators’ comrades for another reason.

As a lifelong teachers (31 years with 13 years teaching journalism to wonderful high school students), I am asking that you join us in the fight instead of taking your impartial stance that allows the well-funded but misguided reformers to keep on keeping on.

Any takers?

[1] How many mainstream journalists covering the Vergara ruling in California addressed that the judge issuing the ruling has job security himself?

[2] As “fair,” we are occasionally allowed to rebut the “reformers” somewhere in the middle or bottom third of the coverage, but even then we are framed as “critics.”

U.S. and Education Reform Need a Critical Free Press

Few things are worse than mainstream media coverage of education.

Except for that sentence above, which stretches hyperbole beyond credibility.

But that is exactly where the mainstream media finds itself when covering education. Journalists, in their quest to maintain the traditional commitment to “fair and balanced” journalism [1], consistently endorse and perpetuate organizations without credibility (such as NCTQ) and baseless claims (such as cries of “bad” teacher, “bad” teacher certification, and “bad” unions).

With yet another report released by NCTQ, that failure of the mainstream media has been highlighted once again—notably at NPR [2] and Education Week: Study Delivers Failing Grades For Many Programs Training Teachers, Claudio Sanchez and Juana Summers; Alternative Certification Deemed Weak by NCTQ in New Teacher-Prep Report, Stephen Sawchuk; Most Teacher Preparation Falls Short on Strategies for ELLs, NCTQ Finds, Lesli A. Maxwell.

First, the mainstream coverage of NCTQ’s reports remains trapped inside assumed crises that have no basis in fact; NCTQ’s reports and then the media begin with the givens that education suffers under the burden of “bad” teachers, “bad” teacher certification, and “bad” unions. However, at the very least, these claims are disproportional, if not outright erroneous:

  • If we maintain the current context that student achievement is accurately reflected in test scores (and it isn’t), then we must acknowledge that teacher quality (10-15%) and school quality account for only about 20% of that measurement, but “60 percent of achievement outcomes is explained by student and family background characteristics (most are unobserved, but likely pertain to income/poverty),” as Di Carlo details.
  • If we accept that value-added methods (VAM) can accurately and consistently identify “good” and “bad” teachers (and the evidence is that it cannot) and if we accept the much repeated claim by Chetty et al. that teacher quality can add $50,000 to the lifetime earning potential of a student (and that also is a significantly contested claim, as well as another example of advocacy and media hyperbole since that lifetime earning figure equates to about 1.5-2 tanks of gas per month), the enormity of the claims about “bad” teachers and the urgency expressed about creating and implementing huge and expensive test-based systems to address teacher quality are at best overstated. No rational person would endorse the cost-benefit analysis of such schemes.
  • Finally, claims that teachers unions are primary or significant negative influences on educational quality are powerfully refuted by the historical and current fact that the states in the U.S. with the lowest standardized test scores tend to be those that are right-to-work (non-union) states. Unionization correlates positively with measurable student achievement, in fact, while poverty is the greatest correlation with low measurable student outcomes (for the record, union bashing is a straw man because U.S. public education has a poverty problem, not a union problem).

Next, NCTQ has established a sort of immediate appearance of credibility through three strategies: partnering itself with U.S. News & World Report, garnering significant and influential sources of funding, and bombarding the mainstream media with a series of reports without vetting those reports as is common in traditional scholarship (which slows down and greatly harnesses higher-quality research from reaching the public [3]. But scholars don’t issue press releases, and apparently, journalists respond primarily to press releases instead of conducting investigative journalism [4].)

Finally, once I engaged Sawchuck (EdWeek) and then Summers (NPR) on Twitter, several key aspects of this phenomenon were highlighted. Both journalists argued that their pieces on NCTQ were fair, and even critical—which I will examine below—but even more significant is a comment on Twitter from Summers:

My two reactions to Summers deferring from examining the credibility of NCTQ are, first, to strongly disagree, and second, note that no journalists need to do any real investigative journalism to uncover that NCTQ has no credibility because all of that work has been done already by a number of scholars (see those critiques catalogued here and here).

As disturbing, however, as that stance is, examining carefully the coverage of NCTQ reveals that the mainstream media does in fact endorse NCTQ implicitly (despite claims of impartiality) and also marginalizes the credible critiques of NCTQ.

All three articles (see above) have headlines that establish immediately for any reader that NCTQ’s report is worthy of major media coverage. Next, all three articles have ledes that also present NCTQ positively:

The nation’s teacher-preparation programs have plenty of room for improvement, according to a new report. (Sanchez and Summers)

Alternative-certification programs for preparing teachers suffer from many of the same problems that the National Council on Teacher Quality has identified in traditional, university-based programs, the Washington-based group concludes in a new pilot study. (Sawchuk)

More than 75 percent of elementary teacher-preparation programs are failing when it comes to readying future teachers to work effectively with English-language learners, a new report from the National Council on Teacher Quality contends. (Maxwell)

Sanchez and Summers (again, note that Summers argues it isn’t her job to assign credibility to the study) certainly imply that the study is credible by using this language: “The study is a dismal read, given that the U.S. spends more than $6 billion each year to prepare teachers for the classroom.”

The NCTQ study is only a “dismal read” if it is accurate (and it isn’t). NCTQ has been carefully discredited in scholarship (for example, see Fuller here and here) for serious conflicts of interest (Teach For America and KIPP leaders sit on the Advisory Board, for example), for a flawed study design, and for shoddy methodology [5].

So how are credible academic critiques of NCTQ characterized in the journalism that claims not to take evaluative positions?:

When NCTQ released a version of this report last year, it was met with some skepticism among educators and those responsible for preparing teachers. Critics said the advocacy group should have visited individual teacher-prep programs and talked to graduates and students, rather than relying on syllabi. (Sanchez and Summers)

Last year’s inaugural teacher-prep review was immediately rejected by most teacher colleges and, especially, by their main membership body, the American Association of Colleges for Teacher Education.

Criticism focused on the NCTQ’s tack of reviewing syllabi and other course materials rather than visiting institutions; its use of open-records requests and current students to obtain documents; the complaint that its standards weren’t agreed to by the profession; and the fact that its research products aren’t peer reviewed. Additionally, critics have claimed that the project is ideologically driven, given NCTQ’s role as incubator of an alternative-certification group, the American Board for Certification of Teacher Excellence (ABCTE), which received federal funding from the George W. Bush administration.

The latter complaint seems less viable now that the NCTQ has turned its green eyeshade toward alternative-certification programs. (Sawchuk)

“Some skepticism” and “critics” clearly position credible scholarship negatively while maintaining the implied endorsement of NCTQ as an organization and NCTQ’s reports. And while Sawchuk appears to address more directly NCTQ’s lack of credibility, he still marginalizes scholars as “critics” and then in the last paragraph above, simply discounts the criticism [6].

Further in Sawchuk’s piece, the contrast between lacking credibility (NCTQ) and credibility (scholarship discrediting NCTQ) is reduced to a simple misunderstanding and a matter of tone (not substance):

Notably, the report’s introduction this year contains a number of mea culpas regarding the bad blood between the NCTQ and teacher colleges. And Walsh agreed that her group bore some of the blame.

“At times we were a bit arrogant about what it is we think teacher education should be doing,” she said. “Even if we agree to disagree, we can be more respectful.”

Again, this trivializes criticism of NCTQ and further equates NCTQ (an advocacy think tank) with scholarship—while also painting NCTQ as apologetic (despite the organization maintaining its threat of ranking programs whether they cooperate or not; a powerful tool afforded NCTQ because of its media partnership with U.S. News & World Report).

One of my strongest criticisms of teachers is that we far too often allow ourselves to be trapped within traditional calls that we take neutral stances; however, the U.S. needs critical teachers (political teachers) if our public schools are to be a foundation for our democracy.

What I have detailed above is that journalists in the U.S. have bowed to the same call for neutrality, one that cannot be accomplished but can serve as a shield for maintaining the status quo.

The U.S. needs critical journalists, ones who see their job as maintaining a commitment to seeking out and identifying the credibility of things they report. Only those in power benefit when the free press is mostly free of taking to task those in power.

Nowhere is that more apparent than in how the mainstream media fails the education reform debate.

[1] Journalists and teachers share the burden of traditional expectations that they should never be “political,” but taking a neutral stance is, in fact, taking a passive stance endorsing the status quo. In other words, taking a dispassionate pose is a political stance (see The Politics of Calling for No Politics).

[2] See a similar example with NPR’s coverage of “grit.”

[3] See the following in terms of how the mainstream media disproportionately reports on think tank (non-peer reviewed) reports as compared to peer-reviewed and university-based research:

Molnar, A. (2001, April 11). The media and educational research: What we know vs. what the public hears. Milwaukee, WI: Center for Education Research, Analysis, and Innovation. Retrieved from http://epsl.asu.edu/epru/documents/cerai-01-14.htm

Yettick, H. (2009). The research that reaches the public: Who produces the educational research mentioned in the news media? Boulder, CO and Tempe, AZ: Education and the Public Interest Center & Education Policy Research Unit. Retrieved from http://epicpolicy.org/publication/research-that-reaches

[4] I make this claim not as a direct attack on any journalists, but teachers and journalists now experience very similar and negative influences on their ability to conduct their professions. While education reform tends to impeded good teaching, the contracting media market has tended to overburden journalists. As a result of newspapers and magazines disappearing and contracting their staffs, many journalists resort to press-release journalism as a survival technique, similar to teachers teaching to the tests. The conditions of both professions, teaching and journalism, are stark reasons why both teachers and journalists must exert their political selves in their professional work.

[5] As a brief glimpse into NCTQ’s accidental admission of their methodology, in an effort to twist criticism of their practices, this post adds at the end “As one teacher candidate who is working on our office this summer said,” highlighting that anecdote is enough for NCTQ, as long as it matches their advocacy.

[6] Sawchuk fails to recognize that NCTQ is working within a scorched-earth policy as part of the large disaster capitalism driving education reform in the U.S. For a vivid example of how this works, and why NCTQ, TFA, and KIPP benefit once the traditional education system is dismantled, see the events that have occurred since Katrina in New Orleans where the public school system has been replaced by charters schools, many KIPP and many staffed by TFA recruits.

South Carolina Officially Vamboozled

If anyone is wondering why the “bad” teacher crisis remains central to what political leaders want the public to hear, South Carolina offers a heaping dose of why: Political leaders are enormously incompetent and need us all to remain distracted from that fact.

Case in point: SC has now been officially vamboozled, passing a new teacher evaluation system that includes the new sham word “growth” nearly 50 times in a little over 30 pages—notably (in part):

The changes described in this document will result in a support and evaluation system that is valid, reliable, and fair and that will

…use multiple valid measures (including but not limited to observations, professional practice, and student growth) in determining performance levels, with data on student growth for all students (including English Language Learners and students with disabilities) being a significant factor in the calculation of the overall effectiveness score (growth measure for teachers of tested grades and subjects include growth based on statewide assessments as a component)….

Of course, the most important point here is that VAM (teacher evaluations based on student test scores, or now euphemistically “student growth”) has been proven again and again to fail against the measures of “valid, reliable, and fair.”

SC political leadership has emphasized that their hands are tied because of federal mandates linked to opting out of NCLB and the lure of “filthy lucre” promised therein—a baffling stance taken by a state infamous for taking the most preposterous stances just to shun the federal government.

It is well past time to stop listening to political leaders who have no credibility (hint: If you hear a politician or read a politician, you are in the presence of one who has no clue) and to turn our gazes away from the distractions (hint: “bad” teachers are not the problem).

It is time to invoke the Oliver Rule about claims about and policies including VAM; thus, here is the enormous body of evidence so far refuting VAM (see related research cited within the following):

There is, however, a growth we should be concerned about because it is cancerous; that growth is VAM. Let’s remove it before it spreads.

What We Tolerate (and for Whom) v. What the Rich Demand: On Teacher Quality

Two of the best teachers and education advocates I respect and listen to carefully, Nancy Flanagan and Susan Ohanian, had an important exchange on Twitter recently:

Prompted by the Vergara court case in California and a related article in The Atlantic, Flanagan and Ohanian share a concern about the political, popular, and media claim that “bad” teachers work in “bad” schools with “bad” students (all of which is the result of a flawed misreading of the conditions common in high-poverty schools that serve high-poverty communities).

Before examining the issue of teacher quality raised above, please take note of a Big Picture problem at the heart of the Vergara ruling: All professions, including teaching, have elite, solid, and weak practitioners, but to suggest that teacher quality is the or one of the primary causes of educational problems is simply false, a manufactured scapegoat to keep the public view off the real problems facing us—mostly social and educational inequity.

Let’s look carefully, now, at how the complex issue of teacher quality is further complicated by high-poverty schools:

  • Teacher quality represents only about 10-15% of measurable student outcomes (test scores); thus, since high-poverty students are disproportionately likely to produce low test scores, it is a mistake to read raw tests scores as a reflection of teacher quality (or even student learning) since those test scores are primarily a reflection of out-of-school influences. (Also note that outliers to that dynamic do not disprove a generalization and do not suggest what the norm should be; as well, all claims of such “miracle” schools have been discredited.)
  • High-poverty schools and high-poverty students are staffed and taught by teachers who are disproportionately new, early career, and/or under-/un-certified. That dynamic does contribute to “teacher quality,” but not in ways popular claims mean (for example, a new teacher is not “bad,” but experience certainly contributes to greater teacher effectiveness—although that is not necessarily reflected in measurable student outcomes).
  • High-poverty students and their parents value education as much as more affluent students and parents, but living in poverty is existing in a state of scarcity that taxes adults and children physically, psychologically, and mentally—often leaving them less capable of advocating for education and benefitting for learning opportunities. Conversely, students and parents from relative affluence are afforded the slack necessary to advocate for education and fully engage with learning experiences. The important point here is that teaching, parenting, and learning in conditions influenced by poverty negatively impact nearly everyone involved, but by changing those conditions (and not the people), those same people would perform differently. (This contradicts claims that we need to rid schools of “bad” teachers and instill poor students with “grit”—both of which focus the problems in the people and not the system.)

Flanagan and Ohanian, then, are right that high-poverty schools are not overburdened by “bad” teachers; many wonderful teachers work in high-poverty schools (but some genuinely weak teachers are protected by the mask of affluent students, affluent schools, and affluent communities)—and high-poverty students are potentially as capable of learning as their more affluent peers (impoverished students don’t need to be “fixed” and don’t need to be taught “grit”).

But those facts are harshly clouded by the knee-jerk conclusions drawn from raw test scores—just as most people erroneously believe private schools are superior to public schools simply because the outward evidence appears to suggest so.

What do the data show, then, when we look at teacher characteristics (interpreted as “quality”) in high-poverty schools along with student test scores as that contrasts with teacher characteristics and student test scores in affluent public and private schools?

As a society, we tolerate the worst possible conditions for the students with the most need (teacher characteristics, school conditions and funding, and high percentage of children living in poverty are the failure of adequate public response and policy) while students living in relative affluence have the advocacy of their parents demanding that they do not suffer the same conditions experienced by children in poverty (disproportionately children of color): high student-teacher ratios, inexperienced and un-/under-certified teachers, underfunded schools, facilities in disrepair, and worksheet and rote instructional practices aimed at test-prep.

The cruel reality is that high-poverty and high-minority schools are not staffed by “bad” teachers forced to teach “bad” students, but that those teachers and students have the most challenging conditions in their lives and schools to overcome while affluent students have slack in their lives and the best conditions in their schools.

Instead of misguided commitments to value-added methods (VAM) for finding the best teachers (which do not work), policies that address equity would serve us much better:

  • No child should be taught by un-/under-certified teachers.
  • No child should have new or early career teachers for several years in a row.
  • Teacher quality must be re-imagined as an effective matching of teachers with students and not reduced to measurable outcomes.
  • Student achievement must no longer be reduced to measurable outcomes.

As with much of the flawed education reform agenda, the teacher quality issue reflects the overuse and misuse of test scores.

Until we address the scarcity in children’s lives and schools, addressing teacher quality is a futile distraction, just as continuing to change standards and tests is a futile distraction.

Instead of labeling, ranking, and then firing teachers, our first best step would be to end the cult of high-stakes testing because the problems of education are mostly systemic (social and educational) and not the adults who choose to teach or the children we seek to serve.

Gates Moratorium Another Scam: Beware the Roadbuilders pt. 2

The road to hell is not paved with good intentions. [1]

The road to hell in the U.S. of the 21st century is paved with the appearance of good intentions fostered by billionaires.

Billionaires are our roadbuilders, and in education reform the main roadbuilder is Bill Gates.

Gates is a billionaire education hobbyist who started a road to small schools, only to bail, but has since shifted his roadbuilding to value-added methods (VAM) for evaluating teachers and his tour de force superhighway, Common Core.

Now that Gates has issued a call for a moratorium on the intersecting roads to hell (VAM linked to next-generations high-stakes tests of the Common Core), we must return to two important points:

  1. I cannot stress or repeat often enough: If Bill Gates had no money, who would listen to him about education reform? No one–the same as who should listen to him now.
  2. Beware the roadbuilders.

To the first point, Gates has never had and does not now have any credibility as an authority on education or education reform. Zero. His commentaries linked to his huge bribes should be ignored when he advocates for policy, and his call for a moratorium should be ignored as well.

Delaying a road to hell still means we will have a road to hell.

To the second point, as Nettie and the Olinka learn in the Color Purple, the roadbuilders have an agenda to be done to those in their way and to benefit the roadbuilders. Words such as moratorium, philanthropy, and entrepreneur are thinly veiled code for not good intentions but the self-interests of the roadbuilders.

The roadbuilders are powerful because money speaks louder than words; however, the option before us is not a moratorium but a collective non-cooperation to end their roadbuilding.

[1] The best version of this cliche is in Ernest Hemingway’s The Sun Also Rises when Bill responds to Jake with the wonderfully ambiguous nod to the corrosive power of materialism: “‘Road to hell paved with unbought stuffed dogs’” (p. 78). In a capitalist society, the consumer’s hell is all the fake crap that consumer does not or cannot buy. The consumer doesn’t need the fake crap, of course, and there is never an end to the fake crap dangled before the consumer.

Invoking “Oliver Rule (Expanded)” for Education Reform Debate

While it is becoming increasingly common and frustrating that the most perceptive views of political and public claims and policies come from satirical programs and their figure-heads (as I have noted recently), John Oliver’s skit about the current ways the mainstream media present the climate change debate can serve well how we proceed with the education reform debate.

Oliver’s climate change debate skit highlights that by always having one FOR and one AGAINST, popular media debates distort the current state of the field; for example, Oliver notes that 97% of climate scientists support that humans impact climate change, thus he sets up a debate between 97 FOR and 3 AGAINST. The journalistic quest for “fair and balanced” (one FOR and one AGAINST), ironically, creates a distorted view of fields of knowledge.

Guideline 1 for the “Oliver Rule,” then, is that all public debates must accurately represent the current ratio of perspectives within the field.

Embedded in Oliver’s skit, however, is another recognition of the tension between democracy (everyone deserves a voice) and expertise.

Guideline 2 for the “Oliver Rule” must be a slight expansion of Oliver’s initial premise: All who wish to contribute to the debate must be allowed to contribute, but each voice must also establish his/her context for contributing, his/her expertise and experience with the topic, and his/her investment in or potential for benefitting from the stated position.

If we apply, for example, the “Oliver Rule (Expanded)” to the on-going debate about value-added methods (VAM) for evaluating and retaining teachers, the debate would look quite different.

For anyone making a public claim (such as Op-Eds or blogs) or advocating new policy (such as by political leaders or think tanks) endorsing VAM, those advocates should have to establish the following:

  • Formal study and/or experience in education.
  • Investments in research or policy related to VAM.
  • Potential for profiting from VAM policy.
  • Whether or not he/she will be professionally or personally impacted by the policy.

But, more importantly, as with the Oliver skit on climate change, the VAM debate must accurately represent the current knowledge-base on VAM. And that would have to reveal the following:

  • The research base still identifies VAM-based teacher evaluation as experimental—not ready for wide-scale or high-stakes implementation.
  • The research base also recommends when ready for implementation, VAM should be only a small part (often endorsed at only about 10-15%) of a larger range of evidence for evaluating teachers.
  • Commitments to VAM must be revealed as a large investment in a greatly increased testing program that will pre- and post-test every student for every teacher; otherwise, the process can never be equitable. (This huge investment must be framed in the context of “return on investment” for, now, an experimental process.)

If we begin to demand the “Oliver Rule (Expanded)” for the wide range of education reform policies currently being endorsed—VAM, charter schools, Common Core, high-stakes testing, etc.—the debates themselves would look  much different, and as a result, public understanding and support for those policies would likely change dramatically as well.

Just as the overwhelming majority of climate science supports human-made climate change, the overwhelming majority of educational research reveals that charter schools are nearly indistinguishable from public schools (both in student outcomes and in the trend toward re-segregation), that VAM is inconsistent and not ready for high-stakes implementation, that standards reform has not and likely will not address our greatest educational needs, and that high-stakes testing causes more harm than good. But the public and political debate—again as with the climate change debate—greatly misrepresents those bodies of research.

Also, as Oliver detailed brilliantly, climate science has actually been dealt a disservice by Bill Nye because his expertise and relentless commitment to his field (all of which must be commended) have been distorted in the simplistic FOR and AGAINST format in the mainstream media.

In the education reform debate, Diane Ravitch has become our Nye. We need Ravitch to continue, of course, and she has made huge strides in our message.

But we must now make sure that when Ravitch (or any single representative of evidence-based education policy) speaks, the public knows what ratio of the evidence-base she represents.

As Oliver has highlighted, yes, everyone voice counts, but not every voice counts the same.

Devaluing Teachers in the Age of Value-Added

“We teach the children of the middle class, the wealthy and the poor,” explains Anthony Cody, continuing:

We teach the damaged and disabled, the whole and the gifted. We teach the immigrants and the dispossessed natives, the transients and even the incarcerated.

In years past we formed unions and professional organizations to get fair pay, so women would get the same pay as men. We got due process so we could not be fired at an administrator’s whim. We got pensions so we could retire after many years of service.

But career teachers are not convenient or necessary any more. We cost too much. We expect our hard-won expertise to be recognized with respect and autonomy. We talk back at staff meetings, and object when we are told we must follow mindless scripts, and prepare for tests that have little value to our students.

During the 1980s and 1990s, U.S. public schools and the students they serve felt the weight of standards- and test-based accountability—a bureaucratic process that has wasted huge amounts of tax-payers’ money and incalculable time and energy assigning labels, rankings, and blame. The Reagan-era launching of accountability has lulled the U.S. into a sort of complacency that rests on maintaining a gaze on schools, students, and test data so that no one must look at the true source of educational failure: poverty and social inequity, including the lingering corrosive influences of racism, classism, and sexism.

The George W. Bush and Barack Obama eras—resting on intensified commitments to accountability such as No Child Left Behind (NCLB) and Race to the Top (RTTT)—have continued that misguided gaze and battering, but during the past decade-plus, teachers have been added to the agenda.

As Cody notes above, however, simultaneously political leaders, the media, and the public claim that teachers are the most valuable part of any student’s learning (a factually untrue claim), but that high-poverty and minority students can be taught by those without any degree or experience in education (Teach for America) and that career teachers no longer deserve their profession—no tenure, no professional wages, no autonomy, no voice in what or how they teach.

And while the media and political leaders maintain these contradictory narratives and support these contradictory policies, value-added methods (VAM) of evaluating and compensating U.S. public teachers are being adopted, again simultaneously, as the research base repeatedly reveals that VAM is yet another flawed use of high-stake accountability and testing.

When Raj Chetty, John N. Friedman, and Jonah E. Rockoff released (and re-released) reports claiming that teacher quality equates to significant earning power for students, the media and political leaders tripped over themselves to cite (and cite) those reports.

What do we know about the Chetty, et al., assertions?

From 2012:

[T]hose using the results of this paper to argue forcefully for specific policies are drawing unsupported conclusions from otherwise very important empirical findings. (Di Carlo)

These are interesting findings. It’s a really cool academic study. It’s a freakin’ amazing data set! But these findings cannot be immediately translated into what the headlines have suggested – that immediate use of value-added metrics to reshape the teacher workforce can lift the economy, and increase wages across the board! The headlines and media spin have been dreadfully overstated and deceptive. Other headlines and editorial commentary has been simply ignorant and irresponsible. (No Mr. Moran, this one study did not, does not, cannot negate  the vast array of concerns that have been raised about using value-added estimates as blunt, heavily weighted instruments in personnel policy in school systems.) (Baker)

And now, a thorough review concludes:

Can the quality of teachers be measured the way that a person’s weight or height is measured? Some economists have tried, but the “value-added” they have attempted to measure has proven elusive. The results have not been consistent over tests or over time. Nevertheless, a two-part report by Raj Chetty and his colleagues claims that higher value-added scores for teachers lead to greater economic success for their students later in life. This review of the methods of Chetty et al. focuses on their most important result: that teacher value-added affects income in adulthood. Five key problems with the research emerge. First, their own results show that the calculation of teacher value-added is unreliable. Second, their own research also generated a result that contradicts their main claim—but the report pushed that inconvenient result aside. Third, the trumpeted result is based on an erroneous calculation. Fourth, the report incorrectly assumes that the (miscalculated) result holds across students’ lifetimes despite the authors’ own research indicating otherwise. Fifth, the report cites studies as support for the authors’ methodology, even though they don’t provide that support. Despite widespread references to this study in policy circles, the shortcomings and shaky extrapolations make this report misleading and unreliable for determining educational policy.

Similar to the findings in Edward H. Haertel’s analysis of VAM, Reliability and validity of inferences about teachers based on student test scores (ETS, 2013), the American Statistical Association has issued ASA Statement on Using Value-Added Models for Educational Assessment, emphasizing:

Research on VAMs has been fairly consistent that aspects of educational effectiveness that are measurable and within teacher control represent a small part of the total variation in student test scores or growth; most estimates in the literature attribute between 1% and 14% of the total variability to teachers. This is not saying that teachers have little effect on students, but that variation among teachers accounts for a small part of the variation in scores. The majority of the variation in test scores is attributable to factors outside of the teacher’s control such as student and family background, poverty, curriculum, and unmeasured influences.

The VAM scores themselves have large standard errors, even when calculated using several years of data. These large standard errors make rankings unstable, even under the best scenarios for modeling. Combining VAMs across multiple years decreases the standard error of VAM scores. Multiple years of data, however, do not help problems caused when a model systematically undervalues teachers who work in specific contexts or with specific types of students, since that systematic undervaluation would be present in every year of data.

Among DiCarlo, Baker, Haertel and the ASA, several key patterns emerge regarding VAM: (1) VAM remains an experimental statistical model, (2) VAM is unstable and significantly impacted by factors beyond a teacher’s control and beyond the scope of that statistical model to control, and (3) implementing VAM in high-stakes policies exaggerates the flaws of VAM.

The rhetoric about valuing teachers rings hollow more and more as teaching continues to be dismantled and teachers continue to be devalued by misguided commitments to VAM and other efforts to reduce teaching to a service industry.

VAM as reform policy, like NCLB, is sham-science being used to serve a corporate need for cheap and interchangeable labor. VAM, ironically, proves that evidence does not matter in education policy.

Like all workers in the U.S., we simply do not value teachers.

Political leaders, the media, and the public call for more tests for schools, teachers, and students, but they continue to fail themselves to acknowledge the mounting evidence against test-based accountability.

And thus, we don’t need numbers to prove what Cody states directly: “But career teachers are not convenient or necessary any more.”

VAMboozled by Empty-Suit Leadership in SC

Rep. Andy Patrick, R-Hilton Head Island (SC), has made two flawed claims recently, one about leadership and another about teacher evaluation (“S.C. lawmaker proposes teacher evaluation plan,” Charleston Post and Courier, December 10, 2013).*

First, and briefly, Patrick asserts that SC needs leadership for superintendent of education, discounting the importance of experience or expertise. As I will address below, Patrick’s lack of experience and expertise is, ironically, evidence that leadership is not enough. In fact, leadership begins with experience and expertise; it doesn’t replace those essential qualities.

Next, and more importantly, Patrick’s and current Superintendent Mick Zais’s pursuit of test-based teacher evaluation reform is deeply flawed and discredited by research on value added methods (VAM) of evaluating teachers.

Endorsing VAM-heavy teacher evaluation joins grade retention, charter schools, and Common Core as a series of policy decisions in SC that are countered by the research base—resulting in a tremendous waste of time and funding that should be better spent for our students and our state.

For example, Edward H. Haertel’s Reliability and validity of inferences about teachers based on student test scores (ETS, 2013) now offers yet another analysis that details how VAM fails, again, as a credible policy initiative. Haertel’s analysis offers the following:

  • First, Haertel addresses the popular and misguided perception that teacher quality is a primary influence on measurable student outcomes. As many researchers have detailed, teachers account for about 10% of student test scores. While teacher quality matters, access to experienced and certified teachers as well as addressing out-of-school factors dwarf narrow measurements of teacher quality.
  • Next, Haertel confronts the myth of the top quintile teachers, outlining three reasons that arguments about those so-called “top” teachers’ impact are exaggerated.
  • Haertel also acknowledges the inherent problems with test scores and what VAM advocates claim they measure—specifically that standardized tests create a “bias against those teachers working with the lowest- performing or the highest performing classes” (p. 8).
  • The next two sections detail the logic behind VAM as well as the statistical assumptions in which VAM is grounded, laying the basis for Haertel’s main assertion about using VAM in high-stakes teacher evaluations.
  • The main section of the report reaches a powerful conclusion that matches the current body of research on VAM:

These 5 conditions would be tough to meet, but regardless of the challenge, if teacher value-added scores cannot be shown to be valid for a given purpose, then they should not be used for that purpose.

So, in conclusion, VAM may have a modest place in teacher evaluation systems, but only as an adjunct to other information, used in a context where teachers and principals have genuine autonomy in their decisions about using and interpreting teacher effectiveness estimates in local contexts. (p. 25)

  • In the last brief section, Haertel outlines a short call for teacher evaluations grounded in three evidence-based “common features”:

First, they attend to what teachers actually do — someone with training looks directly at classroom practice or at records of classroom practice such as teaching portfolios. Second, they are grounded in the substantial research literature, refined over decades of research, that specifies effective teaching practices….Third, because sound teacher evaluation systems examine what teachers actually do in the light of best practices, they provide constructive feedback to enable improvement. (p. 26)

Haertel’s concession that VAM has a “modest” place in teacher evaluation is no ringing endorsement, but it certainly refutes the primary—and expensive—role that VAM is playing in proposals to reform teacher evaluation in SC and across the U.S.

Would SC benefit from focusing on teacher quality—as well as insuring all children have equitable access to experienced and certified teachers? Absolutely.

But current calls by leaders with no experience or expertise in education are failing that possibility by rushing to implement policy that is contradicted by a growing body of research discounting the value of VAM as a key element of teacher evaluation.

SC students, teachers, and schools cannot afford doubling-down on a failed test-based education culture, and certainly, SC cannot afford more leadership without expertise, which is what Representative Patrick is offering.

* Submitted to and unpublished in, so far, Charleston Post and Courier.

Please see VAMboozled web site for research refuting VAM.