South Carolina Officially Vamboozled

If anyone is wondering why the “bad” teacher crisis remains central to what political leaders want the public to hear, South Carolina offers a heaping dose of why: Political leaders are enormously incompetent and need us all to remain distracted from that fact.

Case in point: SC has now been officially vamboozled, passing a new teacher evaluation system that includes the new sham word “growth” nearly 50 times in a little over 30 pages—notably (in part):

The changes described in this document will result in a support and evaluation system that is valid, reliable, and fair and that will

…use multiple valid measures (including but not limited to observations, professional practice, and student growth) in determining performance levels, with data on student growth for all students (including English Language Learners and students with disabilities) being a significant factor in the calculation of the overall effectiveness score (growth measure for teachers of tested grades and subjects include growth based on statewide assessments as a component)….

Of course, the most important point here is that VAM (teacher evaluations based on student test scores, or now euphemistically “student growth”) has been proven again and again to fail against the measures of “valid, reliable, and fair.”

SC political leadership has emphasized that their hands are tied because of federal mandates linked to opting out of NCLB and the lure of “filthy lucre” promised therein—a baffling stance taken by a state infamous for taking the most preposterous stances just to shun the federal government.

It is well past time to stop listening to political leaders who have no credibility (hint: If you hear a politician or read a politician, you are in the presence of one who has no clue) and to turn our gazes away from the distractions (hint: “bad” teachers are not the problem).

It is time to invoke the Oliver Rule about claims about and policies including VAM; thus, here is the enormous body of evidence so far refuting VAM (see related research cited within the following):

There is, however, a growth we should be concerned about because it is cancerous; that growth is VAM. Let’s remove it before it spreads.

Devaluing Teachers in the Age of Value-Added

“We teach the children of the middle class, the wealthy and the poor,” explains Anthony Cody, continuing:

We teach the damaged and disabled, the whole and the gifted. We teach the immigrants and the dispossessed natives, the transients and even the incarcerated.

In years past we formed unions and professional organizations to get fair pay, so women would get the same pay as men. We got due process so we could not be fired at an administrator’s whim. We got pensions so we could retire after many years of service.

But career teachers are not convenient or necessary any more. We cost too much. We expect our hard-won expertise to be recognized with respect and autonomy. We talk back at staff meetings, and object when we are told we must follow mindless scripts, and prepare for tests that have little value to our students.

During the 1980s and 1990s, U.S. public schools and the students they serve felt the weight of standards- and test-based accountability—a bureaucratic process that has wasted huge amounts of tax-payers’ money and incalculable time and energy assigning labels, rankings, and blame. The Reagan-era launching of accountability has lulled the U.S. into a sort of complacency that rests on maintaining a gaze on schools, students, and test data so that no one must look at the true source of educational failure: poverty and social inequity, including the lingering corrosive influences of racism, classism, and sexism.

The George W. Bush and Barack Obama eras—resting on intensified commitments to accountability such as No Child Left Behind (NCLB) and Race to the Top (RTTT)—have continued that misguided gaze and battering, but during the past decade-plus, teachers have been added to the agenda.

As Cody notes above, however, simultaneously political leaders, the media, and the public claim that teachers are the most valuable part of any student’s learning (a factually untrue claim), but that high-poverty and minority students can be taught by those without any degree or experience in education (Teach for America) and that career teachers no longer deserve their profession—no tenure, no professional wages, no autonomy, no voice in what or how they teach.

And while the media and political leaders maintain these contradictory narratives and support these contradictory policies, value-added methods (VAM) of evaluating and compensating U.S. public teachers are being adopted, again simultaneously, as the research base repeatedly reveals that VAM is yet another flawed use of high-stake accountability and testing.

When Raj Chetty, John N. Friedman, and Jonah E. Rockoff released (and re-released) reports claiming that teacher quality equates to significant earning power for students, the media and political leaders tripped over themselves to cite (and cite) those reports.

What do we know about the Chetty, et al., assertions?

From 2012:

[T]hose using the results of this paper to argue forcefully for specific policies are drawing unsupported conclusions from otherwise very important empirical findings. (Di Carlo)

These are interesting findings. It’s a really cool academic study. It’s a freakin’ amazing data set! But these findings cannot be immediately translated into what the headlines have suggested – that immediate use of value-added metrics to reshape the teacher workforce can lift the economy, and increase wages across the board! The headlines and media spin have been dreadfully overstated and deceptive. Other headlines and editorial commentary has been simply ignorant and irresponsible. (No Mr. Moran, this one study did not, does not, cannot negate  the vast array of concerns that have been raised about using value-added estimates as blunt, heavily weighted instruments in personnel policy in school systems.) (Baker)

And now, a thorough review concludes:

Can the quality of teachers be measured the way that a person’s weight or height is measured? Some economists have tried, but the “value-added” they have attempted to measure has proven elusive. The results have not been consistent over tests or over time. Nevertheless, a two-part report by Raj Chetty and his colleagues claims that higher value-added scores for teachers lead to greater economic success for their students later in life. This review of the methods of Chetty et al. focuses on their most important result: that teacher value-added affects income in adulthood. Five key problems with the research emerge. First, their own results show that the calculation of teacher value-added is unreliable. Second, their own research also generated a result that contradicts their main claim—but the report pushed that inconvenient result aside. Third, the trumpeted result is based on an erroneous calculation. Fourth, the report incorrectly assumes that the (miscalculated) result holds across students’ lifetimes despite the authors’ own research indicating otherwise. Fifth, the report cites studies as support for the authors’ methodology, even though they don’t provide that support. Despite widespread references to this study in policy circles, the shortcomings and shaky extrapolations make this report misleading and unreliable for determining educational policy.

Similar to the findings in Edward H. Haertel’s analysis of VAM, Reliability and validity of inferences about teachers based on student test scores (ETS, 2013), the American Statistical Association has issued ASA Statement on Using Value-Added Models for Educational Assessment, emphasizing:

Research on VAMs has been fairly consistent that aspects of educational effectiveness that are measurable and within teacher control represent a small part of the total variation in student test scores or growth; most estimates in the literature attribute between 1% and 14% of the total variability to teachers. This is not saying that teachers have little effect on students, but that variation among teachers accounts for a small part of the variation in scores. The majority of the variation in test scores is attributable to factors outside of the teacher’s control such as student and family background, poverty, curriculum, and unmeasured influences.

The VAM scores themselves have large standard errors, even when calculated using several years of data. These large standard errors make rankings unstable, even under the best scenarios for modeling. Combining VAMs across multiple years decreases the standard error of VAM scores. Multiple years of data, however, do not help problems caused when a model systematically undervalues teachers who work in specific contexts or with specific types of students, since that systematic undervaluation would be present in every year of data.

Among DiCarlo, Baker, Haertel and the ASA, several key patterns emerge regarding VAM: (1) VAM remains an experimental statistical model, (2) VAM is unstable and significantly impacted by factors beyond a teacher’s control and beyond the scope of that statistical model to control, and (3) implementing VAM in high-stakes policies exaggerates the flaws of VAM.

The rhetoric about valuing teachers rings hollow more and more as teaching continues to be dismantled and teachers continue to be devalued by misguided commitments to VAM and other efforts to reduce teaching to a service industry.

VAM as reform policy, like NCLB, is sham-science being used to serve a corporate need for cheap and interchangeable labor. VAM, ironically, proves that evidence does not matter in education policy.

Like all workers in the U.S., we simply do not value teachers.

Political leaders, the media, and the public call for more tests for schools, teachers, and students, but they continue to fail themselves to acknowledge the mounting evidence against test-based accountability.

And thus, we don’t need numbers to prove what Cody states directly: “But career teachers are not convenient or necessary any more.”

Conservative Leadership Poor Stewardship of Public Funds

In South Carolina and across the U.S., conservative leadership of education reform has failed to fulfill a foundational commitment to traditional values, good stewardship of public funds. [1]

The evidence of that failed stewardship is best exposed in commitments to three education reform policies: Adopting and implementing Common Core State Standards (CCSS), designing and implementing new tests based on CCSS, and proposing and field-testing revised teacher evaluations based on value-added models (VAM).

SC committed a tremendous amount of time and public funding to the accountability movement thirty years ago as one of the first states to implement state standards and high-stakes testing. After three decades of accountability, SC, like every other state in the union, has declared education still lacking and thus once again proposes a new round of education reform primarily focusing on, yet again, accountability, standards, and high-stakes testing.

Several aspects of committing to CCSS, new high-stakes tests, and teacher evaluation reform that are almost absent from the political and public debate are needs and cost/benefit analyses of these policies.

More of the Same Failed Policies?

If thirty years of accountability has failed, why is more of the same the next course of reform? If thirty years of accountability has failed, shouldn’t SC and other states first clearly establish what the problems and goals of education are before committing to any policies aimed at solving those problems or meeting those goals?

Neither of these questions have been adequately addressed, yet conservative political leadership is racing to commit a tremendous amount of public funding and public workers’ time to CCSS, an increase in high-stakes testing never experienced by any school system, and teacher evaluations proposals based on discredited test-based metrics.

Just as private corporations have reaped the rewards of tax dollars in SC during the multiple revisions of our accountability system, moving through at least three versions of tests and a maze of reformed state standards, the only guaranteed outcomes of commitments to CCSS, new tests, and reformed teacher evaluations are profits for textbook companies, test designers, and private consultants—all of whom have already begun cashing in on branding materials with CCSS and the yet-to-be designed high-stakes tests that will eventually be implemented twice a year in every class taught in the state.

SC as a state and as an education system is burdened by one undeniable major problem, inequity of opportunities in society and in schools spurred by poverty.

Numerous studies in recent years have shown that schools across the U.S. tend to reflect and perpetuate inequity; thus, children born into impoverished homes and communities are disproportionately attending schools struggling against and mirroring the consequences of poverty.

Commitments in SC to CCSS, new high-stakes tests, and reforming teacher evaluations based in large part on those new tests are at their core poor stewardship of public funding in a state that has many more pressing issues needing the support of state government.

A further problem with conservative leadership endorsing these education reforms is that much of the motivation for CCSS, new test, and reforming teacher evaluations comes from funding mandates by the federal government.

Misguided education reform is not only a blow to conservative economics but also a snub to traditional trust in local government over federal control.

Recently, as well, a special issue on VAM from Education Policy Analysis Archives (EPAA) includes two analyses that should give policy makers in SC and all states key financial reasons to pause if not halt commitments to education reform based on student test scores—the potential for legal action from a variety of stakeholders in education.

Baker, Oluwole, and Green explain: “Overly prescriptive, rigid teacher evaluation mandates, in our view, are likely to open the floodgates to new litigation over teacher due process rights. This is likely despite the fact that much of the policy impetus behind these new evaluation systems is the reduction of legal hassles involved in terminating ineffective teachers.”

Further, Pullin warns: “For public policymakers, there are strong reasons to suggest that high-stakes implementation of VAM is, at best, premature and, as a result, the potential for successful legal challenge to its use is high. The use of VAM as a policy tool for meaningful education improvement has considerable limitations, whether or not some judges might consider it legally defensible.”

Do schools across SC need education reform? Yes, just as social policy in the state needs to address poverty as a key mechanism for supporting those schools once they are reformed.

But in a state driven by traditional values and conservative political leadership, current commitments to CCSS, new high-stakes tests, and reforming teacher evaluations are neither educationally sound nor conservative.

[1] Expanded version of Op-Ed published in The State (Columbia, SC), March 8, 2013: “Conservatives poor stewards of education funds”

NFL again a Harbinger for Failed Education Reform?

During the impending NFL strike in 2011—the act of a union—I drew a comparison between how the public in the U.S. responds to unionization in different contexts:

“I am speaking about the possible NFL strike that hangs over this coming Super Bowl weekend: a struggle between billionaires and millionaires, which, indirectly, shines an important light on the rise of teacher and teacher union-bashing in the US. Adam Bessie, in Truthout, identifies how the myth of the bad teacher has evolved.”

Once again, the NFL is facing a situation that I believe and even hope is another harbinger of how education reform can be halted: A suit filed by the family of Junior Seau:

“The family said the league not only ‘propagated the false myth that collisions of all kinds, including brutal and ferocious collisions, many of which lead to short-term and long-term neurological damage to players, are an acceptable, desired and natural consequence of the game,’ but also that ‘the N.F.L. failed to disseminate to then-current and former N.F.L. players health information it possessed’ about the risks associated with brain trauma.”

This law suit has prompted a considerable amount of debate concerning whether or not the NFL as we currently know it could be dramatically reconfigured under the pressure of more law suits. In other words, the inherent but often ignored or concealed dangers of football are now being exposed by legal action, in much the same way as the tobacco industry was unmasked and thus the entire culture of smoking has radically changed in the last couple decades.

With the release of the Education Policy Analysis Archives (EPAA) Special Issue on “Value-Added Model (VAM) Research for Educational Policy,” a similar question should now be raised about the future of implementing high-stakes accountability policies that focus on teacher evaluation and retention through VAM-style metrics.

“High-Stakes Implementation of VAM,…Premature”

Two articles in the special issue from EPAA examines the validity and reliability of VAM-based teacher evaluation in high-stakes settings and then places these policies in the context of legal ramifications faced by districts and states for those policies.

“The Legal Consequences of Mandating High Stakes Decisions Based on Low Quality Information: Teacher Evaluation in the Race-to-the-Top Era” (Baker, Oluwole, & Green, 2013) identifies the current trend: “Spurred by the Race-to-the-Top program championed by the Obama administration and a changing political climate in favor of holding teachers accountable for the performance of their students, many states revamped their tenure laws and passed additional legislation designed to tie student performance to teacher evaluations” (p. 3). Because of the political and public momentum behind reforming teacher evaluation, Baker, Oluwole, and Green seek “to bring some urgency to the need to re-examine the current legislative models that put teachers at great risk of unfair evaluation, removal of tenure, and ultimately wrongful dismissal” (p. 5).

While Baker, Oluwole, and Green offer a detailed and evidence-based examination of the VAM-based and student growth model approaches to high-stakes teacher accountability, they ultimately place the weaknesses of reform policies in the context of potential challenges from teachers who believe they have been wrongfully evaluated or dismissed:

“In this section, we address the various legal challenges that might be brought by teachers dismissed under the rigid statutory structures outlined previously in this article. We also address how arguments on behalf of teachers might be framed differently in a context where value-added measures are used versus one where student growth percentiles are used. Where value-added measures are used, we suspect that teachers will have to show that while those measures were intended to attribute student achievement to their effectiveness, the measures failed to do so in a number of ways. That is, where value-added measures are used to assign effectiveness ratings, we suspect that the validity and reliability, as well as understandability of those measures would need to be deliberated at trial. However, where student growth percentiles are used, we would argue that the measures on their face are simply not designed for attributing responsibility to the teacher, and thus making such a leap would necessarily constitute a wrongful judgment. That is, one would not necessarily even have to vet the SGP measures for reliability or validity via any statistical analysis, because on their face they are invalid for this purpose.”

The analysis ultimately discredits both the use of narrow metrics to determine teacher quality and the high-stakes policies being implemented using those metrics, concluding with the ironic consequences of these policies: “Overly prescriptive, rigid teacher evaluation mandates, in our view, are likely to open the floodgates to new litigation over teacher due process rights. This is likely despite the fact that much of the policy impetus behind these new evaluation systems is the reduction of legal hassles involved in terminating ineffective teachers” (pp. 18-19).

In “Legal Issues in the Use of Student Test Scores and Value-added Models (VAM) to Determine Educational Quality” (Pullin, 2013), the rapid increase of VAM-based accountability is further examined in the context of “a wide array of potential legal issues [that] could arise from the implementation of these programs” (p. 2).

Pullin notes the motivation for reforming teacher evaluation:

“VAM initiatives are consistent with a highly publicized press from the business community and many politicians to make government services more like private business, data-driven to measure productivity and accountability (Kupermintz, 2003). VAM approaches are in part a response to concerns that the current system of selecting and compensating teachers based their education and credentials is insufficient for insuring teacher quality (Corcoran, 2011; Gordon, Kane & Staiger, 2006; Hanushek & Rivkin, 2012; Harris, 2011). There have been increasing expressions of concern that teacher evaluation practices are not robust and do not improve practice (Kennedy, 2010). In the contemporary public policy context, much of the support for the use of student test scores for educator evaluation comes from a concern that the current system for evaluation is ineffective and that the current legal protections for teachers are too cumbersome for schools seeking to terminate teachers (Harris, 2009, 2011).”

While a business model for addressing quality control of a work force may seem efficient, Pullin highlights that legal ramifications are likely with these new models.

Pullin’s analysis offers a detailed and useful examination of previous court cases involving the use of test scores to evaluate educators, including recent cases involving VAM, concluding that the picture is not clear on how the courts may rule in the future, but that a pattern exists of “heavy judicial deference to state and local education policymakers and the allure of using test scores to make decisions about education quality” (p. 5).

Further, Pullin notes “there are differences of perspective among social scientists about VAM and the defensibility of using it to make high-stakes decisions about educators,” further complicating the concerns of legal action (p. 9).

While raising many other complications, Pullin also notes that students and parents may enter legal battles using VAM metrics “to substantiate their own legal claims that schools are not meeting their obligations to provide education” (p. 14).

Pullin concludes with a sobering look at teacher quality reform built on VAM and implemented in high-stakes environments:

“In the broad contemporary public policy context for education reform, the desire for accountability and transparency in government, coupled with heavily financed criticisms of public school teachers and their unions, may mean that VAM initiatives will prevail. The concerns of education researchers about VAM, coupled with legal obligations for the validity and reliability of education and evaluation programs should require judges and education policymakers to take a closer look for future decision-making. At the same time, the social science research community should be generating substantial new and persuasive evidence about VAM and the validity and reliability of all of its potential uses. For public policymakers, there are strong reasons to suggest that high-stakes implementation of VAM is, at best, premature and, as a result, the potential for successful legal challenge to its use is high. The use of VAM as a policy tool for meaningful education improvement has considerable limitations, whether or not some judges might consider it legally defensible.” (p. 17)

Like the NFL, federal and state governments may soon be compelled to reform the reform movement under the threat of legal action from a variety of stakeholders since the science of teacher evaluation remains far behind the curve of implementation, particularly when teacher evaluation is high-stakes and based on VAM and other metrics linked to student test scores.

The special issue from EPAA is yet another call for political leadership to pause if not end wide-scale teacher evaluation and retention models that pose legal, statistical, and funding challenges that those leaders appear unwilling to acknowledge or address.

VAM: A Primer

Education reform has existed at some policy and public levels since at least the 1890s in the U.S. The current reform movement grounded in state-based accountability began in the early 1980s with a Nation at Risk, and then was nationalized in 2001 with No Child Left Behind.

In the first decades of the recent accountability era, standards and high-stakes testing were implemented and periodically revised at the state level with the primary focus being on student and school accountability. The current cycle, however, has seen an increase in policies and practices aimed at teacher accountability and increasing teacher quality—despite a solid research base showing that teacher quality constitutes only 10-15% of measurable student outcomes (test data).

The focus on increasing teacher quality and accountability has included both experiments and policies with value added methods (VAM) of determining teacher quality in order to label, rank, sort, and retain or dismiss teachers. VAM claims to isolate teacher quality through pre- and post-testing methods that seek to identify teacher quality and isolate that from the other factors reflected in test scores.

Before policy-makers and stakeholders in education commit to reforming teacher evaluation and retention, foundational questions must be addressed, and then the current facts about VAM must be acknowledged.

First, the foundational questions:

(1) What evidence exists identifying teacher quality as a primary or significant problem facing a school or district? Where, then, does teacher quality rank as a priority for a school, district, or state in terms of cost effectiveness in committing funding to the reform?

(2) Are all elements of implementing VAM at any percentage to the revised teacher evaluation process valid for determining teacher quality? In other words, what measures are taken to account for using student test scores (designed to reflect student learning, and not designed to reflect teacher quality) as data points for teacher quality?

(3) Have the teaching and learning environments for students’ home and schools been addressed to insure equitable teaching and learning environments in which determining teacher quality becomes valid?

Next, what is the current knowledge base about VAM*?:

(1) Including VAM at any percentage in reformed teacher evaluation models is currently in the experimental phase. Data and the validity of including VAM are being tested, but almost no researchers currently claim VAM (at any percentage but certainly at high percentages such as 40-50%) to be ready for widespread implementation.

(2) VAM models for labeling teacher quality are highly unstable. Teacher rankings tend to shift with new populations of students.

(3) Researchers agree that VAM is unlikely ever to be completely stable; thus, it is possible that VAM will never be a practical or fair element in teacher evaluation, particularly at the individual teacher level or in any single year.

(4) The statistical and practical requirements to isolate teacher quality in student test scores pose tremendous costs in time and funding that also may prove to be not cost effective in the context of most school systems’ priorities. In order to implement a fair and equitable teacher evaluation system that includes VAM at any percentage, states must create and implement pre- and post-tests to all students in all teachers’ courses, creating a new and costly commitment to education funding.

(5) Decades of research on high-stakes testing have shown many negative unintended consequences to accountability measures focusing on students and schools; including VAM in high-stakes accountability policies focusing on teachers is likely to have similar unintended negative consequences such as discouraging high-quality teachers from working with high-needs populations of students.

Thus, policy-makes and stakeholders are strongly cautioned to consider education reform priorities, the experimental nature of VAM, and the current knowledge base on VAM before committing tax dollars to either field testing or implementing new teacher evaluation policies built in any way on VAM.

* See a recent review of teacher evaluation reform for links and citations to numerous research studies on VAM.

Assembled Pieces Reveal Disturbing Reform Picture

Every time I write about Michelle Rhee, as I noted in a recent post, I feel like I should reenact the shower and wire-brush scene in Silkwood to purge myself of participating in the ceaseless media attention disproportionately afforded Rhee while the voices, daily efforts, and expertise of K-12 practitioners are not just ignored, but marginalized and even demonized.

So it is with a shared reservation (see Jose Vilson’s excellent post) that I once again wade into the Common Core State Standards (CCSS) debate—not to rehash my unequivocal opposition to the CCSS movement, but to offer a brief look at the picture revealed once all the pieces of the corporate/ “no excuses” reform movement puzzle are assembled. First, then, let me identify the primary pieces of that puzzle:

CCSS

National high-stakes tests built on CCSS

Reformed teacher evaluation driven by VAM-based teacher ranking

Teach for America

Charter schools

These various pieces are an effective strategy with a common thread because separately each reform element creates a focal point of debate; for an educator or researcher to challenge any one of these policies is a seemingly endless task since the reform agenda is being set by those with political and financial power. Refuting the need for new standards, much less the flaws with implementing those new standards, immediately positions educators as reactionary and allows the self-appointed reformers to characterize those challenges as being for the status quo and against reform and accountability.

For example, teachers in my home state of South Carolina who have spoken against VAM-style teacher evaluation reform have been publicly labeled by the state superintendent of education, Mick Zais, as trying to avoid being held accountable for their work.

The picture these reform pieces show is not a patchwork of evidence-based and innovative strategies for improving public education, but a carefully unified process of infusing even more deeply the power of high-stakes standardized testing into the fabric of public schools. Look beneath any of the elements listed above and find the allure of new and better tests, as Secretary of Education Arne Duncan (2010) celebrated himself:

Today is a great day! I have looked forward to this day for a long time–and so have America’s teachers, parents, students, and school leaders. Today is the day that marks the beginning of the development of a new and much-improved generation of assessments for America’s schoolchildren. Today marks the start of Assessments 2.0. And today marks one more milestone, testifying to the transformational change now taking hold in our nation’s schools under the courageous leadership and vision of state and district officials.

Duncan’s entusiasm doesn’t stop there:

This new generation of mathematics and English language arts assessments will cover all students in grades three through eight and be used at least once in high school in every state that chooses to use them. In addition, the PARCC consortium will develop optional performance tasks to inform teachers about the development of literacy and mathematics knowledge and skills in kindergarten through second grade.

I am convinced that this new generation of state assessments will be an absolute game-changer in public education. For the first time, millions of schoolchildren, parents, and teachers will know if students are on-track for colleges and careers–and if they are ready to enter college without the need for remedial instruction. Yet that fundamental shift–re-orienting K-12 education to extend beyond high school graduation to college and career-readiness–will not be the only first here.

For the first time, many teachers will have the state assessments they have longed for– tests of critical thinking skills and complex student learning that are not just fill-in-the-bubble tests of basic skills but support good teaching in the classroom.

And what provides the basis upon which Duncan makes these claims?:

Yet existing assessments are only part of the problem. An assessment system and curriculum can only be as good as the academic standards to which the assessments and curriculum are pegged. We want teachers to teach to standards–if the standards are rigorous, globally competitive, and consistent across states. Unfortunately, in the last decade, numerous states dummied down their academic standards and assessments. In effect, they lied to parents and students. They told students they were proficient and on track to college success, when they were not even close.

The Common Core standards developed by the states, coupled with the new generation of assessments, will help put an end to the insidious practice of establishing 50 different goalposts for educational success. In the years ahead, a child in Mississippi will be measured against the same standard of success as a child in Massachusetts.

Even if we account for the sort of soaring rhetoric associated with political discourse, Duncan clearly envisions policy that must include a staggering and unprecedented commitment to testing that rises to the level of parody. But for all stakeholders in public education, the results of all the policies linked to standardized testing must include a brave new world of testing that boggles the mind in terms of the amount of time and funding required to design, field test, implement, and manage pre- and post-tests aligned with CCSS for every single course and teacher year after year after year.

As Yong Zhao has detailed carefully in an exchange with Marc Tucker, commitments to education reform policy linked to CCSS and the high-stakes tests built on these new standards are not anything new, are not justified by any clearly identified problems or needs, and are not consistent with the larger democratic goals of universal public education:

[L]et me restate my main point: it is impossible, unnecessary, and harmful for a small group of individuals to predetermine and impose upon all students the same set of knowledge and skills and expect all students progress at the same pace (if the students don’t, it is the teachers’ and schools’ fault). I am not against standards per se for good standards can serve as a useful guide. What I am against is Common and Core, that is, the same standards for all students and a few subjects (currently math and English language arts) as the core of all children’s education diet. I might even love the Common Core if they were not common or core.

Classroom teachers, educational researchers, and educational historians have offered and continue to offer a clear and valid voice that Duncan’s claims and the resulting policies are deeply flawed, but as Brian Jones asks, “If all of this testing is so bad for teaching and learning, why is it spreading?” According to Jones, the answer detailed in the full picture is clear:

As the tests spread and the consequences associated with them rise, absurdities abound….

The shift toward using student data to evaluate teachers is part of a larger trend of restructuring public education to align it with the rest of the economy. As one of the last heavily unionized groups of workers in the country, teachers stand in the way of privatization. And to the extent that they are self-governed, self-motivated and enjoy professional autonomy, teachers are a ‘bad’ example for other workers.

Even though it may not make for great teaching or genuine learning, high-stakes standardized testing is spreading because it is the perfect tool for controlling and disciplining teachers–and for training the next generation to internalize the priorities of the system.

The attempt to quantify and track every aspect of an employee’s ‘performance’ is not new.

Standardized testing—the inevitable consequence of commitments to CCSS, reformed teacher evaluation, and each piece of the corporate reform puzzle—combines the veneer of objectivity with the power of perpetual control over schools, teachers, and students, what Foucault characterized as “…entering the age of infinite examination and of compulsory objectification” (p. 200):

The exercise of discipline presupposes a mechanism that coerces by means of observation; an apparatus in which the techniques that make it possible to see induce effects of power in which, conversely, the means of coercion make those on whom they are applied clearly visible….

[T]he art of punishing, in the regime of disciplinary power, is aimed neither at expiation, nor precisely at repression….It differentiates individuals from one another, in terms of the following overall rule: that the rule be made to function as a minimal threshold, as an average to be respected, or as an optimum toward which one must move. It measures in quantitative terms and hierarchizes in terms of value the abilities, the level, the ‘nature’ of individuals….The perpetual penalty that traverses all points and supervises every instant in the disciplinary institution compares, differentiates, hierachizes, homogenizes, excludes. In short, it normalizes….

The examination combines the techniques of an observing hierarchy and those of normalizing judgment. It is a normalizing gaze, a surveillance that makes possible to qualify, to classify, and to punish….

In discipline, it is the subjects who have to be seen. Their visibility assures the hold of the power that is exercised over them. It is the fact of their being constantly seen…that maintains the disciplined individual in his subjection. And the examination is the technique by which power…holds them in a mechanism of objectification. (pp. 177, 170, 197, 199)

Now, in the context of whether or not the U.S. is committed to universal public education as a central element of a commitment to democracy and individual liberty, and then whether or not education reform is seeking that foundational goal, time has come to set aside the puzzle-piece-by-puzzle-piece dismantling of the corporate reform agenda and confront directly the central flaw with the picture itself, as Jones acknowledges:

The solution to this dilemma is not to develop better tests, but to tear down the whole enterprise of high-stakes standardized assessment and replace it with authentic assessments that are organic to the process of real teaching and learning.

In sum, the attempt to quantify learning and teaching in a standardized manner is extremely expensive; takes up weeks and, in some places, months of time in school; narrows the curriculum; undermines the intrinsic joy of learning; and leads to a culture of corruption and cheating. As a measure of student learning, standardized tests are an extremely limited instrument. As a measure of teacher effectiveness, they are even more flawed.

Measuring, labeling, ranking, and then sorting students, teachers, and schools is an anti-democratic process, a dehumanizing process, and a mechanism for control. At the center of this process being antithetical to both our democracy and our faith in education is the fundamental flaw of high-stakes standardized testing.

Do many of the puzzle pieces of the corporate reform puzzle misuse standardized tests and the data drawn from those tests? Yes.

But we must not fall prey to the simplistic claim that the problem is how tests are used and not the tests themselves.

The ugly full picture of corporate reform shows that the problem is testing. Period.

Review [UPDATED]: “How to Evaluate and Retain Effective Teachers” (League of Women Voters of SC)

The League of Women Voters of South Carolina has released a report entitled “How to Evaluate and Retain Effective Teachers” (2011-2013) with the identified purpose “to examine the growing movement toward ‘results based’ evaluation nationally and in South Carolina.”

Before examining the substance of the report, several problems with the larger context of teacher evaluation and retention as they intersect with important challenges facing SC need to be identified:

• The report fails to clarify and provide evidence that teacher quality and retention are primary problems facing SC. Without a clear and evidence-based problem, solutions are rendered less credible. However, SC, like most of the US, has a teacher assignment problem that has been clearly documented: students of color, students from poverty, English language learners (ELL), and special needs students are disproportionately assigned to un-/under-certified and inexperienced teachers (Peskey & Haycock, 2006). As well, the implied problem of the report marginalizes the greatest obstacles facing SC schools, poverty and the concentration of poverty (notably the Corridor of Shame along I-95).

• The report compiles and bases claims on a selection of references that are not representative of the body of research on teacher quality and value-added methods (VAM) or performance-based systems of identifying teacher quality. As detailed below, the claims and research included in this report misrepresent the reports themselves as well as the current knowledge-base on teacher quality and retention.

In that context, the report fails the larger education reform needs facing SC as well as the stated purpose of the study.

Do Teachers Matter?

The opening claim of the report asserts “the most important school-based factor is an effective teacher,” and then cites Hanushek, among others. While the report is careful to note teacher quality is an important in-school factor, it repeatedly overstates teacher quality’s impact with terms such as “overwhelmingly” and fails to clarify that in-school factors are dwarfed by out-of-school factors. Matthew Di Carlo offers a balanced picture of the proportional impact of teacher quality, including an accurate interpretation of many of the same references (such as Hanushek):

“But in the big picture, roughly 60 percent of achievement outcomes is explained by student and family background characteristics (most are unobserved, but likely pertain to income/poverty). Observable and unobservable schooling factors explain roughly 20 percent, most of this (10-15 percent) being teacher effects. The rest of the variation (about 20 percent) is unexplained (error). In other words, though precise estimates vary, the preponderance of evidence shows that achievement differences between students are overwhelmingly attributable to factors outside of schools and classrooms (see Hanushek et al. 1998Rockoff 2003Goldhaber et al. 1999Rowan et al. 2002Nye et al. 2004).” [1]

Along with misrepresenting the impact of teacher quality on measurable student outcomes, the report lends credibility to a misrepresented and flawed study by Chetty, Friendam and Rockoff  (2011):

“[T]hose using the results of this paper to argue forcefully for specific policies are drawing unsupported conclusions from otherwise very important empirical findings.” (Di Carlo)

“These are interesting findings. It’s a really cool academic study. It’s a freakin’ amazing data set! But these findings cannot be immediately translated into what the headlines have suggested – that immediate use of value-added metrics to reshape the teacher workforce can lift the economy, and increase wages across the board! The headlines and media spin have been dreadfully overstated and deceptive. Other headlines and editorial commentary has been simply ignorant and irresponsible. (No Mr. Moran, this one study did not, does not, cannot negate  the vast array of concerns that have been raised about using value-added estimates as blunt, heavily weighted instruments in personnel policy in school systems.)” (Baker)

The teacher quality impact is misrepresented in this report and perpetuates popular and agenda-driven research myths such as the need for consecutive years of high-quality teachers; the claim is inaccurate and should not drive policy:

“This is important, because the ‘X consecutive teachers’ argument only carries concrete policy implications if we can accurately identify the ‘top’ teachers. In reality, though, the ability to do so is still extremely limited [emphasis added].

“So, in the context of policy debates, the argument proves almost nothing. All it really does – in a rather overblown, misleading fashion – is illustrate that teacher quality is important and should be improved, not that policies like merit pay, higher salaries, or charter schools will improve it.

“This represents a fundamental problem that I have discussed before: The conflation of the important finding that teachers matter – that they vary in their effectiveness – with the assumption that teacher effects can be measured accurately at the level of the individual teacher (see here for a quick analogy explaining this dichotomy)….

“But the ‘X consecutive teachers’ argument doesn’t help us evaluate whether this or anything else is a good idea. Using it in this fashion is both misleading and counterproductive. It makes huge promises that cannot be fulfilled, while also serving as justification for policies that it cannot justify. Teacher quality is a target, not an arrow.” (Di Carlo)

Has Traditional Teacher Evaluation Failed?

One of the implied and cited reasons for addressing teacher quality and retention rests on wide-spread criticism of traditional teacher evaluation policies and practices. This report lends a great deal of credibility to those criticisms while relying on The Widget Effect from The New Teacher Project (TNTP). [2] However, a review of this report calls into question, again, the credibility of the study’s claims as well as using it as a basis for policy decisions:

“Overall, the report portrays current practices in teacher evaluation as a broken system perpetuated by a culture that refuses to recognize and deal with incompetence and that fails to reward excellence. However, omissions in the report’s description of its methodology (e.g., sampling strategy, survey response rates) and its sample lead to questions about the generalizability of the report’s findings.”

Di Carlo has also documented that TNTP continues to misrepresent and overstate the impact of teacher quality and the effectiveness of identifying high-quality teachers:

“I just want to make one quick (and, in many respects, semantic) point about the manner in which TNTP identifies high-performing teachers, as I think it illustrates larger issues. In my view, the term ‘irreplaceable’ doesn’t apply, and I think it would have been a better analysis without it….

“Based on single-year estimates in math and reading, a full 43 percent of the NYC teachers classified as ‘irreplaceable’ in 2009 were not classified as such in 2010. (In fairness, the year-to-year stability may be a bit higher using the other district-specific definitions.)

“Such instability and misclassification are inevitable no matter how the term is defined and how much data are available – it’s all a matter of degree – but, in general, one must be cautious when interpreting single-year estimates (see herehere and here for related analyses).

“Perhaps more importantly, if you look at how they actually sorted teachers into categories, the label irreplaceable,’ at least as I interpret it, seems inappropriate no matter how much data are available.”

Performance-Based Teacher Evaluations: Is VAM Credible?

A significant portion of the report makes claims about value-added methods (VAM) of teacher evaluations in the context of performance-based approaches to identifying teacher quality. Nationally, VAM and other performance-based policies are being implemented quickly, but with little regard to the current understanding of the effectiveness and limitations of those policies; this report fails to represent the current state of research on VAM accurately and depends on research and think tank advocacy (National Council on Teacher Quality [NCTQ]) that distorts the importance of teacher quality and the effectiveness of identifying teacher quality based on measurable student outcomes.

The report remains supportive of performance-based policy recommendations, but does identify cautions about test-based teacher evaluations while also encouraging teacher evaluations include multiple measures and include teachers in the creation of a new evaluation system.

Two failures, however, of this report’s endorsement of VAM and/or performance-based teacher evaluations systems include couching that endorsement in the distorted claims about teacher quality’s impact on measurable student outcomes and depending on reports and claims made by NCTQ and the Bill and Melinda Gates Foundation. [3]

What, then, are the current patterns from the research on VAM and performance-based models and how should those patterns shape policy? [4]

• VAM and test-based evaluations for teachers remain both misleading about teacher quality and misrepresented by research, the media, and political leadership. Numerous researchers have detailed that teachers identified as high-quality or weak one year are identified differently in subsequent years: Numerous factors beyond the control of teachers remain reflected in test scores more powerfully than the individual impact of any specified teacher. The debate over teacher quality and measuring that quality, then, is highly distorted, as Di Carlo explains: “Whether or not we use these measures in teacher evaluations is an important decision, but the attention it gets seems way overblown.” This report makes that mistake.

• VAM and performance-based teacher evaluations in high-stakes settings distort teaching and learning by narrowing the focus of both teaching and learning to teaching to the test and test scores. VAM and test-based data  are likely valuable for big picture patterns and in-school or in-district decision making regarding teacher assignment, but VAM and performance-based evaluations of individual teachers remain inaccurate and inappropriate for evaluation, pay, or retention.

Particularly in a state such as SC where poverty and state budget concerns burden the state and the public school system, VAM and performance-based systems that rely on extensive retooling of standards, testing, and teacher evaluation systems are simply not cost effective (Bausell, 2013): “VAM is not reliable or valid, and VAM-based polices are not cost-effective for the purpose of raising student achievement and increasing earnings by terminating large numbers of low-performing teachers” (Yeh, 2014).

And further, rejecting VAM and using significant percentages of student test scores to evaluate and retain teachers is not rejecting teacher accountability, but confronting the misuse of data. Ewing (2011) clarifies that VAM is flawed math and thus invalid as a tool in teacher evaluation:

“Of course we should hold teachers accountable, but this does not mean we have to pretend that mathematical models can do something they cannot. Of course we should rid our schools of incompetent teachers, but value-added models are an exceedingly blunt tool for this purpose. In any case, we ought to expect more from our teachers than what value-added attempts to measure.”

If SC choses to reform teacher evaluation—which remains a project far less urgent than other problems being ignored—the state would be guided better by Gabriel and Allington (2012), who have analyzed and challenged the Gates Foundations MET Project, which has prompted misguided and hasty implementation of VAM-style teacher evaluation reform:

“Although we don’t question the utility of using evidence of student learning to inform teacher development, we suggest that a better question would not assume that value-added scores are the only existing knowledge about effectiveness in teaching. Rather, a good question would build on existing research and investigate how to increase the amount and intensity of effective instruction.”

Gabriel and Allington (2012) recommend five questions to guide teacher evaluation reform, instead of VAM or other student-outcome-based initiatives:

“Do evaluation tools inspire responsive teaching or defensive conformity?…

“Do evaluation tools reflect our goals for public education?…

“Do evaluation tools encourage teachers to use text in meaningful ways?…

“Do evaluation tools spark meaningful conversations with teachers?…

“Do evaluation tools promote valuable education experiences?”

Again, these guidelines are evidence-based alternatives to discredited and experimental commitments to the misrepresented evidence in the report from LWV SC, but SC remains overburdened by issues related to equity and opportunity that outweigh the need to reform teacher evaluation at this time.

How Should SC Proceed with Teacher Quality and Retention?

On balance, this report misrepresents teacher quality and overstates the need and ability to identify high-quality teachers using VAM and other performance-based policies. The flaws in this report grow from an over-reliance on misguided and misrepresented research and advocacy while ignoring the rich and detailed evidence from the full body of research on teacher quality. Finally, the report concludes by discrediting SC’s current teacher evaluation system (ADEPT) in the context of the inaccurate and distorted claims in the report.

Ultimately, the report encourages SC to spend valuable time and resources on policies that are dwarfed by more pressing needs facing the state and its public schools—a failure of state leadership replicated in the perpetual retooling of state education standards (Common Core State Standards adoption) and high-stakes testing based on revised standards. In short, SC has a number of social and educational challenges that need addressing before the state experiments with revising teacher evaluation and retention policies, including the following:

• Identify how better to allocate state resources to address childhood and family poverty, childhood food security, children and family access to high-quality health care, and stable, well paying work for families.

• Replace current education policies based on accountability, standards, and testing with policies that address equity and opportunity for all students.

• Address immediately the greatest teacher quality issue facing SC’s public schools—inequitable distribution of teacher quality among students in greatest need (high-poverty children, children of color, ELL, and special needs students).

• Address immediately the conditions of teaching and learning in the state’s schools, including issues of student/teacher ratios, building conditions and material availability, administrative and community support of teachers, equitable school funding and teacher salaries, teacher job security and academic freedom in a right-to-work state, and school safety.

Any policy changes that further entrench the culture of testing in SC as a mechanism for evaluating students, teachers, and schools perpetuate the burden of inequity in the state and schools.

SC does not need new standards, new tests, or a new teacher evaluation system. All of these practices have been implemented in different versions with high-stakes attached for the past thirty years—with the current result being the same identified failures with public schools that were the basis of these policies.

SC, like much of the US, needs to come to terms with identifying problems first before seeking solutions. The problems are ones of equity and opportunity, and no current teacher evaluation plan is facing those realities, including this report.

[1] For additional examinations of out-of-school factors compared to teacher quality and in-school factors see Berliner, 2009, 2014; Hirsch, 2007; Rothstein, 2010; Traub, 2000.

[2] Also see Di Carlo, 2011, on TNTP.

[3] For evaluations of NCTQ and Gates see the following: (a) NCTQ: Benner, 2012; Merrill & Ingersoll, 2010; Baker, 2010, January 29; Baker, 2010, October 8; Baker, 2010, December 4, and (b) Gates: Baker, 2010, December 13; Baker, 2011, March 2; Libby, 2012; [Gates MET Project] Gabriel & Allington, 2012; Rubinstein, 2013, January 13; Rubinstein, 2013, January 9; Baker, 2013, January 9; Glass, 2013, January 14; Rothstein & Mathis, 2013

[4] For comprehensive examinations of research on VAM, see Di Carlo, 2012, November 13*; Di Carlo, 2012, April 12 (reliability); Di Carlo, 2012, April 18 (validity); Baker, 2012, May 28 (misrepresentations of VAM); Baker, et al., 2010, August 27; Ewing, 2011, May. For recent concerns about legal action and VAM-based teacher evaluation, see Baker, Oluwole, & Green, 2013; Pullin, 2013

* Di Carlo remains a proponent of including VAM in teacher evaluations.

References

Berliner, D. C. (2014). Effects of inequality and poverty vs. teachers and schooling on America’s youth. Teachers College Record, 116(1). Retrieved from http://www.tcrecord.org/content.asp?contentid=16889

Berliner, D. C. (2009). Poverty and potential: Out-of-school factors and school success. Boulder, CO and Tempe, AZ: Education and the Public Interest Center & Education Policy Research Unit. Retrieved from http://nepc.colorado.edu/publication/poverty-and-potential

Gabriel, R., & Allington, R. (2012, November). The MET Project: The wrong 45 Million dollar question. Educational Leadership, 70(3), 44-49. Retrieved from http://216.78.200.159/Documents/RandD/Teacher%20Evaluation/MET%20Project%20-%20Wrong%2045%20Millon%20Question%20-%20Gabriel.pdf

Hirsch, D. (2007, September). Experiences of poverty and educational disadvantage. York, North Yorkshire, UK: Joseph Rowntree Foundation. Retrieved from http://www.jrf.org.uk/knowledge/findings/socialpolicy/2123.asp

Peske, H. G., & Haycock, K. (2006, June). Teaching inequality: How poor and minority students are shortchanged on teacher quality. Washington DC: The Education Trust, Inc. Retrieved from http://www.edtrust.org/sites/edtrust.org/files/publications/files/TQReportJune2006.pdf

Rothstein, R. (2010, October 14). How to fix our schools. Issue Brief 286. Washington DC: Economic Policy Institute. Retrieved from http://www.epi.org/publications/entry/ib286

Thomas, P. L. (2011, December). Teacher quality and accountability: A failed debate. Daily Censored. Retrieved from http://www.dailycensored.com/teacher-quality-and-accountability-a-failed-debate/

Traub, J. (2000, January 16). What no school can do. New York Times Magazine. Retrieved from http://local.provplan.org/pp170/materials/what%20no%20school%20can%20do.htm