L.A. Times stands by its teacher ratings

An article in last Monday’s Times has come under fire from critics who say it misrepresents the results of a review of the “value-added analysis” of L.A. Unified teachers that was published in print and online last August.

The review, conducted by the National Education Policy Center at the University of Colorado at Boulder, looked at the LAUSD data that The Times used in its “Grading the Teachers” series. The Feb. 7 article by Jason Felch said the review “confirms the broad conclusions of a Times analysis of teacher effectiveness in the Los Angeles Unified School District while raising concerns about the precision of the ratings.”

The policy center issued a news release taking issue with the article, saying its researchers believed The Times’ teacher-effectiveness ratings were based on “unreliable and invalid research.” Therefore, the release continued, the study “confirms very few of The Times’ conclusions.”

Several readers e-mailed The Times, questioning the reporting.

The article “distorted the study's findings for self-serving purposes,” said one reader. 

“It smacks of either shock journalism or a deliberate attempt to mislead the public on behalf of big business and privatizers,” said another.

Readers raised two basic questions about the Colorado study and The Times’ handling of it: Did the  article accurately reflect the findings of the study? Does the study invalidate the “Grading the Teachers” series?

The article said this about the study:

The authors largely confirmed The Times' findings for the teachers classified as most and least effective. But the authors also said that slightly more than half of all English teachers they examined could not be reliably distinguished from average. The general approach used by The Times and the Colorado researchers, known as "value added," yields estimates, not precise measures.

The article listed the percentages of teachers who the study said had been incorrectly labeled as more effective or less effective. Altogether, 22% of English teachers and 14% of math teachers ended up on different sides of the “effective” line in the center's study as compared with The Times’ analysis.

National Education Policy Center publications director Alex Molnar said the article should have included an additional finding:

NEPC researchers demonstrated that the inclusion of three additional sets of variables in the model [The Times] used –- a longer history of a student’s test performance, peer influence, and school -- leads to dramatic changes in teacher ratings. For reading test outcomes in particular, as many as half of all teachers would be rated differently.

On what basis did the article say that the Colorado study “confirms the broad conclusions” of The Times’ earlier work?

“The huge public-policy question that folks have been arguing about since we first published our ratings is whether there is such a thing as a ‘teacher effect’ that can be measured statistically –- whether teachers have a significant impact on what their students learn or whether student achievement is all about demographics, differences among schools, family background and other factors outside of teachers’ control,” said Assistant Managing Editor David Lauter, who oversees the California reporters and editors.

“The Colorado study comes down on our side of that debate. Their study said the teacher impact that they found was actually ‘slightly larger’ than the effect found by Dick Buddin, the economist who did the underlying work for The Times.

“For parents and others concerned about this issue, that’s the most significant finding: the quality of teachers matters. So although they disagree with us about how to measure the teacher effect, it was entirely accurate to say that their study confirmed some parts of our work and criticized others.”

Molnar also questioned the assignment of Felch, one of the reporters on “Grading the Teachers,” to cover the center’s study that “directly criticized the research” used in the earlier reporting.

Lauter said that he’d considered the appearance of conflict , but that “it seemed to me that any Times reporter who wrote about the study could be accused of a conflict. So, it seemed to me we were best off having the person who understood the subject best write about it.” However, Lauter acknowledged, “maybe that was the wrong call.”

Does the Colorado study invalidate The Times’ original series?

The Colorado researchers took data from the Los Angeles school district on student test scores and ran it through a different value-added model than the one used by Buddin. 

When the Colorado researchers compared the results of their model with the results of Buddin’s model, they found that the two “correlate” with each other 92% of the time on math scores and 76% of the time on English. But they also found that as many as half the English teachers might have ended up in different categories under their model than under the model The Times used.

It’s important to note that the Colorado study was not based on entirely the same data Buddin used. Roughly 93,000 student test scores that Buddin used were excluded in the Colorado study -– that’s roughly 15% of the data. The Colorado researchers did not contact The Times before preparing their report in order to compare data sets, and they have not explained the reasons for the dropped data. They also have not disclosed basic information about some of their statistical techniques that would allow outside researchers to assess their work.

In at least one case, the Colorado study contained a finding that the researchers concede overstated the differences between their model and the one used by The Times.  The researchers based their work on a pool of roughly 11,000 teachers. But The Times published scores for only 6,000 teachers because the project excluded scores from any teacher who had not taught at least 60 students. When informed of that discrepancy over the weekend, Derek Briggs, the lead author of the Colorado study, said in an e-mail to The Times that the use of the 60-student limit “serves to mitigate” some of the shortcomings his study had alleged. Briggs and his colleagues, however, have not made that information public.

It also is worth noting that the policy center is partly funded by the Great Lakes Center for Education Research and Practice, which is run by the top officials of several Midwestern teachers unions and supported by the National Education Assn., the largest teachers union in the country and a vociferous critic of value-added analysis.

The fact that the two models differ does not tell you which is “right” and which is “wrong,” Lauter said.  As Professor Thomas Kane of Harvard noted in a Washington Post article about the Colorado study, “we still don't know yet which [model] was the right one.”

In the more than five months since the publication of “Grading the Teachers,” The Times has received suggestions and critiques from many experts on value-added analysis. In the next few weeks, The Times plans to publish version two of the database, updated with a new year of data and incorporating a number of changes based on that feedback.

[Updated 3 p.m.: The Times has also released a statement addressing criticism of its analysis.]

-- Deirdre Edgar

Comments

[Richard Wexler]

So, let me see if I understand this. Because both the Colorado study and the Times stories show that teachers have some kind of effect on student achievement, that means the Colorado study “confirms the broad conclusions” of the Times stories.

That’s like saying if I hand you a mushroom and say “Eat this, you’ll really enjoy the taste” while a mushroom expert says “If you eat that, it will kill you” the mushroom expert confirms my broad conclusion because we both agree the mushroom will have some kind of effect on you.

Not only does the L.A. Times cover education the same way it covers child welfare – sloppily – it also uses the same kind of feeble knee-jerk defense. There’s more about the parallels in my organization’s child welfare blog here: http://bit.ly/hxjFUw

Richard Wexler
Executive Director
National Coalition for Child Protection Reform
http://www.nccpr.org

[Mike Rose]

I grew up with The Times, know and admire people on the paper, and have been fortunate to contribute to its opinion page. But I have to ask: What is happening at The Times? With each article in Value-Added methods it gets deeper and deeper into a mess of its own making and displays further hubris or ignorance.

Mr. Lauter claims that the big public policy question people have been arguing since The Times published its first value-added article is whether "teachers have a significant impact on what their students learn or whether student achievement is all about demographics." This statement is a little like claiming that a recently published article on, let's say, a lawsuit against a tobacco company got the public arguing about the link between tobacco and lung disease.

The complex web of issues involving parental income, student achievement, and the role of the teacher has been a central topic in education research and practice for a very long time and in many countries. I was writing about it over two decades ago, and a half-century worth of work preceded mine. To say that The Times study (or the one from Colorado) finds that the quality of the teacher matters, is perhaps news to Mr. Lauter, but not to others who know about schools.

In its eagerness to justify itself, The Times staff resorts to parsing out the details of The Times and Colorado studies -- something, by the way, that journalists always accuse researchers of doing -- but miss the big point. The paper acknowledges but finally sets aside the well-known fact that Valued-Added scores are not stable, and that the results one gets will significantly depend on the models and assumptions one uses. Given the way these measure work, it is unlikely that the instability will be refined away. These problems, therefore, will have a marked effect on the accuracy of the evaluation of teachers -- including the vast range in the middle -- and on their employment and career advancement. Furthermore, just as important, the use of this problematic method will affect the nature of the profession and recruitment into it.

I cringed at the cheap insinuation that the Colorado study is influenced by the source of some of its funding. Shall we consider the vested interest of Mr. Lauter, Mr. Felch, etc. in this project? Or the fact that Thomas Kane, who Mr. Lauter approvingly quotes, is a high-level official at the Gates Foundation, overseeing a project which has invested heavily in Value-Added methods? The point is that there are all kinds of personal, professional, and institutional investments in this debate, so if you're going to lay them out, lay them all out. And if you suspect a biasing influence, do the reporter’s job of demonstrating it.

But the big, big question for me is how is it that this newspaper moved so strongly toward advocating a particular technology in school reform? The Times is not just editorializing that we need reform, but within its news department is taking a side on a technique. The paper is no longer reporting the news, but creating it and spinning it.

Here's a thought experiment. The PSA test for prostate cancer has many supporters and detractors in the medical community. The Times has run excellent articles reporting on this debate. Imagine, though, if the Times decided to take a side in it, and hired a medical researcher to run a study for The Times in support of its stance. Preposterous, right?

But that is analogous to what The Times has done.

Mike Rose
Social Research Methodology Division
UCLA Graduate School of Education and Information Studies

[Tim Jones]

Before my job as a teacher, journalism was my trade where a very wise editor once said, "statistics were numbers in search of an argument." So true. Funny. No statistic found in the LA Times tells the value I've added to students by teaching them music, art, or history. Where are the statistics about those subjects?

Only my English and math scores are value-added material. Plus, if my lower grade colleagues, were brilliant or baffoons my "value-added" score would also include a reflection of their talents possibly eclipsing mine. Some students are team taught. Where is that data separated and reported? It's not.

It's a shame and a sham the LA Times has lowered its standards to make up and report junk news based on statistics looking for trouble. Having been a journalist and a teacher, its my opinion the LA Times doesn't have value in reporting educational issues and should be dismissed in this category of reporting.

[Hannes Minkema]

How many brain cells does it take to understand the immorality of publishing personal assessments of teachers that are mainly artefacts of the applied statistical method?

Apparently more brain cells than some LA Times journalists possess.

The LA Times should know better than to undermine the integrity and safety of the very people who stick out their necks to offer America's children a proper education.

Hannes Minkema
Amsterdam, The Netherlands