Daily Dish

The inside scoop on food in Los Angeles

« Previous Post | Daily Dish Home | Next Post »

Wine Spectator reveals Top 100 Wines of 2009, but... are all wine rating systems flawed?

Photo: Diana Hirst, general manager of Hi-Time Wine Cellars in Costa Mesa, with a bottle of 2005 Araujo Cabernet that retails for $265. Credit: Don Bartletti / Los Angeles Times.

Now that Wine Spectator has finished dragging out the reveal of its Top 100 Wines of 2009 -- a 2005 Columbia Crest Cabernet Sauvignon was ranked No. 1 -- over a yawn-inducing three days, we have to ask: Are wine ratings an accurate and useful guide for consumers? Or are they merely a series of wildly subjective impressions based more on context and expectation than the actual qualities of the wines? That's the question Leonard Mlodinow explores in a recent Wall Street Journal story, "A Hint of Hype, A Taste of Illusion."
Given the high price of wine and the enormous number of choices, a system in which industry experts comb through the forest of wines, judge them, and offer consumers the meaningful shortcut of medals and ratings makes sense.

But what if the successive judgments of the same wine, by the same wine expert, vary so widely that the ratings and medals on which wines base their reputations are merely a powerful illusion? That is the conclusion reached in two recent papers in the Journal of Wine Economics.

He's referring to findings published by Robert Hodgson, a retired statistics professor and the proprietor of Fieldbrook Winery. A few years ago, Hodgson joined the California State Fair wine competition advisory board, which allowed him to run a controlled scientific study of its tastings.

The results, published in the Journal of Wine Economics, showed that the judges' ratings varied by ±4 points on a standard 100-point rating scale. And "only about one in 10 [judges] regularly rated the same wine within a range of ±2 points."

In September in the private wine newsletter the California Grapevine, Hodgson discussed his analysis of the complete records of several wine competitions. "The distribution of medals," he wrote, "mirrors what might be expected should a gold medal be awarded by chance alone.'" Ouch. Many winemakers feel vindicated by Hodgson's findings, while wine experts generally dismiss them.

If you want to try a similar, albeit much less scientific, experiment on your own, serve a group of friends the same wine in different bottles: first as a humble table wine and then as an expensive, award-winning wine. You'll likely get a very different set of responses. When we know that something is highly valued by others, we tend to value it more highly. When we know that something is expensive, we're more inclined to enjoy it. Almost nobody is immune from that. Apply that knowledge to the powerful tastemakers of the multi-billion-dollar wine industry, and where does that leave wine buyers?

At recently opened wine shop Domaine LA on Melrose near Highland Ave., owner Jill Bernheimer, who has run an online wine store for years, holds an annual contest for consumers to guess Wine Spectator's No. 1 zinfandel. The winner receives a six-pack of wines that are rated 85 points or less, "just to prove a point," she says. Namely, that there are plenty of underrated wines that are worth drinking. For Bernheimer, it's about figuring out whose palate you trust.

"You have to start somewhere and the easiest place is reading wine publications and scores," she says. "But as people gain more experience and confidence in themselves, the best thing consumers can do is find a sommelier, a retailer or a critic who they feel they share a palate with. That can often be a better guide than scores on their own."

--Elina Shatkin

--Photo: Diana Hirst, general manager of Hi-Time Wine Cellars in Costa Mesa, with a bottle of 2005 Araujo Cabernet that retails for $265. Credit: Don Bartletti / Los Angeles Times
Comments () | Archives (6)

The comments to this entry are closed.

How wonderful that an affordable wine, "Columbia Crest Cabernet Sauvignon" was named Top Wine in the World. So often these finest wines are out of the normal consumer's price range !

The thing about such wide ranging scores (8 points) is that it can make or break a wine and winery. That kind of difference, let's say between a 96 point wine and the same wine being given 88 points can mean the difference of thousands of cases of wine sales and hundreds of thousands of dollars price-wise too.

Over in the beer world, the American Homebrewers Association has a 50 point rating scale. It breaks the total rating down into several very short scales judging for color, flavor, smell, etc. Guess what? Consistent ratings across judges, and from the same judge on the same beer.

A 100 point scale may make you feel good when you get 90points, but it just implies a false precision. Who can really tell an 89 point wine from a 90 or 91 point wine with any consistency? Get real about the rating method and stop fooling yourself. http://www.bjcp.org/docs/SCP_BeerScoreSheet.pdf

Wait a minute with the debunking. The rating of different judges for the same wine can vary as much as +/-4 points (i.e., 8 points range)? Judges who repeatedly taste the same wine in a blind tasting vary as much as 2 points (a four point range) ?

If this proves anything, it proves either that the judges have been tipped off or that they do have an accuracy of judgment approaching that of God himself.

It is notorious that different teachers grading the same paper can grade it from an A to an F (roughly 100 - 60, a range of +/- 20 points). Anyone who has ever graded a paper knows that to come within 4 points out of 100 on a second reading is near miraculous.

Why doesn't some statistician start talking about how remarkably similar these wine judges' results are?

John D. Madden

I don't understand the relationship of the photo with the story.

You omitted the x factor in all of this ratings jive: the out and out favoritism, politicking, yea, even corruption that figures into the process. Just as the music trade papers are perennially infected by advertising money, leading them to "juice" the chart numbers on selected songs, so is the wine biz similarly rife to influence. Objectivity? Leave that to the white coats -- anytime it's humans judging their peerage, look out for the action under the table as well as in the glass.


Recommended on Facebook


In Case You Missed It...


Recent Posts
5 Questions for Thi Tran |  August 6, 2012, 8:00 am »
SEE-LA hires new executive director |  July 31, 2012, 9:34 am »
Food FYI: Actors reading Yelp reviews |  July 31, 2012, 9:16 am »
Test Kitchen video tip: Choosing a bread wash |  July 31, 2012, 6:04 am »



About the Bloggers
Daily Dish is written by Times staff writers.