Netflix awards $1-million prize to recommendation wizards, announces 2nd contest

This article was originally on a blog post platform and may be missing photos, graphics or links. See About archive blog posts.

It took three years and 40,000 teams from 186 countries, but the $1-million Netflix prize has finally been awarded.

The competition, first launched in October 2006, asked engineers and scientists around the world to solve what might have seemed like a simple problem: improve Netflix’s ability to predict what movies users would like by a modest 10%.

Credit: Paul Sakuma / Associated Press

As it turned out, it took an international brain trust led by a pair of AT&T Labs researchers nearly 36 months to reach that milestone. The team -- called ‘BellKor’s Pragmatic Chaos’ and made up of researchers and computer scientists from the United States, Austria, Canada and Israel -- turned in the winning algorithm less than 30 minutes before the contest’s deadline in July. The submission improved Netflix’s existing system by 10.06%, just enough to secure the prize.

An explanation of the algorithm by AT&T said their researchers had won the contest by making advances in two technical fields of taste prediction. With ‘neighborhood modeling,’ the scientists, including Robert Bell and Chris Volinsky, improved methods for finding films that share certain characteristics. For example, the neighbors of ‘X2’, (The sequel to ‘X-Men’) might be ‘Spiderman 2’ -- another comic book sequel -- or ‘Wolverine,’ which also starred Hugh Jackman; or even ‘Valkyrie,’ the latest feature film from ‘X2’ director Bryan Singer.

The team’s second approach dealt with what the researchers called ‘latent factors.’ These are movie characteristics that are identified mathematically rather than by human evaluation. In an article in IEEE Spectrum magazine explaining their entry, the team described latent factors this way:

Because the factors are determined automatically by algorithms, they may correspond to hard-to-describe concepts such as quirkiness, or they may not be interpretable by humans at all. ... The model may use 20 to 40 such factors to locate each movie and viewer in a multidimensional space. It then predicts a viewer’s rating of a movie according to the movie’s score on the dimensions that person cares about most.

The capacity to guess user preferences has been a major theme of both artificial intelligence and online retailing for most of this decade. In addition to Netflix, sites such as Amazon.com, Digg.com and the music site Pandora feature oft-used ‘recommendation engines,’ and heaps of other sites are in the taste-guessing business as well.

“Right now, we’re driving the Model T version of what is possible,’ in recommendations, said Netflix Chief Executive Reed Hastings in 2006, when the prize was launched. ‘We want to build a Ferrari and establishing the Netflix Prize is a first step.”

Knowing that it would take more than a 10% improvement to make a Model T into a Ferrari, Netflix has already launched a second contest. This time entrants have been asked to predict the tastes of customers who have not taken the time to actually rate movies. Instead, the predictions will be based on demographic information -- age, gender, ZIP Code -- as well as the movies users have rented in the past. Netflix will award $500,000 to the team that has built the best algorithm after six months, and another $500,000 after 18 months.

-- David Sarno