Kevin Rose and Jay Adelson on Digg ‘throttling back,’ power users and dupe detection*

This article was originally on a blog post platform and may be missing photos, graphics or links. See About archive blog posts.

Rose and Adelson at the L.A. DiggNation meetup. (Photo: David Sarno)

Digg just announced today that given the current economic climate, it’s going to be chilling out on growth plans, and maybe even reducing staff. I caught up with Digg founder Kevin Rose and CEO Jay Adelson last week at the L.A. Diggnation meetup, and asked them about the state of the San Francisco company, features in the works, the power-user problem and their take on the media landscape.

What’s been happening since the $28 million funding round last September?

Adelson: The world changed a little bit. A little bit of an economic shift. We’re looking at 2009 and 2010 as riskier years from the advertising world and so on, just like everyone else does. We originally said we were going to double our staff in 2009 and go international. We’ve definitely pulled the throttle back on a lot of that. So we’re focusing hard on getting to profitability as fast as possible so that that question is over with.

Rose: We still want to be the best social news site on the Web. That means continuing to evolve what we already have, taking feedback from the users and blending in some new features that we haven’t talked about yet.

So are you still going to move to a new headquarters?

Adelson: No, we’re not moving the company. We’re going to stay right where we are on Potrero Hill. We’re not going to double the staff -- well, we may eventually, but not right now. The idea is to be conservative about our spending like everyone else and make sure we’re as efficient as possible, but not to slow down our engineering or ability to roll out these new features.

What Digg features from 2008 are you happiest with?

Rose: The recommendation engine is the big one. We realized a long time ago that there were so many new stories per day that it was impossible for any one user to go through all those. When we first launched the site there were maybe 3-, 4-, 500,000 stories. We had some hard-core Diggers that would go through them all and find the best. And then as we grew larger people would rely on their friends networks. But soon we realized we needed a better way to comb through all the stories and present users with relevant information....

Adelson: Don’t wait for them. Be proactive and push it to them.

Rose: Exactly.

Are people using it?

Rose: Yes, Digging in the Upcoming section is up 500% since we rolled out recommendations.

What’s been going on behind the scenes?

Adelson: One of the things we did that no one really sees is that we rewrote 65- to 70% of the code base while the jet was flying, and no one really noticed.

Rose: It’s always been a challenge for us to scale the site. Getting up to 34 million unique visitors, where we are now -- staying ahead of that growth takes a whole set of engineers working on that nonstop. Finally we have an infrastructure that we can scale. That’ll let us do all these crazy geeky things like sharding that’ll allow us to scale for quite a long time.

Adelson: I think it’s a big misconception about Digg that...

... because it’s an algorithmic and automated system, that it wouldn’t take a lot of expertise to make that work. To do that at scale with hundreds and hundreds of millions page views and that many unique visitors, and the numbers of transactions per second on the database....

Rose: We’re talking hundreds and hundreds of servers just to handle the capacity for this stuff. When you’re talking about millions of votes being cast — it’s insane. It’s a really heavy, write-intensive MySQL application.

A lot of Digg users seem to complain about the high percentage of front-page stories submitted by top users. Should the average user realistically expect to be able to get a story to the front page, or should they understand that it’s way more complex than just pressing ‘submit’ and crossing your fingers?

Rose: You have to realize that out of the tens of thousands of stories that are on the Web, we only promote 100 to 130ish a day. So your shot at getting on the homepage is already pretty small. Then you have to look at, who are the people that are the taste makers and the ones that are living and breathing this stuff, and the first to find a story and submit it -– there’s a certain dedicated core group of users that are out there doing that kind of thing.

But every single day we promote people on the front page that have never had a story up there -- people who managed to find a really unique, strange article that no one else has seen before.

Adelson: Or if they break news –- there are lots of first-time promotions from people who found or submitted a story at the moment of publication. If that happens, that person might get promoted first, just because it’s the first version of that story.

But how would anyone notice the submission if it was from an unknown user?

Adelson: They see it in the Upcoming sections. There’s another section of Diggers combing through the new submissions.

Rose: They also get the recommendation based on who that user is and who they’re tied to.

Their friend network…

Rose: Yeah, but even if they don’t have a friend network, it’s linked to their ‘similar user’ network. That’s where the recommendation engine is. So let’s say you’re digging a story on the Audi R8 — and you don’t have any friends on Digg, but you love Audi R8 stories. We’re going to know that and spread that story to similar diggers, and they’ll see it when they go into the Upcoming section.

And that’s where we’ll be going in the future -– we’ll be providing a more customized view of our front page, and promotion at a different level to different interest groups. So if my mom comes in and she’s huge into cross stitch, she should have a place where she can submit a story and it’ll spread to all those similar users. Where we’re going will be a more personalized experience that everyone can participate in -– not just a certain niche like we have today –- we want to expand it into smaller niches.

Adelson in L.A., drinking it all in. (Photo: Sarno)

Any other ways you’ll be promoting upcoming content?

Rose: On the R&D side, we’re getting pretty good at predicting how popular things will eventually become. We call this our ‘prescient users’ -- these are users that are really good at digging things early on that eventually become very popular. There are people out there that just have an amazing eye for news, and they exist in all these different niches, they just don’t know it. And we want to expose those users and the stories they digg to the community.

Adelson: And these aren’t necessarily people who even submit stories. What’s really surprising is that a lot of those prescient users are actually not what people have thought of as ‘top diggers’ -– they’re different people. It’s interesting stuff.

How about the problem of duplicate story submission? People seem frustrated when they submit a story and it goes nowhere, but then a stronger user submits it later and it becomes popular.

Rose: We’re doing a big fix to this. We have a new duplicate detection algorithm that’s coming out as our first big overhaul of dupe detection –- it’ll probably come out in the next couple months. So basically what we do, at the time of submission when you first put a URL into the site, we go out and crawl the story in real time, grab the relevant “meat” of the story, and then use that in our algorithm to compare against other stories that have been recently submitted. And it’s extremely accurate. It’s awesome.

Adelson: If it works well, we may get more aggressive about enforcing [duplicate prevention] at the time of submission.

Rose: It’ll be able to throw back a set of stories instantly and say, hey by the way, this story has already been submitted -- even before you take the time of going all the way through the submission process. With this, right away, we’ll say: Before you even go to the next step, this story has already been submitted, and you can digg it right here.

But even if you solve the duplication problem, you still have the issue of the unknown guy submitting what could have been a big story, and because it doesn’t get noticed, the story kind of dies away…

Adelson: Well, so, let’s say the second [more powerful user] goes to submit it, the duplicate detector will draw his attention to the original story.

Rose: That way, he’ll see it right there and it’s like, Boom, here’s the story. You can’t submit it, but you should digg it. Then when he diggs it, we use his [user power] to propel it forward.

There are a lot of conversations about revenue sources drying up -- for old media and new. You guys depend on the fact that reading online news is basically free. What happens if some of the top entities that feed free information to Digg just dry up?

Rose: I feel that we do a really good job now of pushing a lot of attention toward certain authors and writers of content. So if we’ve done anything, it’s kind of level the playing field when it comes to different sites so we can drive just as may viewers to someone’s personal blog as we can to a top-tier news site. I’ve seen friends of mine that have worked in traditional media for a long time move to their own blogs, and they make a living doing that. And they’re extremely talented and awesome writers. Om Malik did that -- he left Red Herring for GigaOm to start doing his own thing.

Adelson: And the other side of it is that, as traditional media and print publications find that their subscribers are moving to online news consumption, I think that we’re helping these companies develop online audiences. Remember that online audiences aren’t geographically limited like traditional print audiences. So by creating these tools and these symbiotic relationship where we drive traffic to regionally specific sites, I think we’re actually helping with the online transition.

So news aggregation can help. It’s certainly been driving a lot of traffic to traditional publications -- it’s one of the top five referrers of traffic to these guys.

Rose to Adelson: Have we talked about some of the tools that we’re going to be launching for publishers?

Adelson: No, Kevin, we haven’t!

Hint maybe?

Adelson: I’ll say that we consider publishers to be our partners.

Rose: Absolutely -- anything we can do to help them, we’re willing to do.

Adelson: So look in the future for things we’ll be doing to assist them with that transition and monetize their own users.

Sounds good to me.

Corrected, 5:50 p.m.: Because of a dropped comma, a previous version of this post’s headline suggested Digg was throttling back power users. In fact, Digg is throttling back on some of its growth plans.

-- David Sarno