Gmail still down for some users; bug in software storage update blamed
For a third day, Google is working on fixing problems with its Gmail service that have left some users without access to their accounts and undelivered messages.
In a Monday night blog post, Google addressed the problems, explaining that a software storage update contained a bug that has caused the outages.
Google also assurred that no e-mails have been lost forever -- just temporarily.
The problems were first reported by Google and Gmail users on Sunday.
"Imagine the sinking feeling of logging in to your Gmail account and finding it empty," said Ben Treynor, Google's vice president of engineering and site reliability, said in the blog post.
"That's what happened to 0.02% of Gmail users yesterday, and we're very sorry.
"The good news is that email was never lost and we've restored access for many of those affected. Though it may take longer than we originally expected, we're making good progress and things should be back to normal for everyone soon."
Google say service outages such as the one it's experiencing now are rare -- particularly because it saves every file that a user has in Gmail (or any of its other cloud-based services) in multiple data centers.
"In some rare instances software bugs can affect several copies of the data," Treynor said. "That's what happened here. Some copies of mail were deleted, and we've been hard at work over the last 30 hours getting it back for the people affected by this issue."
"To protect your information from these unusual bugs, we also back it up to tape. Since the tapes are offline, they're protected from such software bugs. But restoring data from them also takes longer than transferring your requests to another data center, which is why it's taken us hours to get the email back instead of milliseconds."
Google said it halted its software storage update as soon as it realized there was a bug that was causing the deletion of e-mails and reverted to an older version of the storage software.
Treynor said, for those affected by the problems, "it's important to note that email sent to you between 6:00 PM PST on February 27 and 2:00 PM PST on February 28 was likely not delivered to your mailbox, and the senders would have received a notification that their messages weren’t delivered."
-- Nathan Olivarez-Giles