Amazon cloud crash endangers federal websites

Parts of Amazon's cloud computing service crashed Wednesday night, bringing down or disrupting several popular websites and endangering some federal sites that purchase space in the cloud.

By Thursday afternoon, the only federal site affected by the crash was an Energy Department website devoted to sharing clean energy practices with industry. A note on the site said it was "temporarily down" because of the Amazon crash and that engineers were "working aggressively to restore service."

Recovery.gov, which tracks money spent on President Obama's stimulus plan, was located inside the affected cloud -- the Amazon Web Service EC2 cloud in Northern Virginia -- but was engineered to jump to another location as soon as it spotted trouble and so service was never disrupted, a spokesman for the Recovery Accountability and Transparency Board, said on Thursday.

The EC2 cloud also hosts other federal websites that were operating smoothly on Thursday, including Sigtarp.gov, which tracks money spent on the 2008 bank bailout, and several other Treasury Department sites.

It wasn't clear whether those sites had also jumped to another cloud or whether they were in an uninfected part of the EC2.

A Treasury spokesman said no problems had been reported with any of the department's websites between Wednesday and Thursday.

Amazon engineers still hadn't located the source of the crash by 5 p.m. Thursday and it was still causing trouble for several high traffic websites, including reddit.com, a popular site for ranking other web content, which was operating in an "emergency read-only mode."

Quora.com, a popular question-and-answer site was still completely down at 5 p.m.

Amazon did not respond to Nextgov requests for comment. Other federal agencies that rent space in the Amazon cloud also did not respond to requests for comment, including the Energy Department.

Cloud computing consists of large blocks of privately owned and managed server space that entities can rent out on an as-needed basis. Renting cloud space has become increasingly popular in recent years because it saves businesses the cost of building their own data centers and allows them to ramp up or down the amount of server space they use without paying for the unused space.

Migrating large amounts of federal data and services to the cloud is a major part of the Obama administration's 25-point plan to reform how the government uses information technology and is expected to help the government save millions of dollars and reduce its 2,100 federally owned data centers by more than a third by 2015.

Darrell West, founding director of the Brookings Institution's Center for Technology Innovation, called the Amazon crash a "freak occurrence" and told Nextgov he doesn't expect federal agencies to rethink their move to the cloud as a result.

"This crash is unusual because the cloud industry has had a very strong record of service reliability," West said. "Indeed, that's one of the strong points of cloud storage. So this is unusual, one, that it happened at all, and, two, that Amazon has had difficulty figuring out what the problem is."

West said he thinks it's unlikely any data was lost in the crash because cloud providers have extensive redundancies and backup systems to protect stored information.

The EC2 Amazon cloud houses the Treasury Department sites MyMoney.gov, TIGTA.gov, and IRSOversightBoard.treasury.gov, as well as the website for a State Department video contest aimed at promoting intercultural understanding, all of which were operating smoothly on Thursday.

The EC2 cloud also stores some geographic information systems mapping services for the Agriculture Department and some software NASA uses to process high-resolution satellite images, according to company statements.