NSA Hid Explosions, Delays at $1.2 Billion Spy Compound
NSA's Utah Data Center, shown in June, occupies 200 acres in Utah. Rick Bowmer/AP file photo
'This is why acceptance testing is done,' spokeswoman says of the Utah supercomputer center meltdowns.
While the government was nearing a shutdown and bracing for a default, a $1.2 billion National Security Agency supercomputer homestead in Utah, built to ingest surveillance data, reportedly was spending hundreds of thousands of dollars to replace fried equipment.
NSA officials masked the woes out west, at a time when the agency already was facing accusations of lying about domestic spying practices.
As recently as June 19, agency officials told Nextgov the spy center -- 15 times the size of MetLife stadium -- would be "operational" by Oct. 1. That month, NSA spokeswoman Vanee Vines also said in an email the "exterior is done. . . It's now time for IT guts to be installed,” referring to information technology systems.
But, behind the security perimeter, persistent electrical surges apparently sparked explosions and a year-long delay that NSA officials did not disclose. On Monday night, the Wall Street Journal reported there have been 10 meltdowns during the past 13 months that have prevented NSA from running computers.
The revelations come after ex-NSA contractor Edward Snowden this spring leaked files to the media detailing the agency’s mammoth collections of Internet and phone data, revelations later acknowledged by U.S. officials. The 200-acre Utah computer fortress was designed to warehouse this data.
In June, Harvey Davis, a top agency official, who handles military installations, as well as power, space and cooling for all NSA data centers, said in an interview that the construction – undertaken by the Corps of Engineers – has been an open project.
"The military construction process by design is a very, very transparent process,” he said. “It's a public discourse. When we give out our request for proposal, that's through FedBizOpps.gov," a website that advertises jobs for prospective vendors, so "by its nature, military construction is a very, very open process."
Project contract papers called for a 65-megawatt center with a chiller plant, fire suppression systems, electrical generators and an uninterruptible power supply backup capacity.
On Tuesday, NSA and Army Corps of Engineers officials declined to answer questions about the failure to disclose that there would be some sort of delay.
Norbert Suter, the Corps chief of construction operations, said in an email that power connectivity remains incomplete. "The final commissioning of the entire electrical system is still ongoing," he said, declining to address cost overruns. "The Corps is ensuring it functions as required and that it will be completely reliable prior to releasing the project to the customer,” meaning NSA. Suter did not say when the site would turn on.
He acknowledged there have been a series of glitches. "During the testing and commissioning of the Utah data center, problems were discovered with certain parts of the electrical system. Issues such as these can arise in any project, and are the reason the Corps tests and reviews every aspect of any project prior to releasing it to the customer," he said.
More Corps professionals were summoned and the source of the problems was discovered, Suter said, without elaborating further.
"Because of the scope of this project, a specialized team of experts from the Corps' headquarters was sent to review and assist with addressing these issues. The cause of the electrical issues was identified by the team and is currently being corrected by the contractor," he said.
The team also uncovered ventilation troubles. In order to function, data centers need an air management system able to cool IT equipment and remove waste heat. Suter said, "An airflow issue was discovered with the generators and the contractor is correcting the problem."
On Monday night, Vines said, "On the whole, this is why acceptance testing is done - so that issues can be found and resolved before the government takes possession."
While final assessments continue, Suter said the Corps is "making preparations to release it to the customer.”
The final tab for the ultimate customer, the taxpayer, is a question mark. Davis this summer said, "I came in under cost,” referring to $100 million in savings gained partly by refusing to let contractors adjust the plan.
This story has been updated with minor edits.