‘Catastrophic’ Power Outage Nearly Destroyed MSPB’s Electronic Records System

A little-reported but “catastrophic” power outage that hit the Merit Systems Protection Board staff June 30, 2015, has prompted a series of reforms to the federal employee complaint adjudication agency’s in-house electronic records system.

Details are outlined in an October 2015 consultant’s report recently obtained by Government Executive under the Freedom of Information Act, supplemented by comment this week from MSPB’s acting chief information technology officer.

“The catastrophic failure of the entire virtual environment on 30 June 2015, a key component to the e-Adjudication strategy, and the resulting loss of data, configuration and confidence of the user community has halted much of the e-Adjudication strategy in the near term,” wrote IT auditors from Cask LLC in the Oct. 30, 2015, report after assessing the MSPB’s Information Resource Management team’s efforts to move to a paperless records system.

The consultants interviewed staff and analyzed the agency’s IT infrastructure, virtualization strategy and operational processes and procedures, making 42 specific observations and recommendations. Among the observations:

• Basic system and network security practices had not been implemented, “leaving the organization open to multiple vulnerabilities,” the report said: “MSPB has no safeguards in place to prevent an unauthorized user from plugging a random laptop or other device into the network."
• The guest Wi-Fi access point was tied into the headquarters production network, thus “allowing anyone with the proper tools on their laptop to get an accurate map of all devices on the network, opening up the organization for further malicious intrusions.”
• Operational processes weren’t documented, leaving the infrastructure vulnerable to failures, the report said: “The lack of documentation and independent configuration backups prior to the virtual environment failure set back the [virtual desktop infrastructure] implementation a number of months.”
• Analysts found that key network security and management processes weren’t in place, and there were “significant technical obstacles” that probably warranted outside professional services, along with “significant organizational acceptance obstacles.”

Among the recommendations was that MSPB find the funding to bring in a third-party IT services provider and increase its help desk hours (staff are located across several time zones).

“Some users have gotten into the habit of bypassing the help desk and calling technicians directly,” the report said. “This is not best practice and reduces visibility across the enterprise.”

Overall, however, “upon consideration of all of the organizational, process and technology assessment observations, we conclude that although there are significant obstacles, with sufficient resourcing IRM can meet the vast majority of e-Adjudication goals,” Cask concluded.

In a statement to Government Executive in late November, acting Chief Technology Officer William Spencer said MSPB has completed 60 percent of the Cask report recommendations.

“We continue to follow up on the remaining recommendations. Some of those require substantial effort, e.g., updating our core business applications,” he said. The agency is actively pursuing two internal goals that align with the Cask recommendations, Spencer wrote in an email: “Improving the stability and reliability of our IT environment; and modernizing our core business applications and migrating our data center to the cloud.”