Examiners trying to access files stored in the cloud experienced significant delays.
The Veterans Affairs Department’s $491 million disability claims processing system, which is used in 18 out of 57 regional offices, slowed to a crawl last month due to sluggish response times for claims examiners trying to access veterans’ files stored on remote servers, Nextgov has learned.
The department plans to deploy its Veterans Benefits Management System to 14,400 claims examiners at the Veterans Benefits Administration this year. But last month only 721 examiners -- just 5 percent of the nationwide total -- were using the system, according to internal emails obtained by Nextgov.
VBMS slowed when claims examiners tried to perform simple transactions with claims files, such as search, update, save or retrieve, one email said. VA’s Office of Information Technology determined sluggish response times resulted from “the way data is being read from disk storage. Since the impact was considerably more read/write work for each transaction, it had a greater impact as more users attempted to perform work on VBMS,” an email said.
VA Chief Information Officer Roger Baker acknowledged the problems with VBMS in an email to Nextgov. “In mid-December, volume of VBMS usage grew rapidly as users from the 18 [regional offices] were added. VBMS began to experience dramatic slowdowns in response time for some users, especially during peak usage hours. This also affected the customer relationship management (CRM) application, a new desktop application for all VBA call center users that uses the same physical infrastructure as the VBMS system,” Baker said.
VBMS is a Web-based application that accesses files stored on remote servers. It supports other programs in a shared infrastructure “meaning other applications experienced the same issue,” the internal email said. Department officials attempted to alleviate what one VBA official termed “the impact of excessive read write activity” by removing “all non-essential, non-productive activities for the shared VA environment.”
Shutting down non-essential work “will improve VBMS performance [but] it should not be misconstrued as a singular remedy [emphasis included],” the official said, adding that the Office of Information Technology needs to develop a plan to permanently fix the problem as VA rolls out VBMS nationwide.
Baker said his staff determined that the slowdown did not result from read/write problems but from an inefficiently written service call that is part of a system benefits enterprise platform (BEP) on which VBMS and CRM both rely. He said “Once the root problem was identified, we added hardware overnight to increase capacity on BEP, and re-coded and tested the service call over the next week to operate more efficiently. As a result, VBMS is operating normally in production at this time.”
Baker added, “VBMS is constantly evolving to meet end-user needs, business requirements, and performance benchmarks. Many software optimizations have been put in to reduce end-user wait times to include: client and server side caching, optimized database queries, Wide Area Network Accelerators, compression. Additionally, upgraded hardware (servers, network equipment) has been installed to improve both VBMS and VBMS dependencies through the VA enterprise.”
Last June, Richard Dumancas, deputy director of the American Legion’s Claims, Veterans Affairs and Rehabilitation Commission, warned of latency problems with VBMS when it was deployed to only four regional VBA offices.
He told a hearing of the House Veterans Affairs Committee that the initial experience with VBMS shows some delays in opening files that cumulatively could create significant delays over the course of a workday. “These lag issues are showing up with a relatively small number of users in pilot sites, and when the whole system goes nationwide, system demands will presumably be far greater.”
Paul Sullivan, a Gulf War veteran, former VBA claims examiner and now managing director for public affairs and veteran outreach at law firm Bergmann & Moore LLC in Bethesda, Md., said the problems with VBMS with only 5 percent of claims examiners online shows the system “is not ready for prime time.”
The VA’s disability claims backlog hit 900,121 at the end of 2012, with 608,365 -- 68 percent -- stuck in the system more than 125 days. VA Secretary Eric Shinseki views VBMS as key to eliminating the backlog by 2015,
VA has farmed out much of the work on VBMS to the Space and Naval Warfare Systems Center Atlantic in Charleston, S.C., Nextgov reported Wednesday.