recommended reading

Commentary: How to net big gains from big data

A recent MeriTalk survey of federal information technology professionals reinforces how harnessing reams of agency data can go a long way to help decision-makers. In the report, “The Big Data Gap,” 59 percent of respondents noted the biggest gain from managing big data is efficiency, followed by speed and accuracy of decisions (51 percent) and the ability to forecast challenges and resources (30 percent).

Unlocking operational data is becoming more critical -- and challenging -- as agencies struggle to balance a growing demand for quick information with tight budgets, aging hardware and legacy systems. Agencies are realizing that building a big data capability in-house would consume significant amounts of time and money. While 64 percent of IT professionals surveyed said their agency’s data management system could be easily expanded or upgraded, they estimated it would take an average of 10 months to double their capacity. That doesn’t even include the costs involved in new or upgraded IT infrastructure and training staff to manage and analyze the data.

An alternative approach is to leverage years of private sector investment in research and development. Technology companies can help federal managers analyze their organizations’ data to find opportunities for cost savings, operational efficiencies, informed decision-making and data transparency.

As IT providers address early-stage concerns about storing big data in the cloud --including access controls, privacy and security -- the concept is increasingly attractive to agencies.

The General Services Administration, for example, recently moved the USASpending.gov website to a big data cloud provided by technology services provider GCE. USASpending.gov is a one-stop destination for federal procurement information, such as grants, contracts, loans and payments. As one of the most widely used sites for tracking agencies’ buying habits, USASpending.gov is a vital mechanism for ensuring government transparency. It was a prime candidate for a big data platform since it incorporates analytics, search tools and large data feeds, and serves an extensive user base of citizens, media professionals, lawmakers and resellers.

USASpending.gov’s transition to a big data cloud yields several valuable lessons for agencies looking to analyze and manage data more effectively.

The Right Questions

In the current budget climate, it is unrealistic to address big data needs by investing in massive in-house systems. The cloud has made it easier for agencies to shed weighty IT infrastructure in favor of vendor-provided services.

But there is a learning curve in collaborating with companies on big data projects. Agency managers and contracting officers, in some cases, can look to industry experts on how to construct effective requests for proposals based on what has worked and what hasn’t. At the same time, they should coordinate with subject matter experts inside the agency to determine what information they want from big data tools. Asking the right questions will ensure the product can deliver the right answers.

The Optimal Starting Point

A big data journey begins with identifying the organization’s needs and opportunities. Leaders must determine which areas they can derive the most value from having a new way to analyze data. Once the starting point is identified, the following steps are essential to weaving big data into existing infrastructure:

  • Start small. Build a pilot program in the cloud, where a subset of users can test out an analytical tool. Enabling users to see how their data behaves on a certain platform drives better decisions about what data is most valuable.
  • Start clean. Determine whether the data is in a condition that is useful. Is cleansing required before transitioning to a big data framework? Data quality is hard to measure in absolutes and should be evaluated in relation to the community using it. Develop a thorough understanding of what people want out of the data, rather than simply building something and force-feeding it to them. Recognize the needs of internal and external audiences. If the big data tool will be used exclusively by an internal audience, an agency can probably get away with a few warts as long as the functionality is there and users see value. If the data will be exposed to external audiences, the bar is far higher.

All Clouds Are Not Created Equal

After deciding to move forward in the big data cloud, IT managers must understand their options and determine which provider aligns with their needs. Agencies might be tempted to exert more control through a proprietary product, but there are significant benefits to open source software like Hadoop.

With open source technologies agencies can avoid being locked into the flavor of the month, which requires painful extraction if leaders choose a different direction. Based on a Hadoop framework, for example, products from various vendors can be easily right-sized and computing capacity can be scaled up and down as needs change. In such an environment, computing power can be five to 10 times faster than mainstream disk and switch technologies.

Agencies should seek out experienced vendors with knowledge of government data and the applicable requirements, attributes and policies. This will ensure agency and contractor officials have a common understanding and speak the same language.

For all the rhetoric surrounding big data, there are real projects delivering significant benefits to agencies. As the federal government grows comfortable leveraging investments, expertise and resources from the private sector, more agencies will be able to weave a big data cloud service into their existing IT infrastructure.

Ray Muslimani is chief executive officer at GCE, which provides cloud and big data services.

Threatwatch Alert

Thousands of cyber attacks occur each day

See the latest threats

JOIN THE DISCUSSION

Close [ x ] More from Nextgov
 
 

Thank you for subscribing to newsletters from Nextgov.com.
We think these reports might interest you:

  • Featured Content from RSA Conference: Dissed by NIST

    Learn more about the latest draft of the U.S. National Institute of Standards and Technology guidance document on authentication and lifecycle management.

    Download
  • PIV- I And Multifactor Authentication: The Best Defense for Federal Government Contractors

    This white paper explores NIST SP 800-171 and why compliance is critical to federal government contractors, especially those that work with the Department of Defense, as well as how leveraging PIV-I credentialing with multifactor authentication can be used as a defense against cyberattacks

    Download
  • Toward A More Innovative Government

    This research study aims to understand how state and local leaders regard their agency’s innovation efforts and what they are doing to overcome the challenges they face in successfully implementing these efforts.

    Download
  • From Volume to Value: UK’s NHS Digital Provides U.S. Healthcare Agencies A Roadmap For Value-Based Payment Models

    The U.S. healthcare industry is rapidly moving away from traditional fee-for-service models and towards value-based purchasing that reimburses physicians for quality of care in place of frequency of care.

    Download
  • GBC Flash Poll: Is Your Agency Safe?

    Federal leaders weigh in on the state of information security

    Download
  • Data-Centric Security vs. Database-Level Security

    Database-level encryption had its origins in the 1990s and early 2000s in response to very basic risks which largely revolved around the theft of servers, backup tapes and other physical-layer assets. As noted in Verizon’s 2014, Data Breach Investigations Report (DBIR)1, threats today are far more advanced and dangerous.

    Download

When you download a report, your information may be shared with the underwriters of that document.