recommended reading

Commentary: How to net big gains from big data

A recent MeriTalk survey of federal information technology professionals reinforces how harnessing reams of agency data can go a long way to help decision-makers. In the report, “The Big Data Gap,” 59 percent of respondents noted the biggest gain from managing big data is efficiency, followed by speed and accuracy of decisions (51 percent) and the ability to forecast challenges and resources (30 percent).

Unlocking operational data is becoming more critical -- and challenging -- as agencies struggle to balance a growing demand for quick information with tight budgets, aging hardware and legacy systems. Agencies are realizing that building a big data capability in-house would consume significant amounts of time and money. While 64 percent of IT professionals surveyed said their agency’s data management system could be easily expanded or upgraded, they estimated it would take an average of 10 months to double their capacity. That doesn’t even include the costs involved in new or upgraded IT infrastructure and training staff to manage and analyze the data.

An alternative approach is to leverage years of private sector investment in research and development. Technology companies can help federal managers analyze their organizations’ data to find opportunities for cost savings, operational efficiencies, informed decision-making and data transparency.

As IT providers address early-stage concerns about storing big data in the cloud --including access controls, privacy and security -- the concept is increasingly attractive to agencies.

The General Services Administration, for example, recently moved the USASpending.gov website to a big data cloud provided by technology services provider GCE. USASpending.gov is a one-stop destination for federal procurement information, such as grants, contracts, loans and payments. As one of the most widely used sites for tracking agencies’ buying habits, USASpending.gov is a vital mechanism for ensuring government transparency. It was a prime candidate for a big data platform since it incorporates analytics, search tools and large data feeds, and serves an extensive user base of citizens, media professionals, lawmakers and resellers.

USASpending.gov’s transition to a big data cloud yields several valuable lessons for agencies looking to analyze and manage data more effectively.

The Right Questions

In the current budget climate, it is unrealistic to address big data needs by investing in massive in-house systems. The cloud has made it easier for agencies to shed weighty IT infrastructure in favor of vendor-provided services.

But there is a learning curve in collaborating with companies on big data projects. Agency managers and contracting officers, in some cases, can look to industry experts on how to construct effective requests for proposals based on what has worked and what hasn’t. At the same time, they should coordinate with subject matter experts inside the agency to determine what information they want from big data tools. Asking the right questions will ensure the product can deliver the right answers.

The Optimal Starting Point

A big data journey begins with identifying the organization’s needs and opportunities. Leaders must determine which areas they can derive the most value from having a new way to analyze data. Once the starting point is identified, the following steps are essential to weaving big data into existing infrastructure:

  • Start small. Build a pilot program in the cloud, where a subset of users can test out an analytical tool. Enabling users to see how their data behaves on a certain platform drives better decisions about what data is most valuable.
  • Start clean. Determine whether the data is in a condition that is useful. Is cleansing required before transitioning to a big data framework? Data quality is hard to measure in absolutes and should be evaluated in relation to the community using it. Develop a thorough understanding of what people want out of the data, rather than simply building something and force-feeding it to them. Recognize the needs of internal and external audiences. If the big data tool will be used exclusively by an internal audience, an agency can probably get away with a few warts as long as the functionality is there and users see value. If the data will be exposed to external audiences, the bar is far higher.

All Clouds Are Not Created Equal

After deciding to move forward in the big data cloud, IT managers must understand their options and determine which provider aligns with their needs. Agencies might be tempted to exert more control through a proprietary product, but there are significant benefits to open source software like Hadoop.

With open source technologies agencies can avoid being locked into the flavor of the month, which requires painful extraction if leaders choose a different direction. Based on a Hadoop framework, for example, products from various vendors can be easily right-sized and computing capacity can be scaled up and down as needs change. In such an environment, computing power can be five to 10 times faster than mainstream disk and switch technologies.

Agencies should seek out experienced vendors with knowledge of government data and the applicable requirements, attributes and policies. This will ensure agency and contractor officials have a common understanding and speak the same language.

For all the rhetoric surrounding big data, there are real projects delivering significant benefits to agencies. As the federal government grows comfortable leveraging investments, expertise and resources from the private sector, more agencies will be able to weave a big data cloud service into their existing IT infrastructure.

Ray Muslimani is chief executive officer at GCE, which provides cloud and big data services.

Threatwatch Alert

Thousands of cyber attacks occur each day

See the latest threats

JOIN THE DISCUSSION

Close [ x ] More from Nextgov
 
 

Thank you for subscribing to newsletters from Nextgov.com.
We think these reports might interest you:

  • Modernizing IT for Mission Success

    Surveying Federal and Defense Leaders on Priorities and Challenges at the Tactical Edge

    Download
  • Communicating Innovation in Federal Government

    Federal Government spending on ‘obsolete technology’ continues to increase. Supporting the twin pillars of improved digital service delivery for citizens on the one hand, and the increasingly optimized and flexible working practices for federal employees on the other, are neither easy nor inexpensive tasks. This whitepaper explores how federal agencies can leverage the value of existing agency technology assets while offering IT leaders the ability to implement the kind of employee productivity, citizen service improvements and security demanded by federal oversight.

    Download
  • Effective Ransomware Response

    This whitepaper provides an overview and understanding of ransomware and how to successfully combat it.

    Download
  • Forecasting Cloud's Future

    Conversations with Federal, State, and Local Technology Leaders on Cloud-Driven Digital Transformation

    Download
  • IT Transformation Trends: Flash Storage as a Strategic IT Asset

    MIT Technology Review: Flash Storage As a Strategic IT Asset For the first time in decades, IT leaders now consider all-flash storage as a strategic IT asset. IT has become a new operating model that enables self-service with high performance, density and resiliency. It also offers the self-service agility of the public cloud combined with the security, performance, and cost-effectiveness of a private cloud. Download this MIT Technology Review paper to learn more about how all-flash storage is transforming the data center.

    Download

When you download a report, your information may be shared with the underwriters of that document.