Is Your Data Vision 20/20?

For government to see data clearly, with perfect 20/20 vision, first you have to remove the barriers that prevent analysts from accessing and using big data.

In fact, most of the public sector operates with significant blind spots when it comes to big data, said Webster Mudge, senior director of technology solutions at Cloudera. Particularly for agencies, it can be difficult to get at all the details because they are stored on different systems, some online, some offline and across several different departments, if not divisions.

Building an operational data store (ODS), Mudge said, with open source Apache Hadoop as part of an enterprise data hub can be an effective approach for data optimization, especially when it comes to budget and strategy. With a Hadoop-powered ODS in place, an agency can consolidate, manage and see through the vast amounts of data that it collects.

Optimized data processing also means that government can more readily see the bigger picture through big data. In an interview, Mudge explains how an ODS can help lead government down a clear path toward more efficient and innovative work.

Why is there a need for an operational data store (ODS) in government?

The public sector is facing a number of challenges and transformations. First, most current IT systems are largely inflexible and disjointed, so there’s a real need for data visibility and analysis in government. Second is that there’s a need for contiguous security and compliance. We’re seeing a change in security approach. The old way of doing data security — command and control — is faltering. Security is inverting, really. The strategy is shifting to “trust, but verify.” There’s just so much data out there that the old ways of protecting it just don’t work anymore. Third it’s all about funding and economics. You’re being forced to maximize your infrastructure on a limited budget. An ODS is one way that you can bring technology innovation to address these challenges. But in the bigger context, this is about disruptive technology and how organizations can take advantage of this change. An operational data store built for the rigors of big data is essential in today’s government. It will help agencies complete their mission.

How do you see an ODS working within state and local governments?

A Hadoop-powered ODS is really about improving your data visibility and analysis. It helps you to target processing and access inefficiencies and constraints and allows you to get to data that you weren’t able to get to before. That’s the critical piece. As mission and business continue to demand a growing body of disparate data, you will need to process, cleanse, transform and present it to your end user at greater speeds. Traditional ODS systems run into issues when trying to do this for large, diverse data volumes. The results? Operations get backed up, data gets archived and ETL/ ELT processes fail.

An ODS built on an enterprise data hub eliminates the ‘where’ and the ‘how’ questions for reaching data analytics and accelerating discovery and time-to-action. It is ultimately how you optimize data processing and is your foundation. And that’s what a Hadoop-powered ODS offers organizations – a unified landing zone that allows for large volumes of disparate data to be transformed, cleansed and served in extremely short order. You get access to larger volumes of data at greater speeds for operational and analytical purposes, yet you continue to store data without having to archive it.

Are there particular parts of the public sector where an ODS is needed right now?

Like industry, what we’re seeing in government is that there are certain kinds of workloads that need the depth and variety and processing of an operational data store. But the workloads are broad. One thing that ODS is not: It’s not uniform. It can meet a variety of needs, so we see it used to remedy missed SLAs due to ETL bottlenecks and helping analysts look at not just one year, but seven or even 10-plus years of data, for instance.

Can you give an example where an ODS helped in data optimization?

An early example that we’ve had in the private sector was the auto insurance industry. One company that has used this is Allstate. They adopted an ODS to crunch risk profiles on a state-by-state basis. To get the full gamut of risk reports, it used to take Allstate almost two months to complete. Now, with an enterprise data hub used as their ODS in place, they can run the reports on all 50 states in less than 16 hours.

So this sounds like an efficiency tool. Is that how you look at it?

The enterprise data hub architecture is all about efficiency. It has the open-source software framework Apache Hadoop as its core. Yet there’s a big ecosystem of databases and analytical applications that already exist in your computing environment. In the end, an enterprise data hub is about maximizing those investments and using that data to see what it can do for you. Government, horizontally across all the agencies, has a lot of data. Using an enterprise data hub as a highly efficient ODS, the public sector has a unique opportunity to tap into a full range of data, discover new correlations with the data and, finally, to operationalize those findings.

About Cloudera

Cloudera is revolutionizing data management with the first unified platform for big data, an enterprise data hub built on Apache Hadoop™. Agencies now have a central, secure, and cost-efficient place to store and analyze all their data, empowering them to derive new insights and correlation while extending the value of existing investments. Cloudera was the first and still is the leading provider and supporter of Hadoop for the public sector. Government organizations can tackle their mission critical data challenges and goals including storage, cloud, security, management, and analysis with Cloudera and its more than 1,300 hardware, software, and services​ partners. Visit: clouderagovt.com.

This content is made possible by our sponsor. The editorial staff of Nextgov was not involved in its preparation.