Big Data, Not Quick Data: Most Agencies Don’t Need Real-Time Analytics Yet

For consumer products companies, such as smart-thermostat maker Nest, the ability to immediately respond to commands, or to relay temperature in real-time, is a selling point. For health records companies, up-to-the-minute changes in a patient's electronic file could dramatically impact their quality of care.

While some industries embrace real-time data analysis, most parts of the federal government — aside from defense and security-related agencies — really don't need it for internal purposes, according to officials at the departments of Health and Human Services and Transportation. 

The federal government, and especially HHS, has amassed large volumes of data, said Damon Davis, director of the department's health data initiatives. "You automatically think the next step in data is towards being real-time — and it's not practical in all instances," he said. 

In a Q&A at an analytics conference in Washington, Davis and Dan Morgan, chief data officer at Transportation, discussed where real-time data fits into their departments' missions.

Outside of select examples — the Centers for Disease Control's real-time tracking of Ebola cases, for example — there is little need for up-to-the-minute, dynamic data collection and analysis, Davis said. 

"At a national level, it's important to recognize that while our data may be huge, it is also collected in the form of surveys and other kinds of things that are long-time aggregations of data about the American populace," he said. "It's not data we have from a real-time perspective."

Instead, he said, HHS data would be more valuable when added to predictive systems. For instance, Centers for Medicare and Medicaid claims data might eventually help a local hospital prepare for incidents based on their specific population.

"But you're not getting those claims at the CMS level until . . . it's been processed, and by then, the hospital will get on to your next thing."

At a largely regulatory department such as DOT, Morgan said, "there's very few 'right now' decisions we need to make. The 'right now' decisions may happen in the operation of the national airspace, but the rest of it is pretty slow," he said. 

"We still can build these models and get more better data from lots of different sources to make better and more precise decisions, but we don't need to make them in milliseconds or minutes, nine times out of 10," Morgan added.

Even in cases in which real-time collection would help, often information-sharing protocols between the public and the federal government have yet to be formalized.

Better information about trucks and their average speed might help DOT improve the freight system, Morgan said. But "we also regulate that industry, and trucking companies and truck drivers can be put out of business for breaking the federal carrier safety regulations . . . Those companies don't really have an incentive to share detailed, granular data with their regulator."

While there's a lot of value in such real-time data, "there's much trickier questions around what the information sharing frameworks and brokerage models need to be," he added. 

Forrester analyst Noel Yuhanna said he didn't expect the majority of federal agencies to move toward real-time analysis for at least five to 10 years.

In the commercial sector, businesses are trying to reach customers, or process large volumes of their own data for internal use, so that it can be accessed in real-time on users' mobile phones, he added. 

"For government agencies, traditionally, you [still] have a legacy mainframe system . . . it will take a while to get to that next level."

(Image via HerrBullermann / Shutterstock.com)