Cybersecurity index aims to penetrate the fog of marketing hype

It's impossible to fix something if you can't even gauge that it's broken. It's a classic problem that systems engineers and defense contractors face: they are staring into a fog of elusive threats made worse by marketers trying to make a sale on security hype.

Frustrated by this lack of clarity, Dan Geer, chief information security officer of CIA venture capital arm In-Q-Tel, launched a personal project that aims to measure threats in a meaningful way. Teaming up with a financial industry professional, he built a monthly sentiment index to capture the security community's impressions on whether risks to IT systems and networks were rising or falling.

The project, called The Index of Cyber Security, highlights a young, growing movement within the security community to craft metrics that can give professionals direction if they are groping in the dark. The experiment was driven in part by "the despair of the security metrics guy thinking, 'Where am I going to get the kind of aggregate data that allows us to get the big picture?' " Geer said.

"What perpetuates the fog is when different people who try to quantify technology risks may have an ax to grind," added his partner Mukul Pareek, a risk professional working in New York. "So they want to present a numeric representation of risk to sell a product or create marketing gimmicks."

Here's how the year-old index is constructed. Every month, roughly 200 people are polled on how they feel about myriad security threats from industrial espionage to insider threats. The duo doesn't rate actual risks, but evaluates if perceptions of risks are growing or decreasing. "In this way, we do not have to calibrate one respondent to the next such as to ensure that each of them has identical definitions and tastes," Geer said.

To protect the anonymity and privacy of those polled, the survey is electronically configured to not allow anybody -- including Geer and Pareek -- visibility into any respondent's answers. Many are CISOs and risk officers from banks, government agencies and institutions that house sensitive data. The two cast their net by getting referrals from people on the front lines of security. Their goal is to reach 300 respondents. Those who fill in the survey get a monthly aggregate snapshot and analysis of what the pool has submitted, as well as excerpts of comments from respondents, who sometimes reveal how they are tweaking their security budgets.

When the index was first conceived, the pair envisioned that cybersecurity insurance providers could use it to bring transparent pricing models to a market that is notoriously opaque. Another possibility was to propose the index as the basis of a tradable financial product that companies could buy and sell as a hedging tool in an investment portfolio. But they will not fully develop those applications until the index is more mature.

The duo behind The Index of Cyber Security plans to do a formal review of what they have learned in the past year. "We designed the index in a way that adjustments could be made to its components without damaging its continuity," said Geer, "There is a lot of maturity that can happen in the meantime."

Geer is no stranger to the difficulty of creating security metrics. In a separate 2007 collaboration, he created a price index for stolen passwords to routers and credit card information auctioned on websites. His goal was to measure how much hacked data was worth and see how financial incentives for hacking were changing over time. That became tricky when law enforcement officials started shutting down the websites that were put in the limelight, he recalls. It is also difficult to put a price tag on data sought by politically motivated actors.

While the appetite for sounder metrics is growing in the security industry, Geer admits, "it is not the roar of the crowd, but more like the hum of a cocktail party, frankly as evidenced by the reception our index has received."

The index is up 26.4 percent since it was launched, and has risen every month during the past year.

Dawn Lim, a financial reporter in New York, was formerly an intern at NextGov.