recommended reading

Cybersecurity index aims to penetrate the fog of marketing hype

It's impossible to fix something if you can't even gauge that it's broken. It's a classic problem that systems engineers and defense contractors face: they are staring into a fog of elusive threats made worse by marketers trying to make a sale on security hype.

Frustrated by this lack of clarity, Dan Geer, chief information security officer of CIA venture capital arm In-Q-Tel, launched a personal project that aims to measure threats in a meaningful way. Teaming up with a financial industry professional, he built a monthly sentiment index to capture the security community's impressions on whether risks to IT systems and networks were rising or falling.

The project, called The Index of Cyber Security, highlights a young, growing movement within the security community to craft metrics that can give professionals direction if they are groping in the dark. The experiment was driven in part by "the despair of the security metrics guy thinking, 'Where am I going to get the kind of aggregate data that allows us to get the big picture?' " Geer said.

"What perpetuates the fog is when different people who try to quantify technology risks may have an ax to grind," added his partner Mukul Pareek, a risk professional working in New York. "So they want to present a numeric representation of risk to sell a product or create marketing gimmicks."

Here's how the year-old index is constructed. Every month, roughly 200 people are polled on how they feel about myriad security threats from industrial espionage to insider threats. The duo doesn't rate actual risks, but evaluates if perceptions of risks are growing or decreasing. "In this way, we do not have to calibrate one respondent to the next such as to ensure that each of them has identical definitions and tastes," Geer said.

To protect the anonymity and privacy of those polled, the survey is electronically configured to not allow anybody -- including Geer and Pareek -- visibility into any respondent's answers. Many are CISOs and risk officers from banks, government agencies and institutions that house sensitive data. The two cast their net by getting referrals from people on the front lines of security. Their goal is to reach 300 respondents. Those who fill in the survey get a monthly aggregate snapshot and analysis of what the pool has submitted, as well as excerpts of comments from respondents, who sometimes reveal how they are tweaking their security budgets.

When the index was first conceived, the pair envisioned that cybersecurity insurance providers could use it to bring transparent pricing models to a market that is notoriously opaque. Another possibility was to propose the index as the basis of a tradable financial product that companies could buy and sell as a hedging tool in an investment portfolio. But they will not fully develop those applications until the index is more mature.

The duo behind The Index of Cyber Security plans to do a formal review of what they have learned in the past year. "We designed the index in a way that adjustments could be made to its components without damaging its continuity," said Geer, "There is a lot of maturity that can happen in the meantime."

Geer is no stranger to the difficulty of creating security metrics. In a separate 2007 collaboration, he created a price index for stolen passwords to routers and credit card information auctioned on websites. His goal was to measure how much hacked data was worth and see how financial incentives for hacking were changing over time. That became tricky when law enforcement officials started shutting down the websites that were put in the limelight, he recalls. It is also difficult to put a price tag on data sought by politically motivated actors.

While the appetite for sounder metrics is growing in the security industry, Geer admits, "it is not the roar of the crowd, but more like the hum of a cocktail party, frankly as evidenced by the reception our index has received."

The index is up 26.4 percent since it was launched, and has risen every month during the past year.

Dawn Lim, a financial reporter in New York, was formerly an intern at NextGov.

Threatwatch Alert

Software vulnerability

Malware Has a New Hiding Place: Subtitles

See threatwatch report


Close [ x ] More from Nextgov

Thank you for subscribing to newsletters from
We think these reports might interest you:

  • Modernizing IT for Mission Success

    Surveying Federal and Defense Leaders on Priorities and Challenges at the Tactical Edge

  • Communicating Innovation in Federal Government

    Federal Government spending on ‘obsolete technology’ continues to increase. Supporting the twin pillars of improved digital service delivery for citizens on the one hand, and the increasingly optimized and flexible working practices for federal employees on the other, are neither easy nor inexpensive tasks. This whitepaper explores how federal agencies can leverage the value of existing agency technology assets while offering IT leaders the ability to implement the kind of employee productivity, citizen service improvements and security demanded by federal oversight.

  • Effective Ransomware Response

    This whitepaper provides an overview and understanding of ransomware and how to successfully combat it.

  • Forecasting Cloud's Future

    Conversations with Federal, State, and Local Technology Leaders on Cloud-Driven Digital Transformation

  • IT Transformation Trends: Flash Storage as a Strategic IT Asset

    MIT Technology Review: Flash Storage As a Strategic IT Asset For the first time in decades, IT leaders now consider all-flash storage as a strategic IT asset. IT has become a new operating model that enables self-service with high performance, density and resiliency. It also offers the self-service agility of the public cloud combined with the security, performance, and cost-effectiveness of a private cloud. Download this MIT Technology Review paper to learn more about how all-flash storage is transforming the data center.


When you download a report, your information may be shared with the underwriters of that document.