recommended reading

Cybersecurity index aims to penetrate the fog of marketing hype

It's impossible to fix something if you can't even gauge that it's broken. It's a classic problem that systems engineers and defense contractors face: they are staring into a fog of elusive threats made worse by marketers trying to make a sale on security hype.

Frustrated by this lack of clarity, Dan Geer, chief information security officer of CIA venture capital arm In-Q-Tel, launched a personal project that aims to measure threats in a meaningful way. Teaming up with a financial industry professional, he built a monthly sentiment index to capture the security community's impressions on whether risks to IT systems and networks were rising or falling.

The project, called The Index of Cyber Security, highlights a young, growing movement within the security community to craft metrics that can give professionals direction if they are groping in the dark. The experiment was driven in part by "the despair of the security metrics guy thinking, 'Where am I going to get the kind of aggregate data that allows us to get the big picture?' " Geer said.

"What perpetuates the fog is when different people who try to quantify technology risks may have an ax to grind," added his partner Mukul Pareek, a risk professional working in New York. "So they want to present a numeric representation of risk to sell a product or create marketing gimmicks."

Here's how the year-old index is constructed. Every month, roughly 200 people are polled on how they feel about myriad security threats from industrial espionage to insider threats. The duo doesn't rate actual risks, but evaluates if perceptions of risks are growing or decreasing. "In this way, we do not have to calibrate one respondent to the next such as to ensure that each of them has identical definitions and tastes," Geer said.

To protect the anonymity and privacy of those polled, the survey is electronically configured to not allow anybody -- including Geer and Pareek -- visibility into any respondent's answers. Many are CISOs and risk officers from banks, government agencies and institutions that house sensitive data. The two cast their net by getting referrals from people on the front lines of security. Their goal is to reach 300 respondents. Those who fill in the survey get a monthly aggregate snapshot and analysis of what the pool has submitted, as well as excerpts of comments from respondents, who sometimes reveal how they are tweaking their security budgets.

When the index was first conceived, the pair envisioned that cybersecurity insurance providers could use it to bring transparent pricing models to a market that is notoriously opaque. Another possibility was to propose the index as the basis of a tradable financial product that companies could buy and sell as a hedging tool in an investment portfolio. But they will not fully develop those applications until the index is more mature.

The duo behind The Index of Cyber Security plans to do a formal review of what they have learned in the past year. "We designed the index in a way that adjustments could be made to its components without damaging its continuity," said Geer, "There is a lot of maturity that can happen in the meantime."

Geer is no stranger to the difficulty of creating security metrics. In a separate 2007 collaboration, he created a price index for stolen passwords to routers and credit card information auctioned on websites. His goal was to measure how much hacked data was worth and see how financial incentives for hacking were changing over time. That became tricky when law enforcement officials started shutting down the websites that were put in the limelight, he recalls. It is also difficult to put a price tag on data sought by politically motivated actors.

While the appetite for sounder metrics is growing in the security industry, Geer admits, "it is not the roar of the crowd, but more like the hum of a cocktail party, frankly as evidenced by the reception our index has received."

The index is up 26.4 percent since it was launched, and has risen every month during the past year.

Dawn Lim, a financial reporter in New York, was formerly an intern at NextGov.

Threatwatch Alert

Credential-stealing malware / User accounts compromised / Software vulnerability

Android Malware Infects More than 1M Phones, Adds 13,000 Devices a Day

See threatwatch report


Close [ x ] More from Nextgov

Thank you for subscribing to newsletters from
We think these reports might interest you:

  • Featured Content from RSA Conference: Dissed by NIST

    Learn more about the latest draft of the U.S. National Institute of Standards and Technology guidance document on authentication and lifecycle management.

  • PIV- I And Multifactor Authentication: The Best Defense for Federal Government Contractors

    This white paper explores NIST SP 800-171 and why compliance is critical to federal government contractors, especially those that work with the Department of Defense, as well as how leveraging PIV-I credentialing with multifactor authentication can be used as a defense against cyberattacks

  • Toward A More Innovative Government

    This research study aims to understand how state and local leaders regard their agency’s innovation efforts and what they are doing to overcome the challenges they face in successfully implementing these efforts.

  • From Volume to Value: UK’s NHS Digital Provides U.S. Healthcare Agencies A Roadmap For Value-Based Payment Models

    The U.S. healthcare industry is rapidly moving away from traditional fee-for-service models and towards value-based purchasing that reimburses physicians for quality of care in place of frequency of care.

  • GBC Flash Poll: Is Your Agency Safe?

    Federal leaders weigh in on the state of information security

  • Data-Centric Security vs. Database-Level Security

    Database-level encryption had its origins in the 1990s and early 2000s in response to very basic risks which largely revolved around the theft of servers, backup tapes and other physical-layer assets. As noted in Verizon’s 2014, Data Breach Investigations Report (DBIR)1, threats today are far more advanced and dangerous.


When you download a report, your information may be shared with the underwriters of that document.