recommended reading

Hacking Tools Get Peer Reviewed, Too


In September 2002, less than a year after Zacarias Moussaoui was indicted by a grand jury for his role in the 9/11 attacks, Moussaoui’s lawyers lodged an official complaint about how the government was handling digital evidence. They questioned the quality of the tools the government had used to extract data from some of the more than 200 hard drives that were submitted as evidence in the case—including one from Moussaoui’s own laptop.

When the government fired back, it leaned on a pair of official documents for backup: two reports produced by the National Institute of Standards and Technology that described the workings of the software tools in detail. The documents showed the tools were the right ones for extracting information from those devices, the government lawyers argued, and that they had a track record of doing so accurately.

It was the first time a NIST report on a digital-forensics tool had been cited in a court of law. That its first appearance was in such a high-profile case was a promising start for NIST’s Computer Forensics Tool Testing project, which had begun about three years prior. Its mission for nearly two decades has been to build a standardized, scientific foundation for evaluating the hardware and software regularly used in digital investigations.

Some of the tools investigators use are commercially available for download online, for relatively cheap or even free; others are a little harder for a regular person to get their hands on. They’re essentially hacking tools: computer programs and gadgets that hook up to a target device and extract their contents for searching and analysis.

“The digital evidence community wanted to make sure that they were doing forensics right,” said Barbara Guttman, who oversees the Software Quality Group at NIST. That community is made up of government agencies—like the Homeland Security Department or the National Institute of Justice, the Justice Department’s research arm—as well as state and local law enforcement agencies, prosecuting and defense attorneys, and private cybersecurity companies.

In addition to setting standards for digital evidence-gathering, the reports help users decide which tool they should use, based on the electronic device they’re looking at and the data they want to extract. They also help software vendors correct bugs in their products.

Today, the CFTT’s decidedly retro webpage—emblazoned with a quote from an episode of Star Trek: The Next Generation—hosts dozens of detailed reports about various forensics tools. Some reports focus on tools that recover deleted files, while others cover “file carving,” a technique that can reassemble files that are missing crucial metadata.

The largest group of reports focuses on acquiring data from mobile devices. Smartphones have become an increasingly valuable source of evidence for law enforcement and prosecutors, because they’re now vast stores of private communication and information—but the sensitive nature of that data has made the government’s attempts to access it increasingly controversial.

“It’s a very fast-moving space, and it’s really important,” Guttman said. “Any case could potentially involve a mobile phone.”

It’s an odd feeling to flip through these public, unredacted government reports, which lay bare the frightful capabilities of commercially available mobile-extraction software. A report published just two weeks ago, for example, describes a tool called MOBILedit Forensic Express, which is made by San Francisco-based Compelson Labs. The tool works on Apple iPhones 6, 6S, and 6S Plus, two versions of Apple’s iPads, as well as several Samsung Galaxy smartphones and tablets. It can extract the following types of information from a mobile device:

… deleted data, call history, contacts, text messages, multimedia messages, files, events, notes, passwords for wifi networks, reminders and application data from apps such as Skype, Dropbox, Evernote, Facebook, WhatsApp, Viber, etc.

The product page for MOBILedit Forensic Express claims the software is capable of cracking passwords and PINs to get into locked phones, but it’s not clear how effective that feature is. Getting into a locked, encrypted smartphone—especially an iPhone—is difficult, and it’s unlikely MOBILedit can bypass every modern smartphone’s security system.

When the FBI tried to break into an iPhone 5C it found at the scene of the 2015 San Bernardino shooting, it initially wasn’t able to access the phone’s data, and asked Apple for help. (Presumably, the FBI would have had access to MOBILedit and other commercial tools.) Apple refused, and the FBI brought a lawsuit against the company—but withdrew it when agents finally found a way in.

Guttman says NIST doesn’t address phone encryption in its testing.

“Encryption is certainly an issue for law enforcement access to phones and other digital media, but that issue is outside of our expertise and the type of work we do, which is focused on software quality and software understanding,” she said.

The NIST report on MOBILedit describes how the tool fared against different combinations of smartphones and mobile operating systems. It found, for example, that the tool only obtained the first 69 characters in particularly long iOS notes. Besides that issue and five others, though, the tool largely behaved as it promised it would on iOS devices, the report says.

“None of the tools are perfect,” Guttman said. “You really need to understand the strengths and limitations of the tools you’re using.”

Unlike some more complex tools, MOBILedit doesn’t require an investigator to open up a smartphone and manipulate its internals directly—the software connects to the target phone with a cord, just like a user might to update his or her device. But law enforcement doesn’t necessarily need to force its way into a phone it’s interested in searching, either by cracking open its case or by brute-forcing its passcode.

In certain cases, officers can just ask—or pressure—the phone’s owner to open it. That’s what happened when Sidd Bikkannavar, a NASA engineer, was stopped by a customs agent on his way back to his native United States from a vacation: The officer just asked Bikkannavar to turn over his PIN, wrote it down and took his smartphone to another room for about half an hour. When the agent returned the phone, he said he’d run “algorithms” to search for threats. It’s possible Bikkannavar’s phone was searched with one of the mobile acquisition tools DHS has tested.

The government’s growing library of forensic tool reports is supplemented by other testers. Graduate students at the Forensic Science Center at Marshall University in West Virginia, for example, do some of the same sorts of testing NIST does. They often work with West Virginia State Police, which runs its own digital forensics lab on campus, to test extraction tools before they’re deployed. They post their results online, just like NIST does, to grow the body of shared knowledge about these tools.

“If we weren’t validating our software and hardware systems, that would come up in court,” said Terry Fenger, the director of Marshall’s Forensic Science Center. “Part of the validation process is to show the courts that the i’s were dotted and t’s crossed.”

A new NIST project called “federated testing” will make it easier for others to pitch in with their own test reports. It’s a free, downloadable disk image that contains all the tools needed to test certain types forensic software, and automatically generate a report. The first report from the project came in recently—from a public defender’s office in Missouri, an indication digital forensics isn’t just the realm of law enforcement.

I asked Fenger if the technical information being made public in these validation reports could help hackers or criminals circumvent them, but he said the validation data probably wouldn’t be of much value to a malicious hacker.

“It’s more or less just the nuts and bolts of how things work,” Fenger said. “Most of the hackers out there are way beyond the level of these validations.”

Threatwatch Alert

Thousands of cyber attacks occur each day

See the latest threats


Close [ x ] More from Nextgov

Thank you for subscribing to newsletters from
We think these reports might interest you:

  • Modernizing IT for Mission Success

    Surveying Federal and Defense Leaders on Priorities and Challenges at the Tactical Edge

  • Communicating Innovation in Federal Government

    Federal Government spending on ‘obsolete technology’ continues to increase. Supporting the twin pillars of improved digital service delivery for citizens on the one hand, and the increasingly optimized and flexible working practices for federal employees on the other, are neither easy nor inexpensive tasks. This whitepaper explores how federal agencies can leverage the value of existing agency technology assets while offering IT leaders the ability to implement the kind of employee productivity, citizen service improvements and security demanded by federal oversight.

  • Effective Ransomware Response

    This whitepaper provides an overview and understanding of ransomware and how to successfully combat it.

  • Forecasting Cloud's Future

    Conversations with Federal, State, and Local Technology Leaders on Cloud-Driven Digital Transformation

  • IT Transformation Trends: Flash Storage as a Strategic IT Asset

    MIT Technology Review: Flash Storage As a Strategic IT Asset For the first time in decades, IT leaders now consider all-flash storage as a strategic IT asset. IT has become a new operating model that enables self-service with high performance, density and resiliency. It also offers the self-service agility of the public cloud combined with the security, performance, and cost-effectiveness of a private cloud. Download this MIT Technology Review paper to learn more about how all-flash storage is transforming the data center.


When you download a report, your information may be shared with the underwriters of that document.