DARPA Wants AI to Tell You When It's Dumb

Scientists in the Pentagon’s research office are working to build artificial intelligence systems capable of something that stumps most humans: owning up to their own incompetence.

On Tuesday, the Defense Advanced Research Projects Agency kicked off the Competency-Aware Machine Learning program, an effort to build AI tools that can model their own behavior, recall past experiences and apply knowledge to new situations.

Given these skills, officials said, AI could ultimately assess its own expertise for a given task, and let people know if it doesn’t know what it’s doing.

In general, AI systems work best when they’re applied to very explicit, narrow tasks, and even the most finely tuned tech could fail if situations are slightly changed. A tool that classifies dogs might work flawlessly in broad daylight, but mistake a golden retriever for a black lab when it’s cloudy outside, for instance.

The tech itself doesn’t know how accurate it will be in a given situation or whether it’s properly trained for the task at hand. And in the high-stakes, rapidly changing world of military operations, this uncertainty could be particularly problematic, DARPA said in the solicitation.

“In order to transform machine learning systems from tools into partners, users need to trust their machine counterparts,” officials wrote. “One component to building trust is understanding machine competence” and gaining “an accurate insight into the machine’s skills, experience, and reliability.”

During the four-year program, participants will create tools that can both build memories of past experiences and analyze their own decision-making processes, according to the solicitation. Ultimately, teams will deploy and test the tech on Defense Department platforms, officials said.

Interested teams must submit proposals by April 22.

Today, transforming AI from a useful tool to a “collaborative partner” is one of DARPA’s top priorities. Over the next five years, the agency plans to invest $2 billion in building so-called “third-wave AI,” technology that can reason its way through the world and supply humans with new ideas.

Included in DARPA’s wide-ranging AI portfolio are efforts to automate the scientific process, build computers with common sense, build tech that mirrors insect brains and link military systems to the human body.