US Spies Train Computers to Spot Suspicious Activity in Live Videos

Andrey VP/Shutterstock.com

A research project will attempt to automatically detect suspicious activities, with the help of live video pouring in through multiple camera feeds.

Story has been updated with comments from an ODNI official. 

The intelligence community is working on amping up people-recognition power to spot, in live videos, shooters and potential terrorists before they have a chance to attack.

Part of the problem with current video surveillance techniques is the difficulty of recognizing objects and people, simultaneously, in real-time.

But Deep Intermodal Video Analytics, or DIVA, a research project out of the Office of the Director of National Intelligence, will attempt to automatically detect suspicious activities, with the help of live video pouring in through multiple camera feeds. 

ODNI’s Intelligence Advanced Research Projects Agency is gathering academics and private sector experts for a July 12 "Proposers’ Day," in anticipation of releasing a work solicitation.

“The DIVA program will produce a common framework and software prototype for activity detection, person/object detection and recognition across a multicamera network,” IARPA officials said in a synopsis of the project published June 3. “The impact will be the development of tools for forensic analysis, as well as real-time alerting for user-defined threat scenarios.”

» Get the best federal technology news and ideas delivered right to your inbox. Sign up here.

In other words, the tech would scour incoming video surveillance and body-camera imagery from areas of interest for people and objects who could present a threat, or individuals and items that might have been involved in a past crime. 

This is the type of video-recognition system that might have been used for identifying would-be suicide bombers before the Paris and Brussels attacks, some video analytics experts say.

Privacy laws in the United States and Europe differ, so it's tricky to know whether such activity-recognition software would have been legal to use on video around the time of the 2013 Boston Marathon bombings.

On Tuesday, an ODNI official said on background the regulations governing video capture in open areas, stateside, are complicated.

"The legalities of how video recorded from public spaces is used is complex, depends on both federal and local statutes, and may change over time," he said. "Technology from DIVA should be used for legal purposes, and in these cases, it is intended to enhance national security and public safety."

The official added that IARPA is not an authority on how Boston or other jurisdictions may use video recorded in public spaces. The new program's goals are "aiding a timely review of video footage after an attack, or by detecting evidence of a planned attack, and tipping further investigation," he said.

What Is Sticking out of That Spectator’s Backpack?

The envisioned system will provide multiple levels of granularity, according to IARPA. 

One perspective would flag "primitive activity," like people getting into or out of a car, or someone carrying an object, the synopsis states. In this experimental scenario, the video would be collected from security cameras. 

Another feature would key in on complex activities: someone carrying a firearm or two people exchanging an object, according to IARPA.

The most sophisticated capability would recognize people and things in live footage from many angles, showing different perspectives of areas of interest.

This pinnacle of the program involves “person and object detection and recognition across multiple overlapping and non-overlapping camera viewpoints," IARPA officials said. 

These last two experiments will take advantage of body-cam video feeds and handheld video camera images, the synopsis states. Some of the sensors also might capture infrared data and video from other portions of the electromagnetic spectrum not visible to the human eye. 

Participating teams are expected to consist of experts from many technical disciplines, including artificial intelligence, probability, person re-identification, and 3-D reconstruction from video.

The intelligence community anticipates academic institutions and private sector companies from around the world to join in. 

Correctly identifying individuals before they can attack requires a system with not only keen recognition but also lots of video, data profiles and pictures of faces.

The technology must have access to a bad-guy database that already contains the would-be perpetrator’s face, cameras that can capture usable images of people approaching an area, and a way to signal guards or otherwise cut off access to the target, Defense One reported shortly after 32 people were killed in bombings on the Brussels metro and at the city's international airport March 22.

A failure to stop the same extremists behind last November's deadly terrorist attacks in Paris from carrying out the Brussels bloodshed, highlighted inadequate government intelligence on terrorists and communities’ lack of trust in police.

Tagging Faces and Things in A Crowd

Last year, a facial recognition system was used on live video from surveillance cameras at the European Games, in Baku, Azerbaijan, according to the tool's developer. During the June 2015 event, organizers watched a webpage that could issue an alert if a face in the crowd matched that of an individual on a watch list, explained John Waugaman, president of Tygart Technology, the company that deployed the technology. 

A match that scores above a certain level of confidence will generate an alert.

Sometimes, the vast amount of faces in a highly-populated area can bog down the scanning process, so high-performance computers are available for support on the back-end, said Waugaman, whose customers include the U.S. intelligence community, Pentagon and law enforcement agencies. 

If an agency needs to screen more faces per minute, for example, it can grab more computing bandwidth from cloud computer providers. 

The IARPA research will focus on creating a scalable framework that can function in an open cloud environment, the synopsis states.

The scope of the intelligence project -- the intertwining of real-time person identification, object recognition and activity detection -- is the next wave of video surveillance, Waugaman said. 

"Easily within the next two years, you’ll see pairing of facial and object recognition in operational use," he said. 

With "accurate object detection capabilities, you can broaden the use cases from known subjects to just people that are behaving oddly," Waugaman said. Instead of the software program "being trained to find faces, it’s trained to find people with backpacks or it’s training to find people carrying guns.”

There could be backlash surrounding the use of activity recognition software from privacy or gun rights groups, Waugaman acknowledged.

But perhaps paradoxically, combining all the identification modes could cut the number of false alarms, also called “false positives.”

“We might know that the person is a threat," Waugaman said. "They have an object that might be a threat and they are acting in a manner that appears to be threatening. That will really help reduce the false positive rates in these systems." 

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.