State Dept. looks to AI for declassification, FOIA processing 

James Leynse/Getty Images

The agency’s enterprise AI strategy is expected next month.

The State Department is testing out the use of artificial intelligence for declassifying decades-old cables and helping members of the public pinpoint what they want in public records or through the Freedom of Information Act process.

Next month, the State Department is launching its “first-ever enterprise AI strategy” that will be “laying out the framework so that the department can responsibly, safely and securely harness the capabilities of AI to advance our work,” said Giorleny Altamirano Rayo, chief data scientist at the department’s Center of Analytics and State’s responsible AI official, during a Sept. 7 meeting of the FOIA Advisory Committee.

The department already released an AI policy in the Foreign Affairs Manual in April 2023, and officials are working on two AI pilots started earlier this summer.

The department is testing the use of AI in Freedom of Information Act searches to combine similar requests, cut out duplicative work or find potentially responsive records, said Eric Stein, deputy assistant secretary for the department’s Office of Global Information Services.

The department’s presentation also describes using AI to identify classified, sensitive or otherwise FOIA-exempt materials in search results, as the agency finds responsive records through the pilot, although Stein said that the department isn’t yet at the point of using the technology for redactions.

“I think those are the fears that people have in the public: that we’re just going to apply a model, just [exempt] everything, and nothing will ever come out again,” he said. “That’s not what we want either.”

Another ongoing pilot is meant to make it easier for the people looking for information to find what they want in public records on the State Department’s website, said Stein. The department will “find and direct customer to existing released documents” and “automate customer engagement early in the request process,” the presentation states.

“So you come to State’s site hypothetically… looking for a specific country, topic and so forth, and as you type it in, it pops up, ‘Here are some records that have been released,’ which may either satisfy your request or help you narrow or identify what you’re looking for,” said Stein.

Already, the department recently operationalized the use of machine learning for declassifying cables between the State Department and overseas posts, Stein said.

The project started as a pilot last fall meant to augment the manual declassification process for information once it reaches a certain age, usually 25 years. Until recently, that had happened via people sitting at computers and going through records one-by-one, the vast majority of which are usually declassified, said Stein.

The department used cables from 1997 for the pilot after training a model on human decisions from 2020 and 2021 that concerned cables marked as confidential and secret in 1995 and 1996, said Rayo. 

The model sorts cables into three categories: those it’s confident should be declassified, those it’s confident shouldn’t be declassified and those that need manual review. 

For the 1997 pilot group of over 78,000 cables, there was “around 96% agreement with human reviewers,” said Rayo. The pilot “[reduced] burden up to 63%” by culling the number of cables that need manual review, she said.

The model was fully operationalized for the 1998 batch, said Rayo, who emphasized that people are still part of the process in both the cables tagged for manual review and a random subset from the other two categories for quality control.

“This AI machine learning technology does not replace human reviewers, but we can augment our work,” she said. “We can leverage this AI technology to do the tedious parts, but leaving the critical decision making to our staff.”

Freeing up resources is particularly important, said Stein, as the department stares down a tidal wave of records coming up for declassification.

“We have a big growth in cable traffic that’s going to occur in the next several years, and the question is, ‘How will we ever address this declassification review demand, which is also directly related to growing FOIA requests and the volume of electronic records that we have as well?’” he explained. “Without a change… we were going to have big problems meeting demands, which we’re already struggling to do as it is.”

“Machine learning, a subfield of AI, [will] be required if we’re going to automate FOIA searches for the billions and billions of records that government agencies hold,” said Debra Steidl Wall, deputy archivist. But “concerns exist about standards of use and thoughtful legal analysis.”

Another coming change the department will need to reckon with is the type of records up for declassification, which might not necessarily work well with the recently operationalized pilot. The number of classified emails up for that review is going to skyrocket as the department moves into the 2000s, hitting 12 million for the year 2018, said Rayo. 

One meeting attendee asked about the potential for errors in the algorithm to lead to the release of records that shouldn’t have been unclassified. Stein’s response: “Should we just not release anything then and keep the status quo?”

Risk appetite can be a “real problem,” he said. “We never want to release anything we shouldn't, but we also know there's an obligation to be transparent with the public of these records.”

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.