Inside a Hacking Competition to Take Down a Water-Treatment Plant

BeeBright/Shutterstock.com

Just before 2:30 p.m. last Friday, the water-treatment plant lost power. For minutes, as technicians scrambled to bring it back to life, it was still, its pumps quiet, its purifying chemicals inactive, its ultraviolet lamps dark.

The plant had been attacked by hackers, who had bombarded its power-control system with an unrelenting stream of data until, overwhelmed, it shut itself off. This same technique had been trained on a vital piece of the internet’s infrastructure earlier that very day—albeit at a much bigger scale—preventing millions of Americans from accessing vast swaths of the internet.

For the next two and a half hours, the water-treatment plant remained under siege from several different groups of hackers, who were attacking each other even as they delved deeper and deeper into the plant’s controls, causing absolute mayhem. At 2:45, a pair of revolving sirens threw blue beams around the room: The system that maintained the plant’s water levels had been disabled, and one of its tanks began to fill at an alarming rate.

“The float’s been submerged!” a technician called out from near the tanks. The float was supposed to cut off water flow the moment it became immersed.

“Is it still filling?” asked another, hunched over a laptop perched on his knees.

“Yes!”

“That’s bad.”

The workers powered down the plant again in order to drain the tank to a safe level.

Later, in a moment of calm after the attacks had subsided, I caught up with Joe Needleman, the technician who’d been wrestling with the plant’s settings. “It’s been insane,” he told me, nodding his head. But he wasn’t too shaken—actually, he was pretty excited.

Needleman had intended the plant to be attacked. In fact, it was built specifically for that purpose: He and four other students at California State Polytechnic University, Pomona, had spent the last month assembling it and writing 5,000 lines of code to control it, hewing as close as was practical to reality.

The result was a desk-sized model made up of three large plastic tubs, several aquarium water pumps, and PVC pipe. It was arranged on a table in a large, airy room at a co-working space in Washington, D.C., that was, on this particular Friday, completely overrun with wires. The plant was just one in a buffet of targets that had been set out for participants in a capture-the-flag-style hacking competition.

All afternoon, 13 teams of three or four hackers—mostly made up of college students, with a few professional security researchers sprinkled in—raced each other to accrue the most points by solving trivia questions, reverse-engineering computer programs to un-encrypt files, finding hidden messages encoded in digital images and songs, and, of course, attacking the model water-treatment plant.

The event was hosted by Passcode, the Christian Science Monitor’s cybersecurity-focused publication, but the challenges were designed by security researchers at Uber and the students at Cal Poly Pomona.

Alex Levinson, who leads Uber’s incident-response team, said he tried to create challenges that reflected the sort of work security researchers do in the real world: frantically responding to a cyberattack as it unfolds, plugging holes and trying to prevent any data from being stolen—or, in quieter moments, probing computer systems for vulnerabilities to patch before the bad guys find them. Levinson said he intentionally created too many challenges for teams to solve in order to force them to manage their time and prioritize their strengths.

When it came to the water-plant challenge—which the event’s organizers said was a unique feature among hacking competitions because of its physical presence in the room—one team dominated. It wasn’t the professionals from Tenable Network Security, but rather three 19-year-old sophomores from Carnegie Mellon University, playing under the moniker “Plaid Parliament of Pwning.”

The unassuming trio made their way through the plant’s control systems, accessing them from their laptops at a table just 10 yards away. They frowned at their machines as they toggled between terminal windows and browsers, Googling commands and downloading programs as they explored the network set up specifically for the event.

“I have literally complete control of this host but I have no idea what to do with it,” said Zach Wade, his skinny frame scrunched into a futuristic red swivel chair, to nobody in particular. He had broken into one of the water-treatment plant’s control systems, but didn’t quite know it yet. He jumped from database to database, probing them for weaknesses and searching for flags that would win his team points.

Wade left a trail of destruction as he worked. After accessing one system, he deleted every user account except his own, changing the administrator password to “zachpwn.” In another, he found the controls that limit the water temperature in one of the treatment tanks, and raised the maximum from 100 to 1,000 degrees Fahrenheit. Stressful dubstep pounded in the background.

“Zach just went in and started burning things,” laughed Matthew Savage, one of his teammates. As the final half hour of the competition ticked down, Savage and the team’s third member, Corwin de Boor, tried to complete as many of the smaller challenges as possible, while Wade continued trouncing through the water plant’s control systems.

The way the team from Carnegie Mellon found vulnerabilities in the control systems and exploited them mirrored real-world patterns. Especially when it comes to sensitive infrastructure, hackers who break in may not even be intentionally targeting them, Levinson said. Instead, they might just be scanning the internet for vulnerable devices and networks, and not realize until they dig a little further that they’ve stumbled upon something bigger than a computer or a server—say, for example, a dam near New York City.

Keeping hackers out of critical infrastructure has become a priority for government agencies worried about a catastrophic cyberattack on an energy grid or, well, a water-treatment plant. Many installations still run on outdated computer systems, Levinson said. He’s concerned both about their capacity for holding up to attack, given the systems’ age, but also about the prospect of a bumpy transition to modern technology. If newer control systems aren’t installed and secured correctly, they could be just as vulnerable to attack.

The hacking competition played out in a friendly atmosphere: To Levinson’s surprise, the teams largely heeded his warning not to hack one another. But when many teams had trouble connecting to the website where the challenges had been posted, it was an ominous reminder of the cyberattack that was still making it hard to access the internet up and down the East Coast of the United States.

With threats like Friday’s disruptive hack emerging more and more often, private companies and the government are scrambling to hire and train the brightest young hackers to defend against cyberattacks. To that end, recruiters from Uber, Tenable, and Northrop Grumman were roaming the room as the competitors hacked away, dispensing advice and collecting resumes.

Before the competition began, Phyllis Schneck, the Department of Homeland Security’s top cybersecurity official, spoke about the importance of bringing computer-security skills into the government. One of the main obstacles to luring hackers into the public sector was illuminated during the question-and-answer session that followed, however, when one particularly distraught participant stood to protest the ways hacking laws can put white-hat researchers in jeopardy.

As the final minutes of the competition ticked off, the Plaid Parliament of Pwning was the only team to have put any points on the board for completing various elements of the water-treatment plant challenge. But at the buzzer, the trio came in second, falling to a team from the University of Virginia, who had accrued more points on other challenges. “We lost to the guys that taught us,” Wade told me: The three friends had gone to the same high school as some of the members of the UVA team.

“We gotta get ’em next time, Zach!” de Boor said. The team tossed Snapple bottles and 7-Up cans into the recycling, packed up their laptops, and left, turning around only to snatch up a Northrop Grumman business card they’d forgotten on the table.

X
This website uses cookies to enhance user experience and to analyze performance and traffic on our website. We also share information about your use of our site with our social media, advertising and analytics partners. Learn More / Do Not Sell My Personal Information
Accept Cookies
X
Cookie Preferences Cookie List

Do Not Sell My Personal Information

When you visit our website, we store cookies on your browser to collect information. The information collected might relate to you, your preferences or your device, and is mostly used to make the site work as you expect it to and to provide a more personalized web experience. However, you can choose not to allow certain types of cookies, which may impact your experience of the site and the services we are able to offer. Click on the different category headings to find out more and change our default settings according to your preference. You cannot opt-out of our First Party Strictly Necessary Cookies as they are deployed in order to ensure the proper functioning of our website (such as prompting the cookie banner and remembering your settings, to log into your account, to redirect you when you log out, etc.). For more information about the First and Third Party Cookies used please follow this link.

Allow All Cookies

Manage Consent Preferences

Strictly Necessary Cookies - Always Active

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data, Targeting & Social Media Cookies

Under the California Consumer Privacy Act, you have the right to opt-out of the sale of your personal information to third parties. These cookies collect information for analytics and to personalize your experience with targeted ads. You may exercise your right to opt out of the sale of personal information by using this toggle switch. If you opt out we will not be able to offer you personalised ads and will not hand over your personal information to any third parties. Additionally, you may contact our legal department for further clarification about your rights as a California consumer by using this Exercise My Rights link

If you have enabled privacy controls on your browser (such as a plugin), we have to take that as a valid request to opt-out. Therefore we would not be able to track your activity through the web. This may affect our ability to personalize ads according to your preferences.

Targeting cookies may be set through our site by our advertising partners. They may be used by those companies to build a profile of your interests and show you relevant adverts on other sites. They do not store directly personal information, but are based on uniquely identifying your browser and internet device. If you do not allow these cookies, you will experience less targeted advertising.

Social media cookies are set by a range of social media services that we have added to the site to enable you to share our content with your friends and networks. They are capable of tracking your browser across other sites and building up a profile of your interests. This may impact the content and messages you see on other websites you visit. If you do not allow these cookies you may not be able to use or see these sharing tools.

If you want to opt out of all of our lead reports and lists, please submit a privacy request at our Do Not Sell page.

Save Settings
Cookie Preferences Cookie List

Cookie List

A cookie is a small piece of data (text file) that a website – when visited by a user – asks your browser to store on your device in order to remember information about you, such as your language preference or login information. Those cookies are set by us and called first-party cookies. We also use third-party cookies – which are cookies from a domain different than the domain of the website you are visiting – for our advertising and marketing efforts. More specifically, we use cookies and other tracking technologies for the following purposes:

Strictly Necessary Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Functional Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Performance Cookies

We do not allow you to opt-out of our certain cookies, as they are necessary to ensure the proper functioning of our website (such as prompting our cookie banner and remembering your privacy choices) and/or to monitor site performance. These cookies are not used in a way that constitutes a “sale” of your data under the CCPA. You can set your browser to block or alert you about these cookies, but some parts of the site will not work as intended if you do so. You can usually find these settings in the Options or Preferences menu of your browser. Visit www.allaboutcookies.org to learn more.

Sale of Personal Data

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Social Media Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.

Targeting Cookies

We also use cookies to personalize your experience on our websites, including by determining the most relevant content and advertisements to show you, and to monitor site traffic and performance, so that we may improve our websites and your experience. You may opt out of our use of such cookies (and the associated “sale” of your Personal Information) by using this toggle switch. You will still see some advertising, regardless of your selection. Because we do not track you across different devices, browsers and GEMG properties, your selection will take effect only on this browser, this device and this website.