Locking down on water system security
When Milwaukee Water Works began planning for security upgrades in 1999, terrorism was not part of the picture. “We were thinking about vandals and graffiti artists and unhappy current and former employees,” says Carrie Lewis, superintendent for the agency. “We thought that the most serious attacker could be a local with a political agenda, somebody like Timothy McVeigh. The international, organized, funded terrorist was not something we thought about.”
The Water Works” main treatment plant is situated close to Lake Michigan and abuts soccer fields, beaches and a marina. The city lately has improved public access to the lakefront, increasing visitation to the area — and visibility of the water plant. “Every now and then, a bicyclist would end up on plant grounds, or a soccer player would walk in to see what the place was,” Lewis explains. “The fact that those people were getting onto the plant grounds made us realize that we had to tighten up.”
By the time Sept. 11, 2001, arrived, Milwaukee Water Works was ahead of the security curve. It had completed a system-wide evaluation and had begun implementing measures to mitigate its risks.
Most public water systems (there are 168,000 in the United States) were not as well positioned as Milwaukee”s. As a result, those communities are gearing up for their own security evaluations, which will include identifying assets, assessing vulnerability and risk, considering countermeasures and weighing costs.
Three major concerns
Assessing vulnerability begins with identifying assets (e.g., people, equipment, facilities) and potential threats to those assets. Evaluation must be comprehensive, says Brian Ramaley, director of Newport News (Va.) Waterworks. “[Utilities] have to look at their entire systems, from stem to stern, from the headwaters to the customer”s tap, and determine where their systems are most vulnerable,” he notes.
Furthermore, the evaluation should focus on activity rather than on the attacker. As Lewis noted, attackers can come in many forms — from employees and vandals to renegades and terrorists — but threats to the water system are limited to what those people can do.
Activities can range from low-level intrusion, such as that described by Lewis, to more serious acts that interrupt operations or even sicken or kill residents. The major threats fall within the realm of physical damage to facilities, cyber attack and system contamination.
Physical damage.
Cyber attack.
“Most water systems in America are able to run in a fully manual mode, although perhaps not as efficiently as we operate with our computer monitoring and control systems,” Ramaley says. “Were those systems compromised through some form of cyber attack, there”s little doubt that we could continue to function and meet the needs of our customers.”
“For most of us, if the system shuts down, we can run it manually,” confirms John Sullivan, chief engineer for the Boston Water and Sewer Commission and president of the Association of Metropolitan Water Agencies, based in Washington, D.C. “The real question is: Does anybody know how to do it? When you build a new plant and everything runs by computer, do the operators ever simulate [manual operations]?”
Sullivan also notes that communities with automated security sometimes integrate those systems with the SCADA network. Having both systems on one server presents a two-for-one hit for a hacker.
Contamination.
“People are worried that a small amount of some chemical or biological agent — a few drops, for instance — could result in significant threats to the health of large numbers of people,” she said. “That scenario just can”t happen. It would take large amounts of contaminants to threaten the safety of a city water system.”
Vulnerability to contamination still has to be assessed, according to Knudson. “If we look at source water, there might be a concern about contamination that”s going to find its way to intake,” he notes. “If [the source] is a well field, there are geohydrology issues: What happens if [we discover a contaminant] in one well? Will it show up in another one? In the transmission system, contamination is an issue because there are probably places where you could contaminate an open channel or aqueduct.”
Assessing vulnerability
Vulnerability studies can be performed by utility staff or by a third party, but, ideally, they are done by both. “Outside studies are more likely to pick up things that you would miss in house,” Ramaley says. “But you can”t just sit back and wait to bring in the experts. You have to look at things yourself and then bring in the outside, fresh eyes to look at your system.”
“It really has to be both,” Knudson says. “Nobody knows the system as well as the people who have to make it work. Those are the people that are the operators and the engineers that deal with the system on a day-to-day basis. At the same time, because they deal with it every day, they are [not necessarily objective]. You need the objective opinion of the consultant.”
When Milwaukee Water Works conducted its vulnerability assessment, it worked with Sandia National Laboratories, based in Albuquerque, N.M. The laboratory had been hired by the Denver-based American Water Works Association Research Foundation to study water system vulnerabilities and produce a guide for utilities to conduct their own assessments. (The guide will be published this month; see “Water on the Web” on page 30.) It selected Milwaukee as its “guinea pig,” Lewis says.
“We were in the right place at the right time,” she says. “For two years, we had budgeted money to upgrade security at our facilities. Sandia came [to Milwaukee] in March of this year, and they gave us some specific advice on what they had seen while they were here.
“Some of the recommendations were easy fixes, like making sure that hatch locks are not the kind that bolt cutters can cut,” Lewis notes. “Others, like upgrading electronic surveillance and alarm systems, [were more involved].”
Sullivan favors bringing in a third party, too. “I”m not saying it has to be a consultant”s paradise, but you need to bring in an outside consultant to get a fresh pair of eyes. You need a Doubting Thomas,” he says. “You need the other person there to ask, ‘Why do you leave all your tools for your pump right in front of the pump?’ Well, you do that because [the pump has] odd-sized bolts, and the tools are only good for that pump. You”re thinking the tools are there to fix the pump, and the consultant is thinking, ‘No. Those tools are there to destroy the pump.’”
Nationwide, consulting firms are gearing up to assist with vulnerability assessments. “You”ve got security experts who know nothing about water and water experts who know nothing or very little about security,” Ramaley cautions. “It”s really critical that anybody who does this type of work has both sets of expertise and knows how to marry that up so you get useful results.”
Determining risk
Every water system has vulnerabilities, but vulnerability does not equal risk. To determine risk, utilities have to review their vulnerability findings and measure the probability of an attack.
“You have to decide what”s important to you and then determine the chances of people getting at it,” Sullivan says. He notes that an earthen dam may pose vulnerability, but, if it is not easily accessible, it is not necessarily a risk.
Furthermore, utilities should look at risk in light of potential consequences. “You need to take a look at your system and ask, ‘What would happen if this went out?’” Sullivan explains. “For example, ‘If I lose this pump, what happens?’ Maybe the answer is, ‘Nothing. I”ve got another one over on the other side of town, and I”ll pump on that one.’ But that is where you have to measure your risk.”
Having determined their risk, utilities must be prepared to live with some of them, Knudson says. “We”re not going to be able to completely eliminate all the risks associated with potential vulnerabilities,” he notes. “We just can”t afford to do it. We could spend billions of dollars on any given system, and we would still not be there. You have to ask — and here”s the key to the whole thing —‘What level of risk are we willing to accept?’”
More than anything, that exercise can help utilities focus their countermeasures. “It gives you the information you need to make an informed decision about where you want to spend your money,” Knudson explains. “You might decide that it doesn”t make sense to put armed guards out at the wellhead because you don”t think the wellhead [poses a great risk]. But that becomes a conscious decision in terms of accepting risk.”
Delay and detect
Armed guards? Actually, many of the recommendations for mitigating risk are far more mundane.
“Everyone”s facilities were built with doors and gates and windows that lock,” Lewis says. “The first thing is to use those things that are already available to us, and the second thing is to get the buy-in of your employees. Those are the people who are at the facilities around the clock, and they can assess whether something they see or hear is unusual and alert others [for response].”
Employees also have to buy in to security procedures, Knudson says. “They have to understand the necessity of keeping doors locked and understand that, if somebody walks up, you don”t just let them in.”
Public access is a thing of the past, according to Ramaley. Utilities that offered facility tours have, since Sept. 11, locked down. “Many systems have eliminated that,” Ramaley notes. “They”ve shut their gates; they”ve locked their doors.”
In addition to making those changes, water utilities are upgrading security by installing surveillance equipment, and expanding fencing and setbacks, with emphasis on detecting the adversary and delaying him. “I think it”s unreasonable to expect every single water utility to upgrade its defenses to successfully defend against organized, armed terrorists,” Lewis says. “I think we need to tighten up our facilities so that the casual and lower-level attackers will either go someplace else, or we can delay them long enough to detect them.”
Detection is part of the plan in Newport News. There, the Waterworks is considering expansion of electronic surveillance, and it has stepped up watershed patrols. “We serve five jurisdictions, and our facilities touch seven jurisdictions. We”ve coordinated with local law enforcement in each of those jurisdictions to visit our facilities on a regular basis,” Ramaley says. “We”ve also [contacted] the local FBI office, and I recommend that all water utilities consider doing that. If there is a problem, you”re going to have to work with those folks, and it”s good to know them ahead of time and have those communication channels open.”
Newport News also has worked with chemical suppliers to boost security on deliveries. “Alum deliveries come to us in large tanks. Those trucks are now sealed before they leave the chemical shipping lot, and the seal is not broken until it gets to our site,” he says. Similarly, prior to receiving chemicals, Milwaukee Water Works gets the serial number of the tanker and the driver”s name from the supplier, and it checks both before accepting delivery.
Although many mitigation measures focus on prevention, others focus on recovery. “Mitigation is sometimes a matter of keeping a spare piece of equipment,” Lewis says. “Historically, our inventory was [managed on an as-needed basis]; if you”ve got eight pumps in a room and one goes out, the other seven will run, so you can wait two months to get parts for the broken one. But now we”re operating under the scenario of someone taking out all the pumps, so we will keep the parts that we can in inventory.”
Design implications
In addition to upgrading security in the short term, water utilities are anticipating changes in technology and facility design that could help them mitigate risk. The water industry is pushing for research and development of microtoxicity monitors and gas chromatographs that could help utilities pinpoint contaminants in their systems. Additionally, new plants will be built with security in mind.
“I think every facility built from now on will be looked at totally different,” Sullivan says. “You”ll see brick walls, sky lights with bars. You won”t just have door alarms; you”ll have backup systems — maybe a door and a camera or an infrared [detection device].”
“Security will be incorporated much more completely in terms of alarms and lighting and typical security-type measures,” confirms Les Lampe, vice president and director of water resources for Black & Veatch, a Kansas City, Mo.-based consultant. “You”re going to see a lot more focus on setback spaces; you”re going to see fewer entrances; and you”re going to see more control over the roadways, gates, fences and lighting.”
Utilities will be looking for optimum safety of source supplies, making them less open and available for contamination, Lampe says. He also expects to see more attention to alternative treatment processes that might provide a higher level of protection than do conventional processes.
“There are a lot of design implications as a result of security concerns,” Ramaley says. “We have to look at our facilities as needing to be able to withstand or continue to provide service under potential attack, or at least to know where, if we were attacked, we would be most likely to fail and how we would recover from that quickly. It”s not going to be possible to harden all of our facilities so that they can”t be attacked, but we can lay our systems out so that recovery can be much quicker.”
Quick recovery requires redundancy, and Lampe reports that he is seeing increased emphasis on redundancy, from SCADA backup to power feeds for pump stations. He also notes that physical separation of facilities can assist with recovery. “If you have one pumping facility that is critical to your whole system, [it may be possible] to add smaller pumping stations in different locations, giving you reliability without putting everything in one location,” he explains.
A price to be paid
From vulnerability and risk assessments to mitigation, shoring up the nation”s water systems will be expensive. Where will the money come from?
Congress is debating two bills that could help substantially with payment for assessments and future technology. The first, passed by the House in November, appropriates more than $109 million to assist communities with vulnerability assessments. The second bill, which Sullivan “absolutely expects” to be passed, will fund $60 million for research and development of monitoring technology.
Those sums will not cover security costs for every water system, however. As a result, Sullivan anticipates that cities will share information, with bigger cities assisting smaller ones. “The small guys, who might not have the money for consultants, could work off the big guys,” he suggests.
Water utilities may have to raise rates to recoup some of their costs, and they will definitely have to weigh the feasibility of security spending against the capital requirements for infrastructure repairs, adding capacity and complying with regulations.
There is a long road ahead, Ramaley says. “Securing our water systems is going to be a long process,” he notes. “It”s one that we need to begin immediately, and there is going to be an economic price to be paid. Utilities are going to have to begin down the path and keep moving towards more and more secure facilities.”
Water on the Web
For more information about water system security, visit the Web sites for the following organizations.
American Water Works Association, Denver; www.awwa.org
Awwa Research Foundation, Denver; www.awwarf.com
Association of Metropolitan Water Agencies, Washington, D.C.; www.amwa.net
Environmental Protection Agency, Washington, D.C.; www.epa.gov
Federal Bureau of Investigation, Washington, D.C.; www.fbi.gov
