IT departments learn from misfortune

Natural disasters such as hurricanes and tornadoes can knock out an area’s power and infrastructure, leaving local officials and first responders out of touch and hamstrung. Drawing on personal experiences during storms and learning from others, city and county information technology (IT) departments are bolstering plans for keeping their communication and computer systems up and running in case similar incidents occur in their communities.

After a massive tornado completely destroyed Greensburg, Kan., on May 4, emergency management workers from Sedgwick County, Kan., located 100 miles away, went to help the town. In addition, the county’s IT staff closely watched the recovery and determined how they would handle their operations if such a storm hit their county and its major city, Wichita.

The county already has several safeguards in place to assure continued communication and protect data during such disasters. IT staff regularly list all of their services and technologies to determine how they would bring the systems back into service if one was destroyed or incapacitated, says Sedgwick County Chief Information Officer Richard Vogt. “The number one way we do that is to ensure redundancy [of systems],” Vogt says. “So, we have multiple data centers scattered around the community that are all connected together by fiber optic cable.” If one data center fails, in most cases, the function is switched to another center. IT staff also backup data on separate disk drives and tapes.

The redundancy also applies to personnel, an approach made necessary by contingencies such as an outbreak of avian flu. “The question from our manager was [to] decide what you were going to do if 40 percent of your staff dies or was otherwise unavailable for work,” Vogt says. Therefore, area managers have assigned a backup operator for every system, two backups for 75 percent of the important functions and then, for the most critical functions, a third backup operator. “Hopefully, not all of those people on that backup list are going to be out of action,” he says.

Because of the disaster in Greensburg, Vogt says, Sedgwick County has learned the importance of distance. Currently, the county’s multiple data centers are within a few blocks of each other in Wichita. “What we learned from Greensburg is that a big area, a mile-by-mile square area, can be impacted by a storm,” Vogt says. “Having another data center, at least with your really critical information, more than a mile away is a good idea.” The county now is moving its backup systems to its juvenile detention facility three miles away.

Collier County, Fla., IT officials learned valuable lessons firsthand when the county was hit by Hurricane Wilma in October 2005 and its public safety communications systems were disabled. After that, maintaining communications became IT Director Barry Axelrod’s biggest concern. The county’s public safety radio system was the top priority, followed by the telecommunication system that officials use to update residents. “What we learned from Katrina is that when communication breaks down, command and control breaks down,” Axelrod says.

In response, nine months ago, the county began providing HAM radio equipment to staff members who were certified to operate it. During emergencies, the equipment users will be assigned to different facilities, such as fuel depots, water and sewer plants, and shelters. “In case everything else breaks down, point-to-point HAM radios always work,” Axelrod says.

After being sideswiped by Hurricane Charlie in 2004, the county learned the limits of its phone system and changed to a centralized network. Now, all phone lines run through one of the phone company’s eight central offices in the county, which are built with generator backups.

Nearly three years later, the county is within a year of completing the project, which should make it well prepared for this year’s hurricane season, Axelrod says. “It’s just one of those things where, until you suffer the losses, you don’t really understand how to strengthen your systems,” he says.