With flames lapping at the eaves and 500,000 people sprinting for safety in Southern California this week, it might make sense to literally pull the plug on the server, toss the rack in the trunk and drive out to the desert to avert the loss of equipment and data.
That’s just what one business did, said George Vahle, a partner and head of marketing at The I.T. Pros, a San Diego-based managed services provider. A full 80 percent of IT Pros customers have been affected by the wildfires in one way or another, many just by being displaced, Vahle said.
Other customers just removed tape backups to off-site locations, or shut down and moved pieces of equipment. One customer, fearful that smoke in the area would set off the sprinkler system in his building in the hard-hit neighborhood of Rancho Santa Fe, removed his most vital IT equipment, Vahle said.
The peak of the crisis thus far occurred the night of Monday, Oct. 22, into Tuesday morning, Vahle said. From its network operating center, the I.T. Pros skeleton office crew monitoring its customers’ IT systems could see customers dropping offline one by one.
“Many customers went offline due to power outages,” Vahle said. “Customers were then operating on battery backups, but then the batteries got depleted.”
The company’s biggest task right now is getting those systems, interrupted in one way or another, back up and running.
I.T. Pros started seeing higher call volume from users on Tuesday, as people turned away from watching coverage of the fires on television and tried to return to work. Many found they couldn’t log into their e-mail or their VPNs.
“There are still parts of the county that don’t have power,” Vahle said. “We are interfacing with those customers and by the end of the day tomorrow most customers will be back.”
San Diego emergency services had asked residents and businesses in the area to stay off the roads and cell phones to leave those free for those responding to the crisis.
“On Monday there was nobody on the road,” Vahle said. “The people who came in to the office are the ones who live close by.” That left most companies, including I.T. Pros, running with just essential staff members. Still, many of the company’s engineers were able to log into the system from home and provide remote support to customers.
“The thing that is hard to understand for people who aren’t here is the scope of the disaster,” Vahle said. “Hundreds of homes were burned, the county of San Diego came to a screeching halt, over 600,000 people were evacuated.”
Frustration in the ‘Danger Zone’
Further north in Los Angeles, solution provider Dynamic Systems was also identifying customers it thought might be “in the danger zone.” The company, which caters to the IT needs of federal government and higher education customers, offered help with disaster relief associated with IT needs, said Dynamic Systems President Mardi Norman.
“We have heard from some customers,” she said. “They are interested in knowing if we have some systems that they can port some of their data to. They are trying to get a last-minute disaster plan together that they hadn’t quite put together yet. Luckily, however, we have not had anyone who is truly in danger of losing anything.”
The fire and its aftermath will spur new interest in putting disaster plans and disaster recovery plans into place, Norman said.
Dynamic Systems’ customers include military sites, which “have some of the most sensitive data in our country, and it needs to be the most secure and readily available,” Norman said, pointing out that these types of clients use co-location sites.
In addition to the security of data, the continuing crisis “also affects people’s moods,” Norman said. “I open my car door at LAX and ash is falling on me. It’s a constant and real reminder that there are facilities and people in danger.”
Advisories across southern California have warned people to stay inside because of the unhealthy air quality. And there is also the sheer frustration of not being able to move around.
“I was in California yesterday on a potential client site, and the fires were right around me. We couldn’t make it through,” said Lester Keizer, president of Connecting Point Technology Center, a Las Vegas-based VAR. “We had to find a hotel room, or try to. They were all full, because the main artery that goes from Las Vegas to Los Angeles, the I-15, was closed down.”
At I.T. Pros, while the disaster has not affected day-to-day business tremendously, there is concern for the people who work there.
“Our primary concern has been our own employees,” said CEO Doug Ford. “We are having conference calls twice a day to see where everyone is. We have at least one employee who thinks they may have lost their house.”