House of IT Horrors

By Deborah Rothberg  |  Print this article Print


Desktop-as-a-Service Designed for Any Cloud ? Nutanix Frame

Who needs ghosts and goblins, haunted houses, and smashed pumpkins, when sometimes your most harrowing horrors are only steps away from your technology desk?

Ghosts and goblins, haunted houses, and smashed pumpkins—sometimes the most harrowing horrors are only steps away from your cubicle or office.

Just ask Steven Calderon.

Calderon was in his second week working as security guard for Fry's Electronics when Anaheim, Calif., police walked in and arrested him for crimes including child molestation and rape.

Fry's had requested a background check on Calderon, which was done by The Screening Network, a service of ChoicePoint, the $1 billion-a-year data broker based in Alpharetta, Ga. When it came up with criminal warrants and felony charges, nobody—not Fry's, not the police—stopped to ask if the data supplied by ChoicePoint was accurate.

Calderon spent a week in jail for crimes he didn't commit because an identity theft report he'd filed in Norwalk, Calif., in 1993 wasn't connected with the criminal files that were created in his name.

He went to jail for an IT error.

While, fortunately, not all IT disasters are of these cataclysmic proportions, every weathered IT professional has an all-too-real eerie tale about a day when everything went wrong. Even 10 and 50 years later, these pros retell their horror stories with a startling sharpness, more haunting than ghost stories.

Invasion of the Inept VP

John Mitchell (not his real name), a senior support analyst with an insurance company in Madison, Miss., like many worker bees, had his most chilling IT calamity at the hands of a higher-up who believed he had it all figured out.

"A senior VP, who fashions himself a 'programmer,' decided to install the latest version of Visual Studio 2005 before we, the IT staff, had a chance to test it. Once installed, his code no longer worked. His solution? He uninstalled a previous version of Visual Studio," said Mitchell.

Peter Coffee looks back at almost 25 years of IT screwups. Click here to see the slide show.

Mitchell's IT group came to find out that he had not one or two, but three different versions of Visual Studio on his machine.

"Yes, we should be able to control this since we're IT, but we've been warned against 'stifling a user's creativity,' which loosely translates into 'He's a VP. What are you going to do, tell him no?'"

Adding insult to injury, the VP put in a Severity 1 trouble ticket, which was supposed to mean that he was incapable of continuing the duties of his job until it was fixed, when it was really more of a Severity 3. Mitchell's group, however, had no choice but to drop everything and make this machine their first priority.

"Our programming staff attempts to decipher his code but can't because it seems they're not trained in early Egyptian hieroglyphics," said Mitchell. "It then gets transferred to me. I turn his machine inside out trying to make this thing work. I uninstalled and reinstalled every VB application on his computer multiple times, manually cleared the registry, Googled every variation of 'broken Visual Studio 2005' I could think of, lit a few candles, said a few prayers, but nothing worked.

"Meanwhile, the VP, who took the day off, made sure to phone every 2 hours for updates and to offer 'suggestions.' For two solid days, I monkeyed with the problem, but couldn't fix it. His code continued to crash."

So what happened? Mitchell came in to work on the morning of the third day to a voice mail from the VP who was driving him up the wall, in which he said that while he had been at the office that weekend, he realized that he'd mistyped a character in his data collection string. Once he made the change, the code worked perfectly.

"'You can close the ticket,' he told me," Mitchell said. "Fortunately, my office has double-insulated walls; if not, I'd be telling you this from the unemployment line."

Next page: Dawn of the Dismal Data Conversion.

Dawn of the Dismal Data Conversion

Stuart Robbins, founder and director of The CIO Collective, a nonprofit association of senior IT executives providing strategic guidance to emerging businesses, and author of "The System is a Mirror" (John Wiley & Sons, 2006), had his worst professional nightmare come true almost 10 years ago, when his group attempted to migrate from a very old to the then-current version of Sybase.

"We were attempting to leap six versions [4.x to 11.x] because our third-party vendor promised it was the only way to maintain the viability of our Scopus data, which depended upon the Sybase upgrade," said Robbins.

So, just how badly did it go? Cataclysmically, it could be argued.

"There were so many problems that we needed a triage team dedicated to resolve issues that we had caused by the work on the previous day, and after delays of more than a year and many hundreds of thousands of dollars over budget, we backtracked and moved the data to SAP," said Robbins.

Robbins added that in almost every nightmare project he's heard about from colleagues, custom data conversions, such as his Sybase disaster, were the root cause.

"They never work and should be outlawed," he said. "No matter what the vendor promises, major vendor changes involving critical corporate data should simply be re-entered by hand using inexpensive Kelly Temp resources."

Night of the Living Zero-Divide

Jerry Luftman, now an associate dean and distinguished professor at Stevens Institute of Technology's School of Technology Management, was a young IBM systems engineer supporting one of the major television networks during the 1976 Carter-Ford presidential election when disaster struck.

"Like today, even in ancient times, everyone was glued to their televisions (yes there was color, but we did have to get up to change the channel) watching the returns unfold. The networks competed heavily on who could predict the outcomes the fastest and the most accurate," said Luftman.

A new security survey reveals that the majority of IT pros are most afraid of hacker attacks. Click here to read more.

Suddenly, at the network where Luftman was supporting, each of the three mainframes—one primary and two backups—within 5 seconds of each other crashed at the peak hour of 9 p.m.

"At first, the team from IBM and the television network looked at each other in fear and puzzled. We then looked at the TV monitor and saw that our [television] star did not skip a beat. They kept looking at their terminal, and without any sign of concern kept relaying the status of the election," said Luftman.

As the stars were reciting the election results from memory, Luftman's team scrambled to find the cause of the outage, frantically trying to figure out how to get the systems back up quickly.

"Within 5 minutes the core dump was printed and we were analyzing the problem. There was a local election in the Southwest that had a candidate running unopposed. Did you figure it out yet? There was a 'zero divide,' and the systems just could not tolerate it," said Luftman.

As the clock still ticked and the broadcasters continued to present the results, someone came up with the ingenious idea to add a fictitious candidate to the local race and give them one vote.

"It worked. The three computers came back up, and the broadcaster had current information on their screen. ... And we were the first to predict that Carter won, and we were the most accurate."

Additional reporting by Debra D'Agostino, Debra Gage and John McCormick


Submit a Comment

Loading Comments...