Patching systems is always painful, but Tuesday's scenario of two major cumulative patch sets on the same day—from
Oracle and
Microsoft—is downright sadistic.
When will this patching madness end? Not soon. Although there's a concerted effort on the part of ISVs to develop more secure code, analysts predict that it will take generations before this scrubbed-up, buttoned-down code catches up with the Swiss cheese that is legacy applications.
"The problems will only get worse, not better, anytime soon," said Jon Oltsik, an analyst with Enterprise Strategy Group. "We'll have more software, more integration, more functionality [and] new protocols, and all that stuff will, by its nature, be insecure out of the chute."
If you're running an Oracle Corp. database on top of a Microsoft Corp. platform, you have double the patching delight. A sizable number of people do indeed host that scenario, with Oracle being the No. 2 database to run on the Windows platform, right after Microsoft's SQL Server database.
To ensure that patches don't break anything, enterprises that have the resources and the ability to take their systems offline put patched systems into a testing environment. With regression testing, they replicate their production systems and run them through technical and financial transactions to ensure that production won't grind to a halt because of a patch set.
It's the safest way to go, and you'd think all enterprises would do it. "Those who test in production usually die by their methods," said Mike Herald, a consultant at Pronto North America, an ISV that markets an ERP (enterprise resource planning) management system. "We've got the same issues with our products."
Herald's company learned the hard way that taking customers live on their software releases, without walking them through regression testing, translates into multiple nights of misery.
"I was in here working 10-hour days trying to repair the mess," Herald said.
Thanks to that learning experience, Pronto now declines to roll out software in live production environments. "We've since not done upgrades unless they name a project to the upgrade that dedicates resources, time and money," he said. "In this case here, as a software company, it's tough to say that, but we've come out and done that."
Click here to read about a fake Microsoft patch that triggered a virus attack.
Still, plenty of businesses forgo testing, given the laundry list of resources and the breathing room for system downtime that it requires. After all, enterprises need adequate space in server boxes to replicate their environments. They need manpower. They need to plan, coordinate and time potential disruption phases when the system can be down.
Herald's wife, for example, who works as a project manager at another company, turned down Oracle's April patch set because her employer didn't make the necessary resources available to test the system outside the production environment.
"They basically turned down a patch set to 10g for one of the applications they were running, because a) they didn't have the resources to provide a separate test instance, and b) they weren't willing to take the risk to what the patch test would do to what they had in production," Herald said.
"She, as project manager, said to the business, and to IS, 'We're not going to do it, not on my watch. You can't give me the dedicated resources, both computer and people, to thoroughly test the patch set.'"
After all, Herald said, at least his wife's employer had been living with the broken application. "What you don't want is for it to break something else," he said.
As far as testing the Microsoft-Oracle duo, Oltsik's advice is to first test the platform—i.e., Windows—for two reasons. First, the application is only as stable as the platform on which it sits. And second, there are by far more attacks launched against Microsoft than against Oracle.
Regarding Oracle flaws, Oltsik recommends prioritizing patching according to severity. That's not always an easy task, given the limited amount of information on severity that ISVs such as Oracle include in their alerts, and thus it requires digging into third-party sites that rate severity.
Also, Oltsik recommends keeping an eye on those who have access to the database. "I would … make sure anyone with access to the database is monitored, because they're the most likely to go in and exploit those vulnerabilities," he said.
For those shops doing any Oracle communications over the Internet, make sure the perimeter is secure as well. "[You don't want to] poke a hole in the firewall that lets people over the Internet find my Oracle servers," Oltsik said.
Check out eWEEK.com's for the latest database news, reviews and analysis.
Channel News and Analysis 
