Channel Insider content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

A Microsoft presentation at this week’s RSA Conference has some observers concerned about a recurring nightmare of computer security: Rootkits.

A rootkit is a malicious program that uses system hooks to conceal its presence on the system. For instance, it monitors if the user opens the Windows Task Manager in order to keep itself out of the list of processes. It filters directory listings to remove its own files from them. The rootkits could be everywhere, living among us, and we wouldn’t know!

Rootkits are not new by any means. When I first wrote about them they were just beginning to appear for Windows and had been around on UNIX for at least a couple of years.

To the best of my knowledge, full-blown rootkits haven’t been a major problem for Windows users (how would I know?), but many malware programs have dabbled in rootkit-like behavior, such as using legitimate-looking process names (almost a kind of phishing) and even disabling security software like antivirus programs.

Rootkits are scary because they can hide their presence from almost any method of detection. A well-written one can even block some on-demand virus scanners from knowing that the file system has been used.

So how do you find out that you have a rootkit on your system?

Addressing this problem appears to be the goal of a new prototype tool from Microsoft Research named Strider GhostBuster that compares the list of files on the system scanned under the potentially compromised OS to one created by a separate, clean OS scanning the same computer.

Scanning and cleaning a system that potentially has a rootkit is better done while not running the operating system potentially infected. Using a tool based on Windows PE is one way to accomplish this. Since the rootkit isn’t running, it can’t take active measures to conceal itself.

One approach is that as long as the scanner knows about the rootkit and can identify the files involved, removing it from such an offline scan is relatively easy, since the rootkit program isn’t running and in a position to defend itself. There’s also an excellent poor-man’s program that takes the same approach: BartPE.

Then there’s the Microsoft Research approach: Scan the system with its own operating system running; scan it with another operating system, such as a Windows PE-based one booted off of a CD; and then report on the differences. Any files that show up under Windows PE and not the local operating system were probably trying to hide themselves under it.

In one important way rootkits are no different from any other attack: You need to have your system compromised in order to get the rootkit on to it. So if you protect your system well and actively, it’s as difficult for the rootkit to get on as with any other threat. The difference comes after the fact, but any system that is ‘owned’ is compromised.

Even without being concerned about rootkits I would say that selective removal of threats from systems is not a reassuring strategy. Consider the spyware industry where vendors removing well-known threats often leave parts of them on the system.

With a rootkit you really can’t know whether you were successful unless you use an offline scanning method like Microsoft Research’s Strider GhostBuster. This is why restoring images of systems is a good strategy for securing them with a high degree of confidence.

If user profiles and data are stored elsewhere on the network then systems are completely interchangeable. And you can even re-image them periodically just to be sure, if you want to go to the trouble. There are systems for doing this in an automated and managed fashion.

It’s true that if a major rootkit attack were to gain some traction we’d be in trouble. Very few people utilize scanning techniques that would find a rootkit. I’m not really worried because as time goes by, more and more systems are effectively protected, and rootkit-compromised computers are no more dangerous to their neighbors than those infected with more conventional malware.

The real problem with rootkits is that to do things they would want to do—such as send spam or attack another computer— they must expose themselves. A true rootkit is so hidden that it’s useless.

Security Center Editor Larry Seltzer has worked in and written about the computer industry since 1983.

Check out eWEEK.com’s for the latest security news, reviews and analysis. And for insights on security coverage around the Web, take a look at eWEEK.com Security Center Editor Larry Seltzer’s Weblog.

More from Larry Seltzer