Channel Insider content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

SAN FRANCISCO—Vulnerability researchers at penetration testing software maker Core Security claim that a well-known vulnerability existing in CA’s BrightStor backup software can be exploited when the program is running on Microsoft Windows Vista, essentially defeating the purpose of the operating system’s much-publicized security features.

Officials with Core, which is based in Boston, announced the flaw here at the ongoing RSA Conference just as Microsoft Chairman Bill Gates delivered his keynote address. The issue illustrates the fact that unless third-party application vendors go to great lengths to integrate their products with Vista’s security features, the technologies cannot take advantage of the operating system’s malware-defense tools, Core officials said.

Core contends that a previously disclosed vulnerability in CA’s BrightStor ARCserve Backup software, dubbed CVE-2007-0169, can be exploited to compromise systems running the new Vista operating system.

By exploiting the buffer overflow vulnerability in versions 9.01 through 11.5 of the CA software, along with its Enterprise Backup 10.5 and CA Server/Business Protection Suite r2 products, attackers could remotely execute arbitrary code on computers and potentially gain access to other systems, the automated penetration testing company said.

Microsoft executives Bill Gates and Craig Mundie kick off the RSA Conference. Click here to read what they said.

To craft an attack that takes advantage of the flaw, hackers need only manipulate slightly exploits designed to attack the same problem on systems running Microsoft’s earlier Windows XP and 2000 operating systems, Core maintains.

CA already has a security patch available that will allow users of the software to block the loophole.

One of the most significant benefits being touted by Microsoft in Vista is the product’s many security features, which claim stronger protection of the software’s kernel, on-board malware-fighting tools and the program’s User Account Control system—meant to keep viruses from escalating privileges on infected machines to prevent them from proliferating themselves onto other devices.

However, unless application vendors such as CA go to great lengths to integrate with those features, the tools can easily be defeated, as in the case of the BrightStor exploit, said Max Caceres, director of product management at Core. Part of the problem is that building products that can tap into the Vista security protections is not an easy task, according to the vulnerability expert.

“Application vendors need to be diligent about making sure that their products take advantage of Vista’s security features—they don’t integrate with them by default—and as long as developers do not make the necessary adjustments, their products will remain vulnerable to the same issue we saw in Windows XP,” Caceres said. “The exploit we found demonstrates that even if companies are running Vista, they can easily be exposed to third-party flaws.”

The CA vulnerability specifically circumvents Vista’s ALSR (Address Space Layout Randomization) technology, which is meant to prevent buffer overflow exploits, a common mode of malware attack. The technique is also widely used in by developers to secure open-source software programs.

Most independent software vendors are porting their products to run on Vista but would need to completely rewrite sections of the programs to take advantage of the feature, Core maintains.

Who’s inflating Vista security expectations? Click here to read more.

Microsoft officials didn’t immediately return requests seeking comment on the reported Vista-BrightStor vulnerability—however, CA representatives challenged Core’s report, calling the information “misleading” and pointed out that the company has specifically instructed customers not to run the products in question on Vista systems.

“CA goes to great lengths to certify and test its products in a variety of real-world configurations,” company officials said in a statement. “Core made reference to a so-called security vulnerability that could occur if IT organizations use certain versions of a CA product—but this is precisely why CA has not specified that its customers use those software versions with Vista.”

CA said that its first general release of BrightStor ARCserve Backup for Microsoft Vista (ARCserve Backup r11.5 SP3) will arrive in several weeks, and that it will include a patch for the vulnerability mentioned in Core’s report.

Check out eWEEK.com’s for the latest security news, reviews and analysis.