Patch management: Hold it together

Patch management is undoubtedly one of the top bugbears of IT and security managers today. Software vendors, patch management suppliers and consultants have all worked to improve patching, but have they succeeded?

Ultimately, patch management goes back to the twin IT security pillars of effective asset management - determining what software is being run across the business and where the potential vulnerabilities are; and risk management - factoring the exposures and risks of dealing with all patch introductions.

"Each business should strike a balance between spending too much time on patch management, which can be very expensive in terms of manpower and budget, and not spending enough time on it, putting the company's assets at risk," says John Botting, a director at Altiris Security Management. 'IT directors should ask themselves: 'If I don't patch, what's the potential risk to my business?"' He points out that a business's network can be particularly vulnerable when it has just outsourced some functions or has merged with another company and has legacy systems in place as a result.

On the vendor side, the problem shows no sign of abating. Suppliers of operating systems and application software continue to issue code updates and fixes that need to be incorporated into the IT fabric. Microsoft alone has released on average 1.38 patches per week since 2002 to cover vulnerabilities across its product range. Controlling the distribution and effective deployment of all those new bits of code then brings a whole new range of challenges for IT departments.

Patch management software from specialist companies such as PatchLink and Shavlik Technologies can to a certain extent automate the process of finding, downloading and applying patches or correcting system misconfigurations across an enterprise. It does this either immediately once a patch or threat has been identified or on a scheduled basis.

There are a number of products that include internal discovery systems to find network devices, identifying missing patches or misconfigurations, or they can interact with other security products to provide that knowledge. Automated patching systems respond to preset policy on how to react to problems, and they also provide sourcing for patching content.

Automated patch management solutions can do the legwork for you, distributing and applying patches to all the computers in the company, or to selected users, at the click of a button. Rule-based systems can apply patches of critical importance to certain users straightaway to cover those most at risk. These systems frequently use a simple "drag and drop" interface, and log patching activity so that managers can keep track of patching history across the infrastructure.

Tracking patch performance is a useful metric to add in when analysing security posture. USAID, the US government's aid distribution agency, uses the nCircle product to monitor the 1,600 nodes on its network. "We can generate a report card on regional vulnerability this way that helps increase security awareness," says Bill Geimer, the agency's global security programme manager.

Another advantage of automated patch management is that it reduces the risk of human error "There's an element of painting the Forth Bridge here," jokes Simon Perry, European vice-president of security at CA. "It can be very hard to be sure you have nailed down all your machines. I once dismayed a CIO by telling him I'd found a number of HP machines on his network he was sure he'd shut down years earlier." Adrian Davis, a research consultant at the Information Security Forum, agrees: "The average large UK organisation thinks it has 1,500 servers - plus or minus 300."

Could systems be developed that automatically patch themselves? Unlikely, at least in the near future: the IT "stack" in most organisations, public or private, is so dense, with such a heterogeneous set of IT assets, that it's very hard to see how that much intelligence could be introduced to any real-world network.

Some suppliers and IT security players take a different view, arguing that a tightly run network wouldn't need patches as insecure code could not get through in the first place. "Patch management is shutting the door after the horse has bolted," says Pete Simpson, ThreatLab manager at Clearswift. "It won't be an issue if you have secure enough defences in place."

Some recommendations by this camp: use filters to strip off executable files from emails and disable applications such as instant messaging, WinZip and media players to reduce the risk of introducing viruses to the network.

But a rise of zero-day vulnerabilities exploited by attackers, such as the recent PowerPoint flaws, serve to remind us that even the best reactive defence is not absolute. And the time between the announcement of a flaw, or the release of a patch, to a worm doing the rounds is shrinking all the time. Nimda took more than 300 days to come out but Blaster, two years later, took less than 30. In May, email management firm Postini claimed that between two and three per cent of the 22 billion email messages it processed in the previous month were viral in nature.

Vendors are constantly facing demands for faster release cycles. There has even been a nascent but undeniable open source movement for patches, with open source groups releasing patches for Microsoft flaws before the vendor has released an authorised patch.

But most observers feel this is still very much in its infancy. "Software is a very complex product - so complex that even the vendor who created it can't get it right, or there wouldn't be a need for a patch," says Mike Small, director of security strategy, EMEA at CA. "It would be madness to accept a fix from a third party. There is a risk that any fix which does not come directly from the original source could have been compromised."

Wherever the patch is coming from, it is clear that the time you have to patch is shrinking. This means the process itself must become faster and more efficient, so the issue comes down to what extent technology can help in streamlining what is essentially a management rather than a product or technology issue.

Even when rolled out consistently, patches are not always successful. Changes to applications can break other software, even other patches. "Patches destroy the structure of the original code and, even worse, can interfere with other patches and with patches not yet created," says Small. "This is why vendors are increasingly moving to cumulative releases that allow them to put structure back into the fixed code, which will have been comprehensively tested."

Change strategy is key

This also raises the importance of testing. Patch management should always be aligned with any change management plans in place. These are important processes that set out criteria for upgrades and contain their own back-out plans. Patch management usually requires emergency change, and this needs to be factored in.

"There is, in principle, a very strong case for prompt application of patches, where the flaw being patched would otherwise leave systems vulnerable to remote attack," says Clearswift's Simpson. "However, there is a degree of risk in that the patch may have side effects or may not be 100% effective. Financial systems, for example, may be subject to a rigorous and time-consuming quality assurance cycle. It's a difficult balancing act, particularly when the patch is applied to a large number of PCs."

As useful as patch management products are, most commentators still stress the need for a change management-oriented approach, where patches get rolled out in a predictive and orderly fashion. "Teams that adopt best practice combine a touch of human judgement with a good patching infrastructure product to help prioritise patches," says Altiris' Botting. "Once the correct patches have been identified, an IT manager can automate them."

Patch management software is not a miracle fix, it is only as good as the overall security policy. "It's all about defence in depth," says Paul King, senior security adviser at Cisco UK and Ireland. "Patch management is another part of the security architecture you need in place to protect your organisation and its assets.

90% - Gartner Group says that more than 90 per cent of security exploits are carried out through vulnerabilities for which there are known patches.

PREPARING FOR PATCH TUESDAY

Patch Tuesday, the second Tuesday of each month, is Microsoft's designated day for releasing updates. Although urgent security releases are sometimes released between official release days, Patch Tuesday has become synonymous with multiple patches and the heavy lifting required to test and deploy them.

Security firm PatchLink offers this advice for handling Patch Tuesday:

2 weeks before

- Identify all firmware and software on the network; categorise by platform, department, etc

- Determine which systems are most critical to protect and how prone they are to attack

- Determine ownership, permissions needed and responsibilities for threat identification, testing and remediation across security, IT and business units

- Ensure all assets in the network have been fully installed with an automated patch solution. Install new patch management agents where required if this has not yet been fully automated

- Build a representative sample set of each type of machine based on the above

One week before

- Allocate IT resources for the day, but don't neglect other patch release schedules from Adobe, Apple (ad hoc), Oracle, etc

- Reserve time slots to be able to deploy patch updates to any mission-critical servers within 72 hours of Patch Tuesday release

- Monitor security sites for pre-announcements of patches and discussion of vulnerabilities and possible zero-day exploits from sources such as SANS, National Vulnerability Database, etc

- Review and update system records of last patch deployments; make sure that all computers are being regularly scanned. Deploy any missing service packs, Hotfixes or rollups. Remember that some patches won't install if you have missing pre-requisites

On The Day

- Microsoft and other vendors provide webinars, email alerts and online information on all new Patch Tuesday updates - use them

- Use patch impact (critical, important) asset classifications to prioritise systems for patch testing and deployment

- Testing each patch - in each environment of your previously defined groups - is vital; automated deployment is risky and not advised

- Follow any internal planning and approval processes for agreeing on patch deployment

- Stage deployments by system groups and prioritisation. Start with small, low-risk groups, validate that no problems occur, and then work your way to larger and higher risk areas

The Day After

- Maintain good records of all patches deployed

- Measure how long it takes to get all servers, PCs and laptops fully patched in your organisation; this is a great metric to measure against.