Developing A Successful Patch Management Process

Transcription

1 Developing A Successful Patch Management Process White Paper FoxGuard Solutions, Inc. August 2014

2 Introduction Almost every day, new vulnerabilities are discovered and disclosed to software vendors, who then develop and release patches to mitigate those vulnerabilities. In order to protect your systems, you should apply these updated patches, but when and how should you do so? You should develop a patch management process that balances your need for security against your need for system availability and reliability. Develop a patch management process that balances your need for security against your need for system availability and reliability. However, even with a solid, well thought-out patch management processes in place, things do not always go as planned. Sometimes it may be difficult to determine which patches are actually needed. Once you have determined what patches are required, locating the correct files may be difficult because there are many similar products. For example, there are multiple versions of the Microsoft Visual C++ Redistributable packages, and all of the patches have similar filenames, making it difficult to determine what version each patch is for. What do you do when something goes wrong during a patch installation, or the patch causes an adverse effect after installation? Learn from our experience with patch management and discover some of the potential issues to watch for in various stages of your patch management process. Overview of the Patch Management Process While an organization s patch management process will be tailored to its own needs and may vary from what is listed below, the following key elements should be considered: 1. Determination of patch applicability: Determine which patches apply to systems in your environment. While a software vendor may release a large list of patches that apply to software you are using, not all of those patches may apply to your systems, depending on which software components you have installed. 2. Acquisition of all applicable patches: Use a combination of patch management software or vendor websites to locate and download all patches that you have determined to be applicable. 3. Validation of patches in a controlled environment: Set up a separate environment to test and validate patches against the hardware and software used in your production environment. Document the validation results and weigh the risks of applying these patches to your production environment. 1

3 4. Documentation of the validation results: Document the results of your validation testing, including any detected changes (such as port or service changes) or test failures. 5. Deployment of patches to production environments: Deploy all validated and approved patches to your production environment. Using a phased rollout approach is useful to catch any potential issues that were not caught during validation prior to applying the patches to all systems. An example of a phased rollout would be deploying patches to non-critical systems first, and testing for 24 hours or more before deploying patches to the rest of the systems. Difficulties in Determining Patch Applicability Patch Detection Issues and Inconsistencies Many patch management software packages provide an easy to use interface for scanning systems and determining what patches are applicable to those systems. Up-to-date definition files will need to be provided to the application in order for these software packages to function properly. These definition files provide a reference for the patch management software to know what patches are available and how to determine which systems they are to be installed on. In some cases, these definitions provided by the vendor may fail to detect all applicable patches properly. Using a second patch management software package can help uncover such flaws in another software package s definitions. In addition, you should compare the list of patches that the patch management software determined to be applicable to the documentation provided by the appropriate software vendors. If you notice any inconsistencies between the vendor documentation and your patch management software s results, you should investigate further to determine which data set is correct. Application of Security Tools and Revised Patches In addition to security patches, which are the most common type of patch you may see, there are sometimes patches that are classified as security tools. These security tool patches are often created and released to improve the security posture of your system, and are not necessarily directly related to one or more particular vulnerabilities. One example of a security tool patch is an update released for Microsoft Windows systems that allows for the removal of RC4 as an available cipher suite. This update is necessary if you wish to disable the use of RC4, but if you have no plans to do so, then it may not be applicable to your systems. Another difficulty in determining patch applicability arises when a software vendor releases a revised version of a previously released patch. In some situations, the re-released patch includes 2

4 new patch detection logic, which may cause the patch to be detected as applicable on systems that previously did not require the patch. In this case, systems that have successfully applied the patch do not need to be patched with the new version. In other cases, the newly released patch may completely replace the previous patch, and all systems will need to be patched again with this new version. For both security tool patches and revised patches, you should refer to documentation provided by the software vendor to determine whether the patches apply to your systems and your environment. Issues During the Patch Installation Process Determining What Files to Download Once you have determined which patches are applicable to your systems, you will need to obtain the actual patch files. In a traditional IT environment, you may be able to use your patch management software to automatically download all applicable patches via an Internet connection. In an industrial control system or similarly secured environment, lack of Internet access may make this process more difficult if not impossible. Even if you have a list of patch identifiers or version numbers, you may find that there are several versions of each patch due to the wide array of operating systems and other software versions available. One method that can make this process much easier is to set up a test environment in the same manner as those in your live production environment. This test environment can be virtual or physical, and you should use the same operating system, applications, and other components as your production environment. It is critical to ensure that the versions of the operating system, software applications and other components are the same as what is in your production environment. This ensures that the correct patch installation files will be downloaded. In this new test environment, you can use patch management software to scan all systems, and automatically download all applicable patches. From there, you can transfer the necessary patches into the controlled environment, following any applicable procedures determined to be necessary by your organization s information security policies. Dealing with Previously Installed Patches that are Corrupt Sometimes patches may appear to install correctly, but instead something during the installation process may cause the patch to become corrupt. When this happens, issues such as other patches failing to install or certain software features functioning improperly may occur. Until the corrupted patch has been addressed, your ability to further patch and update the system may be limited. 3

5 In some cases, reinstalling the corrupt patch may resolve the issue. In other causes, you may need to run diagnostic tools in order to find and correct the issue. As an example, Microsoft provides such a tool called the System Update Readiness Tool that will automatically fix any corrupt patch files or generate a log file that you can use to determine which patches you need to correct if it is unable to fix them automatically. Other Factors that may Prevent Patch Installation There may be other factors that prevent successful patch installation. In many cases, these factors revolve around permissions within the operating system. A particular file or folder may have access restrictions in place that will prevent particular patches and updates from installing properly, or the user performing the patch may not have the necessary privileges to install the patch. If you suspect a permissions issue is causing patch installation failures, try installing the patch as a user with administrative rights to the system unless vendor documentation says to do otherwise. Also, review any log files generated by the patch installation to determine what went wrong, and investigate accordingly. Undesirable Behavior After Patch Application Anti-Virus False Positives Anti-virus software packages typically use one of two methods to detect malware: behaviorbased detection and signature-based detection. With behavior-based detection, the software looks at what types of actions a program performs and whether those actions are potentially malicious. With signature-based detection, the software scans the contents of files to determine if they are identical or similar to known malware. Sometimes anti-virus software will falsely detect that a file is infected with malware because it exhibits similar behavior to known malware or matches the signature of known malware. For example, a popular anti-virus software package recently released updated virus definitions that caused the software to detect itself as malware. FoxGuard Solutions has also seen anti-virus software falsely detect control system software as malware, as well as critical operating system files. While it is not possible to prevent false positive detections from occurring, you should have a plan in place for handling the situation. As part of this plan, you should always treat malware detections as legitimate until you have taken reasonable steps to prove that it is indeed a false positive detection. You can scan the file with other anti-virus software, as well as compare the hash of the file to a known good copy to determine if it has been modified. You may also be able to submit a sample of the file to the anti-virus software vendor, who can then analyze the 4

6 file and if deemed to be benign, release updated virus definitions that no longer treat the file as malware. Most anti-virus software will let you ignore or whitelist the file, but use caution when doing so, as you may end up whitelisting a malicious file instead of allowing the anti-virus software to protect your system. Patches / Updates with Unintended Side Effects Regardless of how thoroughly a patch has been tested by the software vendor prior to releasing it, not every situation can be accounted for. As a result, some patches may have unintended side effects based on the configuration of a particular system. For example, a particular application may rely on a certain version of a file being present on a system, and a patch that changes the file version may cause the application to stop working correctly. In more extreme cases, a patch may cause fatal system errors due to incompatibility with certain installed software. Validating patches in a dedicated test environment may allow you to address these issues prior to encountering them in your production environment. However, sometimes you are unable to configure your test environment in the same manner as your production environment. Your validation environment may have limited node connectivity, and you may be limited in what you can test versus complete control system functionality that is available in your production environment. As a result, even with a solid validation program you may encounter issues after deploying patches to your production environment. Regardless of whether it is in your validation environment or your production environment, if you encounter unintended side effects as a result of a patch, you may need to follow your backup and recovery procedures to restore the system to a known good state. In addition, you should contact the appropriate software vendors to inform them of the issue and work towards a resolution. Conclusion Patch management does not have to be an overly difficult or complex process; however issues may arise regardless of how careful you are. Be prepared to handle potential failures, and whenever possible use a dedicated validation environment. Defined processes and procedures on how to address the failures will prove invaluable. FoxGuard Solutions creates, maintains, validates, documents and supports full validation for key players in the ICS arena and have on many accounts encountered all of the example issues mentioned above. Discovering and mitigating these issues in a validation environment ensures our customers ongoing patch deployments proceed with minimal impact to the operation of their business. 5

7 About FoxGuard Solutions FoxGuard Solutions develops innovative programs and services to improve the cybersecurity and compliance posture of industrial control systems in critical infrastructure markets. To reduce the likelihood of system downtime related to cyber incidents, FoxGuard provides assistance with patch validation and distribution, software updating, and system hardening for control system devices. Additionally, FoxGuard offers research and development services, engineering services, and field implementation services to support these programs. Author Steven Wirt, Information Security Engineer, FoxGuard Solutions Steven joined the FoxGuard team as an Information Security Engineer in 2009 with a B.S. in Computer Science. He helped develop FoxGuard's validation program and has served as a technical escalation point for issues such those as described in this paper. He also has a background in scripting and software development, and played a key role in the development of FoxGuard's DisPatch product offering. Contributors Levi Akers, Engineering Technician, FoxGuard Solutions Matthew Gilbert, Engineering Technician, FoxGuard Solutions Contact Information If you would like to learn more about patch management and validation, contact a FoxGuard Solutions representative. [email protected] 6