Tyson Jarrett CIP Enforcement Analyst Best Practices for Security Patch Management October 24, 2013 Anaheim, CA
A little about me Graduated from the University of Utah with a Masters in Information Systems Have been with WECC for 2 years 11 months Reviewed 1,407 CIP items Ran my first Marathon this Month 2
Wrong reasons to Patch
Why Patch Management Cyber Security Patches are key to avert many known vulnerabilities to Cyber Assets and the environment Identified by ICS Cert as one of the top security challenges within Industrial Control Systems Most importantly Requested by YOU!! 4
Intent of CIP 007 R3 The intent of R3 is to know, track, and mitigate known software vulnerabilities associated with BPS Cyber Assets. It is not intended as an install every security patch requirement; instead it should be considered more of a be aware of in a timely manner, and manage all known vulnerabilities requirement. 5
CIP 007-3 R3 Security Patch Management The Responsible Entity, either separately or as a component of the documented configuration management process specified in CIP-003-3 Requirement R6, shall establish, document and implement a security patch management program for tracking, evaluating, testing, and installing applicable cyber security software patches for all Cyber Assets within the Electronic Security Perimeter(s). o o R3.1. The Responsible Entity shall document the assessment of security patches and security upgrades for applicability within thirty calendar days of availability of the patches or upgrades. R3.2. The Responsible Entity shall document the implementation of security patches. In any case where the patch is not installed, the Responsible Entity shall document compensating measure(s) applied to mitigate risk exposure. 6
Agenda Tracking, Evaluating, Testing, and Installing applicable cyber security software patches o Common Pitfalls o Best Practices o Audit/Enforcement Approach 7
Tracking
Tracking Common Pitfall #1 Only tracking patches available at the Operating System level Entity only tracks patches with Windows Server Update Service (WSUS) o WSUS does not actively identify or track other applications and/or software running on the Windows box o Thus all third-party applications on the device are not being actively tracked 9
Tracking Common Pitfall #2 Not maintaining appropriate documentation Including: o Incomplete or inaccurate list of devices and applications running on those devices o Not knowing or documenting where patch releases are located. Leads to systems not getting patched for known security vulnerabilities 10
Tracking Best Practices Patch and Upgrade identification Asset identification Patch and Upgrade source identification Patch Tracking Process
Tracking Best Practices 12 Asset Identification o Ensure all applicable cyber assets are documented Leverage other CIP requirements (CIP 002 R3 and CIP 005 R1.6) when identifying assets. Include step to periodically verify accuracy of list o Use combination of automated tools and manual walkthroughs/ verifications to ensure list is accurate Patch and Asset Upgrade identification identificat ion Patch and Upgrade source identification
Tracking Best Practices Patch/Upgrade Identification o Identify and document all applications, Operating Systems, s and firmware on cyber assets Minimize applications on CIP 007 applicable devices to only what is necessary Include steps to periodically verify accuracy of list Asset Patch and Identifica Upgrade tion identification Patch and Upgrade source identification 13
Tracking Best Practices Patch/Upgrade Source identification and notification o Where are the patches located? o How will you get notified of a new patch? Vendor email Manually visiting webpage Automated scanner (WSUS, patch management software) o Implement periodic manual checks to verify automated solutions are functioning properly Patch and Asset Upgrade Identific source ation identification n Patch and Upgrade identification 14
Tracking Best Practices Patch Management tools o Commercial Software o Database o Spreadsheet o Paper Don t do this!! o Brain Don t do this!! 15
Tracking Best Practices Commercial Vendor Pros Cons Built with the intent to track and manage patches and upgrades Evidence presentation and retention may need additional planning Vendor support can reduce need for in-house expertise Can come with useful features Asset identification, automated notifications, baseline creation and update, vulnerability scanning Research needs to be done ahead of time to ensure the right product is chosen See SANS A Practical Methodology for Implementing a Patch management Process for 9 items to consider when picking an automated solution. (PG. 7)
Tracking Best Practices Some Commercial Vendors 17
Tracking Best Practices Database Pros Cons Reduces redundancy Can be complex and difficult to implement Reporting features allow for evidence retrieval Not always practical to build from scratch Can reduce data entry, storage, and retrieval costs May need new personnel familiar with creating and maintaining a database Personnel may need additional training
Tracking Best Practices Spreadsheet Pros Easy to implement Low cost Cons Updating data can be difficult and often requires creating a new spreadsheet to maintain historical evidence Can require repetition with other processes for updating data Current patch version, work orders, etc Useful information usually needs to be stored on another spreadsheet. What devices have what applications/os/firmware Where patches are located How are notifications being received
Tracking Audit Approach Maintain documented procedures for tracking patches and updates Evidence of actively monitoring all available software and firmware o Develop a list of all monitored applications/os/firmware o Identify and document process and location for notifications of updates o All applications/operating Systems/firmware that MAY receive security patches must be accounted for in Patch Management tracking procedures. 20
21 Evaluating
Evaluating Common Pitfall #1 Relying on Industrial Control System (ICS) vendor to evaluate applicability of patches o Due to fear of voiding warranty with ICS vendor entity leaves all patch management responsibilities to the ICS vendor o Entity does not have procedures or timeline in place for evaluating patch applicability 22
Evaluating Common Pitfall #2 Not consistently evaluating patches within 30 days of availability o Entity tracks patches once a month Thus entity continuously misses 30 day deadline as it does not have enough time to evaluate patches o SMEs mis-read documentation and didn t verify if all software had patches available 23
Evaluating Best Practice Identify o How assessment will be documented o Specific personnel responsible for assessing the patches and upgrades Should have collaboration with both IT and operations staff 24
Evaluating Best Practice Implement a peer review process to verify evaluations are done correctly and necessary documentation is maintained Conduct periodic training on evaluation procedures and required documentation 25
Evaluating Best Practice Plan ahead o Track patches at least every two weeks to ensure enough time is available to evaluate patches within the required 30 days of availability 26
Evaluating Best Practice Don t rely on ICS vendor to evaluate patches o Determining a patch is applicable will not void a warranty Entity s may still elect to wait for ICS vendor approval prior to installing a patch o ICS vendors are not required by CIP to assess patches immediately, YOU are! 27
Evaluating Audit Approach Documented process for determining if patches and upgrades are applicable o An assessment should consist of determination of the applicability of each patch to the entity s specific environment and systems. Applicability determination is based primarily on whether the patch applies to a specific software or hardware component that the entity does have installed in an applicable Cyber Asset. 28
Evaluating Audit Approach Must maintain evidence of identification and evaluation of applicability within 30 days of availability o Evidence should include date patch or upgrade was made available, date it was evaluated, and evidence the documented evaluation process was followed 29
30 Testing
Testing Common Pitfall #1 Patches are automatically installed, and thus not tested prior to installation o Entity was unaware device was configured for automated updating 31
Testing Common Pitfall #2 Conducted functional testing only o Entity s testing procedures required security patches be installed in test system for 1 week then deployed to production if nothing broke Entity does not identify or document security controls impacted by the patch being installed 32
Testing Best Practices Use existing CIP 007-3 R1 Test Procedures o Don t re-invent the wheel!! 33 Specific Documentation o Identify what tests are performed when and why? o Who is responsible for conducting the testing? Who is responsible for approving the test? o What do the results of the testing mean? What is a pass or fail?
Testing Best Practices Implement peer review process to verify testing was done per procedures Implement periodic follow-up testing to validate current testing procedures are capturing data needed to make installation decision 34
Testing Best Practice Disable automatic updates o Configure software to notify but not install o Implement verification process to periodically check for any cases where patches aren t being tested 35
Testing Audit Approach At minimum must be compliant with CIP 007 R1 testing procedures o From CIP 007-3 R1 a significant change shall, at minimum, include implementation of security patches o Technical narrative describing testing environment(s) Include how is test environment similar/ dissimilar to production environment 36
Testing Audit Approach 37 Documented testing procedures for each cyber asset (or asset type) within the ESP At minimum testing needs to ensure existing security controls are not adversely affected o Before and after the test identify and document ports and services, user accounts, Logging/Alerting functionality, and anti-virus. Controls should be in place to protect production environment o Separate test environment o Back out plan
Installing
Installing - Common Pitfalls Not updating baseline after the change is made o Personnel were unaware who was supposed to update the baseline documentation o Procedures didn t explicitly call out updating documentation 39
Installing Best Practice Leverage CIP 003 R6 Change Control and Configuration Management procedures o When installing a security patch o Following procedures during install o Updating baseline after the change Identify who is responsible for installing the patch or upgrade and updating documentation afterwards 40
Installing Best Practice 41 Use checklists to help SMEs easily identify all that is required as part of the installation Procedures should identify an acceptable time frame to install an applicable patch o Note: Requirement does not specify a required time to install a patch or upgrade Create a back-out plan o Ensure backups and recovery plan are up to date and tested
Installing - Audit Approach Documented procedures for installing security patches and upgrades o Evidence of installation of patches as defined in documented procedures Evidence Documentation o Cyber Assets patch/upgrade was installed on Who installed the patch o Updated device baseline after installation o Date of installation 42
Installing Audit Approach If patch or upgrade is applicable but not installed, must implement and document Compensating Measures o Additionally a Technical Feasibility Exception (TFE) may need to be submitted if the device will not be installing any new patches. 43
Summary 44 Procedures must include methods for Tracking, Evaluating, Testing, and Installing security patches Take extra time planning how patches will be tracked and documented to lessen burden of patch management further down the road Entity s are responsible for compliance, not the entity s ICS vendor
References Ben Christensen's Root Cause Analysis for Commonly Violated Requirements Sans Practical Methodology for Implementing a Patch management Process ICS CERT Improving Inductrial Control Systems Cybersecurity with Defense-In- Depth Strategies 45
Questions? Tyson Jarrett CIP Enforcement Analyst 801.819.7676 TJarrett@wecc.biz