Dell World Software User Forum 2013

Dell World Software User Forum 2013 December 9-12 Austin, TX Advanced Patching Topics Making Patching Work for You Chris Grim

Advanced Patching Why do we Patch? Survey Says Security Reliability/Stability Not because I hate my users Microsoft sends it to me so it must be good Constant change in my environment keeps things interesting Installing and Uninstalling constantly makes windows more stable 2 Software

Advanced Patching Why do we Patch? Perfect World Software would have no bugs Bad guys would go away Users would be smarter No reboots Real World Your users are your customers Security takes a village There is no Easy Button 3 Software

Advanced Patching Ground Rules Discussion There is no one right answer Patching requires thinking You all collectively know more than I do 4 Software

Agenda Anatomy of Patching Feed Agent Configuration Best Practices / Tips Environment Planning Steady State Assessment Monitoring Progress Common Issues Forensics Working with Tech Support 5 Software

Survey How are you doing Operating System patching today? - KACE - Windows Update - WSUS - Imaging - Not 6 Software

Survey How are you doing 3 rd party application patching today? - KACE - Vendor Auto-update - Imaging - Managed Installations - Not 7 Software

Anatomy of Patching 8 Software

Breakdown of Patching How does patching work? Patch Feed Lists Signatures and Patches Sources Client Library Logs Downloads Evolution 5.3/5.4 5.4sp1 5.5 Lists mcescan.cab 60mb All OS Lists, all languages 100mb each Application lists Non-security list Signatures, Packages and Patches Signatures -.pls files detection definition Packages bulletins Patches individual executables exe or plp Sources https://service.kace.com http://kace.cdn.lumension.com http://download.windowsupdate.com http://servicecdn.kace.com http://go.microsoft.com http://cache.patchlinksecure.net http://cache.lumension.com 9 Software

Breakdown of Patching How does patching work? Patch Feed Lists Signatures and Patches Sources Client Libraries Logs Downloads Evolution 5.3/5.4 5.4sp1 5.5 Libraries kpatch.exe pluginpatching.dll Logs kpatch.out pluginpatching.dll Downloads Patches 10 Software

Breakdown of Patching How does patching work? Patch Feed Lists Signatures and Patches Sources Client Library Logs Downloads Evolution 5.3/5.4 5.4sp1 5.5 5.3/5.4 mcescan.cab 60mb All OS Lists, all languages 100mb each 5.4sp1 Vega 1 Operational (Non-security) Patches ospx files security, dependencies, applications, os files no mcescan.cab 5.5 Vega 2 WSUS parity Only subscribed lists and languages Only changes go to endpoints Compression of detect results back to server 11 Software

Breakdown of Patching Configuration Patch Settings Control Panel Signature Downloads Patch Download Options Patch Download Schedule Offline Update Options What happens when new things are downloaded Notifications - email Parsing of the signature lists Replication Impact Updates to existing files 12 Software

Breakdown of Patching Configuration Subscriptions Operating Systems Languages Many patches are not language specific Subscription check boxes Things to think about Application Installers Disabled Patches Auto Activation Superceded 13 Software

Breakdown of Patching Patches Types Bulletins vs Files Operating System vs Applications Service Packs are in App Feed Subscription check boxes Activation vs Disabled Patch Detail Links/Summary Supercedes/Superceded Labels Files/Feed list 14 Software

Breakdown of Patching Subscriptions How - Detect and Deploy Who - Machine Selection Labels OS Filter What Detect set Deploy set When Schedules Server/Agent Time Suspension Window User Interaction Alerts Reboots Suspend 15 Software

Breakdown of Patching Grouping and Reporting Reporting Built in reports New options in 5.5/6.0 reporting wizard New options in 5.5/6.0 labels Interesting Tables PATCHLINK_MACHINE_STATUS KBSYS.PATCHLINK_PATCH KBSYS.KONDUCTOR_TASK 16 Software

Best Practices / Tips 17 Software

Survey What is your patch approval process? Automatically accept all patches Automatically accept critical patches Test critical patches Test all patches Do you have exceptions? 18 Software

Best Practices The Easy Button Automatically manage the computers to be patched Machine Smart Label Automatically manage the patches Patch Smart Label Automatically Run Patching Detect and Deploy Force Reboot Run on Next Connection when offline 19 Software

Survey Do you have a dictated patch SLA? What is your timeline? - ASAP - 7 days after patch Tuesday - 30 days after patch Tuesday - Longer Do you have exceptions? 20 Software

Best Practices Plan What is your SLA / patch goal? Timeline OS vs. 3rd party What is your change management policy? How do you want to handle out of band or updated patches What is your approval process? What is your environment central site, remote sites, laptops/mobile workforce, servers? What is your user tolerance for reboots? What is your power management policy? What does your image look like? How often is it refreshed? Do you have internal audit with a 3rd party scanner like WSUS, Nexpose or OVAL? How to prepare for that 21 Software

Best Practices Patch Approval Automation vs Manual Getting the right smart label hands off Using manual labels be sure what is going out Know your change mgmt profile and control downloads Exception Handling Machines that can t be patched Patches that can t be deployed Outliers that don t match smart label Minimizing Schedules Test Production 22 Software

Best Practices Patch Approval 4 Labels Subscription Exceptions Outliers Production = subscription (filtered) + outliers exceptions Variations on the theme Automation for end users, Manual for servers Current Month patches and all previous patches 23 Software

Best Practices Other Thoughts Replication Don t download only detected Maybe don t download every day Testing to Beta to Production Set a calendar around Patch Tuesday Wed is go time Aggressively Attacking machines that aren t patched Drop machines from the target label as they get patched Remember to detect again sometime 24 Software

Best Practices Environmental Challenges Imaging Process Reboot Tolerance/Mandate Laptops/Mobile Workers Power Management Remote Sites Internal/External Audit 25 Software

Best Practices Application Feeds Some Microsoft patches are in the Application Feed OS Service Packs Recommend Manual Approval of Application Patches Installers vs Patches! Installers detect as missing Application Feed is Big Most application patches are actually full installers Application Feed has many versions of each app Application compatibility Know if your apps have Flash/PDF/.NET/Java dependencies New versions can break mission critical apps TEST, BETA, DEPLOY 26 Software

Assessment 27 Software

Assessment Phases and what they mean Scheduled Waiting to Connect Not Scheduled Detecting Deploying Downloading Reboot Pending Completed Suspended Error (Patch Download Failed) Error (Log Upload Failed) 28 Software

Assessment Monitoring Progress Patch List View by deployment label Sort by Not patched to find the biggest problems Patch Schedule to see task completion rate Identifies machines that are offline, not finishing, etc Patch Detail Identifies success of specific patch Fail/Not Patch/Patch/All Common Errors Machine Detail Status Errors - When 29 Software

Assessment Monitoring Progress Reporting Useful Reports that aren t in the box Excel Dashboard Example 30 Software

Troubleshooting Common Error Codes and what they mean Feed Issues Patch Download Log Errors that are likely environmental 20 File Extraction 99 download error 102 pending reboot Error Codes that might need a trouble ticket 80 Error in Patch Deploy Script 98 Detect or Deploy Data Caused Crash 31 Software

Working with Support How to get issues Resolved Quickly Agent with debugging enabled Put the problem set of patches in a label Create a test schedule targeting one or more problem machines Detect and deploy them with debugging enabled After it is done, run capture state to get all logs plus Event Logs Execute the patch manually last to get true error message MBSA 2.2 and Windows update screenshot and if the patch is installed (When applicable to the issue) 32 Software

Questions 33 Software

Resources 34 Software

Select to retype header Select to retype paragraph Select to retype bullets 2 nd level bullets 3 rd level bullets 35 Software

Select to retype header Select to retype subhead (don t change color of subhead or bullets) Select to retype paragraph Select to retype bullets 2 nd level bullets 3 rd level bullets 36 Software

Select to type in a 2-line header that would look something like this Select to retype paragraph Select to retype bullets 2 nd level bullets 3 rd level bullets 37 Software

Select to type in a 2-line header that would look something like this Select to retype subhead (don t change color of subhead or bullets) Select to retype paragraph Select to retype bullets 2nd level bullets 3rd level bullets 38 Software

Select to retype header Select to retype paragraph Select to retype bullets 2 nd level bullets 3 rd level bullets Select to retype paragraph Select to retype bullets 2 nd level bullets 3 rd level bullets 39 Software

Select to retype header Select to retype subhead (don t change color of subhead or bullets) Select to retype paragraph Select to retype bullets 2 nd level bullets 3 rd level bullets Select to retype paragraph Select to retype bullets 2 nd level bullets 3 rd level bullets 40 Software

Select to retype header Subject here Lorem ipsum dolor sit amet, consectetur adipiscing elit (Max. 3-lines of copy) Subject here Lorem ipsum dolor sit amet, consectetur adipiscing elit (Max. 3-lines of copy) Subject here Lorem ipsum dolor sit amet, consectetur adipiscing elit (Max. 3-lines of copy) 41 Software

Select to retype header Select to retype subhead (don t change color of subhead or bullets) Select to retype paragraph Select to retype bullets 2nd level bullets 3rd level bullets 45 Software

Select to type in a 2-line header that would look something like this Select to retype paragraph Select to retype bullets 2nd level bullets 3rd level bullets 46 Software

Select to type in a 2-line header that would look something like this Select to retype subhead (don t change color of subhead or bullets) Select to retype paragraph Select to retype bullets 2nd level bullets 3rd level bullets 47 Software

Select to retype header Select to retype paragraph Select to retype bullets 2nd level bullets 3rd level bullets Select to retype paragraph Select to retype bullets 2nd level bullets 3rd level bullets 48 Software

Select to retype header Select to retype subhead (don t change color of subhead or bullets) Select to retype paragraph Select to retype bullets 2nd level bullets 3rd level bullets Select to retype paragraph Select to retype bullets 2nd level bullets 3rd level bullets 49 Software

Select to retype header Select to retype paragraph Select to retype bullets 2 nd level bullets 3 rd level bullets 1 2 3 4 5 6 7 8 9 X X X X 50 Software

Select to retype header Select to retype paragraph Select to retype bullets 2nd level bullets 3rd level bullets Sales 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 51 Software

Select to retype header Select to retype paragraph (Use stock photo from DAM for photo on the right) Select to retype bullets 2nd level bullets 3rd level bullets 52 Software

Select to retype header Select to retype paragraph Select to retype bullets 2 nd level bullets 3 rd level bullets Subject here Lorem ipsum dolor sit amet (Max. 2-lines of copy) Subject here Lorem ipsum dolor sit amet (Max. 2-lines of copy) Subject here Lorem ipsum dolor sit amet (Max. 2-lines of copy) 53 Software

Select to retype header 1 2 3 4 5 6 7 8 9 Select to retype paragraph Select to retype bullets 2 nd level bullets 3 rd level bullets X X X X Select to retype paragraph Select to retype bullets 2 nd level bullets 3 rd level bullets 54 Software

Select to retype header Sales Select to retype paragraph Select to retype bullets 2nd level bullets 3rd level bullets 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 55 Software

Select to retype header Subject here Lorem ipsum dolor sit amet (Max. 2-lines of copy) Subject here Select to retype paragraph Select to retype bullets 2 nd level bullets 3 rd level bullets Lorem ipsum dolor sit amet (Max. 2-lines of copy) Subject here Lorem ipsum dolor sit amet (Max. 2-lines of copy) 57 Software

Select to retype header Select to retype paragraph Select to retype bullets 2 nd level bullets 3 rd level bullets 1 2 3 4 5 6 7 8 9 X X X X 58 Software

Select to retype header Select to retype paragraph Select to retype bullets 2nd level bullets 3rd level bullets Sales 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 59 Software

Select to retype header Select to retype paragraph Select to retype bullets 2 nd level bullets 3 rd level bullets Subject here Lorem ipsum dolor sit amet (Max. 2-lines of copy) Subject here Lorem ipsum dolor sit amet (Max. 2-lines of copy) Subject here Lorem ipsum dolor sit amet (Max. 2-lines of copy) 61 Software

Select to retype header 1 2 3 4 5 6 7 8 9 X X X X Select to retype paragraph Select to retype bullets 2 nd level bullets 3 rd level bullets 62 Software

Select to retype header Sales Select to retype paragraph Select to retype bullets 2nd level bullets 3rd level bullets 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr 63 Software

Select to retype header Subject here Lorem ipsum dolor sit amet (Max. 2-lines of copy) Subject here Select to retype paragraph Select to retype bullets 2 nd level bullets 3 rd level bullets Lorem ipsum dolor sit amet (Max. 2-lines of copy) Subject here Lorem ipsum dolor sit amet (Max. 2-lines of copy) 65 Software

Divider title here 66 Software