AD Account Lockout Investigation and Root Cause Analysis Allen Chin Principal Consultant allen_chin@symantec.com 1
Contents 1 Background Issue 2 What was done 3 What were discovered 4 Recommendations 5 Challenges faced & Lesson Learned 2
Background Issue 3
Background Issue Downadup/Conficker worm was first discovered during end of year 2008. Like many other organizations, <abc> customer was also faced with this worm outbreak then. One of the side effect when this worm entered the network was to cause huge number of user accounts locked out, rendered these user unable to logon to the network and unable to work. Huge number of lockout incidents still occurred today. Customer decided to create a script in all their AD servers that runs periodically to unlock all locked out user accounts despite the locked out was legitimate or were due to illegitimate logon attempt by this worm. The script has been activated since. The consequence of the existence of this unlocking script is a violation to security practices but more critical would be on the violation to customer internal audit policies and audit findings. Ultimately, customer would like to terminate the usage of the script and be able to be in compliance with audit policies and recommendation. Competitor has been approaching the customer throwing FUD that this event was due to Downadup/Conficker infection and SEP is not able to detect it. Customer was getting convinced as there s no evident to proof Symantec s innocent. Customer indicated that potentially they will displace SEP! Therefore timeline of a month was set for Symantec to perform the necessary investigation to identify the root cause for the lockout event and subsequently lead to script termination. 4
What was done 5
What was done A Special Task Force was formed with collaboration from Symantec, Partner and various customer operation teams (Desktop, Server, Network and Security) to assist on the investigate. Data sources collected and analyzed are based on the following, Locked out reports correlated from existing SIEM solution (Arcsight) to identify source machines causing locked out from various AD servers, Raw AD logs (~38GB logs) from 10 AD servers (out of 50+ AD) Copy of AD account lockout policy, Enduser comments and experience, Logs and loadpoint from various Endpoints that caused ID locked out events, SEPM logs and reports, Logs and reports from 3 rd party scanning tool/av, Reports from existing network IPS. 6
Lockout Trending Statistics Top Accounts 6 5 4 3 2 1 12:00:00 0 AM 4:00:00 AM 7:00:00 AM 9:00:00 AM 11:00:00 AM 1:00:00 PM 3:00:00 PM 5:00:00 PM Sample report 1 : Day with high lock out events 7
Lockout Trending Statistics Top Accounts 6 5 4 3 2 1 12:00:00 0 AM 2:00:00 AM 4:00:00 AM 6:00:00 AM 8:00:00 AM 10:00:00 AM 12:00:00 PM 2:00:00 PM 4:00:00 PM 6:00:00 PM 8:00:00 PM 10:00:00 PM Sample report 2 : Day with high lock out events 8
What were discovered 9
Logon Attempts from Web Proxy What were discovered Web Proxy High account logon attempts triggered from Web proxy 7000 Detected numerous failure audit for account logon attempts These did not correlate with account lockout events Investigated AD account lockout policy with Windows team 6000 5000 4000 3000 FAILURE SUCCESS 2000 1000 0 User IDs 10
Likely Cause of High Logon Attempts from Web Proxy Crypt32 function trying to auto update retrieval of third-party root list sequence number http://support.microsoft.com/kb/317541/en-us Event generated if Update Root Certificates component is installed and computer cannot connect to Windows Update server on Internet Symc Internal Only 11
What were discovered Account Lockout Policy Customer Account Policies/Account Lockout Policy Description Value Account lockout duration Account lockout threshold Reset account lockout counter after 0 minutes (forever / manual unlock) 5 invalid logon attempts 99,999 minutes (69 days) Policy set were too strict, even more aggressive than Best Practice at High Security Level! Best Practice Benchmark Settings (CIS) Center for Internet Security (CIS) Account lockout duration Account lockout threshold Reset account lockout counter after Value 15 minutes or more 15 invalid logon attempts 15 minutes or more 12
What were discovered Account Lockout Policy Customer Account Policies/Account Lockout Policy Description Value Account lockout duration Account lockout threshold Reset account lockout counter after 0 minutes (forever / manual unlock) 5 invalid logon attempts 99,999 minutes (69 days) Policy set were too strict, even more aggressive than Best Practice at High Security Level! Best Practice Benchmark Settings (Microsoft) Microsoft Account Lockout Best Practices Low Med High Account lockout duration N/A 30min 0min Account lockout threshold N/A 10min 10min Reset account lockout counter after N/A 30min 30min 13
What were discovered Kerberos Traffic Being Blocked Windows Firewall blocked Kerberos Traffic 14
What were discovered Kerberos Traffic Being Blocked Event ID 40960, 40961 in Windows XP Client Logs 15
Possible Reason #1 http://support.microsoft.com/kb/938457 Symc Internal Only 16
Possible Reasons #2 Symc Internal Only 17
Possible Reasons #2 Symc Internal Only 18
Possible Reasons #3 Symc Internal Only 19
Possible Reasons #3 Symc Internal Only 20
What were discovered Kerberos Traffic Being Blocked Sample Windows XP Client logs 21
What were discovered Cached Credentials Stale Service Account Password on client machines 22
What were discovered Cached Credentials Stale Credential Manager Entries on Client Machines 23
What were discovered - Others Other observed potential root causes of Account Lockout, Mapped network shared drives/folders with wrongly cached or non-updated credentials on endpoints, Scheduled tasks with logon scripts configured with wrongly cached or nonupdated credentials on endpoints, Scheduled scripts running on some servers that used wrongly configured/cached or non-updated credentials, Old or non-updated logon credentials cached by Applications NTLM authentication error in (McAfee) Web Gateway Failure of Active Directory replication between domain controllers No identification of Downadup worm based on the analysis of the collected loadpoint logs from the identified source machines causing lockout. No identification of Downadup worm using 3 rd party scanning tool/av on source machines identified. No identification of Downadup worm from SEPM and Network IPS reports 24
Recommendations Remedy for Web Proxy logon issue, To allow unauthenticated access to Windows Update server, Turn off Update Root Certificates component, Update AD Account Lockout Policy, inline with Best Practices, Plan for purging of stale credentials inclusive of following areas, Service Account Password in Service Control Manager (SCM) Logon credentials cached by Stored User Names and Passwords in Control Panel (Credentials Manager) Persistent drive mappings / network share, Scheduled tasks and/or scripts, Applications, Health check on Microsoft Active Directory inclusive of, Architecture and configuration review (GC, DC, RODC, DNS) Investigation on errors encountered in Server and Client logs 25
Recommendations Ensure Kerberos & AD traffic/port are not being blocked between Server- Server & Client-Server communication, Adopt Microsoft recommendation for addressing Crypt32 events, more info on URL http://support.microsoft.com/kb/317541/en-us Ensure McAfee Web Gateway is updated to latest level to resolve NTLM related issues observed, Expand coverage and improve performance of current SIEM tool, All AD servers Internal and external FW + IPS SEP Managers Web Gateway Gather baseline statistics of typical daily locked out incidents as a moving forward strategy to ascertain legitimate locked out vs abnormal locked out count, such as caused by presence of Downadup worm in the network, To have client OS image standardization to eliminate deviations from unknown configuration, applications and client access. 26
Challenges faced & Lesson Learned SIEM AD locked out reports collected was not wide enough for lock out pattern identification due to configured AD logging limitation. No effective perimeter or network monitoring capability to identify Downadup threats in the environment. Faced difficulty collecting more detail AD locked out reports from network/siem team due to infrastructure complexity and know how issue. The amount of AD logs collected was extremely huge due to high logging level and large number of users (more than 10,000 seats) therefore it was significantly taxing to mine the data. Task force was not familiar with existing customer applications, map drives and inhouse scripts, hence it was challenging to rule out the root causes. No involvement from other principals in the exercise such as Microsoft, McAfee (Web Gateway/Proxy) and Arcsight (SIEM) that would help to expedite some of the investigation processes. Last but not least, despite all the FUDs the competitor (Trend Micro) has thrown in, customer s confidence on Symantec is still retained and there IS NO evident of Downadup infection that SEP is not able to detect so far! 27
Thank you! Allen Chin allen_chin@symantec.com +6019-212 0126 SYMANTEC PROPRIETARY/CONFIDENTIAL INTERNAL USE ONLY Copyright 2010 Symantec Corporation. All rights reserved. 28