Active Directory Forest Disaster Recovery: What You Don t Know Will Hurt You

Transcription

1 Active Directory Forest Disaster Recovery: What You Don t Know Will Hurt You By Gary L. Olsen Sponsored by

2 Table of Contents AD Disasters... 1 AD Forest Failure Causes and Case Studies... 2 The Danger of the Domain Admin Account... 2 Improper Authoritative Restore... 2 Reverse Time Change... 3 Schema Change... 3 Forest Functional Mode reversal... 4 USN Rollback... 4 RID Pool Depletion... 4 Failure of all DCs in forest root... 5 The Case for Backups... 5 Recovery... 5 Recovery Manager for Active Directory Forest Edition... 6 About the Author... 8

3 There are many forms of Active Directory (AD) disasters that will cause a significant impact on your business. Since AD is the core authentication method for many enterprises, mission-critical (and standard) applications, access to files, and many other functions businesses rely on are impacted by AD failure. Needless to say a speedy and efficient recovery from AD failure is critical in order to minimize business impact. Today, with companies reducing IT staff to save cost, IT staff members must wear many hats, making it more difficult to not only monitor AD but to repair problems. In the case of AD disasters, it is critical to recover and get everything back online as soon as possible. A speedy recovery of mistakenly deleted objects, for example, is not only critical to keep business functions online such as , file access, authentication and critical applications, it s good job security for the IT staff. As an IT professional, you don t want to be reading a step-by-step recovery procedure while the CIO watches over your shoulder, waiting for his account to be restored so he can send out a critical . AD DISASTERS While AD is amazingly self-healing, there are AD failures that require recovery and are well documented not only by Microsoft, but by many authors in books, whitepapers, and Internet articles. These failures can be in the form of Domain controller failure Accidental bulk deletion of objects (i.e. deleting an OU with 10,000 users) Domain failure Forest failure Of the failure types in this list, only a complete forest failure should make you lose sleep. A domain controller failure in a properly configured domain with a minimum of two and preferably three DCs in a domain is not a big deal. The other DCs pick up the load, and while you may have a performance hit, it won t be a disaster. Repair/replace the server, re-promote or restore from backup, case closed. Recovery from accidental bulk deletion of objects, more difficult in a multi-domain forest, has been mitigated with NTDSutil features in Windows 2003 and the default condition in Windows 2008 where deletion of an OU object is prohibited (Figure 1). Note that this option is available (though not enabled by default) for any object created in the AD Users and Computers or Sites and Services snap-in. While recovery of objects with native Windows tools is well documented, and can be done, it is not for the faint of heart. Remember you get what you pay for. One of the most time-tested and efficient tools for AD recovery is Quest s Recovery Manager for Active Directory. RMAD takes all the procedures and best practices and presents them in an easy to use menu-driven application. No worrying about restoring Figure 1 backlinks, users before groups, etc. Remember that recovery of AD objects is time sensitive for your company. Some might say the Active Directory Recycle Bin, introduced in Windows Server 2008, is a great recovery tool. This tool certainly has its place but has a number of drawbacks: It must be manually configured or scripted there is no UI You must enable the Active Directory Recycle Bin and it cannot be reversed (except via a forest recovery) It really just lets you recover more attributes than otherwise possible Not really a tool for forest recovery Need to upgrade to forest functional mode Does not cover the recovery of Group Policy Objects For more information on the Active Directory Recycle Bin, read Ned Pyle s TechNet blog at: Any discussion about AD disaster recovery usually describes domain failure as the next level of DR. In a single domain forest, this equates to a forest recovery and requires all DCs to be restored to a known good state. In a multiple domain forest, recovering a single domain in the forest is more complex for reasons such as: Global Catalogs need to be rebuilt Global Catalogs in other domains may hold orphaned objects of the broken domain Lingering objects could occur in the restore of the domain External trusts must be re-established Even so, domain failures can be recovered. Recovery of the entire forest is a challenge in a multiple domain forest and is the focus of this whitepaper. While there are many articles written about AD forest recovery, only a few even touch on causes, and they only mention some abstract idea, without examples. It is important to understand specific causes of AD forest failure in order to properly prepare for it. 1

4 AD FOREST FAILURE CAUSES AND CASE STUDIES While a complete forest failure is somewhat rare, it can and has happened. The danger is often from IT staff that fail to recognize the causes of such a failure and fall short to proactively prepare for such a disaster. If a forest failure only occurs once in the history of the company, failure to recover could mean the business failure of that company. It should be obvious that preparation for such a disaster is worthwhile. For a complete forest failure to happen, the entire AD database would have to be corrupt and irrecoverable, or all DCs in all domains would fail at the same time with no valid backup in a single domain forest, meaning the domain is the forest. At first blush, it would seem that these conditions would be rare if not impossible. Consider the following possible conditions that could exist, requiring a forest recovery. Note that I ve included actual experience and case studies to reinforce these cases. The Danger of the Domain Admin Account Few IT professionals who administer Active Directory understand the power of the Domain Admin account. This account cannot be trumped or limited. Its rights can t be diminished or restricted. Microsoft has long taught that the only security to the Domain Admin account is to trust the administrator. I ve worked many issues with IT managers who want to protect the AD environment from their domain admins who they don t know if they can trust. Tongue in cheek, I suggested polygraph tests. In one case, a domain admin was disabling auditing, performing some destructive operations, and then re-enabling auditing. All we could see is the auditing change was made and when, but not by who. Did you know the Domain Admin account: Can turn auditing off, do something bad, then re-enable auditing? This can t be traced even in a network or ETW (Event Tracing for Windows) trace without third-party software. Can reset the Directory Service Repair Mode (DSRM) password? This allows the administrator to boot a DC offline and use NTDSutil to do things like authoritative restore and run database utilities. NTDSutil allows any domain admin to change the DSRM password without knowing the existing one. Can modify forest-level objects? Read the example in the next section Improper Authoritative Restore. Has privileges in the domain that an Enterprise Admin account doesn t have? In a more serious case, I was reviewing the AD design for a global company who revealed that they had no idea how many users had Domain Admin privileges. This is a disaster waiting to happen. In looking for causes for forest failure, remember a rogue, or untrained Domain Admin could be the root cause of such a failure and will be noted in examples shown in this whitepaper. It is best practice to limit the number of users with Domain Admin privileges, but a disgruntled employee or even an inexperienced employee with Domain Admin privileges can cause catastrophic failure. Improper Authoritative Restore One of the tools Microsoft provides is the Authoritative Restore, available in the NTDSutil command line tool. Authoritative Restore, typically used to recover bulk deletion of objects, rolls AD back in time for a single object, a subtree of objects or the entire AD database. In one case, I worked with a university who had a single forest but about 80 domains. Each college in the university that had its own domain so as to be autonomous. This worked well until one day a domain admin decided to do an Authoritative Restore. Probably due to either ignorance or mistakenly choosing a wrong option, this admin recovered the entire configuration naming context from a two-week-old backup, in addition to his domain objects he was trying to recover. Combined with the university s 3rd level IT staff having removed and added several DCs after that backup was made, it had a disastrous effect of reanimating deleted DCs and losing existing DCs. Since this information is held in the configuration naming context, it affected many of their domains. While they were fortunate that this event did not result in a complete forest failure, it easily could have. The damage stretched across many domains and caused widespread outage, which took many days to repair. Note that with proper tools, such as Quest s Recovery Manager for Active Directory Forest Edition, recovering the entire forest even one of this size could have put the environment back online in hours. Thus the forest doesn t need to completely fail to take advantage of such a tool. The results of this outage were: 1) the administrator responsible was fired and 2) the IT manager wanted to know why a domain admin had the ability to restore the configuration naming Figure 2 2

5 context. This is an example of the power of the Domain Admin account and demonstrates that misuse of the power of this account can indeed cause a failure that requires forest recovery procedures. Note that to access the powerful NTDSutil tool to perform Authoritative Restore, delete server objects and other destructive actions, the DC must be booted into Directory Service Repair Mode (DSRM), which has its own local administrative account and password. If a domain admin doesn t know that password, he or she can simply login to their Domain Admin account, use NTDSutil to change the DSRM password, reboot to DSRM and login. Again, the domain admin has all power. Reverse Time Change Active Directory uses the Kerberos authentication protocol, which relies heavily on accurate time between all domain clients and servers, with the domain controllers being time servers. Figure 2 shows the time services hierarchy in a multiple domain forest. Each domain has a PDC Emulator (PDC) that serves as authoritative time server for all DCs in the domain. When clients join the domain, the DC they authenticate to becomes their authoritative time server. In the multiple domain configuration, the child domain PDCs use the root domain s PDC as their authoritative time server. As a best practice, the root domain PDC synchronizes to a reliable external time source. Kerberos requires each client computer in the domain (and forest) to be in sync with DCs and servers, within five minutes by default in Windows domains. This is configurable but it is not recommended to change it. Thus, if a client computer tries to authenticate to a domain and the client s system time is more than five minutes skewed from the authenticating DC s system time, the authentication will be denied. This applies to applications, file access, and all server resources using authentication for access. The system time is always measured in UTC and is not affected by time zones. Time skew failures can cause: Computer account password failures Trust account password failures AD replication failures Any application requiring Kerberos to authenticate In one instance a company was experiencing a widespread authentication failure of all clients in all domains. The root cause was found in a system log event shown in Figure 3. The external time source has mistakenly set the time for the root domain PDC in reverse one year. This caused massive errors in the system logs, the Netlogon log and Directory Services logs. These included Kerberos service ticket errors notifying of expired passwords, target principal name errors, etc. The rogue time server, about an hour later, reset the time a year forward. This cycle repeated several times. We were able to restore service after many hours of troubleshooting, but then the time reset cycle was repeated and we had to do it over again before finally discovering the cause, which we solved by changing to a reliable external time server. We also discovered a prevention of this situation, described in KB Setting two Figure 3 registry keys (a maximum and minimum value) prevents the time change from occurring greater than a defined amount. We set these keys so that the time could not be changed more than 15 minutes either way. Note that the pre-2008 default was 0xFFFFFFFF, which would accept any change. Windows 2008 reduced this to a 48-hour default. Good protection. While we recovered from this instance, what if the time source had not corrected itself? There would have been widespread lingering and orphaned objects because garbage collection would have purged the tombstoned objects prematurely, and once that happens you cannot recover them from backup. This can cause replication failures, authentication failures, etc., depending on the objects involved. There have been instances reported where the time could not be rolled back (or forward depending on the circumstance), which required a forest recovery. This could be a true disaster. Schema Change Most discussions around forest corruption usually mention schema changes. While the ability to modify classes and attributes in the schema is limited to Schema Admins group (in the forest root domain), accounts can be added. I discussed the possibility of schema corruption with one company, and they were not worried because they have no developers or others who make schema modifications. I then asked about popular AD aware applications not only Exchange but third-party applications and troubleshooting tools. We made a long list of these applications and they were surprised at how many applications had made changes to the AD Schema. This is a great feature allowing AD extensions to leverage AD authentication and tightly bind the application to AD, but the downside is that they modify the schema in an unknown fashion. And what about uninstalling those applications and tools? The classes and attributes they created cannot be purged (though they can be defuncted). The only way to fix schema problems, since all DCs hold a copy of the schema, is to roll back the entire forest to a 3

6 known good state before the schema problems occurred. Schema issues that can require a forest recovery include: Application with a schema extension that corrupts the schema Application or programmatic change that modifies a schema component that other applications or system functions use and stops them from proper operation Clean up schema extensions. (there is no purge capability) It is a best practice to test any application schema extension or programmatic schema change in a test lab that mimics the production environment. A thorough testing of these changes will prevent surprises when it is introduced into production. Don t rely on vendor reports. Forest Functional Mode Reversal Beginning with Windows Server 2003, the Forest Functional Mode (FFM) was introduced, adding to the Domain Functional Mode (DFM) from Windows Server In order to take advantage of new features that apply to the configuration naming context (such as the replication performance improvements in Windows 2003), the forest had to be at Windows 2003 FFM. In order to switch to FFM 2003, all domains must be at Windows 2003 DFM. In order for a domain to be at Windows 2003 DFM, all DCs in the domain must be Windows Server This is referred to as native mode. While a domain or forest can be in mixed mode and contain down-level versions of Windows server, all the functionality of the newer version of Windows won t be available. This is true for Windows Server 2008 and 2008R2 as well. (Note: this applies to domain controllers only not member servers or clients). This action is one-way. Once the DFM in all domains and the FFM in the forest is switched to Windows 2008R2 mode, for example, it cannot be reversed. Occasionally there will be reasons such as a legacy application, or perhaps even a licensing or funding issue, that prevents some DCs from being upgraded for a time, so sometimes it isn t as easy as flipping the switch. If the DFM or FFM is switched to Windows 2008R2 and it is discovered later that there is some requirement that one or more DCs have to be at Windows 2008, the only way to reverse this and put the DFM and FFM back to Windows 2008 mode is to restore the forest. While testing of applications in a lab environment might prevent this, testing every condition with every application is not practical, and adopting new applications that have a requirement to use Windows 2003 DFM, for example, is not always predictable. Remember the discussion on the power of the domain admin? Changing the DFM or FFM could be performed by a rogue or uninformed domain admin. USN Rollback The USN rollback is essentially an unintended authoritative restore. It is caused by a DC being restored from a backup that was not created with an AD-aware backup utility, which does not reset the invocation ID. Microsoft KB microsoft.com/kb/ describes this. This can also happen if a virtual DC is restored by backing up and restoring the virtual machine file rather than backing it up inside the virtual machine instance itself. The result is that the DC loses touch with the other DCs for a time and does not replicate with its partners. This DC reports to its partners that it is up to date (though it s not) and the partners don t replicate updates to it. The only repair is to manually demote the DC, clean up the AD and promote it again. This could easily happen in a domain that has all virtual DCs. If they are improperly backed up and then restored, one by one for various reasons, the problem could spread over time until no DCs are replicating, thus requiring a forest recovery. RID Pool Depletion This is a rare condition but it has happened. The Relative Identifier (RID) is the last part of the Security Identifier (SID): The RID must be unique in the domain. The RID Master is a FSMO role holder in each domain, which has 230 (or about 1 billion) RIDS. The RID Master hands out 500 RIDS at a time to DCs, who in turn assign them to newly created security principals user, computer, etc. If the RID Master runs out of RIDs, there is no recovery but a complete domain recovery. It would seem that depleting one billion RIDs would be difficult. For example if a domain consumed 38,462 RIDS per business day (five days per week) it would take around 100 years to deplete it. But consider: If a DC is restored from system state backup, it takes another 500 RIDs (new pool) RID Master never re-uses RIDs once assigned to a DC Re-promoting a DC takes a new 500 RID pool Provisioning tools can create/delete computer accounts, user accounts, etc. Microsoft discovered one provisioning tool that was creating 4 million computer accounts a month! Lab scripts that create lots of users. I have one that I use in testing that I configure to create 10,000 users every time I run it. Microsoft has worked support cases where the RID pool was exhausted. Microsoft KB is a good article as well as the TechNet Blog at managing-rid-pool-depletion.aspx. The prevention is to monitor the RID pool, find the problem and fix it before RID pool is depleted. But honestly, how many AD admins can really monitor 4

7 the RID pool? They usually have a hard time monitoring AD replication and common failures. Once it is depleted, you have to recover the domain. You can t even migrate it to a new domain because you can t create a new trust without any RIDs available. This means restoring the domain to a pre-depleted state, but the problem needs to be resolved or it will occur again. The most deterministic way to do this is to recover the domain then build a trust to a new domain and migrate the accounts. Failure of all DCs in Forest Root In the case of the multi-domain forest, the forest root holds the forest together. There are certain components that only exist in the forest root domain including but not limited to: GC and domain DNS resource records in the _MSDCS DNS domain in the forest root domain Trust objects for trust relationships with the other domains Security groups such as Schema Administrators Root level objects Obviously, if all DCs in the forest root become inoperable with no available live DCs available, the entire forest would require recovery. While this would seem to be an unlikely probability, I actually assisted a company to recover from just such an incident. In this case the IT staff had a root domain we ll call Corp.net with two child domains, America.corp.net and EMEA.corp.net. Each had a delegated zone and all user accounts, member server accounts, etc. were in the child domains. The root Corp.net domain was a placeholder and held administrative accounts only. While the child domains each had two domain controllers, the root domain had only one. But, they reasoned, they had a RAID 5 configuration across 4 drives so they were safe. Things were fine until one catastrophic day when two drives in the RAID set failed, and when they went to get the backup tape it was 11 months old. In spite of the complete unavailability of the root domain, there was no impact to the users. Of course it had to be fixed. I contacted a number of colleagues, but no one had experienced this and could offer no solution other than a forest recovery. Spending over a week, since users were not affected, they built a lab to reconstruct what the production environment had, then performed the following steps: 1. Built lab using current backups on a private network 2. Set the system time on the root DC to date of the backup 3. Restore 11-month-old backup to root DC 4. Set the tombstone lifetime to 365 days to prevent lingering objects 5. Set Strict Replication Consistency key to 1 to prevent lingering objects. This can be done with the native Repadmin tool using: Repadmin /regkey DC_List +strict. Note that best practice dictates this key to be set to strict in normal production conditions. (see 6. Uncheck the GC option on Corp-DC1 (so it is not a GC any more), then re-enable it after recovery is complete 7. Health Check the DCs to make sure there are no errors (verify trusts, check event logs, etc.) 8. Let replication take place 9. Test cross-domain authentication 10. Re-enable the DC to be a GC 11. Promote at least one additional DC in the root domain 12. Repeat the process in production In my opinion, this company was very lucky, but it demonstrates that it is not all that hard for a forest recovery to be required. There are a number of things that could have made this impossible to recover the way we did. Note that this took about three weeks to complete. It could have been prevented with a second DC or a current backup. The Case for Backups In the several case studies described previously, especially the case with the forest root failure, it should be obvious that backups are critical. Making daily backups are a fundamental operation of any data center of any size. Many organizations even require users to perform daily backups. Yet in the 12+ years I ve worked in Active Directory support, I m amazed at how many admins who call don t have a valid backup. Of course those that do have backups usually don t need to make a support call. Backing up domain controllers requires special considerations that normal server backups don t require. Failure to perform these backups will negate even the most diligent recovery efforts. Here are the criteria for making successful domain controller (AD) backups: Use backup software that is AD aware. That is, the invocation ID must be reset on the restored DC. Without this feature, a USN rollback will occur. It is only necessary to back up/restore system state Backups are only good for the tombstone lifetime (TSL). If the TSL is 180 days, backups are only good for 180 days. Attempting to restore backups older than the TSL will fail. Validate backups occasionally test restoration in a lab. Make sure the backups work! Provide for offsite storage of backups. Natural and military disasters such as 9/11, Hurricane Katrina and others should be enough evidence that offsite storage perhaps out of city or state storage is in order. Also consider putting DCs in geographically disperse sites. This privides an online backup much faster than restoring from media. Recovery Microsoft authored a forest recovery whitepaper in 2006 that was updated in

8 recent backup that did not have the problem that is requiring the forest recovery, and hopefully the PDC, as it will have the latest password changes and serve as the authoritative time server. If it is a DNS server and contains other FSMO roles, it will save time later. Make this DC a GC Use DCpromo to create at least one other DC in the root domain Repeat this process to each of the child domains. Enable GCs in all domains Post-recovery steps including recovering or rebuilding any user accounts that were created after the backup was taken. There could be computer accounts, password resets, etc. required. This sounds pretty straight forward. However, using DCpromo to restore each DC, then performing full sync of all GCs in the forest can be problematic. If the AD sites are in a small geographic region and the AD database is small in size, this could be a valid procedure and certainly allows a clean build. However, if the AD database is 5-8 GB as is the case for many environments, or if the sites are scattered across the globe, or if there are slow network links, using DCpromo across the network could take a long time. Figure 4 Like any other whitepaper from Microsoft, it uses native tools and describes the general idea required to restore an entire forest. While it does represent excellent work on behalf of Microsoft, it is very complex and is 55 pages long. It covers planning phases, how to determine if you do need a forest recovery and how to recover the forest. While this whitepaper has outlined many causes of AD failure that would require forest recovery, the Microsoft document is an excellent read for additional background. Certainly the decision of whether to perform a forest recovery is critical to the process, and Microsoft is lending us their experience in this area. In addition, this is a complex process with a lot of steps. And again, do you want the CIO watching over your shoulder while you and your team attempt to restore this? What is needed is a tool to do this, and Quest offers Recovery Manager for Active Directory (RMAD) Forest Edition for this purpose. Recovery Manager for Active Directory Forest Edition Quest designed this tool using Microsoft s forest recovery whitepaper as a guide and basically wrapped all the steps, Microsoft describes forest recovery to contain several key steps. Of course this is assuming current, valid backups are available. Physically isolate all DCs from the network Use backup to restore a DC in the forest root domain. This should not have been a GC before the failure, contain a writeable copy of the most Figure 5 6

9 console or network share (unlike the native Install From Media) Send and other notification formats in case of backup failure The properties for each DC in the collections show the backup history of the DC. Running a test in the lab, Computer Collections were made for the root domain (Acme.lab) and for two child domains (Finance and Sales). With valid backups created, the Forest Recovery Console will: 1. Use Backup Criteria to define the backup to be used (Figure 5) a. Optionally configure alerts to send notification on progress b. Optionally configure pauses to create delays at specified points 2. Select Verify Settings to verify backup configuration 3. Select Start Recovery Figure 6 configurations and best practices into a menu-driven tool that will allow a very neat, clean forest recovery and ensure that the proper steps were followed. Reviewing the tool, the first thing that jumped out at me was the backup management capability. One of the challenges with AD disaster recovery, as has been described previously, is to create timely, valid backups. A recovery on any level is only as good as the backups to recover it from. Backups have to be taken regularly, and they must be local to the DC that is to be recovered. Imagine a tool that will permit configuration of backups on all DCs in all locations, then use those backups to restore the DCs locally when needed forest recovery or otherwise. RMAD permits full configuration and control of backups. Computer collections are created based on management criteria such as backups, and backup agents are deployed from the console. The properties for each collection permits: Figure 6 shows a warning displayed at the beginning of the recovery process. This is a nice feature to remind the admin of critical steps that should be taken before the recovery is begun. Figure 7 shows a snapshot of the recovery in progress. I was impressed with the detail of the console. Note that each DC in each domain is displayed; showing the site, FSMO roles, backup used and age of the backup. In the lower portion, individual progress can be shown for each selected DC. Note in Figure 7, the DSRM administrator password is reset (configurable). Figure 8 shows the final stages of the recovery. The progress display is helpful to know what is actually being done. It s easy to see that the tool is taking care of all the recovery details required and eliminating not only the time required by the admin, but human error as well. System state configuration Scheduling (including purging of old backups) Password encryption Storage on a DC (Figure 4), Figure 7 7

10 All of the actions noted here can be done from a single or multiple consoles. It might be desirable to install a console in multiple data centers (such as a data center in each region). These consoles can be tied together so that if one is not available, all DCs (backups and recovery) can be managed from any of the consoles, making the consoles themselves redundant. Overall, Active Directory forests can fail for a variety of reasons, including the ones described in this paper. When a full forest recovery is required, it is critical to the business of the company to restore AD as soon as possible. While there are manual processes expertly defined by Microsoft themselves to perform a successful Figure 8 recovery, it can be a long, time-consuming process and is subject to human error. In addition, having properly validated backups taken regularly is crucial to such a recovery. You can t recover what you don t have! Recovery Manager for Active Directory Forest Edition simplifies the recovery of a domain or forest in the event of a major disaster, quickly restoring your entire domain or forest to a point in time before the corruption occurred. By selecting unaffected backups, quarantining the damaged environment and automating all the manual steps required to facilitate the recovery, Forest Edition greatly reduces downtime. This process is closely aligned with Microsoft s native forest recovery approach and provides the option to restore some domain controllers (DCs) from backup and others through demoting and re-promoting them with DCPromo. For more information regarding Recovery Manager for Active Directory Forest Edition, go to hyperlink recovery-manager-for-active-directory-forest-edition. ABOUT THE AUTHOR GARY OLSEN holds a BS in industrial education and a master s degree in computer-aided manufacturing from Bringham Young University. He has worked in the IT industry for more than 20 years, and has served on Microsoft s Beta Technical Support Teams for both Windows Server 2003 and Windows 2000 Server; as an HP/ Compaq consultant on Active Directory design; and as a provider of advanced technical support through HP s Customer Support Center. A frequent speaker at Windows and HP technical conferences, he is author of Windows 2000: Active Directory Design and Deployment. Sponsored by 8