Data recovery Data management Electronic Evidence

Transcription

1 Data recovery Data management Electronic Evidence

2 2 RAID, SAN and Virtual systems RAID, SAN and Virtual systems Data Loss Scenarios: RAID Recovery drive failure Deleted VM recovery Reinstall of ESX on top of VMFS Case Study SAN Deleted Virtual Disks SAN Two Drive Failure SAN Multiple Drive Fail Data recovery tips

3 RAID, SAN and Virtual systems

4 HP MSA 4 Proprietary Kroll Ontrack

5 Traditional Disk Array Approach 5 Proprietary Kroll Ontrack

6 Recovery of Software Defined Storage Many storage platforms now use custom data mapping instead of normal Raids NetApp WAFL HP EVA, 3Par, and Lefthand Dell EqualLogic and Compellent EMC VMAX, VNX and Isilon User Volume Mapping Data Blocks In some recovery scenarios, we must use special tools to assemble the data We have over 100 developers who create custom recovery tools for new systems like these 6 Proprietary Kroll Ontrack

7 SAN Virtual Array Approach 7 Proprietary Kroll Ontrack

8 Cloud?

9 Common Data Loss Scenarios Hardware failures Drive Failure RAID Failure Failed rebuilds Software failures File System Damage Data Corruption Database Corruption VMware Metadata Corruption Human error Deleted VMs, snapshots Overwritten Formatted (Guest and Host level) Lack of good backups Restoring to the same server Swapping wrong disk

10 Virtualized Data Loss Failures Source: Kroll Ontrack

11 Data Loss Scenarios Raid Recovery

12 VMware Recovery Overview RAID Failure Common RAID failures: Multiple drive failure Drives forced online Incorrect RAID rebuild RAID re-configured Disk 0 Disk 1 Disk 2 Disk 3 Disk 4

13 VMware Recovery Overview RAID Failure KO NTFS Recovery Tool KO NSS Recovery Tool KO EXT Recovery Tool KO XFS Recovery Tool Specialized file system recovery tools. All repair applications are developed by Kroll Ontrack Kroll Ontrack VMFS Recovery Tool RAID Device VMFS Volume VMDK1 VMDK2 VMDK3 VMDK4 NTFS NSS EXT XFS Kroll Ontrack RAID Manager Kroll Ontrack Rollback Layer Works like a VMware snapshot to catch any changes or repairs needed Locally attached drives, separated from RAID controller Disk 0 Disk 1 Disk 2 Disk 3 Disk 4

14 Data Loss Scenarios Deleted VM recovery

15 Critical VMFS Metadata Files All VMFS metadata exists in first 1GB of Datastore FDC - File Descriptor Cluster Contains Inodes for all files PBC - Pointer Block Cluster Contains pointers to all large files SBC Sub Block Cluster Contains data for all small files

16 Deleted Virtual Machines When a file on a VMFS volume is deleted, the inode is zeroed, but the PBC pointers and VMDK data still exist until overwritten. VMFS Volume F D C R O O T F D C F B B P B C S B C XP-VM VMDK1 Fragment 1 XP-VM VMDK1 Fragment 3 XP-VM VMDK1 Fragment 2 FDC VXP-VM VMDK1 Inode PBC File Block Addresses

17 Deleted VMDK file recovery Kroll Ontrack engineers can use structures from the guest FS as well as structures Once Using However, When These VMDK a the blocks deleting files deleted from the reference are can data files normally a VMDK file, be within points allocated VMFS file made those it has gathered will to in been obtain up reset resources a of contiguous recovered from the reference file s the storage snapshots points to use in rebuilding can device, remains inode area thousands be of consolidated the engineers there mark of datastore, VMDK VMFS until the can file. the blocks. VMFS on rebuild or resources to fragmented it blocks, and the files VMDK are as well in within reused file s as the it and inode can be analyzed or Kroll PBC thousands present resources Ontrack for it of damage has pieces used a Virtual developed by scattered and the Device repaired file tools across as free to other when search so Kroll necessary. device. they Ontrack tools storage can be for used devices further again. for repairs. VMDK file data in areas no longer referenced by the file system. 1

18 Data Loss Scenarios Reinstall of ESX on top of VMFS

19 Reinstall of ESX on top of VMFS When ESX is installed on top of a VMFS volume, it often overwrites the critical VMFS metadata and often portions of the original VMDK file. EXT VMFS Volume Volume F D C R F F P S New O ESX D boot B and swap B partitions B O C B C C T FDC VXP-VM VMDK1 Inode PBC File Block Addresses Exchange/Oracle/SQL file rebuilt from individual pages.

20 Case Study Deleted Virtual Disks Initial Facts 28 LUNs (28 DataStores) 440 VMDKs snapshots Additional Information Exchange, SQL, Oracle, File Servers all gone Backups also deleted

21 Case Study Deleted Virtual Disk Challenges Customer did not have any documentation Unknown content of virtual disks Changing priorities Recovery Process Connected multiple machines to KO RDR Multiple engineers from around the world processed volumes 24 x 7 3 weeks to recover all of the critical VMs and snapshots

22 Case Study Case Study Deleted Virtual Disks - Closure Defendant sentenced to 41 Months in Prison And over $800,000 in restitution

23 Case Study SAN Two Drive Failure Hardware Details HP EVA drives 124 vdisks EVA hosted a mix of NTFS and VMFS volumes Data Loss 2 drives in the same RSS failed Entire disk group offline

24 Case Study SAN Two Drive Failure Drives shipped to Kroll Ontrack for imaging services 166 good drives imaged 100% 2 bad drives imaged 100% Images of bad drives transferred to new drives provided by HP Imaging services completed in 4 days All drives including new drives returned to customer Drives were replaced in original locations Failed disks were replaced with new drives in original slots Customer worked with HP to bring system online

25 Case Study SAN Two Drive Failure Customer was unable to get the system to recognize new drives Customer attempted to modify the original disks reset and force online Kroll Ontrack engaged to recover vdisks from images Kroll Ontrack virtually rebuilt and extracted the data from all vdisks (VMFS and NTFS) Extracted data returned to customer on encrypted media 29 million files, 50TB of data recovered in 10 days

26 Case Study SAN two drive failure Why this was a success HP suggested data recovery to the customer Imaging all of the disks prevented further damage Failed disks getting worse Rebuild attempts writing bad data or corrupting good data HP assisted customer with rebuild attempts After drives were imaged and returned HP offered continuing support Once data was returned to customer site

27 Case Study Multiple Drive Fail Hardware Details HP EVA 6000 SAN 80 drives, 40TB 2 disk groups, 18 vdisks

28 Case Study Multiple Drive Fail Data Loss Event Server room flooded at the Gothenburg Rescue Service Impact Technology worth millions of Swedish Kroner damaged Rebuild of one SQL database estimated at 20,000 man hours Non-functional backups

29 Case Study Multiple Drive Fail Kroll Ontrack Recovery All but 24 of the drives were recovered in clean room Critical volumes rebuilt even with missing drives Over 4TB of data recovered 100% of critical data recovered 86% of total data recovered

30 Case Study Multiple Drive Fail Differences vs two drive case Supplier recommended against data recovery» Case escalated to senior management Drives were powered on after being submerged» Contaminants caused additional damage including: Head crashes Media damage No images were made before recovery attempts» Rebuild attempts wrote bad data or corrupted good data

31 Data recovery tips

32 Data Recovery Tips How does thick and thin provisioning effect the data recovery possibilities ESX 5 provides three options to provision a VMDK file on VMFS. Flat Disk (Thick Provision Lazy Zeroed) This is the default which creates a normal flat disk. It does not write anything to data blocks until the VM writes to them. Thick Provision (Eagar Zero) This option pre-zeros all blocks in file. It is required to use the ESX Fault-Tolerance feature. It is not good for recovery jobs since it zeros all blocks when the VMDK is created. If an Eager Zeroed VMDK file is created after the deletion of a VMDK file, it can overwrite data that belonged to the deleted file. Thin Provision This option creates a flat disk that does not allocate any blocks on the VMFS volume until they are needed. This usually causes lots of fragmentation, and a deleted recovery of a thin disk is more complex than on a tick disk.

33 Data Recovery Tips Storage Vmotion delete VM s If we are going to recover a deleted VMDK file, many customers need to move their working VMs over to another storage before we start the analysis Storage Vmotion allows a virtual machine s VMDK files to be moved from one VMFS volume to another while the machine is still running.

34 StorrageVmotion move during operation ESX Server 1 - LUN ESX Server 2 - LUN FLAT file SnapShot1 delta SnapShot2 delta SnapShot3 delta CHANGES CHANGES CHANGES MOVE MOVE MOVE MOVE FLAT file SnapShot1 delta SnapShot2 delta SnapShot3 delta 34

35 Data Recovery Tips Storage Vmotion - Example. Deleted VM One VM is deleted, 5 VMs are OK and in use. VMFS don t keep track of the pieces of deleted VMs. So it will be fragmented like this Storage Vmotion deletes the VMs from the source datastore in the same process as they are moved This recovery will be much more complex than if the remaining VMs had not been deleted.

36 Data Recovery Tips Storage Vmotion delete VM s We suggest cloning the VMs instead. Cloning will keep the VMDK files on the original VMFS volume, while the VMs can continue to run on the copies on the destination VMFS volume. We like to have the original VMFS volume kept as original as possible.

37 3 Data Recovery Tips Cloning a VMFS volume If it is not possible to clone the existing VMs over to another VMFS volume, and the original VMFS volume must be kept online, it is possible to clone the entire VMFS volume. This can be done with the command dd in ESX server. If the VMFS volume is presented to a Windows machine, it can be cloned with for example FTK Imager Lite. We can work with both raw clones (disk to disk) and file clones (disk to file). However, if the VMFS volume is cloned to a file, the file must be stored on an NTFS file system that is mounted on a Windows machine for a Remote Data Recovery to be possible.

38 3 VMware Tip Sheet

39 Meer klant cases

40 Contact Jaap Jan Visser Kroll Ontrack Netherlands Holland Office Center Kruisweg 825c 2132 NG Hoofddorp The Netherlands Fax: Mobile: [email protected] 4

41 Data recovery Data management Electronic Evidence