Disaster Recovery Cookbook Guide Using VMWARE VI3, StoreVault and Sun. (Or how to do Disaster Recovery / Site Replication for under $50,000) By Scott Sherman, VCP, NACE, RHCT Systems Engineer Integrated Storage Consulting Services Inc scott@eiscsi.com http://www.eiscsi.com 1
Table of Contents Is this you?... 3 About this Cookbook... 4 Primary Site Setup... 6 Moving your Virtual Machines from Local Storage to the iscsi LUN using VirtualCenter... 16 How to move the virtual machines if you are not using VirtualCenter.... 19 VI3 setup for Disaster Recovery Site... 30 StoreVault setup for your Disaster Recovery / Failover site... 36 StoreVault Replication Setup... 44 Testing the replicated LUN... 47 Solution Pricing and Summary... 59 Revision Notes... 61 2
Is this you? A. Your boss has read an article on his last business trip about Disaster Recovery and tasks you with the job. In the past this would be a monstrous expensive task involving redundant hardware, clustering software and a hope and a prayer that it works. B. You have a few mission critical servers that you want to keep online without breaking the bank. C. Maybe your worried that if you lose connectivity to your primary site for a period of several days that your business would suffer D. You have gone through the personal hell of trying to restore a mission critical server from tape, only to find the last tape is lost, damaged, or eaten by your tape drive. Now you want it replicated. This cook book is for you. This is a step by step guide on how to setup an easy to manage and cost effective Disaster recovery solution for your environment. 3
About this Cookbook For the past year or so I have seen many whitepapers on how to setup Disaster Recovery solutions using various VMWARE and disk solutions. Most of these papers have been based on theory with little showing the how to, or they pitch expensive Professional Services offerings to set it up for you. After some thought and discussions with my peers I realized that this could be done using current offerings from vendors, with a straight forward implementation. So here it is. This is a guide designed to show you how do setup replication to a DR site in a step by step fashion. This cookbook may seem large but it is documented with screenshots and dialogs to get setup going very quickly with readily available technology. Introduction. This paper is broken out in sections which allow the setup of the primary site on SUN Hardware, VMWARE VI3 and StoreVault storage. Then we will take these virtual machines and replicate them via the StoreVault Replication Service to have them run in another site. By using this combination of hardware and software it gives the ability to have a Highly Available server farm without the expense of clustering software. Please remember that these configurations below are baseline configurations and that you may want to change, add, or modify the hardware depending on your Operating System, Applications, Users and Network Speed. High Level Setup Attached is an overview of the basic networking using my primary site behind router #1 with two VMware servers. I have only one VMWARE server in my DR site to reduce costs. Assuming that you are not Depending on your setup you might be able to get by with a single server at your DR site. (Depends on how many virtual machines are mission critical and the performance characteristics of your applications. More servers can be added for your own needs. Also the link between the sites is on business class internet service (8mb down / 1mb up) and not on a dedicated circuit. The sites are tunneled via the routers for security. 4
Assumptions 1. This document assumes that you have basic familiarity with VMWARE and STOREVAULT products 2. Those machines have already been setup on the primary and DR site using VI3. 3. That my mission critical servers are currently in VMWARE and hosted on local disk (They could be on SAN/NAS, we are going to show how to move them to an iscsi LUN which will be replicated) Disclaimer This paper is presented as a cookbook guide on how to setup a DR solution using STOREVAULT, SUN, and VMWARE products. Your environment may require specific changes that are not covered in this paper. These companies did not sponsor nor endorse this paper and are trademarks of their respective companies. Scott Sherman, the companies mentioned nor Integrated Storage assume no liability on the use or implementation of these concepts. For further information please contact us. 5
Primary Site Setup First we need to setup your primary server for access to an iscsi LUN on your primary StoreVault. This is show on in the following section by moving your virtual machines from local disk (residing on the local server) to an iscsi LUN on the StoreVault. Log into the primary site (I am doing this via the VIC server) where we have 2 VMWARE servers (IP 192.168.1.150 and 192.168.1.160). Currently my virtual machines are stored on local disk (Storage1(1)) and NFS mounts. What we need to do is enable iscsi on the systems. Starting with 192.168.1.150 goto the configuration tab, then click on networking 6
After verifying that the VMKernel is configured in the networking section of your server. (for more info about this please refer to the VI3 documentation). This is what allows VI3 to communicate via the iscsi protocol to the StoreVault. A separate IP address is needed for the VMKernel. In our example it has an IP of 192.168.1.151 Then we have to allow iscsi to be passed through the VI3 firewall. From the main screen click on Security Profile Click properties Click on Software iscsi Client Click ok. At this point I reboot the system to ensure all changes take effect. 7
After the reboot log back in to the VI3 client, click on Storage adapters then the iscsi Software adapter Click on properties and the iscsi Initiator properties will come up. Click configure then enable and ok. Click over the Dynamic Discovery tab Click Add and enter the IP of your primary site StoreVault Click Ok. This will tell your VI3 server to send it s initiator to the StoreVault 8
At this point log into your primary site StoreVault and create a LUN big enough for your mission critical Virtual Machines. My LUN is 200gb. (You can check the size of each machine by logging into the VI3 client, and look at the size of each of your virtual machine disks) Click Create LUN(SAN) Click Next 9
Add the IQNS that were in VMWARE to the LUN 10
If your IQN didn t show up you can add it manually as follows Right click on the LUN and choose manage LUN Click on initiators and verify that you can see 2 entries for each machine. 11
Now that your LUN is created via the StoreVault GUI we need to go back to the VI3 client. Click on Initiator group and add the missing system into the group (My missing machine is the x345-ip 192.168.1.150). Click save, then ok at the successful prompt. Go back to the VI3 client Choose rescan under the configuration / storage tab 12
Click on storage (SCSI, SAN and NFS) Choose Add Storage Disk / LUN Choose the LUN we created in the StoreVault GUI Click Next 13
Click Next Name the datastore (Note- on the primary site the Datastore name MUST be the same to allow VMOTION if applicable.) 14
Click Next Click Finish REPEAT THE STEPS TO ADD THE VOLUME TO OTHER MACHINES AT YOUR PRIMARY SITE 15
Moving your Virtual Machines from Local Storage to the iscsi LUN using VirtualCenter Log onto the VMWARE VirtualCenter. Choose your virtual machine right click and choose Migrate Choose the same host 16
Click Next This is the key option. Choose Move virtual machine configuration files and virtual disks. Then the datastore choices will appear. Choose the datastore we created on the StoreVault. Click Next 17
Click Finish Repeat these steps for all virtual machines you want to replicate to your DR site. 18
How to move the virtual machines if you are not using VirtualCenter. Moving the virtual machines is still fairly easy if you don t have the VIC server. It just makes use of the command line. 1. Goto your VI3 console, log in at root and create another user with the useradd command. This is necessary as the default security profile in VMWARE does not allow users to log in remotely as root. You must login as another user then switch user to root. This option can be changed, please refer to www.vmware.com for info about this. Then issue the passwd command with the username right after it. My example has the username of druser. You will then type the password twice then when back at the # prompt. Type exit Download a ssh client from the web. For this paper I m using PUTTY which can be found by googleing putty.exe. When you execute putty you should see this screen 19
Enter the IP of the server we want to migrate the virtual machines from (Mine is 192.168.1.160). Click open. If this is the first time you login to this machine via putty, you will get a message about accepting the key. Choose Yes Login as the user we created Type in su root then the root password Directory structure of VMware / = root /vmfs = where virtual machines are located /vmfs/volumes/storage1/ = is where virtual machines are stored on a local system /vmfs/volumes/xxxxx = XXXXX is another datastore location, it can be NFS, iscsi or Fiber Channel. 20
We need to move the files in the /vmfs/volumes/storage1 directory to the iscsi LUN located at This command copies the directory /vmfs/volumes/storage1/root to the iscsi lun which is located at /vmfs/volumes/missioncritialscsi This WILL TAKE about 3 minutes per gig of data (may vary depending on your network, please be patient). Also this leaves the original files on the local machine. 21
After the copy we verify the files are in the correct directory. We now need to point the VI3 server to the virtual machines which we moved tot the iscsi LUN. Log onto the VI3 client to the machine where the files are Click on the machine, click on New Virtual Machine 22
Choose Custom, click next Name the machine 23
Choose the iscsi LUN Choose the Operating System 24
Processors Memory 25
Network Adapters I/O Adapters Choose Use existing virtual disk 26
Browse to the iscsi LUN and the correct directory (Mine is root). Choose the filename.vmdk Choose Next 27
Click Next Click Finish 28
The new machine is now available the VI3 client. You can delete the old system after you have verified it works correctly. 29
VI3 setup for Disaster Recovery Site Remember to make machine HOST BASED LICENSE, unless you have a license server at the DR site unless you have a license server at your DR site. Click on machine name Configuration Tab Add Networking Choose VMKERNEL, click next Note You can choose to put your VMKERNEL on another virtual switch, for this case I m leaving it on vswitch0 30
Click Next Then click finish on next screen You may get this Click Yes 31
Enter your Default Gateway Click ok From the main screen click on Security Profile Click properties 32
Click on Software iscsi Client Click ok. At this point I reboot the system to ensure all changes take effect. Log back in and choose the configuration tab Storage adapters then the iscsi software adapter Click properties 33
Choose Configure Click Enable then ok Now click on dynamic discovery Click Add 34
Enter the IP of your StoreVault Click ok ISCSI is setup on your VI3 DR BOX. Now we need to see the StoreVault GUI 35
StoreVault setup for your Disaster Recovery / Failover site Login to the DR StoreVault box For testing we will create a LUN Click Create LUN (SAN) 36
Click Next 37
Map over your iscsi initiator by highlighting and clicking the right arrow. You can verify which is the correct iscsi IQN by verifying in the VI3 client under storage adapters. (For our example the x235-595ae094 is our target) 38
Click Commit Success, click finish Go back to the VI3 client 39
Configuration Rescan Click ok Your LUN should be created and appear in the details section 40
Click storage (SCSI, SAN and NFS) and choose Add Storage Disk/LUN Your LUN should appear, click next 41
Click Next Name your DATASTORE 42
Click Next Click Finish Your test lun should appear under storage 43
StoreVault Replication Setup At this point we have both sites setup with VI3 and talking to the StoreVault at each location. Now we will setup the LUN with the virtual machines to replicate from the primary site to the DR site every hour. Login in to the StoreVault Client on the primary site Choose Manage Replication from the LUN we want to replicate Click Next 44
Enter the information for the Destination StoreVault. Choose how often you want the data to be replicated. One of the advantages of the StoreVault is that every hour it s not replicating the whole volume. It s only moving the blocks that changes (in 4k increments). This will reduce the replication time drastically after the initial sync. 45
Successful You can telnet into the source box (This is not supported by SV support) to check the status of your replication with a snapmirror status command For my 200 gig LUN it took about 3 hours (It s better to do the initial replication in the same center then ship the DR unit to the site) 46
Testing the replicated LUN. After the replication has completed, the DR site can now be tested to ensure that your applications and servers are all at the DR site and functioning. This involves disabling the replication of data from the primary site to the DR site as follows. Goto the SV GUI on the DR site box Click on the LUN, right click and choose recover. Lun will move to the Primary Storage section of the GUI. Note This breaks the replication from the primary site to the DR site You then need to setup the DR site machines to access the LUN. You can follow the steps in the previous sections to enable iscsi on the VI3 machine. Then add the IP of the DR StoreVault. 47
Goto the StoreVault GUI and right click on manage LUN 48
Goto the Initiator Group and add in the drsite initiator. Click Save Under the inittators tab confirm that your initator is there. (This example shows the node name drsite). It can still be there if you didn t do the previous step. But will not work if it s not added into the Initator Group. 49
Goto Hardware / Storage Adapters, then click rescan. This will populate the Details field of the GUI. We now have our datastore mapped and ready to add in the machines. The last step is adding the Virtual Machines into the DR sites VI3 software. As of now there is not an automated way to do this, but when Site Recovery Manager comes out from VMWARE sometime in Q1/2008 this may be an automated task. 50
Goto the storage section of hardware, then click add storage, choose Disk / LUN. If should then find your storage on the DR site StoreVault and access the LUN. Now to add the machines choose New Virtual Machine 51
Choose Custom, then next. Enter what you have named the machine at the primary site. 52
Select the LUN from your DR site StoreVault, click next Choose your operating system, click next. 53
Choose the number of processors you want, click next Choose the amount of memory, then click next. 54
Choose the Network and quantity of NIC s you want, click next. Choose the SCSI Adapter you want, click next. 55
Choose Use an existing disk, click next Select your virtual machines disk, click next. 56
Click Next Click Finish 57
Your Machine is now added to the DR site. You can now power up the VM and ensure that everything is functional. Repeat the above steps for all machines that are in the iscsi LUN. Notes: 1. You may need to change the IP s of your guest operating systems, or have DHCP setup at your DR site. 2. The original LUN and site may be up and running and you can test your DR site. 3. A current limitation of the StoreVault is that you cannot resync the LUN. If you have a failover situation and do processing at your DR site, you must then replicate the DR LUN to your primary site and reattach them there. 58
Solution Pricing and Summary Disaster Recovery should be looked at my all company s of all sizes to ensure that they can recover from any disaster (Flood, Weather, Etc) and it can be done without breaking the bank, or the need of massive professional service engagements. I ve chosen the hardware below as I believe they are best of breed, truly mission critical and can scale as needed. I wanted to include these prices so the reader would know a baseline of what hardware and software is needed and approximate costs. This hardware selection may not have the horsepower to keep every server and application running in a disaster, but will provide sufficient processing power for about 10-20 mission critical applications and replicate them to a DR site. All prices are list / web as of 12/20/2007 qty Sun Server #1 $ 5,460.00 Sun Server #2 $ 5,460.00 Sun Server #3 $ 5,460.00 StoreVault 500 #1 $ 9,980.00 StoreVault 500 #2 $ 9,980.00 VMWARE VI3 Standard (3 copies) $ 8,985.00 Support $ 1,797.00 $47,122.00 59
StoreVault S500 SUN X4100 SERVERS Item Description Qty Price (U.S.) Total Sun Fire X4100 Server 1 2 2 A86-FJZ2BH4GCBA $ 4,760.00 Sun Fire X4100 M2 Server, 2 AMD Opteron Model 2216, 2.4 GHz, 1MB, Processor, 4 GB (4 x 1 GB DIMMs) DDR2-667 Memory, 4 Disk Chassis, 2 x 73 GB Disks, DVD- ROM/CD-RW, 2 High Efficiency PSU, Service Processor, 4 x 10/100/1000 Ethernet Ports, 3 USB 2.0 Ports, 2 x 8-lane PCIe Slots, RoHS-5 Compliant X311L $ 0.00 Power Cord Kit, North American/Asian, RoHS Compliant X4225A-Z $ 350.00 2 GB (2 x 1 GB DIMMs) Registered ECC PC3200 DDR2-667 Memory Kit, RoHS-5 Compliant 1 $ 5,460.00 $ 5,460.00. Subtotal $ 5,460.00 60
Revision Notes 12/26/2007 V1.0 Released for initial review and comment. 61