EVault for Data Protection Manager. Course 401 EDPM Troubleshooting Basics

Transcription

1 EVault for Data Protection Manager Course 401 EDPM Troubleshooting Basics

2 Table of Contents Objectives... 3 Scenario... 3 Level 1 Support Expectations Rules of Engagement... 4 Online Resources... 5 i365 Knowledge Base... 5 Microsoft s Technet Site... 5 Basic Network Troubleshooting... 6 Backup Failures / Restore Failures... 7 DPM Backup and Restore Failures... 8 DPM 2010 Administrator Console... 8 Volume Shadow Copy EVault Backup and Restore Failures EVault Console EVault Logs Windows Event Logs Google Slow Backups Anti-Virus Network Throughput Heavy Server Load Slow / Old Hardware Working with i365 Support DPM Related Backup/ Restore Problems EVault Related Backup / Restore Problems Contacting Support EVault for Data Protection Manager i365, A Seagate Company

3 Objectives From this document, you ll learn the basics in troubleshooting your EDPM appliance. Prior to working with i365 Support, make sure you ve made your best efforts at troubleshooting the problem on your own. As a partner, you will be required to offer Level 1 Support to your users. Scenario You re experiencing a backup or restore problem with yours or your client s environment. This document walks you through the various places to go to isolate your problem and steps to try troubleshoot the problem. 3 EVault for Data Protection Manager i365, A Seagate Company

4 Level 1 Support Expectations Rules of Engagement Level 1 support should have performed the following diagnostic and troubleshooting techniques before escalating the issue to Level 2 support. Ensured the EVault products are at the latest version supported by i365 Our current shipping products list can be found here: Our Product Support Policy can be found here: Ensure the OS is at the patch level stated in the EVault product s Release Notes. If the issue started after upgrading to a later patch level, roll that patch back if possible. If it is not possible, contact EVault support per the methods described later in this document. The affected system has been rebooted If a performance issue is involved, standard performance troubleshooting is carried out All cables are confirmed to be securely attached Search the EVault Knowledge Base for the issue and follow its instructions. The Knowledge Base can be found on our Customer Service Portal: You will need a valid account to log into the CSP. If you do not have one, please contact the Primary i365 Contact Person at your company as they will have the ability to create the account for you. If none of these steps resolve the issue, please open a support ticket via the Customer Service Portal. This should include Microsoft and/or Dell online resources if this is relevant to the situation User Guides and Release Notes have been checked. If Hardware Related, the Hardware Checklist has been followed Once all obvious checks are completed, the relevant logs and other forensics should be gathered prior to contacting Support Proper scoping of the issue: how many jobs, systems etc are affected? What is the difference between affected and unaffected system Proper priority is given to the situation based on i365 s definition of Priorities 4 EVault for Data Protection Manager i365, A Seagate Company

5 Online Resources The following two sites will be some of the places you should go to research your issues. On these sites, you ll find FAQs, release notes, product downloads, and user guides. User Guides: Release Notes: Knowledge Base: Provide Installation and Operational procedures Provide information on new features and known issues Provides solutions to known issues, How To instructions, and other information i365 Knowledge Base When troubleshooting, one of the first places you should look for common issues and be able research error messages that you might see. The Knowledge Base is available at i365 s Customer Service Portal. Microsoft s Technet Site When troubleshooting DPM related problems, Microsoft has an extensive troubleshooting section available on their Technet Site. 5 EVault for Data Protection Manager i365, A Seagate Company

6 Basic Network Troubleshooting For EDPM related problems, some of the basic steps should come in the form of network connectivity troubleshooting. With EDPM, there are really 2 main components to be considered. You have either the Agent machine or the EDPM appliance. To test connectivity from the Agent machine to the EDPM appliance, please verify that the following commands can complete successfully. These would be run from a command prompt. ping <EDPM Appliance IP Address or Fully Qualified Domain Name(Hostname)> Sometimes ping will fail if ICMP is disabled on the firewalls/switches. telnet <EDPM Appliance IP Address> 2546 EVault backups use port This test connectivity from the Agent machine to the EDPM appliance, and it also tests whether or not the firewall rules have been setup correctly. The EDPM appliance listens on TCP port If this command completes successfully, you expect to see a blank screen with a blinking cursor. To exit, press CTRL+C. If it were to fail, the command would result in a timeout response. EVault Agents also use TCP ports 8086 and 8087 for communication to the EVault Console on the EDPM Appliance. Telnet is not installed by default on Server 2008 machines, but it can be added as a Feature. 6 EVault for Data Protection Manager i365, A Seagate Company

7 traceroute <EDPM Appliance IP Address> This command uses the ping command. Therefore, it will fail if ICMP is disabled, but the command will indicate latency and how many hops there are between points. netstat an If you ever forget which direction firewalls need to be opened, this command will reveal which ports a machine is listening on. Here, I ve run the netstat command on the EDPM appliance. Backup Failures / Restore Failures To check on the overall health of the EDPM appliance, there s a shortcut to the EVault for DPM Dashboard on the desktop of the EDPM appliance. Once opened, you ll look to verify that you re not receiving any failures. DPM and EVault issues will be handled in different manners though, but the first place you should look is the corresponding log for either the backup or restore failure. 7 EVault for Data Protection Manager i365, A Seagate Company

8 DPM Backup and Restore Failures When troubleshooting DPM backup or restore failures, there s a handful of places you should look to try and diagnose the problem. DPM 2010 Administrator Console For DPM, you ll want to check on the status listed in the DPM 2010 Administrator Console. This will generally give you a sense of the health of your various Protection Groups. You ll want to check both the Protection and Monitoring tabs for any red x s indicating critical issues. Protection tab: Monitoring tab: In the bottom panel, there will be a description of what went wrong and some possible ways to correct the problem. 8 EVault for Data Protection Manager i365, A Seagate Company

9 Sometimes, performing a consistency check will fix the problem, but this process can take some time depending on the amount of data you re protecting. 9 EVault for Data Protection Manager i365, A Seagate Company

10 Volume Shadow Copy Since DPM relies on Volume Shadow Copy (VSS), the status of the service should be checked as part of the troubleshooting process. o o Make sure that the Agent machine is up-to-date with all the latest Windows updates. Make sure the Volume Shadow Copy Service is available. It s normal that the Startup Type is Manual. o Make sure all the associated VSS writers are available and have no errors by running vssadmin list writers from a command prompt on the Agent machine. On a normal file server, you ll need the System Writer, but on Exchange, SQL, SharePoint, etc, there will be specific VSS writers for those applications as well. If these writers are not installed, the Application backups will not work. 10 EVault for Data Protection Manager i365, A Seagate Company

11 EVault Backup and Restore Failures For EVault related backup and restore problems, you should always be reviewing the status of the backup or restore and reviewing the associated logs. EVault Console The EVault console displays the status of all the EVault related Agents you re protecting. On the left hand side, you can get a summary of any Agents with Failures, Missed Protection Runs, Errors, or Warnings. You can highlight any Agent under All Computers and view its job in the bottom pane. Notice that per any job, there is a Logs column. By clicking on this, it will pull up the Log File Viewer to see logs located on the Agent machine. EVault Logs When viewing the logs, first of all, they are sorted by either Protection, Recovery, or Synchronize. You can toggle between these categories at the top of the log viewer. 11 EVault for Data Protection Manager i365, A Seagate Company

12 For backup related logs, a successful backup is indicated by the date and time of backup. For failed backups, you will look for a log named Backup.xlog. When there s a Backup.xlog, it means one of two things. Either the backup failed, or the backup is still running. In other words, you ll have a Backup.xlog for any uncommitted backup. Restore logs are also indicated with the date and time of restore. When viewing an EVault log, you will see a lot of detail. You can filter the data to show Errors, Warnings, etc. by using the Display dropdown located at the top right of the screen. Note that each line has a status indicator. -I- = Informational -W- = Warning -E- = Error -F- = Fatal If you don t know what a particular error message means, you can always try to research the problem in our knowledge base at 12 EVault for Data Protection Manager i365, A Seagate Company

13 Windows Event Logs Whenever troubleshooting your backups with DPM or EVault, you need to consider what else might be happening on your systems at time of backup. DPM and EVault will be tracking information relative to the backups, but it won t report things like random server reboots or maintenance tasks that might be running on the Agent machine. The best way to see and track this kind of information is by viewing the Windows Event Logs. Look for any red explanation marks or warning messages near the point of backup or restore failure. You ll want to view the event logs on both the EDPM appliance and Agent machine. Google Whenever you see a failure message that you don t know how to decipher, one of the best places to research this problem is by looking up the error message / code within Google. The nice thing about Google is that many other people might have had the same problem, and you might find a discussion forum related to the issue. From there, you can find various different options to try and fix your problem. 13 EVault for Data Protection Manager i365, A Seagate Company

14 Slow Backups Backups can be running slowly for a number of different reasons. Below are some of the biggest culprits. Anti-Virus When Anti-Virus is setup with its default settings, a lot of IO can be generated. Make sure that Anti-Virus is set to only scan files when they are written to the system. By default, a lot of Anti- Virus programs will scan files as they re being read and written. We don t want Anti-Virus spawned when a file is being read. Backup programs read files for changes when they re backing up, and if Anti-Virus is spawned on these reads, it will slow down backups considerably. Additional information can be found in Knowledge Base Article found at Network Throughput When troubleshooting slowness in backup performance, you want to make sure it s not the network. Take the backup application out of the equation and make sure that normal file transfer speeds are working at reasonable rates. You can try using some of the following free tools to display throughput when transferring a large file from the Agent machine to the EDPM appliance. Robocopy Part of the Windows Resource Kit. FTP Networx Install on either Agent or EDPM appliance, and it enables to pull up a speed meter while transferring files Some things that can slow down network performance can include the following: Network congestion due to non backup related processes Network negotiation rate mismatch. Make sure that your Network Interface Cards (NICs) configs are set correctly to match your switches ie: 1GB/Full, Auto-Negotiate, etc Heavy Server Load Another thing that can lead to slow backup performance might be other processes running on the Agent machine. By checking things like the Windows Task Manager, Windows Event Viewer, and Windows Scheduler, you can see what else might be running during time of backup that might be the cause of the backup running slowly. Slow / Old Hardware Make sure that the Agent machine is meeting the minimum requirements in terms of server specs. You can always check on the Release Notes to see the Supported Platforms and Minimum Hardware Requirements. 14 EVault for Data Protection Manager i365, A Seagate Company

15 Working with i365 Support After going through the basics of troubleshooting a backup or restore problem on your own, you might need to engage with i365 s Support team. They ll request the following information to be collected by you to open a ticket. Before calling or ing them, it s a good idea to collect this information ahead of time. DPM Related Backup/ Restore Problems Collect the following for DPM related Backup / Restore Problems Detailed description of problem. From the Agent machine, go to Start / Run and run MSINFO32.exe. Save the file as Agent.NFO. This will provide detailed information about the Agent system to i365 Support. Copy of Windows Event Logs (Application / System) saved in a.csv format on both the EDPM Appliance and the Agent machine. Run the following command from Agent machine in a command prompt vssadmin list writers > C:\vss.txt EVault Related Backup / Restore Problems Collect the following for EVault related Backup / Restore Problems. Also check the i365 Knowledge Base for the How to obtain troubleshooting information for EVault Product Support. Detailed description of problem. From the Agent machine, go to Start / Run and run MSINFO32.exe. Save the file as Agent.NFO. This will provide detailed information about the Agent system to i365 Support. Copy of Agent Config files: o Linux / UNIX / ESX Agent directory is located by default in /usr/local/buagent/ o Windows Agent directory is located by default in C:\Program Files\EVault Software\Agent\ o Novell Agent directory is located by default in SYS:\EVault\ o iseries Agent directory is located by default in /buagent/ o Files to collect for Support found in Agent directory: global.vvc <Jobname>.vvc schedule.cfg vvagent-*.xlog inventry.rpt Each job created will have a subfolder with the job name as the folder name. From the job subfolder, you ll want to grab the following files: Backup.xlog for backup related problems RST*.xlog for restore related problems 15 EVault for Data Protection Manager i365, A Seagate Company

16 Copy of Windows Event Logs (Application / System) saved in a.csv format on the Agent machine. Contacting Support Support can be contacted via or phone. Americas Support: EU Support: +44 (0) support@i365.com 16 EVault for Data Protection Manager i365, A Seagate Company