How To Support A Backup At Cornell

Transcription

1 Cornell's Support Model of TSM or How to make sure you have a good backup when you need it Paul Zarnowski psz1@cornell.edu Thanks to Karen Durfee for her assistance in preparing this presentation. Overview Cornell Environment Evolution of TSM use at Cornell TSM Support Issues Cornell TSM Support Model User support Server support 2 Paul Zarnowski

2 Cornell Environment Founded in 1865 by Ezra Cornell and Andrew Dickson White Private endowed university and the federal land-grant institution of New York State Member of the Ivy League Partner of the State University of New York (SUNY) 3 Paul Zarnowski Schools and Colleges In Ithaca, New York (central NY): Seven undergraduate units and four graduate and professional units 260 buildings on 745 acres In New York City: Two medical graduate and professional units 4 Paul Zarnowski

3 The Cornell Community Non-medical Divisions 2000 Faculty - 1,516 Staff - 7,797 Students Undergraduate - 13,590 Graduate/professional - 5,405 5 Paul Zarnowski Overview Cornell Environment History of TSM at Cornell TSM Support Issues Cornell TSM Support Model User support Server support 6 Paul Zarnowski

4 History of Tivoli Storage Manager at Cornell GB 10,000 1, Needs analysis V1.2 QIP, Server Farm backups 2 nd robot, server upgrade Departmental backups (EZ-Backup) Server upgrades (R50) 3 rd robot, 2 nd FTE 4 th robot Expand 4 th robot, Server upgrade (M80), 3 rd FTE 7 Paul Zarnowski Tivoli Storage Manager at Cornell Today EZ-Backup service Used for IT server farm backups Offered to departments 2200 systems 11 TB of managed storage Transfer GB / day 8 Paul Zarnowski

5 TSM at Cornell Today Client Systems Platform # nodes v2 v3.1 v3.7 v4.1 Win Win Win Win ME Win NT Win 2K MacOS AIX Solaris NetWare Linux & SCO OS/2 3 3 XOpen Paul Zarnowski TSM at Cornell Today Server Configuration Tivoli Storage Manager v IBM RS/6000 M80 2 CPUs 2GB RAM AIX v GB SSA disk ATL P6000, ATL P3000 tape robots (et. al.) 12 DLT7000 tape drives, 978 tape slots Gbit switched ethernet adapter Gbit network campus backbone 10 Paul Zarnowski

6 Overview Cornell Environment History of TSM at Cornell TSM Support Issues Cornell TSM Support Model User support Server support 11 Paul Zarnowski Tivoli Storage Manager Support Issues It s all about restoring The best backup is one that s there when you need it. How do you make sure you have the backup protection that you need? 12 Paul Zarnowski

7 Things to consider for supporting TSM Clients Needs Analysis / Backup Requirements Software Installation & Configuration Ensuring Reliability Problem Resolution Detecting backup failures Resolving backup failures Restoring data 13 Paul Zarnowski Things to consider for supporting TSM Server Capacity & Performance Licenses Storage Capacity (Disk, Tape, Database, Log) Bandwidth (Network, Disk I/O, Tape I/O) CPU & Memory Growth trends Problem Monitoring Hardware errors TSM errors Media errors 14 Paul Zarnowski

8 Proactive vs. Reactive Support An ounce of prevention is worth a pound of cure or How to keep the phone from ringing off the hook 15 Paul Zarnowski Overview Cornell Environment History of TSM at Cornell TSM Support Issues Cornell TSM Support Model User support Server support 16 Paul Zarnowski

9 How Cornell supports TSM Cornell TSM support group 3 dedicated staff Leverage existing Cornell IT infrastructure Network Operations Center (NOC) Staffed 24 x 7 Monitors networks and servers Contacts EZ-backup staff for TSM problems Departmental support staff Network engineers Business office 17 Paul Zarnowski Cornell TSM Support Group - Client Support Node administration Testing & custom packaging Web site box Telephone Training classes Interface to vendor support Backup failure notification Chargeback 18 Paul Zarnowski

10 Cornell TSM Support Group - Client Support Node administration User fills out form Admin uses nodeadmin script to process form Used to register, update, and cancel nodes Generates confirming registration points user to installation instructions and places to obtain further help 19 Paul Zarnowski Nodeadmin 20 Paul Zarnowski

11 Nodeadmin 21 Paul Zarnowski Nodeadmin 22 Paul Zarnowski

12 Nodeadmin: Confirmation 23 Paul Zarnowski Cornell TSM Support Group - Client Support Node administration Testing & custom packaging Web site box Telephone Training classes Interface to vendor support Backup failure notification Chargeback 24 Paul Zarnowski

13 Client Support Web Site ( Overview of the service (what is it?) Recent changes Subscription information Technical support Installation instructions FAQs Documentation Contact information 25 Paul Zarnowski Cornell TSM Support Group - Client Support Node administration Testing & custom packaging Web site box Telephone Training classes Interface to vendor support Backup failure notification Chargeback 26 Paul Zarnowski

14 Client Support Backup Failure Notification Rexx script First checks Query Event If event was unsuccessful, then checks Query Filespace (I.e., check for a manual backup) Notification customizable by node # days to notify after failure (or always notify) Who is to receive notification for node Notification policy stored in node s contact field 1 mail file per department administrator Central logging 27 Paul Zarnowski Failure notification example 28 Paul Zarnowski

15 Cornell TSM Support Group - Client Support Node administration Testing & custom packaging Web site box Telephone Training classes Interface to vendor support Backup failure notification Chargeback 29 Paul Zarnowski Chargeback EZ-Backup run as Mini-enterprise Business Plan Self-funded service Helps ensure added resources as demand grows Encourages responsible use Rate components System fee Storage fee (based on Query Occupancy) Data Protection Modules handled separately 30 Paul Zarnowski

16 Overview Cornell Environment Evolution of TSM use at Cornell TSM Support Issues Cornell TSM Support Model User support Server support 31 Paul Zarnowski Cornell TSM Support Group Server Support Hardware & software installations & upgrades Performance & capacity analysis Problem detection & resolution Server monitoring & reporting 32 Paul Zarnowski

17 Server Support Monitoring & Reporting Homegrown scripts (mostly Rexx) Real-time monitor Early detection of potential problems Tied in with , paging system, & NOC Reporting Daily Monthly For users Data collection for ad-hoc reports TSM account log filter 33 Paul Zarnowski Server Support Monitoring & Reporting (cont d) Third-party monitoring tools Network monitoring via MRTG Servergraph / TSM BMC Patrol 34 Paul Zarnowski

18 Server Support Homegrown Real-time Monitor Logic: 5 minute polling Query Request Query Stgpool Query Volume Query Log Query Process Query Session Sends messages to alpha pagers Issues TSM commands Problems detected: Volume checkin requests High backup disk storage pool utilization Read-only or unavailable tapes Sessions waiting for a mount point for a long time High database Recovery Log utilization 35 Paul Zarnowski Homegrown Real-time Monitor Handling Filling Recovery Log Goal Slow down log growth to give in-flight transactions a chance to complete. Discourage repeated incremental DB backups running with no benefit. Alert someone before log fills and TSM crashes. Event Trigger Log Util > 60% Log Util < 50% DB backup running Log Util drops after DB backup completes Log Util > 82% Monitor Reaction to Event Cancel inventory expiration. Restart inventory expiration, if it had been cancelled. Raise dbblogfull to Log Util + 15%, but not more than 80%. Reset dbblogfull to what it was before DB backup start Send page to on-call staff. 36 Paul Zarnowski

19 Servergraph / TSM Example Recovery Log Handling 37 Paul Zarnowski Daily & Monthly Reports Usage summary (daily & monthly) Tape / storage pool health Activity log analysis 38 Paul Zarnowski

20 Daily Reports - Usage Summary 39 Paul Zarnowski Servergraph / TSM Example Chart of Top Nodes 40 Paul Zarnowski

21 Daily Reports Tape / Storage Pool Health Tape volumes Errors Unavailable or read-only volumes High # passes High # mounts Tape storage pools Low storage availability Low # tape volumes Scratch tape availability 41 Paul Zarnowski Daily Reports Activity Log Analysis TSM Activity Log daily.actlog longer term logs (large extract) daily activity log (small extract) tape history log Scans TSM activity log Filters out trivial messages s short extract Saves long extract on disk Appends to tape history log 42 Paul Zarnowski

22 Daily Reports Activity Log Analysis 43 Paul Zarnowski Monthly Reports Usage Summary 44 Paul Zarnowski

23 Monthly Reports R50 vs M80 Comparison 45 Paul Zarnowski Reports for Users Filespace reports Billing summaries 46 Paul Zarnowski

24 Data Collected for Ad-Hoc reports Daily storage usage, by node, and total DB stats (from Q DB F=D) LOG stats (from Q LOG F=D) Tape activity Activity Log extracts 47 Paul Zarnowski Network Monitoring MRTG day year 48 Paul Zarnowski

25 TSM Accounting log filter accntdisp Unix filter converts TSM session account records to human-readable form Used in combination with Unix utilities to do various things Watching session activity in real time Searching accounting log 49 Paul Zarnowski accntdisp Filter Watching active sessions Performance investigations Detailed session information 50 Paul Zarnowski

26 Summary Installing the hardware and software is only part of the solution Think about what type of backup protection is required before you need it Keep an eye on things to help ensure that backups are there when you need them Plan ahead for performance & capacity growth 51 Paul Zarnowski Questions? Resources Servergraph / TSM Applied System Design BMC Patrol MRTG - Multi Router Traffic Grapher Tobias Oetiker, et. al. Cornell EZ-Backup 52 Paul Zarnowski