Keys to Optimizing Your Backup Environment: Tivoli Storage Manager John Merryman GlassHouse Technologies, Inc. Introduction Audience Profile Storage Management Interdependence Backup Pain Points Architecture Server Storage Network Clients Audience Response Audience Profile Your Backup/Archive Environment Number of Years using Number of Administrators
Storage Management Interdependence Storage and Production Backup Growing at 65% CAGR (Gartner) Consuming 48% of IT hardware budgets (Gartner) Disaster Recovery Enabling Recovery RPO/RTO impact Archiving Compliance Storage efficiency Disposal of data Application Performance It s no longer just backup, it s data management. Pain Points Business Problems Backup reliability Overwhelming data growth Database and email data explosion Limited budgets and staff Regulatory / data retention requirements Lack of data management policies Poor operational practices Solution Optimize and right-size existing infrastructure Improve processes and procedures Formalize program management Technology Problems architecture scalability Lack of / OS tuning Network architecture Client-side issues (OS, antivirus, network) Increasingly complex requirements Shrinking backup windows Disconnect between application design and storage management Benefits Lower TCO for environment Proactive management model Lowered risks and exposures Scalable and manageable environment Business Problems Backup reliability Overwhelming data growth Database and email data explosion Limited budgets and staff Regulatory / data retention requirements Lack of data management policies Poor operational practices
Technology Problems architecture scalability Lack of / OS tuning Network architecture Client-side issues (OS, antivirus, network) Increasingly complex requirements Shrinking backup windows Disconnect between application design and storage management Solution Optimize and right-size existing infrastructure Improve processes and procedures Formalize program management Benefits Lower TCO for environment Proactive management model Lowered risks and exposures Scalable and manageable environment
5411 Enterprise Switc h 10 00 BASE X 1 2 AMBER AC = TIV I TY GREEN LINK = O K FLASHING GREEN DISABLED = 10/ 100 BASE TX 4 5 6 7 8 9 10 11 1 2 POWER 1 2 MGMT 4 5 6 7 8 9 10 11 R R HEWLETT PACKARD Pain Points Client Administration 8% 7% 2% 2% % 2% 2% % % Certifications, De- Certifications, Re-Certifications Client Problem Resolution Client Upgrades / Change Management Client Process Improvement Testing and Certifying New Client Technologies Performing Restores 7% Server / Device Problem Resolution Server Administration 61% Server Monitoring, Reporting, Planning Testing and Certifying New Server Technologies Joint Effort Problem Resolution for Client Issues Architecture Overview LAN SAN Server Clients Database Recovery Log Architecture Tape/SAN/Disk Technology Server Clients Network
Server Performance Hardware CPU, Memory, HBAs Server backplane/bus architecture HBA location Operating System Tuning VMTUNE for AIX has a significant impact Must be tuned specifically for environment Asynchronous I/O (requires settings too) Direct I/O (requires settings too) Network options tuning Application Tuning Application Tuning Database Buffer Pool Sizing Server Options bufpoolsize parameter (up to 50%) Selftunebufpoolsize parameter (up to 10%) Relates closely to page space sizing Mirrorwrite db parallel and DBpageshadow Txngroupmax, movebatchsize, movesizethresh Application Tuning, II 64-bit and 2-bit code levels (and OS levels) performance tuning guide http://publib.boulder.ibm.com/tividd/td/ts MM/SC2-9101-01/en_US/HTML/SC2-9101-01.htm Maintain a link between operating system, application, and device performance analysis.
Server Performance Other Considerations Backup versioning policies have an effect. Schedule randomization versus static schedules Spread production workload across backup window. Control batch processes to avoid overlap for resource intensive processes. Low-level policies can have a high-level impact. shrstatic versus shrdynamic Changing retries settings Server Operations Ideal Batch Processing 10/12/200 Backup STG Process 60 Backup STG Process 99 Backup STG Process 400 Database Backup Process 472 Migration Process 65 Migration Process 47 Migration Process 474 Expire Inventory Process 479 Space Reclamation Process 480 Space Reclamation Process 481 Space Reclamation Process 482 Space Reclamation Process 48 0:00 1:00 2:00 :00 4:00 5:00 6:00 7:00 8:00 9:00 10:00 11:00 12:00 1:00 14:00 15:00 16:00 17:00 18:00 19:00 20:00 21:00 22:00 2:00
Backup STG Process 60 Migration Process 65 Migration Process 66 Migration Process 67 Migration Process 68 Migration Process 69 Migration Process 70 Migration Process 71 Migration Process 72 Migration Process 7 Migration Process 74 Migration Process 75 Backup STG Process 76 Backup STG Process 77 Backup STG Process 78 Backup STG Process 79 Migration Process 80 Migration Process 81 Migration Process 82 Migration Process 8 Migration Process 84 Migration Process 85 Migration Process 86 Migration Process 87 Migration Process 88 Migration Process 89 Migration Process 90 Migration Process 91 Migration Process 92 Migration Process 9 Migration Process 94 Migration Process 95 Migration Process 96 Migration Process 97 Backup STG Process 98 Backup STG Process 99 Backup STG Process 400 Backup STG Process 401 Backup STG Process 402 Backup STG Process 40 Backup STG Process 404 Backup STG Process 405 Migration Process 406 Migration Process 407 Migration Process 408 Migration Process 409 Migration Process 410 Migration Process 411 Migration Process 412 Migration Process 41 Backup STG Process 414 Backup STG Process 415 Backup STG Process 416 Backup STG Process 417 Migration Process 418 Migration Process 419 Migration Process 420 Migration Process 421 Migration Process 422 Migration Process 42 Migration Process 424 Migration Process 425 Migration Process 426 Migration Process 427 Migration Process 428 Migration Process 429 Migration Process 40 Migration Process 41 Migration Process 42 Migration Process 4 Migration Process 44 Migration Process 45 Migration Process 46 Migration Process 47 Migration Process 48 Migration Process 49 Migration Process 440 Migration Process 441 Migration Process 442 Migration Process 44 Migration Process 444 Migration Process 445 Migration Process 446 Database Backup Process 447 Migration Process 448 Migration Process 449 Migration Process 450 Migration Process 451 Migration Process 452 Migration Process 45 Migration Process 454 Migration Process 455 Migration Process 456 Migration Process 457 Migration Process 458 Migration Process 459 Backup STG Process 460 Backup STG Process 461 Backup STG Process 462 Backup STG Process 46 Backup STG Process 464 Backup STG Process 465 Backup STG Process 466 Backup STG Process 467 Backup STG Process 468 Backup STG Process 469 Backup STG Process 470 Backup STG Process 471 Database Backup Process 472 Migration Process 47 Migration Process 474 Migration Process 475 Migration Process 476 Migration Process 477 Migration Process 478 Expire Inventory Process 479 Space Reclamation Process 480 Space Reclamation Process 481 Space Reclamation Process 482 Space Reclamation Process 48 Space Reclamation Process 484 Space Reclamation Process 485 Space Reclamation Process 486 Space Reclamation Process 487 Database Backup Process 488 Space Reclamation Process 489 Space Reclamation Process 490 Space Reclamation Process 491 Space Reclamation Process 492 Space Reclamation Process 49 Space Reclamation Process 494 Space Reclamation Process 495 Space Reclamation Process 496 Space Reclamation Process 497 Space Reclamation Process 498 Space Reclamation Process 499 Database Backup Process 500 Migration Process 501 Migration Process 502 Migration Process 50 Migration Process 504 Migration Process 505 Migration Process 506 Space Reclamation Process 507 Space Reclamation Process 508 Database Backup Process 509 Space Reclamation Process 510 Space Reclamation Process 511 Database Backup Process 512 Migration Process 51 Migration Process 514 Migration Process 515 Migration Process 516 Migration Process 517 Migration Process 518 Database Backup Process 519 Space Reclamation Process 520 Migration Process 521 Migration Process 522 Migration Process 52 Migration Process 524 Migration Process 525 Migration Process 526 Database Backup Process 527 Backup STG Process 528 Backup STG Process 529 Backup STG Process 50 Backup STG Process 51 Backup STG Process 52 Backup STG Process 5 Backup STG Process 54 Backup STG Process 55 Migration Process 56 Migration Process 57 Migration Process 58 Migration Process 59 Migration Process 540 Backup STG Process 541 Backup STG Process 542 Backup STG Process 54 Backup STG Process 544 1:0 12:0 1:0 14:0 15:0 10/11/2002 10/12/200 10/1/200 10/14/200 16:0 17:0 18:0 19:0 20:0 21:0 2:0 5411 Enterprise Switch 10 00 BASE X 1 2 5411 Enterprise Switch AMBER AC = TIV I TY 10GREEN 00 BASE LINK = X 1 O K FLASHING GREEN DISABLED = 2 AMBER AC = TIV I TY GREEN LINK = O K FLASHING GREEN DISABLED = 10/ 100 BASE TX 4 5 6 7 8 9 10 11 1 2 POWER 10/ 100 BASE TX 1 2 MGMT 4 5 6 7 8 9 10 11 4 5 6 7 8 9 10 11 1 2 R R POWER 1 2 MGMT 4 5 6 7 8 9 10 11 R R 2:0 0:0 1:0 2:0 :0 4:0 5:0 6:0 7:0 8:0 9:0 10:0 1:0 12:0 1:0 14:0 15:0 16:0 17:0 18:0 19:0 20:0 21:0 2:0 2:0 0:0 1:0 2:0 :0 4:0 5:0 6:0 7:0 8:0 9:0 10:0 1:0 12:0 1:0 14:0 15:0 16:0 HEWLETT PACKARD 17:0 18:0 19:0 20:0 21:0 2:0 2:0 0:0 1:0 2:0 :0 4:0 5:0 6:0 7:0 8:0 9:0 Poorly Optimized Batch Processing Architecture Tape/SAN/Disk Technology Server Clients Network SAN Architecture Disk Fabric Design Dedicated SAN/Disk resources for Database/Logs Dedicated SAN/Disk for storage pools Tape Fabric Design Load Balancing and design is important Develop Ratios based on throughput and SAN fabric 1 GB Fabric w/ 590-E1A Drives 1: Ratio HBA: Drive 2 GB Fabric w/ 590-E1A Drives 1:6 Ratio HBA: Drive
Disk Architecture Disk architecture Fast, redundant, flexible and scalable File system design Create naming conventions and plan for scalability. Raw volumes versus file systems Create file system schema with disk layout in mind. Database/log performance (Random I/O) Storage volume performance (sequential) Disk Architecture, II Storage device architecture Design/layout is key for sequential versus random disk performance requirements is often the most I/O intensive application in the datacenter. Disk performance tuning is often necessary. Disk Storage Pools Capacity Planning for disk storage pools Plan for using Migration Delay for critical client data. A note on storage pool caching and database performance Disk is traditionally used in as a staging area for daily migration to tape. This paradigm is evolving.
Disk as a Primary Storage Pool Considerations Detailed Capacity planning methods for disk storage are required. Backup Policies have to be carefully planned. Formatting and Recovering Terabytes of *.dsm files is incredibly time and resource intensive. Performance-oriented disk storage pool design becomes even more important. Copies to tape will still be required, and simultaneous R/W can be a big issue for S-ATA technologies. Tape Architecture Size and Type of Library Robot speed, Compression, Scalability, etc. Number and Type of Tape Drives R/W Speed, Caching, Mount/Seek Times, etc. Media Formats Create a media migration strategy and plan. Consider emerging tape technologies. Tape is not dead and is not dying. High-end tape I/O is faster than low-end disk I/O. The real issue is random vs. sequential access to data. Tape Storage Pools Stream large database objects data directly to tape. Lowers disk pool requirements Lowers batch processing time Optimizes use of tape resources during backup window Schedule and resource planning is key. Resource Utilization and Maxnummp settings Synchronous Copy Storage Pool Writes ( ) Can reduce batch window significantly Use only for large object backup clients. Recovery optimized tape storage pools Collocation=yes (+ s an s)
5411 Enterprise Switch 10 00 BASE X 1 2 5411 Enterprise Switch AMBER AC = TIV I TY 10GREEN 00 BASE LINK = X 1 O K FLASHING GREEN DISABLED = 2 AMBER AC = TIV I TY GREEN LINK = O K FLASHING GREEN DISABLED = 10/ 100 BASE TX 4 5 6 7 8 9 10 11 1 2 POWER 10/ 100 BASE TX 1 2 MGMT 4 5 6 7 8 9 10 11 4 5 6 7 8 9 10 11 1 2 R R POWER 1 2 MGMT 4 5 6 7 8 9 10 11 R R HEWLETT PACKARD Architecture Tape/SAN/Disk Technology Server Clients Network Network Overview Network Architecture Private / Public network configurations Type of Architecture 100 Mb and Gigabit IP architectures are the defacto standard. Develop Network Architecture Standards. Map network standards to SLAs and to service delivery. Publish metrics on network throughput. Enforce Standards. Advancing a network architecture can be cost prohibitive and time consuming so can poor recovery times and data loss. Network Design Dedicated Private VLAN architectures Logical network segregation and physical network segregation Use jumbo frames and gigabit VLANs. Improves streaming efficiencies to tape 9000 and 9004 MTU size differences DNS Considerations Multiple HBA s for client network load balancing Etherchannel configurations Load balancing for network, switches, server HBAs Don t use auto-negotiate network card settings.
5411 Enterprise Switch 10 00 BASE X 1 2 5411 Enterprise Switch AMBER AC = TIV I TY 10GREEN 00 BASE LINK = X 1 O K FLASHING GREEN DISABLED = 2 AMBER AC = TIV I TY GREEN LINK = O K FLASHING GREEN DISABLED = 10/ 100 BASE TX 4 5 6 7 8 9 10 11 1 2 POWER 10/ 100 BASE TX 1 2 MGMT 4 5 6 7 8 9 10 11 4 5 6 7 8 9 10 11 1 2 R R POWER 1 2 MGMT 4 5 6 7 8 9 10 11 R R HEWLETT PACKARD Architecture Tape/SAN/Disk Technology Server Clients Network Client Performance Fact: Client machines are often sized for application workload, not backup workload. Client Tuning Parameters Resource Utilization (2-10) Memoryefficientbackup Virtualmountpoint Network / Transaction Tuning Parameters Tcpnodelay Tcpwindowsize Tcpbufsize Txnbytelimit Client Configuration File System Design and Type File system size and number impacts memory consumption. client can be memory intensive (00 MB Memory / Million Objects). Include/Exclude list size has a direct impact on processing overhead. Leverage domain statements.
HEWLETT PACKARD Client Configuration, II Journal-based backups Data drives only Journal size is limited to 2 GB Image backups LAN-free backups Client Performance Backup Object Success Rates Chart 2 1,000,000 Average Number of Objects 100,000 10,000 1,000 100 10 Average Objects Inspected Average Objects Backed Up Average Objects Failed 1 INAA2068 INAA207 INDA0001 INDA000 INDA2000 INDA2001 INDA2010 INDA2014 INEA2000 INEA2007 INEA2008 INEA2015 INEA2021 INPA2000 INPSWB01 INSA2005 INSA2006 Node Name Client Performance Additional Considerations Data characteristics Data type, volume, volatility Number of files Client workloads and backup schedule Client Software Version Network load and utilization Communication with application architects and developers is increasingly important.
Client Performance Client Versions Version Release 7 Level 1.0, 1 Clients Version Release 1 Level 0.8, 1 Clients Version 5 Release 1 Level 6.0, 1 Clients Version 4 Release 1 Level 0.0, 2 Clients Version Release 1 Level 0., Clients Version Release 1 Level 0.1, Clients Version Release 1 Level 0.7, 4 Clients Version 4 Release 2 Level 0.0, 4 Clients Version 4 Release 2 Level 1.0, 8 Clients Version 4 Release 1 Level 2.0, 45 Clients Version 4 Release 1 Level 1.0, 11 Clients Version 5 Release 1 Level 5.14, 12 Clients (?), 14 Clients Version 5 Release 1 Level 5.0, 27 Clients Version Release 1 Level 0.5, 21 Clients Version 4 Release 1 Level 2.12, 24 Clients To receive a copy of the charts seen in this presentation and more for your environment go to: www.glasshouse.com/backup For a cheat sheet see the highlighter in your conference bag. Thank you. Questions? Mr. Merryman will be available at the Ask-the-Experts booth in the Exhibit Hall: Tuesday 4-5 PM
Information. Optimized. GlassHouse Technologies, Inc. 200 Crossing Boulevard Framingham, MA 01702 www.glasshousetech.com jmerryman@glasshouse.com