Disaster Recovery Planning in an Era of Mobile Computing & Big Data Part 1 (of 3)

Transcription

1 Disaster Recovery Planning in an Era of Mobile Computing & Big Data Part 1 (of 3) Jon Toigo Managing Principal Toigo Partners International Technical University/Symposia materials may not be reproduced in whole or in part without the prior written permission of IBM.

2 Session objectives Welcome and Introductions Who am I? 30+ year IT veteran Managing principal partner of a consulting firm specializing on data management, data preservation and data protection Author and writer with 15 books, thousands of articles, and too many blogs, posts and tweets to count Industry watchdog, consumer advocate in IT What are we here to talk about? The trend toward high availability architecture in contemporary IT planning Contributions of hyper-converged infrastructure to high availability processing The need for hybrid data availability architecture Please ask questions as we go along 1

3 HA trumps DR! and other novel notions Reading about the wonders of high availability Especially from server hypervisor vendors Focused mainly on server failover and template cut and paste No more need to undertake the expensive and complicated process of disaster recovery planning (DR) Simple active-passive failover is all you really need for the preponderance of disaster potentials in the world today 2

4 We get it When systems are down, the business stops And the customer experience is impaired

5 The quest for the five nines is still going strong Concept rooted in electrical engineering and 80s Olympic diving Loosely related to concepts of availability and reliability Likelihood of a component to fail over time Predictive tool for maintenance planning 90 s metaphor for IT uptime, but not really a valid metric or measure of IT performance

6 To wit Mean Time Between Failure was key to component reliability engineering Any system can have its operation degraded or stopped by the failure of any component Those components that are prone to failure need to be made redundant to contain the impact of failures Other mathematical formulae seemed to underscore the idea that reliability and availability were quantifiable attributes and that they could be used to gauge IT performance

7 Lots of holes in that theory, of course Customer experience depends on more than simple availability Application uptime doesn t say anything about customer experience Customer experience is determined by the integrated operation of many elements and processes Individually, uptime looks great: Give everyone bonuses! But the customer experience Isn t so great: Fire everyone! Source:

8 Other variables may matter more than MTBF What if interruptions occur at different intervals for different durations: These disks reflect same availability, just different downtime event durations How do you find the right balance between the cost of downtime and the cost of redundancy?

9 HA at the component and system level Truth is, redundancy is used at many layers to avoid preventable downtime in critical IT infrastructure technology Enterprise servers, switches and arrays may double up on AC power supplies, fans, cabling and other interconnect components Virtually all storage products leverage media redundancy And only the most reliable subcomponents are used in manufacture

10 Hypervisor vendors say they are just taking HA to the next level Host apps as virtual machines Establish an active-passive cluster with a second server When production server gets into trouble, just migrate VM to passive server Rinse, lather, repeat easy peasy! TEMPLATE FILE CUT & PASTE

11 Actually, they are forgetting a few things LAN MIRROR Hypervisor SW Clustering SW VMs Drivers ETHERNET SAN Hypervisor SW Clustering SW VMs Drivers MIRROR via VAAI compliant array controller WARNING: Subnet failover not 100% reliable

12 Basic ingredients of an HA cluster Two servers interconnected by a LAN Running clustering software with a heartbeat channel Programmed to alert when failover is necessary or to failover automatically in the event of heartbeat failure Failover occurs and second server uses storage interconnect to access data on shared storage repository PRODUCTION SERVER (ACTIVE NODE) INTERCONNECT LOCAL AREA NETWORK STORAGE ARRAY PRODUCTION SERVER (PASSIVE NODE) INTERCONNECT

13 But that s not real HA Shared storage is a single point of failure Resolve by eliminating shared storage Direct-attach separate arrays to each server and establish on-going data mirroring PRODUCTION SERVER (ACTIVE NODE) LOCAL AREA NETWORK PRODUCTION SERVER (PASSIVE NODE) INTERCONNECT INTERCONNECT STORAGE ARRAY 1 STORAGE DATA ARRAY MIRRORING PROCESS STORAGE ARRAY 2

14 But, that doesn t match your configuration You have a basic storage area network (SAN) connecting storage to servers Different dog, same fleas Still require clustering for the servers Still require data mirroring for the data But now, the SAN switch is a single point of failure PRODUCTION SERVER (ACTIVE NODE) INTERCONNECT INTERCONNECT LOCAL AREA NETWORK PRODUCTION SERVER (PASSIVE NODE) INTERCONNECT INTERCONNECT STORAGE ARRAY 1 DATA MIRRORING PROCESS STORAGE ARRAY 2

15 No worries So you add a redundant switch And redundant cabling And redundant host bus adapters And redundant fabric connections on arrays PRODUCTION SERVER (ACTIVE NODE) INTERCONNECTS LOCAL AREA NETWORK PRODUCTION SERVER (PASSIVE NODE) INTERCONNECTS INTERCONNECTS INTERCONNECTS STORAGE ARRAY 1 DATA MIRRORING PROCESS STORAGE ARRAY 2

16 One more single point of failure Your facility MAN/ WAN SITE A MIRRORING OR REPLICATION SITE B

17 But we aren t talking about that kind of business continuity, just the continuous operation of virtualization hosts! Hmm I thought I heard you say that you were trumping DR with HA Truth is HA has always been part of the spectrum of strategies available to DR planners: it was just the most expensive one! You will likely need synchronous mirroring software MAN/ for local data WANmirroring and You need to asynchronous build, buy or replication software for lease a second site remote far data replication enough away from your primary site to avoid being impacted by And the same don t forget about a permanent outage events MAN or WAN connection that must support data replication and stretch cluster heartbeat monitoring as well as geoclustering software for failover! Identical hardware required for local and distance data mirroring. SITE A Any changes to Site A must be implemented at Site B as well. MIRRORING OR REPLICATION SITE B

18 Pretty soon, you are talking about some real money The reason why HA strategies have been used only selectively in DR plans is because of cost not because they didn t exist. Moreover, not every application needed HA protection

19 Well, we can knock some cost out of that equation Pray, tell. Use Software-Defined Storage for one thing

20 Yes, we have heard of that Return to internal or directattached storage Use commodity components or JBODs (just a bunch of disks) Low cost because all (or most) value-add software has been migrated from proprietary array controllers into a software layer residing on the server, ideally as part of the server hypervisor software stack

21 Evolutionary of (De-)evolutionary? The next evolution in storage, or de-evolutionary? (Call me cynical, but we ve seen it before )

22 Opinions aside, two approaches garnering mindshare at present VMware Virtual SAN Hypervisor software layer Minimum 3 node cluster Identical complement of flash and disk JBOD per node, scaling oriented toward disk, flash primarily for cache Components must be pre-certified to work with Virtual SAN Data synchronized via mirroring controlled by Virtual SAN software VAAI not supported Storage not accessible to non-vmware workload Microsoft Cluster Storage Spaces Builds on Storage Spaces introduced in Windows Server 2012 Direct attached and internal SAS disk and flash I/O devices supported Control not strictly abstracted from hardware: SAS used to lock files and volumes. Provides SMB 3.0 access to share storage between workloads (virtualized and non-virtualized) Best used as scale-out file server

23 And for the DIY-adverse, hyper-converged infrastructure A mash-up of sorts Preconfigured SDS nodes using brand name server & storage glued together with an SDS software solution IBM 3090 Mainframe, Systems Managed Storage (SMS), Direct Attached Storage Devices (DASD) c Blue Gene/P Cluster Supercomputer c. 2010

24 A tale of two cobbles A Unified Hardware Appliance Node Components pre-integrated 1 box = 1 node Plug and play Easy clustering with identical nodes One throat to choke A Hypervisor Software Controlled Node Commodity servers and pre-certified disk and memory components Minimum 3 nodes Easy clustering with identical nodes Simplified management

25 Optional: use a 3 rd party SDS vendor ISVs making more flexible SDS wares Two node clusters for lower cost entry to technology Greater flexibility and support for different nodal hardware Superior management of caching and buffering Inclusion of capacity management with other SDS services Big list of players and growing

26 3 rd Party SDS vendors making inroads with non-aligned server vendors Helping server vendors regain relevancy by providing hyperconverged server/storage cobbles that work with any hypervisor stack

27 3 rd party or not, some issues for HA architecture SDS/HCI perpetuate management problems of pre-sds storage Issue #1: Isolated islands of storage that are difficult to manage in common FROM LEGACY SANs with heterogeneous arrays with onarray value-add services to differentiate platforms and obfuscate common management TO MULTIPLE SDS models, each proprietary to server hypervisor software vendors, and no common management

28 Issue #2: Looks likely that we will have multiple proprietary storage hardware/software stacks According to leading analysts, by 2016 Plus, recent surveys suggest that larger enterprises are diversifying, using multiple hypervisors. Each will have its preferred hyperconverged or software-defined storage model Resulting in 3+ storage targets for management and protection So, you will need multiple HA strategies lacking common management or coordination Using two or more hypervisor stacks becoming the norm

29 Issue #3: Even when all nodes support the same hypervisor, components must be identical MIRROR MIRROR MIRROR MIRROR Server running hypervisor-based SDS microkernel; Three storage nodes with identical components/capacities Nodal disk hardware different type or capacity: no mirror! Nodal SSD hardware different type or capacity: no mirror!

30 Issue #4: Preceding issues apply to stretch clusters as well as local mirrors REGIONAL OFFICE BRANCH OFFICE REPLICATION (MIRRORING OVER DISTANCE)

31 Issue #4b: And even to clouds CLOUD HOSTING SERVICE BRANCH OFFICE REPLICATION (MIRRORING OVER DISTANCE)

32 High availability clustering/mirroring are labor-intensive strategies in any case, best used with critical apps Given the cost of HA over distance, do all applications require this service? Are systems of interaction really the most mission critical workload?

33 7 Tips for making the right HA strategy decision Tip 1: Know your workload The volume of change data that will need to be synchronized across targets The tolerance of the business process to lost transactions or data for a period of time Tip 2: Experiment with mirroring and replication software to understand How much CPU the replication service consumes on virtual servers How it copes with the volume of data that are being mirrored How well it uses bandwidth and reacts to network traffic conditions If relevant, how it reassembles incremental data with complete images Your workload determines mirroring product performance!

34 Tip 3: Think it through Get a clue about how your will handle dynamic workload movement and mirroring How do you know where the VM is that you wish to mirror at any given time? How do you know where to replicate the VM s data (the target host)? What about interdependent applications (running as different VMs or as physical databases): how can you be sure that all applications are current and synchronized? The dark side of scalability is process looping and infinite recursion

35 There is nothing easy peasy about HA mirroring and replication whether with legacy or SDS/HCI Looks simple in the abstract: clustering with data mirroring But as nodes scale

36 Tip 4: With HA strategies increasingly in vogue, you need to understand application interdependencies Of increasing concern, the problem of how to synchronize data replicas on new architectures IBM struggling through use cases now One of the biggest challenges many clients face is data consistency between dependency applications. When clients decided to have a hybrid production environment, the data consistency issue can become more difficult to detect and resolve. For example, the following diagram shows application A replicating to a disaster recovery center and application B on the cloud protecting its data in the cloud. Now, when application B on the cloud has an outage and failover to its resilience center, application B s record is only record #37. It is now the production version of application B, and this is what creates data consistency issues.

37 Tip 5: HA only as good as testing Traditionally, mirroring was rarely tested because of The requirement to disrupt normal operations to test and validate the mirrored set The potentially career limiting outcome of a failure of mirrors to re-synchronize after restart Ideally, mirrors (and asynchronous replication processes over distance) can be tested and validated non-disruptively and on an ad hoc basis, with additional cost benefits in the elimination of maintenance downtime once and for all

38 Tip 6: HA is a two-way street What provisions are made in the SDS/hyper-converged infrastructure model for failing back to a normal state following an HA failover? Ideally, writes intended for mirroring are buffered while the cluster is not operating and until replication processes can be re-established. Make sure some provision is made for doing this and that it is adequate for your recovery scenarios

39 Tip 7: Hypervisor and hardware independence should be sought from any SDS/HCI solution To reduce hassles in setting up and maintaining mirroring and replication arrangements To reduce infrastructure cost (even investments in legacy gear -- SAN and NAS -- can be preserved) To streamline management requirements And to support Minimum Equipment Configuration (MEC): the art of provisioning resources in a disaster to best meet the needs of missioncritical application workload

40 The field of hyper-converged infrastructure appliances is growing Here are some criteria to help sort out the options Dedicated to single hypervisor s virtualized workload Designated hardware for server/storage deployments Dedicated to single hypervisor s virtualized workload Flexible server/storage hardware choices in predefined storage deployment model Shareable infrastructure for multiple hypervisor workloads Designated hardware for server/storage deployments Shareable infrastructure for multiple hypervisor workloads Flexible server/storage hardware choices and support for mixed deployment models Shareable infrastructure for multiple hypervisor workloads and nonvirtualized workloads Designated hardware for server/storage deployments Shareable infrastructure for multiple hypervisor workloads and non-virtualized workloads Flexible server/storage hardware choices and support for mixed deployment models

41 Conclusions Don t believe everything you are hearing about software-defined storage and hyper-converged infrastructure reducing or eliminating the need for disaster recovery While appearing to make HA strategies more affordable, there are many gotchas that need to be factored in Emphasis should always be on flexibility and manageability Flexibility means enabling V-to-V, V-to-P, P-to-V, and P-to-P workload reinstantiation locally and over distance in a hardware-agnostic way Manageability refers to the ability to manage many mirroring/replication and cluster failover/failback processes under a common dashboard or console Availability is ultimately a function not only of technology, but also people and processes

42 For the record, I prefer storage virtualization SDS with capacity management DELIVERING HIGH AVAILABILITY CLUSTERS LOCALLY AND OVER DISTANCE WITH CENTRALIZED MANAGEMENT AND ORDERLY RE-SYNCHRONIZATION AND FAIL-BACK Recovery Environment Production Environment CREATING FROM HETEROGENEOUS STORAGE ARRAYS VIRTUAL STORAGE POOLS AND CENTRALIZED VALUE-ADD SERVICES THAT CAN BE READILY ALLOCATED TO ANY WORKLOAD, REGARDLESS OF HYPERVISOR More on this in Part 2

43 Any questions? Thanks! Please keep in touch Linkedin: Jon Toigo Blog: DrunkenData.com Websites: IT-SENSE.org Data Management Institute Toigo Partners International