Building Highly Available OpenZFS Storage Appliances Grenville Whelan

Transcription

1 Building Highly Available OpenZFS Storage Appliances Grenville Whelan November

2 High-Availability.com About High-Availability.com UK-based with global reach and support RSF-1 first independent commercial HA solution for UNIX Heritage in enterprise SPARC/Solaris/Oracle and Linux HA Cluster plugin for many ZFS vendors (Nexenta, Coraid..) More than 2,000 HA ZFS based clusters deployed to date About Me Worked with Unix-based enterprise systems since 1985 Ran Sun/Solaris Enterprise Consulting practice Engineered, configured and deployed HA systems since 1994 Designed RSF-1 and co-founded High-Availability.com in 1995 November

3 ZFS Storage Appliance Design Decisions In Mission-Critical environments, Storage is fundamental and must be always available ZFS is disruptive but needs Enterprise feature set Discuss High-Availability issues Not focusing on performance November

4 Design Decisions: Hardware Commodity but use enterprise-grade (not consumer-grade) components ECC RAM Motherboards Cooling Power Redundant Hardware Dual power Multiple NICs Cluster in a box or separate servers Hardware location November

5 Storage Topology Shared using synchronous replication Separate with Asyncronous replication Design Decisions: Storage Select Storage Components Enterprise SSDs SAS (performance/price balance) Avoid SATA interposers ZIL Accelerators November

6 ZFS Pool Configurations Performance v Complexity ZFS import / export times Design Decisions: Pools Standalone Cluster Hot Spare RAID 0 RAID 1 RAID 5 RAID 0+1 November

7 Design Decisions: Operating System Solaris / OpenIndiana derivatives FreeBSD Linux Others, e.g. OSX Free Which Operating System? Commercial Support Enterprise Proven November

8 Design Decisions: Management Ready to go: Commercial Enterprise Solutions (e.g. Nexenta) Freemium (e.g. Napp-IT, FreeNAS) Roll-Your-Own Management / Administration CLI, GUI November

9 Key HA Objectives High Availability of critical storage services: 1. Data (storage) protection 2. Service availability 3. Designed to cope with failure 4. Fast and seamless non-disruptive failover 5. Disaster Recovery capability Secondary benefits/objectives: 1. System Upgrades / Servicing 2. Load Balancing 3. Administration November

10 Key HA Objectives: Data Protection Above ALL else, protect the data! Avoiding server stand-off ( split-brain ) situations! November

11 Key HA Objectives: Service Availability Maintaining Service availability in event of: Hardware Failures Storage Failures Network Failures Application Failures Human Error / Sabotage Resource Constraints Environment November

12 Key HA Objectives: Failure Tolerant Design to cope with failure not necessarily prevent it November

13 Key HA Objectives: Fast Failover Fast and seamless non-disruptive failover Identify key time-outs: NFS VMWARE November

14 DR incorporation or extension to HA Key HA Objectives: DR Capability November

15 Key HA Objectives: System Upgrades Non-disruptive System Upgrades Move all load to one server whilst other is upgraded, then replace November

16 Key HA Objectives: Load Balancing Load Balancing with multiple independent ZFS pools Redistribute storage load equally between servers November

17 Key HA Objectives: Administration Keep it simple and manageable November

18 Key HA Considerations 1. Data Protection 2. Replication (ZFS mirroring or asynchronous replication) 3. Active/Passive, Active/Active or Active/Active/..N 4. File, Block or File & Block 5. Failover timeouts and methods, automatic, manual, failback? 6. Fault detection 7. Cluster communications (heartbeats) 8. Identify and remove all possible Single Points of Failure November

19 Key HA Considerations: Data Protection Mechanisms for safeguarding data: SCSI2/PGR3 reservations STONITH/SMITH How long to wait? Coping with system freezes.. November

20 Key HA Considerations: Replication ZFS (synchronous) mirroring Shared-storage Uncomplicated failover/failback November

21 ZFS (asynchronous) replication Timed send-receive/rsync on non shared storage topology Complicated failover/failback Key HA Considerations: Replication Timed Replication November

22 Active/Active or Active/Passive or N-way Active/Active/..N Key HA Considerations: Topology November

23 Key HA Considerations: ZFS Services File, Block or File & Block FC Access Services VIPs for NFS / CIFS / SMB iscsi views/mappings ALUA Luns iscsi NFS CIFS November

24 Failover / failback methods: Automatic, Manual Key HA Considerations: Failover November

25 HA Design Challenges: Faults Fault detection heartbeat timeout System freezes False failures November

26 HA Design Challenges: HA Comms Reliable Heartbeats Network only? November

27 HA Design Challenges: SPOFs Identify and remove all single points of failure November

28 HA Implementation Challenges 1. Heartbeat mechanisms 2. Optimising Failover times (pool complexity) 3. Dealing with intermittent resource bottlenecks (system freezes) 4. Disk protection (split brain) 5. Fail back with synchronous replication 6. Identify and remove all single points of failure 7. Service failover within key time constraints 8. Administration November

29 Other Considerations Fire/Test drill Perform regular failovers Alarms Know when failovers have occurred, components have failed Load Balancing Are you getting equal load / optimum performance? Tuning Effect change on both/all server heads November

30 Any Questions? Grenville Whelan November