Running an E-Commerce Database in the Cloud Mark Uhrmacher (CTO) Aaron Brown (Senior Systems Engineer) ideeli
What is ideeli? Fastest growing, members-only online shopping destination" Leader in mass affluent women s market; fast growing men s category 4+ million members First event: December 2007 " 6000+ successful events to date $250 million year-end run rate" Grew 41,000% since launch in 2007 Strong commitment, success in footwear" 189% growth in the past 12 months
Where are we now? 400GB production dataset 3TB reporting dataset more than 24k qps & 80k dynamic rpm at peak A self-inflicted DoS every day at noon 4
System Architecture 5
System Architecture Simple version Ruby on Rails Web Stack: nginx haproxy apache Phusion Passenger Database/Caching memcached Percona Server 5.1 6
AWS Terminology region == data center availability zone == isolated pod EBS == persistent storage (NAS) 7
Replication Strategy Server Locality & Disaster Recovery Master/Master pairs in separate AZs Replica trees stay within same AZ Intraregion replicas in separate AZs Extraregion disaster recovery instance for very bad days 8
Database Failover Semantics (Kinesthetic Learning) 1. In-app failover o db1 unavailable? Try db2! 2. MySQL Proxy (disaster) 3. master/master with manual failover (Finger on the keyboard, pager on the SysEng) 9
EBS RAID The Problem Single EBS volumes - 100-150 iops Disk/Network traffic shared on a multitenant NIC Highly variable disk latency EBS volumes fail in unexpected ways No performance guarantees RAID 10 for performance & reliability Why not RAID0? It must be RAID(1 5 6 10) on the back end, right? Diminishing returns after ~10 EBS volumes Linux software raid (md) 10
Application Tuning Query Reduction Query reduction through code optimization ORMs aren't always so smart Evolution of caching strategies 1. memcached 2. membase 3. back to memcached 4. testing Riak 11
Server Tuning Problems The Problems Limited concurrency Frequent server mini-stalls Slow disk Widely variable disk latency Multitenancy EBS performance 12
Server Tuning "Big" servers production servers are m2.4xlarge - 68 GB RAM / 8x2.66GHz CPU bigger servers == less multi-tenancy == (more) consistent performance innodb_log_file_size = 4GB buffers and optimize writes query_cache_type = 0 caused lockups due to mutex contention 13
Percona Server/Services Multi-second DB lockups at peak Queuing at the load balancer Enlisted Percona Services * See Percona/ideeli case study 14
Percona Server Switched to Percona Server w/ XtraDB Internally caused lockups ceased Response curve flattened 15
Backup Strategies Cold backups Copy backup to S3 Slow recovery time EBS snapshots with XFS filesystems ec2-consistent-snapshot from Alestic works with EBS RAID instant atomic snapshot delayed while snapshot writes nearly instant recovery 16
#ec2pocalypse 17
#ec2pocalypse "stuck" volumes (no iops, 100% utilization) Device: r/s w/s %util sdi2 0.00 0.00 100.10 Stuck volumes were in one AZ, but multiple AZs experienced API failure Reconstruction, an additional benefit of RAIDed EBS volumes 18
#ec2pocalypse Reality check Cross-zone, multi-tiered DR strategy poorly designed Loss of us-east-1a caused both DR replicas to become out of date DR replica should slave directly off of master Binlog retention period was too short 19
#ec2pocalypse What we learned This was a data center outage Something similar could have happened anywhere Possibly with less downtime Data center failure, cloud success Expect Failure Cache heavily No really, expect failure 20
We re hiring ideeli.com/pages/careers jobs@ideeli.com If 24K QPS excites you, come talk to us 21
Appendix 22
AWS Terminology Storage Ephemeral Local storage Free and plentiful Irrecoverable upon instance termination or crash Not well suited for important data EBS - Elastic Block Storage Persistent storage Essentially NAS Bandwidth shared with network Slow, high latency Rich API Maximum individual volume size of 1TB 23
AWS Terminology Topology Regions Separate, geographically isolated data centers Current regions are Virgina, California, Ireland, Tokyo, and Singapore Inter-region network traffic uses public Internet Availability Zones Logically separate zones within a region Failures in one zone should not affect another region Inter-zone, intra-region network traffic on private, high-speed network 24
#ec2pocalypse What happened? Someone pressed the wrong button "re-mirroring storm" of EBS volumes Locked up 13% of volumes in a single availability zone EBS APIs were degraded, making recovery more difficult 25
Lessons Learned Expect failure If you need a database server, you need 2 (maybe 3) in different AZs and regions Use software RAID to protect against individual EBS volume failure. Visually diagram regional and AZ topology Cache heavily Disk is bad, worse on EBS. Try to avoid it. No, really. Expect failure. You will be required to organically test your DR plan. 26