Avoiding Pain Running MySQL in the Cloud

Transcription

1 ! Avoiding Pain Running MySQL in the Cloud Neil Armitage

2 whoami DBA Oracle/Mainframes/MySQL (25 Years) Deployment Continuent 1 or 2 Customer Deployments/Week On Premise or Cloud deployments Developer for Cloud Operations and Continuent 2

3 Quick Continuent Facts Largest Tungsten installation processes over 700 million transactions daily on 225 terabytes of data Tungsten Replicator was application of the year at the 2011 MySQL User Conference Wide variety of topologies including MySQL, Oracle, Vertica, and MongoDB are in production now MySQL to Hadoop deployments are now in progress with multiple customers

4 Aims Make you think 4

5 Aims Make you think Make you question 5

6 Aims Make you think Make you question Make you discuss 6

7 Aims Make you think Make you question Make you discuss Learn from our mistakes 7

8 Why Cloud Everyone wants servers now. Waiting months to provision a database environment is no longer acceptable. The business would have failed before new physical servers could be provisioned. Initial capital outlay is low - long term cost can be higher. Less internal skills are required to maintain and deploy servers. 8

9 Before Cloud Came Along (a year or so ago) 9

10 On-premise/Colo deployments Raise a project Get Approval Order hardware Rack and cable hardware Connect to network Install Operating System Connect to SAN 10

11 11

12 Cloud deployments Enter credit card number call API nova boot --image centos6-x86_64 --flavor m1.large db1 nova boot --image centos6-x86_64 --flavor m1.large db2 nova boot --image centos6-x86_64 --flavor m1.large db3! ec2-run-instances ami-xxxxx -k ${EC2_KEYPAIR} -t m1.large ec2-run-instances ami-xxxxx -k ${EC2_KEYPAIR} -t m1.large ec2-run-instances ami-xxxxx -k ${EC2_KEYPAIR} -t m1.large! 12

13 13

14 What is the Cloud? 14

15 Cloud Providers Public Clouds Amazon AWS Google Cloud RackSpace HP Cloud Many others On Premise Clouds VMware Openstack 15

16 Where is a cloud server located? Oregon (3 AZ) California (3 AZ) Northern Virginia (5 AZ) Ireland (3 AZ) Tokyo (3 AZ) Singapore (2 AZ) Sao Paulo (2 AZ) Sydney (2 AZ) 16

17 AWS Availability Zones Availability Zone Availability Zone Availability Zone Availability Zone Availability Zone Region Region 17

18 Why is working in the Cloud Different? 18

19 Where is my server? db1 db2 db3 Provision 3 nodes DB1->DB3 19

20 Where is my server? db1 db3 db2 EC2 Server A EC2 Server B 20

21 Where is my server? db3 EC2 Server B Server A has died 21

22 Where is your data? 22

23 Where is your data? X 23

24 Where is your data? X X 24

25 Where is your data? X X X 25

26 Networking in the Cloud Can be unreliable Where it is? What it is? Security? IP s can change 26

27 High Availability 27

28 High Availability - On Premise Ensure servers in different Racks Redundant Power Redundant Network Redundant Disk Backup Server in DR location Redundant network between Datacenters 28

29 High Availability - Cloud 29

30 Cloud Environments Fail 30

31 Cloud Environments Fail 31

32 Cloud Environments Fail 32

33 Cloud Deployment Options - Simple Separate Nodes across AZ s Slave Master Availability Zone 1B Availability Zone 1C S3 Backups Region US-EAST-1 33

34 Cloud Deployment Options - Better Separate Nodes across AZ s and Regions Slave Master Slave Availability Zone Availability Zone 1B 1C S3 Backups Region US-EAST-1 Availability Zone 1C Region S3 Backups US-WEST-1 34

35 Cloud Deployment Options - Even Better Separate Nodes across Providers Slave S3 Backups Slave Master Availability Zone 1C Availability Zone 1B S3 Backups Region Availability Zone 1C Region US-WEST-1 US-EAST-1 Slave Rackspace Cloud Backups 35

36 What to think about Need to consider locations Have more nodes than you would with on premise Nodes can just disappear or stop functioning Spain cluster has had an issue this afternoon - the current master DB has dropped off the network/out of AWS 36

37 What to think about Plan for and expect node failures Don t assume that you will be able to quickly provision new nodes. Take plenty of backups (S3 costs only $100/ month to store 1TB) Expect a few bad nodes poor cpu performance poor disk performance throw them away - shoot them VS 37

38 Running Databases in the Cloud 38

39 AWS Relational Database Service (RDS) MySQL, Oracle, SQL Service Read Slaves Multi-AZ deployments Simple to set up and use 39

40 AWS Relational Database Service (RDS) AWS Only Limited options for replicating data out Very limited options for replicating in Slow failover with AZ s (10 mins +) 40

41 Trove MySQL Only Single Node No Replication Used by Openstack/Rackspace/HP Cloud 41

42 Google Cloud Google only Synchronous and asynchronous replication Across DC s? 42

43 How we deploy clusters (in the cloud) 43

44 Deploying in the Cloud Automate everything Puppet allows us to work across clouds No need to maintain AMI s/images just use a standard o/s images masterless puppet to ensure we don t have a SPOF Our install tools work across multi-nodes in parallel Heavy use of vagrant with AWS plugin for demos and POC installs 44

45 Need to setup secure connections OpenVPN Need multiple servers to avoid a SPOF Software We now support secure connections Between DB s Between Apps and DB s 45

46 Best Practices RAID EBS Volumes (RAID1) Backups xtrabackup (backed up into S3) EBS Snapshot ec2- consistent- snapshot \ --mysql --freeze-filesystem /vol \ --region eu-west-1 \ --description "$(hostanme) RAID snapshot $(date +'%Y-%m- %d %H:%M:%S')" \ vol-1f9a6446 vol-649a643d Use VPC s and fixed networks 46

47 Best Practices Monitoring Nagios / Opsview Zabbix Alerts New Relic 47

48 Lesson Learnt EC2 Instances fail One of anything is never enough Don t assume you can spin up more resources instantly Think multi-cloud, public/private Resources are disposable - throw away and rebuild if needed 48

49 Lesson Learnt Monitor everything Automate as much as possible Backup everything (twice) 49

50 Questions 50

51 560 S. Winchester Blvd., Suite 500 San Jose, CA Tel +1 (866) Fax +1 (408) Our Blogs: Continuent Website: Tungsten Replicator: 51