Retaining globally distributed high availability Art van Scheppingen Head of Database Engineering

Size: px

Start display at page:

Download "Retaining globally distributed high availability Art van Scheppingen Head of Database Engineering"

Bernice Barrett
8 years ago
Views:

1 Retaining globally distributed high availability Art van Scheppingen Head of Database Engineering

2 Overview 1. Who is Spil Games? 2. Theory 3. Spil Storage Pla9orm 4. Ques=ons? 2

3 Who are we? Who is Spil Games?

4 Facts Company founded in employees world wide 180M+ unique visitors per month 45 portals in 19 languages Casual games Social games Real =me mul=player games Mobile games 35+ MySQL clusters 60k queries per second (3.5 billion qpd) 4

5 Geographic Reach 180 Million Monthly Ac=ve Users(*) Over 45 localized portals in 19 languages Mul= pla9orm: web, mobile, tablet Focus on casual and social games 180M MAU per month (30M YoY growth) Over 50M registered users Source: (*) Google Analy3cs, August

6 Brands Girls, Teens and Family spielen.com juegos.com gamesgames.com games.co.uk 6

7 Foundations The exci2ng theory

8 Retaining globally distributed HA What does it exactly mean? 8

9 What is high availability? Wikipedia: High availability is a system design approach and associated service implementa=on that ensures a prearranged level of opera=onal performance will be met during a contractual measurement period. Oracle: Availability of resources in a computer system 9

10 How do we reach HA with MySQL? Master with (many) slave(s) Master Slave Slave Slave 10

11 How do we reach HA with MySQL? Master with (many) slave(s) Mul= Master Master Master Slave Slave 11

12 How do we reach HA with MySQL? Master with (many) slave(s) Mul= Master Mysqld Clustering Mysqld mgmt ndbd ndbd ndbd ndbd ndbd 12

13 How do we reach HA with MySQL? Master with (many) slave(s) Mul= Master Clustering Geographical redundancy Master local DC Slave local DC Slave Asia Slave US 13

14 What if we keep growing? Scale up Ver=cal Faster CPU/Memory/disks Expensive Costs mul=ply in same rate as # of nodes Scale out Horizontal More (small) machines Inexpensive Par==oning/federa=ng (sharding) 14

15 Scale out Func=onal Shard your database func=onally Reads Add more slaves (keep them coming!) Writes More disks Horizontal par==oning Federated par==ons 15

16 Horizontal partitioning Breaking up tables in small parts on the same host Par==oned on a column Infinite growth (as long as you add diskspace) Less used data to slower (cheaper) disks No stored procedures, func=ons, etc Uneven usage of par==ons (hash par==on may help) Once wrihen, data remains on the par==on 16

17 Federated partitions (sharding) Breaking up your table in parts on mul=ple hosts Par==oned on a column Infinite growth (as long as you add hosts) Less used data on slower hosts Not supported in (standard) MySQL Par==oning on applica=on level (or proxy) Alterna=vely: NDB Uneven usage of par==ons Once wrihen data (mostly) remains on the par==on Parallel queries to retrieve data from all shards 17

18 Amdahl's law Parallel execu=on of sequen=al jobs Limited by the weakest link As fast as the slowest node Fix: nonsequen=al (asynchronous) execu=on 18

19 Typical LAMP stack Client Loadbalancer Webserver Webserver PHP Memcache PHP MySQL 19

20 A-typical LAMP stack Client Loadbalancer Webserver Webserver PHP Memcache PHP MQ Jobs MySQL 20

21 Spil Storage Platform Abstrac2ng the storage layer

22 What was our wishlist? Dependent on one storage pla9orm No more pla9orm- specific query language Differen=ate writes Op=mis=c (asynchronous) Pessimis=c (synchronous) Shard data beher Par==on on user and func=on Cluster informa=on by users, not by func=on Global expansion Par==on on geographic loca=on Solve uneven usage of data storage Move data from shard to shard Anything may/could/will fail eventually Not designed for the happy flow 22

23 Old architecture overview 23

24 New architecture overview 24

25 New architecture overview Presentation layer Client-side API Server API Application Model Storage platform Physical storage 25

26 Our building blocks Everything wrihen in Erlang Piqi as protocol binary JSON XML SSP u=lizes local caching (memcache) Flexible (persistent) storage layer MySQL (various flavors) Membase/Couchbase Could be any other storage product MQs (DWH updates) 26

27 Why choose MySQL? Predictable Reliable Decent performance Easy to comprehend Excellent eco system Libraries Monitoring tools Knowledge 27

28 Why Erlang? Func=onal language High availability: designed for telecom solu=ons Excels at concurrency, distribu=on, fault tolerance Do more with less! Other companies using Erlang: 28

29 How do we shard? What is the bucket model? Each record has one unique owner ahribute (GID) GID (Global IDen=fier) iden=fying different types Bucket(s) per func=onality Bucket is structured data Ahributes contain data of records Ahributes do not have to correspond to schema 29

30 Example bucket $ curl - X POST - H 'Accept: applica=on/json' - H \ 'Content- Type: applica=on/json' - - data- binary "{\"gid\": \ }" hhp:// :8777/demobucket/get { "records": [ { "gid": , "given_name": "g", "registered_on": 1, " ": "mail1", "gender": "m", "birthdate": { "year": 1963, "month": 6, "day": 21 } } ], "meta_info": { "total_ct": 1 } } 30

31 Example bucket MySQL 1 CREATE TABLE demobucket ( gid bigint(20) unsigned not null, given_name varchar(64) not null, registered_on =nyint(3) unsigned default 0, varchar(255) not null, gender enum( m, f, u ) not null default m, birthdate date not null, PRIMARY KEY(gid) ); 31

32 Example bucket MySQL 2 CREATE TABLE demobucket ( gid bigint(20) unsigned not null, user_name varchar(64) not null, user_register =mestamp on update CURRENT_TIMESTAMP(), user_ address varchar(255) not null, user_gender char(1) not null default m, user_dob varchar(10) not null, PRIMARY KEY(gid) ); 32

33 Example bucket Cassandra CREATE COLUMNFAMILY demobucket ( gid int PRIMARY KEY, given_name varchar, registered_on =mestamp, varchar, gender varchar, birth_date varchar ); 33

34 Example Erlang filters demobucket:get( #demobucket_get_input{ gid=12345, filters= [ #filter{ ahr= <<"gender">>, op= <<"=">>, parms= {string, <<"f">>}}, #filter{ ahr= <<"registered_on">>, op= <<"sort">>, parms=asc }, #filter{ ahr= <<"gid">>, op= <<"limit">>, parms={int, 10 }} ]} ) 34

35 Pipeline flow of a bucket 35

36 Global distribution Nearest datacenter (DC) to the end user Satellite DC Processing and caching Do not own/store data Storage DC Processing, caching and persistent storage Store all same user data in same DC Par==on on user globally Global IDen=fier per user 36

37 The lookup server Contains GIDs and their master DC GIDs master DC predefined Migrated GIDs get updated 37

38 How does this work? Globally sharded on GID (local) GID Lookup GID lookup Persistent storage Shard 1 Shard 2 38

39 Master/Satellite DC example 39

40 Why do we need data migration? Spread data even on shards Migra=on of buckets between shards GID migra=on between DCs Crea=ng a new storage DC needs data migra=on Users will automa=cally be migrated a er visi=ng another DC many =mes 40

41 Seamless schema upgrades Versioning on bucket defini=ons GIDs are assigned to a bucket version Data in old bucket versions remain (read only) New data only gets wrihen to new bucket version Updates migrate data to new bucket version Migrates can be triggered 41

42 Seamless schema upgrades Demobucket v1 Demobucket v2 GID name Roy Moss Jen Douglas Denholm Richmond GID name gender Patricia Moss f m 1236 Jen f 42

43 Multi Master writes Every cluster (two masters) will contain two shards Data wrihen interleaved HA for both shards SSP No warmup needed Both masters ac=ve and warmed up Slaves added (other DC) for HA and backup Shard 1 Shard 2 43

44 Where do we stand now? SPAPI is in place SSP is (mostly) running in shadow mode GID buckets running in produc=on Ac=vity feed system first to produc=on Satellite DC in early 2013! 44

45 45

46 Questions?

47 Thank you! Presenta=on can be found at: hhp://spil.com/perconalondon2012 If you wish to contact me: Don t forget to rate my talk! 47

Data Management in the Cloud: Limitations and Opportunities. Annies Ductan

Data Management in the Cloud: Limitations and Opportunities Annies Ductan Discussion Outline: Introduc)on Overview Vision of Cloud Compu8ng Managing Data in The Cloud Cloud Characteris8cs Data Management