Scaling To 1 Billion Hits A Day. Chander Dhall [email protected]

Transcription

1 Scaling To 1 Billion Hits A Day Chander Dhall [email protected]

2 About Me Microsoft MVP Tech Ed Speaker Asp.NET Insider Web API Advisor Pluralsight Author Dev Chair - Dev Connections

3 About Me Conference Organizer - jssaturday Leader NodeLA user group Leader.NET user group at UTDallas Owner Chander Dhall, Inc. Conference Organizer MVPMIX.com Chander Tech Podcast

4 Free Resharper

5 Why? Amazon claim Just an extra 1/10 th of a second on their response times will cost them 1% in sales. Google ½ a second increase in latency caused traffic to drop by a fifth.

6 Theory of Scaling #devconnections

7 Practice of Scalability #devconnections

8 Agenda Why is it important to scale? Creating a scalable solution (in incremental steps) Propose an Architecture Identify Failures and Bottlenecks Identify Downtime Apply a better solution Repeat until we solve (in 10 steps) Then some bonus stuff (a better solution)

9 Unfortunate Solution Load Balancer S S S S S Services

10 Gilbert and Lynch white paper Network 1 Network 2 A Write Algorithm { name : Chander, gender : m } B Read Algorithm { name : Chander, gender : m }

11 Update Message SCALING TO 1 BILLION HITS A DAY Happy path scenario Network 1 Network 2 A Write Algorithm { name : Dhall, gender : m } B Read Algorithm { name : Chander, gender : m }

12 Happy path scenario Network 1 Network 2 A Write Algorithm { name : Dhall, gender : m } B Read Algorithm { name : Dhall, gender : m }

13 Update Message SCALING TO 1 BILLION HITS A DAY Network partitions Network 1 Network 2 A Write Algorithm { name : Dhall, gender : m } B Read Algorithm { name : Chander, gender : m }

14 CAP Theorem Consistency Availability Partitioning

15 Brewer s CAP Theorem Consistency (or more appropriately Atomic) Availability Partition Tolerance No set of failures less than total network failure is allowed to cause the system to respond incorrectly Gilbert & Lynch

16 Just FYI Consistency (in CAP theorem) Atomicity (in ACID) Consistency(in ACID) Means any transaction will bring the database from one valid state to another.

17 Fallacies of Distributed Computing Network is reliable. Latency is zero. Bandwidth is infinite. Network is secure. Topology doesn t change. There is one administrator.

18 Fallacies of Distributed Computing Transport cost is zero. Network is homogenous.

19 Why is Scalability Important Instant success Thanks to Social networking Twitter: 200 billion tweets per year Facebook: 1.23 billion active monthly users a month Billions of devices (desktops, tablets, mobile) Need: Millions of hits with Zero downtime

20 Why is Scalability Important The website was working great UNTIL we launched Instagram was down on the launch day

21 The Variables Scalability - Number of users / sessions / transactions / operations the entire system can handle Performance Optimal utilization of resources Responsiveness Time taken per operation Availability Probability of the application being available at any given point in time Downtime Impact - The impact of a downtime of a server/service/resource - number of users, type of impact etc

22 Major Factors Platform selection Hardware Application Design Database/Datastore Structure and Architecture Caching strategy Asynchronous processing Deployment Process and Architecture Monitoring mechanisms and more

23 Step 1 Appserver & DBServer App Server Database Server 23

24 Step 2 Vertical Scaling Appserver & DBServer App Server Database Server Throw more RAM and CPU

25 Step 2 - Vertical Scaling. or Scale up Increasing the hardware resources without changing the number of nodes Disadvantages Law of diminishing returns Downtime Increases Downtime Impact Incremental costs increase exponentially

26 Step 3 Vertical Partitioning (Services) Introduction Deploying each service on a separate node Advantages Increases Availability (per app) Easy to tune and optimize Reduces context switching Simple to implement App Server Db Server

27 Step 3 Vertical Partitioning (Services) Disadvantages Sub-optimal resource utilization May not increase overall availability Finite Scalability App Server Db Server

28 Vertical Partitioning Distribute the responsibilities. Increased number of nodes. Each node (or cluster) performs separate Tasks Each node (or cluster) is different from the other

29 Step 4 Horizontal Scaling Load Balancer DB Server

30 Horizontal Scaling Replication of nodes Nodes perform the same tasks Nodes are identical Scale out

31 Sticky Sessions Subsequent requests from a user are sent to the original server Asymmetrical load distribution Downtime Impact Loss of session data User 1 Load Balancer

32 Central Session Store Session store is a single point of failure Session reads and writes generate Disk + Network I/O Load Balancer Session Store Ap p S E R V E R

33 Clustered Session Management No Single point of failure Session reads are instantaneous Session writes generate Network I/O Increase in number of nodes increases Network I/O exponentially What happens when? User request arrives before intranode communication finished Intra-node communication fails Clustered Session Management Load Balancer

34 Recommendations Use scaled version of a Central Session Store (Recommended) Use Clustered Session Management ONLY if you have Smaller Number of App Servers Fewer Session writes Don t use sticky sessions if you want to scale

35 Load Balanced App Server Cluster Active-Active assumes that each LB is independently able to take up the load of the other Load Balancer Users Load Balancer

36 Step 5 Vertical Partitioning (Hardware) Load Balancer Load Balancer DB Server SAN

37 Step 5 Vertical Partitioning (Hardware) Advantages Allows Scaling Up the DB Server Boosts Performance of DB Server Disadvantages Increases Cost

38 Step 6 Horizontal Scaling (DB) Introduction Increasing the number of DB nodes Referred to as Scaling out the DB Server Options Shared nothing Cluster Real Application Cluster (or Shared Storage Cluster)

39 Step 6 Horizontal Scaling (DB) Load Balancer Load Balancer DB Server SAN

40 Step 6 Horizontal Scaling (DB) Load Balancer Load Balancer DB Replica DB Server DB Server DB Server SAN

41 Step 7 Vertical / Horizontal Partitioning (DB) Introduction Increasing the number of DB Clusters by dividing the data Options Vertical Partitioning - Dividing tables / columns Horizontal Partitioning - Dividing by rows (value)

42 Step 7 Vertical / Horizontal Partitioning (DB) Load Balancer Load Balancer DB Cluster DB Server DB Server DB Server SAN

43 Step 7 Vertical / Horizontal Partitioning (DB) App Cluster Vertical Partitioning Db Cluster 1 Db Cluster 2 Twitter Table Facebook Table Users Table Products Table

44 Step 7 Vertical / Horizontal Partitioning (DB) App Cluster Horizontal Partitioning 1st Million Users 2nd Million Users Db Cluster 1 Db Cluster 2 Twitter Table Facebook Table Twitter Table Facebook Table

45 Step 7 Vertical / Horizontal Partitioning (DB) Load Balancer Load Balancer Hash Map SAN DB Cluster DB Cluster DB DB DB DB DB DB

46 Step 8 Separating Sets Global Redirector Global Look up Hash Map Load Balancer Load Balancer Load Balancer Load Balancer Hash Map Hash Map DB Cluster DB Cluster DB Cluster DB Cluster DB DB DB DB DB DB DB DB DB DB DB DB Set 1-10 Million Users Set Million Users

47 Step 9 Caching Add caches within App Server Object Cache Session Cache API cache Page cache Software Memcached Redis Azure Cache (App Fabric)

48 Step 10 HTTP Accelerator A good HTTP Accelerator / Reverse proxy performs the following Redirect static content requests to a lighter HTTP server (lighttpd) Cache content based on rules Use Async Non blocking IO Maintain a limited pool of Keep-alive connections to the App Server Intelligent load balancing Solutions Nginx (HTTP / IMAP) Perlbal Hardware accelerators plus LBs

49 More Important Stuff CDNs IP Anycasting Async Nonblocking IO (for all Network Servers) If possible - Async Nonblocking IO for disk Incorporate multi-layer caching strategy where required L1 cache in-process with App Server L2 cache across network boundary L3 cache on disk Grid computing

50 Scalability and Performance SCALING TO 1 BILLION HITS A DAY NoSql Vs Relational Memcached Key Value Document Databases Relational Databases Depth of Functionality

51 NoSql vs Relational No Joins Do you need them though? Transactions RDBMS great for concurrency, integrity or data type validity.

52 Relational -> NoSql Ever increasing users. Scalability needs. Highly structured data to structured, semistructured and unstructured data. Advent of high speed data networking. Distributed computing. Cheap and plenty memory.

53 Relational -> NoSql

54 Scaling RDBMS RDBMS sharding Highly disruptive to re-shard. Lose benefits of relational model. Create and maintain schema on every server.

55 Scaling RDBMS Denormalizing Why use a RDBMS?

56 Scaling RDBMS Distributed caching for RDBMS (eg: memcached) Speed up reads only. Cold cache thrash. Management costs.

57 Relational -> NoSql Schemaless. Auto-sharding. Distributed querying. Integrated caching.

58 No SQL Types Key Value Ordered key value Wide Column Store Document Store/Full Text Search Graph DBs Object DBs

59 Key-Value Store Pros Simple. Programmer friendly. Powerful. Fast. Cons Key range support not good. Aggregation support lacking. Key Key Key Key Key Value Value Value Value Value

60 Ordered Key-Value Store Pros Processes key ranges. More powerful. Cons No framework for value modeling. Key Key Key Key Key Value Value Value Value Value

61 Big Table Key Value Pros Model values as maps of maps of maps. Key Value Key Value Cons Key Value Not appropriate for schemes arbitrary complexity. Key Value

62 Big Table Pros Model values as maps of maps of maps. Cons Not appropriate for schemes arbitrary complexity.

63 Big table Key Column family Key Column family

64 Document/Full-Text Pros Collection of documents which contain keyvalue collections. Natural data modeling. Programmer friendly. Web based. Mostly REST/Json friendly.

65 Document/Full text search databases Key Key Key Val Val Val Person : { name : Chander Dhall, address : { city : los angeles, state : CA, zip : } }

66 Graph databases Key Key Key Key

67 Step 12- Finalizing Caching Load Balancer Load Balancer Hash Map Offline Processing DB Cluster DB Cluster Master DB DB DB DB DB DB Master Slave Slave Slave Slave No Sql SAN Search Db

68 osql Paradigm - Denormalization Data duplication and denormalization First class citizens. Increases total data volume. Simplifies query processing esp. in a distributed environment.

69 NoSql Paradigm - Atomic Aggregates Checking Id Min bal Account Id Account No. Savings Id Interest rate Account { Type : Checking, Id : chk123, Min Bal : 10000, } Account { Type : Savings, Id : sav123, Interest Rate : 5%, }

70 No Sql Paradigm No joins Sql Joins query time. Hence, performance penalty. Handled in application instead.

71 No Sql Paradigm Enumerable keys Sequential Ids for composite keys eg. DeptId_employeeId. Group into buckets sorted by timestamp, day, week etc.

72 No sql paradigm Index table Employe e Id Details [email protected]; State: CA; Dept: IT [email protected]; State: TX; Dept: Sales State Employee Id Dept Employee Id 2234 CA 1234, , [email protected]; 1236, 1244 State: AL; Dept: IT IT 1234, 1235, 1236, 1244 TX 8000, 8100, 8235, 8266 Sales 8000, 8100, 8235, AL 2212, , [email protected]; 2234, 2256 State: WA; Dept: Sales Acc 2212, 2221, 2234, 2256

73 No sql paradigm Tree Index Country - USA State - CA City - LA { property : [{ facilityname : abc, facilityid : 111 }, { facilityname : xyz, facilityid : 222 }] } Properties Facilities

74 No sql paradigm Composite Key E M P L O Y E E S IT Employees Sales Employees IT: Software: 1123 IT: Software: 2323 IT: Hardware: 6767 Sales: Online: 832 Sales : Online: 423 Sales : Store : 556 EmpName: John; Address: Los Angeles EmpName: Kevin; Address: Dallas, TX EmpName: Matt; Address: San Francisco EmpName: Katie: Address: Austin, Tx EmpName: Karen: Address: Irvine, CA EmpName: Richard; Address: San Diego Dept= IT:* or Dept= Sales:Online:*

75 No sql paradigm - Grouping U123: O111 Product Ids: [ Surface, xbox ] U124:O123 U124:O234 U124:O999 U125:O789 U125:O945 Product Ids: [ Win 8, xbox ] Product Ids: [ Win phone, surface ] Product Ids: [ office, azure sub ] Product Ids: [ msdn, office ] Product Ids: [ surface, xbox ] Colocation of a users data.

76 nverted search & direct aggregation EmpId, dept, city,. Dept-IT: [111, 123, 234.] Dept-Sales:[673, 343, 434.] 111: Dept-Sales, City: LA 222: Dept-IT, City: Dallas. City: Dallas City: LA

77 No sql paradigm Materialized paths Electronics TV Phones Computers Cameras Samsung Apple LG LCD LED

78 No sql paradigm Materialized paths TV { entity : TV, category : Electronics } { entity : Samsung, category : Electronics, TV } Samsung Apple LG LCD LED { entity : Samsung, category : Electronics, TV, LCD }

79 No sql paradigm Nested sets Electronics TV Phones Samsung Sony Cell Landline

80 No sql paradigm Nested sets Electronics TV Phone Samsung Sony Cell Landline

81 Flattening nested documents Name: Chander Hadoop: Expert { name : chander, skills : hadoop, nodejs, Spanish, level : expert, expert, novice } Nodejs: Expert Spanish: Novice Skills:hadoop AND level:expert

82 Flattening nested documents Name: Chander Hadoop: Expert Nodejs: Expert Spanish: Novice { name : chander, skills_1 : hadoop, skills_2 : nodejs, skills_3 : spanish, level_1 : expert, level_2 : expert, level_3 : novice }

83 References Highly scalable blog

84 References Building Scalable Architecture