Scaling Teams, Processes and Architectures

Transcription

1 Lorenzo Scaling Teams, Processes and Architectures Managing growth Atmosphere Conference Poznan, May

2 Lorenzo Alberton Chief Technical Architect, DataSift 2

3 Lorenzo Alberton Chief Technical Architect, DataSift 2

4 Scalable Architectures 3

5 Scalability Is About... Technology People Processes 4

6 Scalability Is About... Technology People TRUE FOUNDATION Processes 4

7 Staffing, Roles, Management, Teams People 5

8 Staffing Never compromise. Only hire people smarter than you. Find the best role for talented individuals, not vice-versa. 6

9 Staffing Hire people who can fit the company culture. Promote fun in your working environment. 7

10 Staffing Beware of toxic people 8

11 Staffing Don t hire experts Technologies come and go 9

12 Staffing Don t hire experts Technologies come and go 9

13 Teams Try Cross-functional feature teams 10

14 Processes 11

15 Why are processes critical? Ease management of teams/projects Standardise actions in repetitive tasks Reduce mundane decisions to focus on grander ideas Allow the team to react quickly to crisis 12

16 Processes: don t go overboard 13

17 Processes: don t go overboard The right amount 13

18 Processes: don t go overboard The right amount of the right process 13

19 Processes: don t go overboard The right amount of the right process at the right time 13

20 Determining Headroom Capacity Current Load 14

21 Determining Headroom Why? Capacity Planning annual budget Current Load Hiring plan Prioritisation 14

22 Controlling Change: Determine Risk 15

23 Controlling Change: Determine Risk 15

24 Risk Management Risk is cumulative Determine limits and tolerance 16

25 Load / Stress Testing Load testing - identify, document and eliminate bottlenecks through a strict controlled process of measurement and analysis - measure system s response and stability - verify the app can meet the desired performance objectives (SLA) Stress testing - determine the app s stability when subjected to above-normal loads - verify the app s behaviour when close to the breaking point - test the application recoverability (negative testing) 17

26 Load / Stress Testing 18

27 Load / Stress Testing 18

28 Load / Stress Testing 18

29 Load / Stress Testing JMeter The Grinder Load Runner Avalanche 18

30 Barrier Conditions Code reviews Manual and automated QA processes Performance and stress testing Release documentation checks ( runbook ) Dev, Test, Stage and Live environments Instrumentation checks Protection from significant failures 19

31 Architecting Scalable Solutions Technology 20

32 Architectural Principles 21

33 Architectural Principles +1 N + 1 design 21

34 Architectural Principles +1 N + 1 design for rollback 21

35 Architectural Principles +1 N + 1 design for rollback to be disabled 21

36 Architectural Principles +1 N + 1 design for rollback to be disabled to be monitored 21

37 Architectural Principles +1 N + 1 design for rollback to be disabled to be monitored for multiple live sites 21

38 Architectural Principles +1 N + 1 design for rollback to be disabled to be monitored for multiple live sites use mature technology 21

39 Architectural Principles +1 N + 1 design for rollback to be disabled to be monitored for multiple live sites use mature technology asynchronous design 21

40 Architectural Principles +1 N + 1 design for rollback to be disabled to be monitored for multiple live sites use mature technology asynchronous design stateless systems 21

41 Architectural Principles +1 N + 1 design for rollback to be disabled to be monitored for multiple live sites use mature technology asynchronous design stateless systems buy when non core 21

42 Stateless, Asynchronous Systems 22

43 Fault Isolative Structures 23

44 Fault Isolative Structures Increase availability. Limit impact of failures. Easier debugging. 23

45 Fault Isolative Structures Increase availability. Limit impact of failures. Easier debugging. First 23

46 Fault Isolative Structures Functions causing repetitive problems Increase availability. Limit impact of failures. Easier debugging. First 23

47 Fault Isolative Structures Increase availability. Limit impact of failures. Easier debugging. Functions causing repetitive Natural layout or topology of the site problems First 23

48 Avoid complecting things Too many interactions (e.g. dependencies, locking, synchronisation,...) limit the scalability of your application Rich Hickey 24

49 Design for any technology Dell WatchGuard Cisco CSS HP ProLiant DL HP Media Cache Server Appliance 25

50 Design for any technology Dell WatchGuard Cisco CSS HP ProLiant DL HP Media Cache Server Appliance 25

51 Design for any technology Dell WatchGuard Firewall Cisco CSS Load Balancer HP ProLiant DL Application Servers HP Media Cache Server Appliance Media / Cache DB Server 25

52 Caching for Performance and Scale 26

53 Caching for Performance and Scale Object Caches Usually serialized (marshalling / unmarshalling) get() / set() / replace() APC, Memcached 26

54 Caching for Performance and Scale Object Caches Application Caches Usually serialized (marshalling / unmarshalling) Proxy caches Reverse proxy caches get() / set() / replace() APC, Memcached HTTP headers ISP/Uni proxies Squid, Varnish, mod_cache 26

55 Caching for Performance and Scale Object Caches Application Caches CDNs Usually serialized (marshalling / unmarshalling) Proxy caches Reverse proxy caches Multiple locations / backbones get() / set() / replace() APC, Memcached HTTP headers ISP/Uni proxies Squid, Varnish, mod_cache CNAME entries Akamai, Coral, Limelight... 26

56 Scaling directions - Splitting work M. L. Abbot, M. T. Fisher, The Art Of Scalability, Addison Wesley 27

57 Scaling directions - Splitting work cloning of entities or data - unbiased distribution of work x M. L. Abbot, M. T. Fisher, The Art Of Scalability, Addison Wesley 27

58 Scaling directions - Splitting work cloning of entities or data - unbiased distribution of work x y separation of work by activity or data M. L. Abbot, M. T. Fisher, The Art Of Scalability, Addison Wesley 27

59 Scaling directions - Splitting work cloning of entities or data - unbiased distribution of work x y separation of work by activity or data z separation of work by person for whom the work is done M. L. Abbot, M. T. Fisher, The Art Of Scalability, Addison Wesley 27

60 Scaling directions - Splitting applications M. L. Abbot, M. T. Fisher, The Art Of Scalability, Addison Wesley 28

61 Scaling directions - Splitting applications mirroring x + scale transactions - scale data M. L. Abbot, M. T. Fisher, The Art Of Scalability, Addison Wesley 28

62 Scaling directions - Splitting applications mirroring x + scale transactions - scale data + fault isolation + scale function data - scale customer data y split by service M. L. Abbot, M. T. Fisher, The Art Of Scalability, Addison Wesley 28

63 Scaling directions - Splitting applications mirroring x + scale transactions - scale data + fault isolation + scale function data - scale customer data y + fault isolation + scale customer data - scale function data z split by service split by need / location / value M. L. Abbot, M. T. Fisher, The Art Of Scalability, Addison Wesley 28

64 Scaling directions - Splitting data M. L. Abbot, M. T. Fisher, The Art Of Scalability, Addison Wesley 29

65 Scaling directions - Splitting data data cloning (replication / clustering) x + easy to implement + scale transaction volume - scale data size and growth M. L. Abbot, M. T. Fisher, The Art Of Scalability, Addison Wesley 29

66 Scaling directions - Splitting data data cloning (replication / clustering) x + easy to implement + scale transaction volume - scale data size and growth + fault isolation + reduce query time - more difficult - data migration y split by service / resource / data affinity M. L. Abbot, M. T. Fisher, The Art Of Scalability, Addison Wesley 29

67 Scaling directions - Splitting data data cloning (replication / clustering) x + easy to implement + scale transaction volume - scale data size and growth + fault isolation + reduce query time - more difficult - data migration y split by service / resource / data affinity + balanced demand + fault isolation + scale data and trans. - more costly z split by modulus / hash-based lookups M. L. Abbot, M. T. Fisher, The Art Of Scalability, Addison Wesley 29

68 DataSift Architecture 30

69 DataSift Architecture SOA - loosely coupled, independently scalable services. Simple APIs 30

70 DataSift Architecture SOA - loosely coupled, independently scalable services. Simple APIs example 30

71 SOA - Scale Each Component 31

72 SOA - Scale Each Component Distribute processing load among workers. Lightweight orchestration, heavy lifting in separate, asynchronous processes 31

73 Asynchronous communication Workload distribution Queues 32

74 Message queues as Buffers (Decoupling) P Unpredictable load spikes C 33

75 Message queues as Buffers (Decoupling) P Unpredictable load spikes C P C Load normalisation / smoothing 33

76 Message queues as Buffers (Decoupling) P Unpredictable load spikes C P C Load normalisation / smoothing Batching higher throughput 33

77 0mq PUSH-PULL (workload distribution) Consumer 1 pull Producer push pull Consumer 2 (blocking operation, until delivered to one worker) pull Consumer 3 [Round-Robin-ish] 34

78 0mq PUSH-PULL (Mux) Producer 1 Producer 2 push R1, R2, R3 push R4 pull Consumer Producer 3 push R5, R6 fair-queuing: R1, R4, R5, R2, R6, R3 35

79 0mq PUB-SUB (High Availability) Listener 1 Publisher 1 Listener 2 Publisher 2 Listener 3 [Broadcast] [Dynamic Subscriptions] 36

80 0mq PUB-SUB (High Availability) Publisher 1 DC 1 Publisher 2 DC 2 37

81 Internal Firehose Publishers X Y Z subscribe to topic X Alice s timeline Subscribers John s Inbox subscribe to topic Y Data Bus System Monitor Fred s Followers Tech Blog Feed 38

82 Internal Firehose Publishers X Y Z Data Feeds, User-generated content, System events,... subscribe to topic X Alice s timeline Subscribers John s Inbox subscribe to topic Y Data Bus System Monitor Fred s Followers Tech Blog Feed 38

83 Internal Firehose Publishers X Y Z Applications, Services, Monitors, Routers, Repeaters,... subscribe to topic X Alice s timeline Subscribers John s Inbox subscribe to topic Y Data Bus System Monitor Fred s Followers Tech Blog Feed 38

84 Internal Firehose Publishers X Y Z subscribe to topic X Alice s timeline Subscribers John s Inbox Data Bus Everyone connected to the data bus, no directed graph subscribe to topic Y System Monitor Fred s Followers Tech Blog Feed 38

85 Internal Firehose Publishers X Y Z subscribe to topic X Alice s timeline Subscribers John s Inbox subscribe to topic Y Data Bus System Monitor Fred s Followers Tech Blog Feed 38

86 Monitoring 39

87 Monitoring: Measure Everything 40

88 Monitoring: Measure Everything 1. Is there a problem? 2. Where is the problem? 3. What is the problem? User experience / Business metrics monitors System monitors (threshold - variance) Application monitors 40

89 Monitoring: Measure Everything 1. Is there a problem? 2. Where is the problem? 3. What is the problem? User experience / Business metrics monitors System monitors (threshold - variance) Application monitors Keep Signal vs. Noise ratio high 40

90 Monitoring: Measure Everything StatsD 1. Is there a problem? 2. Where is the problem? 3. What is the problem? User experience / Business metrics monitors System monitors (threshold - variance) Application monitors Keep Signal vs. Noise ratio high 40

91 Instrumentation 41

92 Look! Monitors! 42

93 Look! Rib cages! 43

94 Metrics, Metrics everywhere 44

95 We re Hiring! 45

96 References M. L. Abbot, M. T. Fisher, The Art Of Scalability, Addison Wesley London

97 Lorenzo Thank you! Questions? 47