Velocity and Volume (or Speed Wins)

Size: px

Start display at page:

Download "Velocity and Volume (or Speed Wins)"

Jodie Robertson
10 years ago
Views:

1 Velocity and Volume (or Speed Wins) Flowcon November hhp://

2 "This is the IT swamp draining manual for anyone who is neck deep in alligators." - Adrian Cockcro-, Cloud Architect at Ne5lix

4 Web- scale Cloud Commodity Client- Server Mainframe

5 Goal of TradiQonal IT: Reliable hardware running stable sobware

7 SCALE Breaks hardware

8 .SPEED Breaks sobware

9 SPEED at SCALE Breaks everything

11 Incidents Impact and MiQgaQon Public RelaQons Media Impact High Customer Service Calls Affects AB Test Results PR X Incidents CS XX Incidents Metrics impact Feature disable XXX Incidents Y incidents miqgated by AcQve AcQve, game day pracqcing YY incidents miqgated by beher tools and pracqces YYY incidents miqgated by beher data tagging No Impact fast retry or automated failover XXXX Incidents

incidents miqgated by AcQve AcQve, game day pracqcing YY incidents miqgated by beher tools and

12 Web Scale Architecture AWS Route53 DynECT DNS UltraDNS DNS AutomaQon Regional Load Balancers Regional Load Balancers Zone A Zone B Zone C Zone A Zone B Zone C Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas

Zone B Zone C Zone A Zone B Zone C Cassandra Replicas Cassandra

13 CIO Says Speed IT Up!

14 Get inside your adversaries' OODA loop to disorient them Colonel Boyd, USAF

15 Land grab opportunity CompeQQve Move Engage customers Measure Customers Observe Customer Pain Point Deliver Implement Act Colonel Boyd, USAF Get inside your adversaries' OODA loop to disorient them Orient Analysis Model Hypotheses Commit Resources Decide Get Buy- in Plan Response

Boyd, USAF Get inside your adversaries' OODA loop to disorient them

16 Territory Expansion Foreign CompeQQon Print Ad Campaign Measure Revenue Observe Customer Pain Point Upgrade Mainframe Customize Vendor SW Act Mainframe Era - 1 year cycle Orient Systems Analysis Capacity Model Vendor EvaluaQon Decide 5 year Plan Board Level Buy- in

Vendor SW Act Mainframe Era - 1 year cycle Orient Systems Analysis

17 Mainframe InnovaQon Cycle Cost $1M to $100M DuraQon 1 to 5 years Bet the whole company Cost of failure bankrupt or bought Cobol and DB2 on MVS

18 Territory Expansion Foreign CompeQQon TV Advert Campaign Measure Revenue Observe Customer Pain Point Install Servers Customize Vendor SW Act Client/Server Era 3 month cycle Orient Data Warehouse Capacity EsQmate Vendor EvaluaQon Decide CIO Level Buy- in 1 year Plan

Vendor SW Act Client/Server Era 3 month cycle Orient Data

19 Client Server InnovaQon Cycle Cost $100K to $10M DuraQon 3 12 months Bet a product line or division Cost of failure revenue hit, CIO s job C++ and Oracle on Solaris

20 Territory Expansion CompeQQve Moves Web Display Ads Measure Sales Observe Customer Pain Point Install Capacity Code Feature Act Commodity Era 2 week agile train Orient Data Warehouse Capacity EsQmate Feature Priority Decide Business Buy- in 2 Week Plan

Feature Act Commodity Era 2 week agile train Orient Data

21 Commodity Agile InnovaQon Cycle Cost $10K to $1M DuraQon 2 12 weeks Bet a product feature Cost of failure product mgr reputaqon Java and MySQL on Linux

22 Train Model Process Hand- Off Steps Product Manager Developer QA IntegraQon Team OperaQons Deploy Team BI AnalyQcs Team

23 What Happened? Rate of change increased Cost and size and risk of change reduced

24 Cloud NaQve Construct a highly agile and highly available service from ephemeral and assumed broken components

25 Real Web Server Dependencies Flow (NeDlix Home page business transacqon as seen by AppDynamics) Each icon is three to a few hundred instances across three AWS zones Start Here Cassandra memcached Web service S3 bucket PersonalizaQon movie group choosers (for US, Canada and Latam)

26 ConQnuous Deployment No Qme for handoff to IT

27 Developer Self Service Freedom and Responsibility

28 Developers run what they wrote Root access and pagerduty

29 IT is a Cloud API DEVops automaqon

30 Github all the things! Leverage social coding

32 Pupng it all together

33 Land grab opportunity CompeQQve Move Launch AB Test Measure Customers Observe Customer Pain Point AutomaQc Deploy Act ConQnuous Delivery on Cloud Orient Analysis Increment Implement Model Hypotheses Share Plans Decide Plan Response JFDI

34 ConQnuous InnovaQon Cycle Cost near zero, variable expense DuraQon hours to days Bet a decoupled microservice code push Cost of failure near zero, instant rollback Clojure/Scala/Python on NoSQL on Cloud

35 ConQnuous Deploy Hand- Off Steps Product Manager A/B test setup and enable Self service hypothesis test results Developer Automated test Self service deploy, on call Self service analyqcs

36 ConQnuous Deploy AutomaQon Check in code, Jenkins build Bake AMI, launch in test env FuncQonal and performance test ProducQon canary test ProducQon red/black push

37 Bad Canary Signature

38 Happy Canary Signature

39 Global Deploy AutomaQon ABernoon in California Night- Qme in Europe If passes test suite, canary then deploy West Coast Load Balancers East Coast Load Balancers Europe Load Balancers Zone A Zone B Zone C Zone A Zone B Zone C Zone A Zone B Zone C Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Cassandra Replicas Canary then deploy Canary then deploy Next day on West Coast Next day on East Coast ABer peak in Europe

40 InspiraQon

41 Takeaway Speed Wins Assume Broken Cloud NaDve AutomaDon Github is your app store hhp://nedlix.github.com

42 A/B Test Hypothesis You would prefer the Halloween Scooby Doo Ending

43 I d have got away with it if it wasn t for you meddlesome kids!

Design For Availability. October 2013 Stevan Vlaovic [email protected] http://www.linkedin.com/in/stevanvlaovic

Design For Availability. October 2013 Stevan Vlaovic svlaovic@netflix.com http://www.linkedin.com/in/stevanvlaovic Design For Availability October 2013 Stevan Vlaovic [email protected] http://www.linkedin.com/in/stevanvlaovic Stevan Vlaovic Director, Membership Infrastructure, Netflix Performance Architect, Display