Continuous Delivery for Linux/Windows/Hadoop

Transcription

1 Continuous Delivery for Linux/Windows/Hadoop Wisely Chen

2 Agenda Background Problem Solution Demo Q & A

3 Who I am Wisely Chen ( thegiive@gmail.com ) Enjoys promoting open source tech Release manager of Yahoo! Taiwan shopping and data team Hadoop Summit 2013 San Jose Ruby and Rails (All in Taiwan): Coscup 2006, Ubisunrise 2007, OSDC 2007 Release Practice (All in Taiwan): Webconf 2013, Coscup 2012,2013, PHPConf 2012, RubyConf 2012

4 Team Sanching Yu Windows Neal Lee Hadoop

5 Yahoo! Taiwan Shopping Top B2C site in Taiwan 8 engineering teams Monday.com acquired in K files changed monthly 100K lines of code changed monthly

6 Platform Frontend (Linux) Backend (Window) Data (Hadoop) Catalog Cash Flow Machine Learning Payment Supplier Data Mart Member Logistic Business Intelligence Mobile CMS CRM

7 Problem Frontend Team WebService DB Hadoop API Backend Team Data Team

8 We need a multi-platform build and release solution

9 Solution Create B&R process Build CI infrastructure Automate everything

10 Solution Create B&R process Build CI infrastructure Automate everything

11 Build and Release Process Commit Unit Test Integration Test Perf Test Check Point Deploy UAT Test

12 Hadoop Build and Release Process Commit Unit Test Integration Test Perf Test Check Point Deploy UAT Test

13 Original Difference Frontend Team Backend Team Data Team Process Scrum Waterfall Waterfall OS Linux Windows Linux Language PHP/JAVA.NET Pig/MapReduce Issue Tracking Bugzilla Custom System N/A Repos Subversion VSS N/A CI N/A N/A N/A Int Test N/A N/A N/A Release Management Routine RE Another RE SDE

14 Multiplatform Complexity Reduction Frontend Team Backend Team Data Team Process Scrum(not finish yet) OS Linux Windows Linux Language PHP/JAVA.NET Pig/MapReduce Issue Tracking Repos CI Int Test Release Management Bugzilla Subversion(migrate to git) Jenkins Selenium Delicated Release Manager

15 Communication Cost Huge Large

16 Solution Create B&R process Build CI infrastructure Automate everything

17 WebSite CI Infrastructure Jenkins Master Windows Jenkins Slave Farm Linux Jenkins Slave Farm Selenium Server Farm Staging Server Farm Beta Server Farm Windows Farm Linux Farm Windows Farm Linux Farm

18 Hadoop CI Infrastructure Alpha Jenkins Master Jenkins Slave Hadoop JobTracker Beta Cluster Hadoop Hadoop node Hadoop node Hadoop node Slave node Node PigServer Jenkins Slave Gateway Prod. Cluster

19 Build and Release Process Commit Unit Test Integration Test Perf Test Check Point Deploy UAT Test

20 PigUnit A simple xunit framework No cluster set up is required in local mode We change default PigUnit behavior All PigUnit tests run on single pig server Speed up PigUnit

21 Build and Release Process Commit Unit Test Integration Test Perf Test Check Point Deploy UAT Test

22 Vaidya Rule based performance diagnosis of M/R jobs Extendable framework You can add your own rules Write complex rules using existing rules

23 Vaidya Performance Test Pig Job Conf Sampling Data Pig Job

24 Vaidya Performance Test Pig Job Conf Sampling Data Pig Job Pig Job History

25 Vaidya Performance Test Pig Job Conf Vaidya Rule Sampling Data Pig Job Pig Job History Vaidya

26 Vaidya Performance Test Pig Job Conf Vaidya Rule Sampling Data Pig Job Pig Job History Vaidya Notify User

27 Vaidya Performance Test Pig Job Conf Vaidya Rule Sampling Data Pig Job Pig Job History Vaidya Next CI Stage Notify User Performance Result

28 Vaidya Rule Example <Diagnos)cTest> <Title><![CDATA[Balanaced Reduce Partitioning]]></Title> //////<ClassName> //////////<[CDATA[org.apache...BalancedReducePar))oning]]> ///////</ClassName> ///////<Descrip)on> ///////////<![CDATA[This/rule/tests/...]]> ///////</Descrip)on> ///////<Importance><![CDATA[High]]></Importance> ///////<SuccessThreshold><![CDATA[0.20]]></SuccessThreshold> ///////<Prescrip)on><![CDATA[advice]]></Prescrip)on> </Diagnos)cTest> Rule/Importance See/if/the/reduce/job/is/ balanced/or/not/ Diagnosis/success/ threshold

29 Solution Create B&R process Build CI infrastructure Automate everything

30 Build and Release Process Commit Unit Test Integration Test Perf Test Check Point Deploy UAT Test

31 Before Deploy Check bug status Documention Create a git tag Auto doc generating on bugzilla Deploy to production cluster Change bug status

32 Auto comment in bugzilla Repos url Release Note Issue status change

33 Auto create git tag Git Tag Release Note [Bug xxx] log...

34 Application Logic Monitor SalesNum Today Same day in last week Same day in last month Notify user of something wrong Time Monitor today s sales revenue with baseline

35 Result

36 11Q4: Original Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Time

37 11Q4: Original High Bug Rate(72%) 50 Low Throughput Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Time

38 11Q4: Original Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Time

39 12Q1 : Create B&R Process Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Time

40 12Q1 : Create B&R Process bug rate drop Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Time

41 12Q1 : Create B&R Process Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Time

42 12Q2 : Build CI Infrastructure Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Time

43 12Q3 : Build CI Infrastructure Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Time

44 12Q4 : Build CI Infrastructure Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Time

45 12Q4 : Build CI Infrastructure Dramatic throughput increase 50 Bug rate down to 22% Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Time

46 12Q4 : Build CI Infrastructure Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Time

47 13Q1 : Automate Everything Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Time

48 13Q2 : Automate Everything Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Time

49 13Q3: Automate Everything Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Time

50 13Q3: Automate Everything Deploy time decrease Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Time

51 13Q3: Automate Everything Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Time

52 Hadoop CI Demo Demo1 : Unit test Demo2 : Performance test Demo3 : Deploy

53

54

55 Conclusion This process can reduce bug rate This process can increase productivity Jenkins can fit in any platform

56 ,

57 Appendix

58 90 ReleaseBugRate ReleaseThroughput DeployCostTime On Demand Bi-weekly Weekly 0 11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3

59 Web Site CI Process Frontend 2 Team (Linux) Catalog Payment Member Backend 3 Team (Window) Cash Flow Supplier Logistic Data 2 Team (Hadoop) Machine Learning Data Mart Business Intelligence Mobile CMS CRM

60 Build and release process PHP Code Commit PHP Unit Test Deploy Linux Beta Beta Auto Test Check Test Status.NET Code Commit.NET Unit Test Deploy Win Beta Deploy PHP Staging Release Meeting Staging UAT Staging Auto Test Deploy.NET Staging Code Freeze Release Deploy PHP Prod Deploy.NET Prod Prod Auto/UAT DONE

61 Build CI infrastructure Throughput increased dramaticly 50 Bug rate down to 22% Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Cost Time

62 12Q1 : Create B&R Process Significant bug rate drop 0 11 Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Cost Time

63 Build CI infrastructure Throughput increased dramaticly 50 Bug rate down to 22% Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Cost Time

64 Automate Everything Deploy cost time drop Q4 12 Q1 12 Q2 12 Q3 12 Q4 13 Q1 13 Q2 13 Q3 Release Bug Rate Release Throughput Deploy Cost Time

65 Two B&R Processes Frontend Team Website Process Data Process Backend Team Data Team

66 Automate Everything Bugzilla Check Bug Status Source Repos Log Bug List Check Point Script Yes Deploy Script Style/Static/ Similarity Analysis No Notify User Script Jenkins Auto Manual