Application Lifecycle Management. Build Automation. Fabrizio Morando Application Development Manger Microsoft Italia

Application Lifecycle Management Build Automation Fabrizio Morando Application Development Manger Microsoft Italia

Application Lifecycle Management Fondamenti di Build Management Do your systems talk business? 2

Key Lean ALM delivery processes Portfolio management Project management JIT Demand Management Production planning closed loop Build and software configuration management Release management Change aware continuous integration Deployment Production control closed loop Change management Testing and quality assurance Service management Do your systems talk business? 3

Result in More Successful Outcomes By: Decreasing Risk Improving Quality Do your systems talk business? 4

Continuous Application Delivery Stakeholder Gives Feedback Feedback Incorporated Daily Cycles Ask for Feedback Telling the story Manage the Backlog Plan a Sprint Run a Sprint Deploy to Stakeholders Do your systems talk business? 5

Application Lifecycle Management Integration Do your systems talk business? 6

Integration Modularization enables team development makes complex systems manageable Integrated Modules do successfully Compile Run Pass test

Integration Challenges

Broken Integration You have a broken integration when: Integration server does not build successfully Shared component works in one system, but breaks others Unit tests fail Code quality fails (coding conventions, quality metrics) Deployment fails Do your systems talk business? 9

Manual Integration Integration becomes expensive if made manual (build, test, deployment, ) with too few checkin s (hours or days ) If integration problems and bugs are detected too late Reduces desire to refactor Do your systems talk business? 10

Application Lifecycle Management Continuous Integration vs Nighly Build Do your systems talk business? 11

What Is CI? Martin Fowler: team integrates work frequently, usually each person integrates at least daily; leading to multiple integration by day. Verified by an automated build & test. Results in significantly reduced integration problems; allows team to develop cohesive software more rapidly. Key pillar of Agile Practices Do your systems talk business? 12

Nightly Build (NB) Fundamentals: Daily build and smoke test process. Every file is compiled, linked and combined. Program is then put through a smoke test ; a set of automated unit tests. Benefits Minimizes integration risk Reduces the risk of low quality Supports easier defect diagnosis Do your systems talk business? 13

What s the Difference? NB is periodic, run overnight when we have more time available. Less frequent, more resource-intensive. CI is on-demand, several times a day. Theoretically for each check-in. Is CI better than NB? Do your systems talk business? 14

Is ASAP Integration Best? Ultimately, ASAP Integration should happen on every check-in But Eliminates use of code repository as backup of code-in progress sandbox Requires that I integrate now, in the middle of development of my development task (disruption of flow) The discussion is really about when to do QA Do your systems talk business? 15

QA To answer question of when, let s define what is QA QA == Verification & Validation Verification: testing that the product implement what s in the functional specification documents Easily automated unit tests Good candidate for either CI or NB Do your systems talk business? 16

The Real Issue When the team should: Perform Verification? Perform Validation? Reduce Integration Costs? Do your systems talk business? 17

Timing of Integration & Test Verification & Validation Verification Reduce Integration Risk Do your systems talk business? 18

Timing of Integration & Test CI Run smoke tests or as much verification as possible on every check-in Resource Limited NB Run all verification nightly Release Build Run as often as QA can run validation testing Iterative Do your systems talk business? 19

Release Codeline Interventi evolutivi nuove features Stabilization & bugfix Mainline Manual Build Esercizio Scheduled Build Ambiente di Accettazione Build Drop Sviluppo Build Drop Ambiente di Sviluppo e Test Do your systems talk business? 20

Application Lifecycle Management C.I. in detail Do your systems talk business? 21

Continuous Integration: definitions Do your systems talk business? 22

Embrace Continuous Integration Integration server monitors source repository Rebuilds with every change Runs all unit and acceptance tests Publishes build results Notifies developers if build breaks Labels successful builds in source repository

Continuous Integration Infrastructure Do your systems talk business? 24

Practices of Continuous Integration Maintain a single source repository. Automate the build (nightly builds) Make your build self-testing Everyone commits every day (at least!) Every commit should build the mainline on an integration machine Keep the build fast Test in a clone of the production environment Make it easy for anyone to get the latest executable Everyone can see what's happening Automate deployment (in UOC it could allow carry out the execution of the whole workflow of an installation in PRO) Do your systems talk business? 25

What is a Successful Build? When is your build successful? When it compiles When all the unit-tests have run When it has been deployed In fact, every failure is a success You have exposed a potential problem early! Do your systems talk business? 26

Benefits Always be aware of current status of the project Less time spent investigating integration bugs Less time wasted because of broken code in version control system Prove your system can build! Increase code quality with additional tasks Discover potential deployment issues

Traffic Light Architecture Do your systems talk business? 28

Il ruolo del Build Master Le attività del build manager sono onerose solamente nella fase iniziale del progetto, quando cioè il Build Manager deve ingegnerizzare il workflow e stabilire le convenzioni di sviluppo e di utilizzo del Source Code Control, in modo tale da avere una gestione uniforme dei progetti Visual Studio e sia facilmente implementabile un processo quotidiano automatizzato. Le principali attività iniziali del Build Master sono principalmente di impostazione, quindi. Si tratta di lavorare e consultare i software architect per comprendere al meglio la struttura e la natura dei deliverables, e con il project manager per comprendere le aree di progetto, le tempistiche di rilascio dei differenti deliverables. Do your systems talk business? 29

Master level solution \ Branches \ Mainline \ release1.0 \ release2.0 \ Build \ DataAccess : \ DBScripts \ BusinessComponents \ UserInterface \ CommonComponents La folder Mainline (da cui nascono successivamente tutte le altre branches) conterrà i progetti Visual Studio (.csproj o.vbproj) organizzati per Area. L area può essere funzionale, o tecnologica, o architetturale (come da esempio). All interno della folder di contestualizzazione dell area, sono presenti i progetti VS con alberatura piatta. Non sono ammessi nesting di progetto, in quanto comporterebbero una difficoltosa gestione da parte delle figure che si occupano della build e della manutenzione del repository. La naming convention illustrata successivamente consente l ordine contestuale dei progetti anche con organizzazione strutturale piatta.

Application Lifecycle Management Build Automation con TFS Do your systems talk business? 31

C.I. nativa di Team Foundation Server notifica Developer Esegue Check In Notification Application Changeset processa Power Tool Version Control Service Check-in Event scatena controlla interroga Database handles Build Service gestisce accoda Build Completion Event Build Definitions Build raises lancia Build Agent gestisce receives Build Queue

Eseguire una Team Build Do your systems talk business? 33

Workflow standard di una Team Build Il client invoca una Build su TFS Se non ci sono errori vengono aggiornati i workitem In caso di errore viene creato un Bug TFS lancia la Build sulla macchina designata Vengono eseguiti test e code coverage (opz.) Vengono salvati i dati nel DB Viene generato un Build ID Vengono compilate le soluzioni indicate Vengono pubblicati i file Vengono scaricati i sorgenti Viene analizzato staticamente il codice (opz.)

Application Lifecycle Management Build Automation Reports Do your systems talk business? 35

Build Management Reports Excel and SQL Server Reporting Services (SSRS) Teams members can use reports to track the quality of the builds and to answer the following questions: How much code is being tested? How much is the code changing every day? Is the quality of the builds improving? Excel Code Coverage Report Code Churn Report Build Status Report SQL Server Reporting Services Build Quality Indicators Report Build Success Over Time Report Build Summary Report Build Dashboard: If Microsoft Office SharePoint Server 2007 is hosting the team project portal, web parts for Excel Web Access display the 3 PivotChart reports. Additional web parts for Visual Studio Team System show lists of work items, work item counts, and other project management data that is derived from Team Foundation databases. Do your systems talk business? 36

Code Coverage Excel Report You can review the Code Coverage report to answer these questions: How much of the code is the team testing? Does the team have sufficient code coverage? Is code coverage increasing or decreasing over time? Required Activities for Tracking Code Coverage For the Code Coverage report to be useful and accurate, team members must perform the following activities: Configure a build system using Team Foundation Build Create build definitions Define tests to run automatically as part of the build Configure tests to gather code coverage data Run builds regularly The team can review the Code Coverage report to determine whether tests cover the code sufficiently and how the coverage changed over time. The report provides a line graph of the build verification test (BVT) code coverage and other coverage over the most recent four weeks. This report is based on a PivotChart report that shows the most recent four weeks of data that was captured for code changes. If the team practices test-driven development or similar techniques, the code coverage should almost always approach 100%. If unit tests are reused as BVTs, the code coverage should be visible in the Code Coverage report. You can update this report by opening the report in Excel and changing the filter options for the PivotTable report or change the start date for the report. You can customize this report to support other views also. Do your systems talk business? 37

Code Churn Excel Report You can review the Code Churn report to answer these questions: How much code is changing? How stable is the product the team is expected to deliver? Required Activities for Tracking Code Churn For the Code Churn report to be useful and accurate, team members must perform the following activities: Configure a build system using Team Foundation Build Create build definitions Define tests to run automatically as part of the build Configure tests to gather code coverage data Run builds regularly The team can review the Code Churn report to determine how volatile the code base is and how many lines of code were modified in the previous week. This report is based on a PivotChart report that provides a stacked area graph of the lines of code that the team added, deleted, or modified in the most recent four weeks. All lines are counted, even lines that contain comments or that are blank. Code churn is a good measure to quantify the amount of change that is occurring in your project. In general, high levels of code churn indicate a less stable project. You should expect high rates of code churn at the start of a product cycle or when the team has implemented many changes. Toward the end of an iteration or before a release, you should expect the level of code churn to decrease, which indicates that your project is more stable. You can update this report by opening the report in Excel and changing the filter options for the PivotTable report or change the start date for the report. You can customize this report to support other views also. Do your systems talk business? 38

Build Status Excel Report You can use the Builds Status report to answer these questions: How is my team's build health changing over time? Do any builds need attention today? Required Activities for Tracking Build Status For the Build Status report to be useful and accurate, team members must perform the following activities: Configure a build system using Team Foundation Build Create build definitions Run builds regularly The team can review the Build Status report to help determine the trend of build health over time and whether any builds need attention today. This report is based on a PivotChart report that provides a stacked column of the number of builds that were run with an outcome of failed, passed, or unknown during the most recent two weeks. You can update this report by opening the report in Excel and changing the filter options for the PivotTable report or change the start date for the report. You can customize this report to support other views also. Do your systems talk business? 39

Build Quality Indicators SSRS Report The Build Quality Indicators report shows test coverage, code churn, Are the tests passing? and bug counts for a specified build definition. You can use this report Is the team is likely to finish based on the code and test metrics? to help determine how close portions of the code are to release How often are tests passing, and how much of the code is being quality. You can review the report to find answers to these questions tested? for any specific build definition: What is the quality of the software? Is the team testing enough of our code? The data that appears in the Build Quality Indicators report is derived from the data warehouse. The X-axis lists the specific builds that the report includes, based on the filters that you have set for the platform, configuration, and build definition. Each vertical bar represents a set of data that was derived from one or more builds. In the code size variant of the report, each vertical bar s length represents the size of the checked in code base. (Unhealthy Version of Report) The bars are scaled so that the largest figure fits into the height of the chart. Manual tests can be run any time after the build, and they are associated with that build. Tests that have not been run yet are counted as "inconclusive. Do your systems talk business? 40

Build Success Over Time SSRS Report The Build Success Over Time report provides a pictorial version of the Build Summary report. The report displays the status of the last build for each build category run for each day. You can use this report to help track the quality of the code that the team is checking in. In addition, for any day on which a build ran, you can view the Build Summary for that day. You can review the Build Success Over Time report to find answers to these questions: How high is the quality of the builds? Is the quality improving, deteriorating, or staying constant? What parts of the project are ready to test? What parts of the project are having trouble with regressions or bad check-ins? How well is the code tested? Build status Color Indicates Passed Tests Passed, Low Coverage Build Succeeded, No Tests Build Failed Green Light green Yellow Red Build succeeded. All tests completed successfully. Code coverage was good. Build succeeded. All tests completed successfully. Code coverage was minimal. Build succeeded. No tests were run. Build ran but did not pass. At least one test failed that did not previously fail. Either the test is new or the test passed in previous test runs. Tests Failed No build Orange White Build failed due to a compile error or other error. Build was not run on this day. The report summarizes build and test results for a set of build definitions in one or more projects over time. The chart shows a separate row for each combination of build definition, platform, and configuration. The report shows only those combinations that fall within the filters that you have specified for the report. At a glance, you can determine the success or failure of builds for the time period under review. You should expect the Build Success Over Time report to vary based on where you are in your product development cycle. Early iterations often exhibit some builds and tests failing. By reviewing the report together with the team early and often, you might be better able to focus efforts toward creating stable builds with high rates of tests passing. Do your systems talk business? 41

Build Summary SSRS Report The Build Summary lists builds and provides information about test results, test coverage, code churn, and quality notes for each build. You can review the Build Summary report to answer questions about the most recent builds. It contains more information than the Build Success Over Time report. Which builds succeeded? Which builds have a significant number of changes to the code? Which builds are ready to install? How much of the code did the tests execute? You can use this report to find answers to these questions: What is the status of all builds over time? Quality indicator Build Progress Build Quality % Tests Passed % Code Coverage Description Specifies the status of the build. A build can be in one of the following states: Failed. The build failed to compile or tests failed to pass. Partially Succeeded. Only some portions of the build successfully compiled. Stopped. The build was manually stopped. Succeeded. The build successfully compiled, and tests ran. Specifies a manually assigned assessment of the quality of the build. You can add or remove the build qualities that are defined for your team project. For more information, see Add or Remove Build Quality Values. The column is empty if the build quality has not been rated. Displays a horizontal stacked bar chart that lists the percentage of tests that passed superimposed on a green bar. The remaining bar segment is red, which indicates the percentage of tests that failed. The total length of the chart always equals the width of the column. Displays a horizontal stacked bar chart that lists the percentage of code that was covered superimposed on a green bar. The remaining bar segment is light blue, which indicates the percentage of code that was not tested in the build. The total length of the chart always equals the width of the column. % Code Churn (lines) Displays a horizontal bar chart that lists the percentage of code churn superimposed on a gray bar. The code churn is calculated by determining the number of lines of code that the team has added, deleted, or modified divided by the total number of lines in the build. The bar length is proportionate to the percentage figure, scaled across the report so that the maximum amount of code churn across all builds equals the width of the column. The report presents a visual display of the percentage of tests that are passing, code that is being tested, and changes in code across several builds. You can review the results for both manual and automatic builds, in addition to the most recent builds and continuous or frequent builds. The report lists the most recent builds first and contains build results that were captured during the specified time interval for all builds that were run, subject to the filters that you specified for the report. Do your systems talk business? 42

2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.