Agile SPL SCM Agile Software Product Line Configuration and Release Management

Transcription

1 Agile SPL SCM Agile Software Product Line Configuration and Release Management Reto Kurmann Phonak AG, Switzerland Abstract Software Product Line development techniques, as well as Agile development techniques are used to rapidly develop high quality, complex software products. The goal is the same, but the techniques to realize it are quite different. Despite the different approaches, agile software development practices can be integrated into Software Product Line development techniques to support them and form leaner processes. In this experience report I present agile Software Product Line configuration and release management. The experience is based on Phonak s implementation of a Software Product Line for PC-based applications used by audiologists to fit hearing instruments to hearingimpaired customers. I will discuss SCM in an SPL environment, main-path-centric development, release strategies and some lean and agile processes related to software configuration management. 1. Introduction As in many other industries, the innovation cycles for hearing aid products have become shorter. This has a big impact on the product lifespan, which forces development organizations to deliver more products in less time. In addition to the short development cycles, there has been a shift from individual product development to the development of complex hearing aid systems, which consist of a set of interrelated products and applications. This experience report is based on Phonak s implementation of a software product line for PC-based applications used by audiologists to fit hearing instruments to hearing-impaired customers. The focus of this report is on agile product line configuration and release management. The emphasis is on lean and agile processes and best practices in all disciplines of SCM in an evolutionary platform development where new features are implemented, verified and released by a single software product to be reused in subsequent released software products. The agility of our software development is reflected in: the lean, but effective processes; the short implementation cycle of a new product line feature; the small, distributed teams involved in the development cycle and last but not least in the business success of Phonak after delivering SPL-based products in the past 18 months. Software Configuration and Release Management plays an integral role in our Software Product Line approach. The experience we made regarding agile SCM is based on the following principles: 1. Main path centric development 2. SCM reflects the static view of the SPL architecture 3. Separate platform (core asset) release from product release is not agile and efficient 4. The first release of a new core asset feature is always with only one product 5. Provide special tools to configure variation points of core assets, SCM tools have other strengths 6. Automate build, deployment and smoke-test for all platform-based products 7. Implement lean and agile SCM processes (Checkin, change packages, branching strategies, defect tracking, verification, etc., but not for requirements engineering!) Especially the principles number 1, 3 and 4 are based on a mix of techniques found in Software Product Line and Agile development. These principles are elaborated in the following chapters. 2. Phonak s SPL Adoption This chapter provides some background information on Phonak s Software Production Line adoption for hearing system Fitting Applications (refer to Figure 1). The products in scope of our SPL are PC-based software applications that are used by audiologists to fit hearing instruments to hearing-impaired customers throughout the world. There are 5 major applications from three different brands, several smaller applications and many tools to support production, end-testing, ordering, validating, development and prototyping hearing instruments. All these applications have to support a fast growing amount of different hearing instrument families and types, like hearing instruments behind the ear, in the ear, inside the ear canal or implants. About four years ago, Phonak s top management decided to spend initial effort to build a new platform for all fitting applications. After analysis paralysis and struggling over two sets of requirements from two 1

2 Figure 1: Phonak s Software Product Line adoption for hearing system Fitting Applications. different companies in a non-spl manner, platform architecture was defined and about 30 developers distributed over three locations built GUI-less, productneutral platform components. After more than 40 manyears were invested, our top management seriously considered killing the whole project. The reason was that there was no visible platform progress and the business pressure to deliver the first products in 2005 was getting bigger. Because of this release pressure, we shifted our focus to build our first product under a high stress-factor. The release of our first platform-based Fitting Application with its hearing instruments was sensational. After the first product release the platform needed some cleanup to be ready for integration by other Fitting Applications. Interestingly, around the same time we became aware of the Software Product Line paradigm. We assessed our implementation and as a result optimized, refactored, educated people and streamlined some processes. At Phonak we still use the term platform, but actually mean by it Core Assets, Variation Points, production infrastructure, OTS-components and tools. Today, the platform is growing by a feature driven development. 3. SCM in Software Product Lines The importance of Software Configuration and Release Management (SCM) discipline in the software engineering world is since the 80s indisputable. SCM tools are in my opinion the most important tool of a software developer: That s where all the source code is and since us software developers think and communicate together in source code, that s where all collaboration takes place. We had to wait for this century until Stephen Berczuk defined and laid-out the core principles and patterns in his book Software Configuration Management Patterns [3]. He describes the patterns of traditional SCM in a clear and easy manner. In an additional article in the Better Software magazine [4], he describes practices that turn these patterns into agile SCM. Some of whose we use in our approach of agile SCM for our product line. What is the role of SCM in Software Product Lines? The Configuration and Release Management practice area [1] is where it all boils together. The intangible product, which is based on core assets and variants, governed by architecture, all of a sudden has a real release schedule. We have to give birth to a real product which will be released on a tangible CD. The fundamental purpose of Software Configuration Management is to establish and maintain the integrity of software products throughout their evolutionary life cycle. Each product has to be 100% reproducible at any time for: a release version of OTS-components, libraries & tools a baseline of core asset components a baseline of production mechanisms and environments a version of product-specific components and variations Software Configuration Management is crucial to the success of core-asset based software development. An over engineered and heavy SCM process with too many rules and procedures will slow down the development speed due to the extra work each developer must perform on a daily base. A well-designed and lean SCM process actually speeds up the project. Different projects need different processes and methodologies [5], therefore carefully select a lean and agile process that suits your project. 4. SCM reflects static SPL architecture The main view of source code in SCM tools is folder and file based. Therefore, a proper structuring of platform core assets and product assets in folders, subfolders and shared folders is necessary. Important is that the structure clearly reflects the static architectural view of the Software Production Line. This results in being easy understandable by developers and allows easy release and configuration management. 2

3 Figure 2: Main path centric development Figure 2: Static SPL architecture is reflected in SCM folder structure As shown in Figure 2, we choose to have a top-folder for the platform that can be shared among products. This allows the shared platform version to be either: A released version A specific development path based on a checkpoint The head, mostly in a main path centric development The platform folder contains sub-folders for platform elements, like OTS components, production mechanisms, production tools and last but not least the core assets. The core assets are further structured according to layers and functionality as defined in the architecture. Each product has a separate folder containing productspecific assets, like GUI components; variation delegates implementation; configuration of core assets (in our case configuration files for each core asset component); database of hearing instrument configuration meta-data; Production-related variations (installation and deployment scripts, etc.). 5. Main path centric development The SCM strategy we have chosen for our software product line is a main path centric development (refer to Figure 3). This means that the main path consist always of the entire set of source code and variation configuration in its best possible quality. For every product release, a release path is created to build and enforce high quality. Our experience shows, that choosing this CRM strategy was one of the fundamental success factors of our software product line. The main path contains both, released and new source code: verified core asset components, verified productspecific assets, verified product-specific variations and production tools, environments and scripts - mixed with source code of new feature development extending core assets, new product variations, etc. New features are developed on the main path, where automated daily tests enforce high quality. Before branching into a release path, feature completeness has to be achieved, followed by a short phase of stabilizing and testing by developers. After the stabilizing phase, the platform and product-specific files are branched into a release path containing the entire set of core-asset and production mechanism together with the product-specific variations and assets for the software product to be released. On the release path, there is no feature development anymore. Only verification, validation and bug-fixing are permitted. Bug fixes are immediately merged back into the main path. They may even be merged up to another release path when required by the respective product release. This pro-active step ensures that no bug-fix gets lost and that the main path achieves a better quality. In this agile environment, software developers are already implementing new features on the main path, while the verification and validation teams test the quality of the product to be released. The testing phase on the release path in our case can last several weeks due to the fact, that hearing instruments need a thorough validation before they are released into the markets. Main path centric development combined with simple rules and lean processes form an agile development environment which encourages fine-grained, frequent check-ins. 3

4 Figure 3: Main path centric development In Steve Berczuk s terms [3][4], our main path centric development includes and combines the practices of the following patterns: a single Active Development Line, few Release Lines, Private Workspaces, automated continuous Integration Build, automated Smoke-, Unitand Regression Tests and a lightweight Codeline Policy. 6. Release strategies 6.1. Align platform and product releases An experience we made with our Software Product Line is that the effort of releasing the platform independent from a product is not agile and efficient, mainly because of the following two reasons: Firstly, verification and validation is often impossible, since for proper validation the product to be released has to be tested with real customers. For new hearing instrument features, the validation phase can be quite long. Platform validation could only be done by creating platform products whose sole purpose would be validation of the core assets this is often too expensive. Secondly, it adds an unnecessary time-to-market delay of a new feature: The feature has to be developed in the core asset first, validated and release, before it can be integrated into a product that has to go through another validation and release phase. As a result, we align core asset releases with product releases. We even try to cluster product releases, so that the verification phase is more efficient and the products benefit from a better quality. Please refer to Figure 2: Main path centric development First release of new feature with 1 product While extending our platform with new core asset features, we realized that the most efficient way to implement and release a new feature with high quality on time is together with one product release. The first release of a new feature is always with only one product, never with more than one. The reason is that during the endphase of the development, requirement changes may arise as a result of validation with end-customers and need to be implemented quickly. Adapting other integrated products to these changes may throw them off schedule or may invalidate the validation efforts of their variation of the feature. These advantages outweigh the fact that all other products will have a delay releasing this particular new feature. The validation effort of an already released core asset feature for a different software product is significantly less than for the first release. Only the product-specific variations of the feature have to be validated Releasing source code rather than binaries Another interesting experience we made is that there is no need or purpose to release core asset components in a binary form (executables, libraries, dll, bin). All our build processes are fully automated and can be configured for each software application product, so that at any time any version of a product can be built from the core asset source code directly. This leads to releasing the source code of core assets rather than their corresponding binary form. Releasing source code has also the advantage of a easy, straight-forward implementation of pre-compile, compile- and build-time variation points. Releasing source code is only possible with agile SCM processes, where production process and test suites are fully-automated. 7. Provide tools to configure variation points Software Configuration Management tools should not be misused to configure variation points of core assets. They have a different purpose, as shown in an earlier chapter. Customization in the domain space can be better handled by a special, mostly domain-specific tool. There are a few commercially available tools [8] [9] geared towards product-specific core-asset configuration. At Phonak, we developed a domain-specific configuration tool, which allows audiological engineers to handle the complexity of more than 8000 variation points exposed by the product metadata of a single hearing instrument. The variation points are grouped into Software Configuration Items, which can be XML files, tables, DSP code interface memory maps, audiological 4

5 parameters etc These configuration items are versioned and linked together to form the product metadata of a hearing instrument type. The metadata for each released version of every hearing instrument type is stored in a catalog database that will be used by Fitting Application software on run-time. The complexity shown above obviously can only be handled with a domain-specific tool. 8. Automate SPL production process Automation is the key aspect for an agile Software Product Line development. Most importantly, the SPL production process has to be fully automated (refer to Figure 4). With a single mouse-click, a specific version of a specific product has to be produced. The production process includes synchronizing files (platform and product assets) in the desired version from SCM, substitute variations of variation points, building executables, creating installation packages and deploying it to a common place. Test automation is also desirable. At least unit- and component-tests as well as a smoke test for each software product should be fully automated. The rapidly increasing amount of products and product releases drove us to invest into building a test infrastructure that allows creating automated system tests. Mainly regression and stability tests are constantly monitoring the quality of the main development and release path. In our Software Production Line we created a production server that offers a web-page to build any SPL product of any released version or SCM path and have it smoke tested. This allows a developer to quickly check and test the products after a major check-in. Of course, automated nightly builds will report the source code integrity to the developers when they start their day in the morning. 9. Lean and agile SCM procedures Processes are only useful if they are carried out and lived by the entire development team. Simple rules and lean procedures are typically better understood and implemented than heavy over-engineered processes. Generally, we choose lean and agile SCM procedures for: check-in, change packages, releasing and branching strategies, defect tracking, functional verification, etc. Here are some (un-)usual practices listed: 9.1. Check-in s Changes to the source code are announced by between developers: Every developer sends an for a check-in to all other developers working on the platform or product integration projects. The contains a brief change summary, impact on other assets, notes to specific developers or other important information. This allows every developer to be up-to-date with the source code on the main path. The daily effort spent on check-in s for a person in a team of about 20 developers is less than 15 minutes Check-in policy To maintain a stable main path, we defined the following check-in policy: a) Resynchronize local SCM sandbox b) Make local build and smoke test c) Check-in files or change package d) Send check-in e) Kick production server (optionally with test suite) f) Have change reviewed (optionally) 9.3. Stabilize main path As soon as the production server informs about a broken build or smoke test error, the highest priority of the team is to stabilize the main path again. Figure 4: SPL production process: Automate it! 5

6 9.4. Daily user deployment test Besides the fully-automated test system that bases its tests on a complete installation package, we also perform a manual deployment and installation test. Every morning a tester installs the nightly-built software deployment package on a PC and executes a typical workflow testcase and some exploratory (a better word for ad-hoc) testing. She reports the quality to all potential in-house users to inform them if this particular build is worth-while to install. These 30 minutes are well spent and paid back many times Typical day of a software developer Here is what a typical day of a software developer at Phonak looks like in terms of SCM patterns they use: First thing in the morning (typically 9 o clock for software developers) the developer checks the report e- mails from the production server and user deployment tester. In case of an error, he would team up with other colleagues to immediately fix the problem and stabilize the main path. Otherwise, with a single mouse-click he resynchronizes the sandbox, his private workspace, from the SCM repository and populates it with the entire code base for the platform core-assets and the applicationspecific assets, as well as external components from a third party codeline. Progressing through the day, the developer implements a new feature or fixes a bug and therefore checks out files from a SCM tool into a change package. Depending on the task, pair programming may be used to implement and review software changes. Because the main path codeline is supposed to be in a stable shape, he commits the change package according to the check-in policy: First he resynchronizes the local SCM sandbox, performs a local system build, executes local automated smoke- and unit tests and when everything works fine, he will check-in the change package. After check-in, he will kick the production server to start an entire build of the installation package and execution of the smoke-, unit- and regression tests. Finally, he will send a partially generated check-in to notify all developers and testers of his change made to a core-asset or product-specific asset. Throughout the day he will be notified by check-in e- mails of all changes his colleagues make on the common software code base Requirements engineering There is especially one practice area where I don t recommend a lean process is requirements engineering. The separation of application and domain engineering as described in [2] is essential to develop core assets that provide common functionality and expose clearly defined variation points. Our requirements engineering process for new core asset features (refer to figure 5) allows input from all product managers (variability), domain specialists (commonality), project leaders and architects. Sometimes after several rounds of reviews by the review board, feature requirements get approved and if desired released by the platform steering group to be implementation in the next platform development iteration. Figure 5: Feature requirement engineering process 6

7 A feature requirement is a document that captures the requirements of a single feature. It consists of an overview including business rational, the actual requirements in use-cases, business rules and nonfunctional requirements and a project planning section for estimates and architectural impact. Most importantly, in all mentioned sections above, variation points are identified and possible variations are specified as part of the requirement. A feature requirement is typically written by a domain expert with input from product managers to define the variability. The review process involves product managers and project leaders from product development as well as domain experts, project leaders, architects and test managers from the core asset development. A steering board is the approving authority to release the requirements for a new feature. The implementation of approved features will be schedule by the platform project manager according to its priority and demand by product releases. The feature development will follow the process described in chapter 5: main path centric development. After a feature requirement has been frozen and scheduled for implementation, changes to it must follow our change request process, also not agile. Using this not-so-lean requirements process brought our development more stability, better planning and commitment to milestones, better handling of commonality and variability, a better domain and implementation understanding and also better quality. Generally speaking, our development of software features is more predictable, faster and cheaper since we introduced this requirements engineering process. 10. Related Work The foundation of the Software Product Lines development paradigm can be found in [1] and [2]. Both books are an excellent source about technical and organizational aspects of SPL, describing the fundamental concepts like scoping, domain-/application engineering, commonality versus variability, architecture, core assets, production environment, testing aspects, as well as organizational structures, practice areas, processes and success criteria. Regarding Software Configuration and Release Management, Stephen Berczuk s book [3] is a must to read. He defines widely accepted SCM patterns and describes them in a clear and easy understandable way. In a recent article [4], he describes an agile SCM process based on the patterns defined in his book. By coincidence, the agile process described in this article is actually very similar to the agile SCM process we have chosen for our software product line at Phonak which discussed here in this paper. The following books offered input to our software development process and are a great source on agile software development [5], best practices in software development [6] and feature driven development [7]. [7] is a hands-on book that describes the combination of speed and flexibility of agile methods with enterprise class scalability. 11. Conclusion Yes, I m convinced that the techniques of Agile and Software Product Line development can be used in the same software development process. The heavier Software Product Line approach acts as an overall development strategy and is combined with some agile software development techniques to create leaner and more agile software development processes. This experience report shows some agile practices in the SPL configuration and release management practice area which have been proven in a Software Product Line for hearing instrument fitting applications. Similar examples of agile processes and procedures could be found in other Software Product Line practice areas as well. Experiences taught us, that the requirements engineering practice area is not suitable for lean and agile processes. A last remark regarding people- versus process-/toolfocused development: People should always be valued above processes or tools, no matter if the process is agile or not. Tools help people do their work more efficient and processes help people to work together more efficient. If they don t increase efficiency, they should be changed or removed. Valuing people is done through the company culture and not by tools. A core value of Phonak s culture for example is as defined by one of its founders: Ooni lüüt gaat nüüd! Nothing works without people! 12. References [1] P. Clements & L. Northrop: Software Product Lines: Practices and Patterns. Addison Wesley 2001 [2] Klaus Pohl, G. Böckle, F. Linden: Software Product Lines Engineering. Springer 2005 [3] Stephen P. Berczuk with Brad Appleton: Software Configuration Management Patterns. Addison Wesley. [4] Stephen P. Berczuk: Breaking with tradition. An Agile Spin on Software Configuration Management. Article in Better Software issue February 2006 [5] Alistair Cockburn: Agile Software Development. Addison Wesley [6] Steve McConnell: Rapid Development. Microsoft Press 1996 [7] Stephen Palmer: A practical Guide to the Feature- Driven Development. The coad series, Prentice Hall 2002 [8] Tool: Gears by BigLever Software Inc. (Ch. Krüger) [9] Tool: pure::variants by pure systems (D. Beuche) 7