PERFORMANCE ENGINEERING IN SCRUM Balasubramanian, Infosys Technologies Limited This paper describes how performance engineering as a software discipline should be planned and executed in an agile development cycle. The intention of this paper is not to explain in detail the agile methodology or Performance engineering, even though it provides a high level description of the same. It is expected that the readers have a basic understanding of agile methodologies and performance engineering. Introduction Agile as a software development methodology is fast becoming a popular approach due to its ability to react to business changes. While there are still fears about adopting an agile approach to software development, the industry is clearly seeing a rise in agile adoption. Every year, thousands of dollars are being spent to fix poorly performing applications. This has made the software industry to relook at the way performance engineering is executed. A more proactive approach to architect and design for performance is now planned than a reactive post-mortem like approach. There needs to be a clear direction in identifying the various activities of performance engineering and executing them in logical sequence during an agile development process. This paper aims at suggesting one such approach by indicating various performance engineering activities across an agile development process. Overview of Agile Traditionally software has been developed using a waterfall approach. While this suited the initial days, as business complications grew and demand for time to market increased, waterfall model did not deliver the required results. Agile methodology was born out the need to respond to rapidly changing business requirements and deliver increments of shippable application with time to market as the primary focus. In the book, Lean Software Development: An Agile Toolkit for Software Development Managers", Mary and Tom Poppendieck bring out the concept of eliminating waste in software projects. Agile as a methodology constantly looks at engaging the customer to prioritize scenarios that should be developed in each cycle, so as to eliminate waste. There have been many flavors of Agile developed over the past years. Agile Modeling, Scrum, Extreme Programming (XP) and Agile Unified Process (AUP) are some of the well known Agile Methodologies. For the context of this paper, we will analyze how performance engineering activities can be planned within a Scrum project cycle. Scrum An Agile Methodology Scrum A term derived from the game rugby, indicates a methodology of using small cross functional teams to deliver shippable software using cycles called sprints. This approach was described initially by Takeuchi and Nonaka, in the year 1986, and has been evolved into a widely practiced agile methodology.
Figure 1: SCRUM Methodology The cores of the process are the sprints which are typically a 15-30 day period where the development team focuses on producing production ready software. The various phases of a scrum project are explained in the sections below. Planning and System Architecture Every Scrum project typically begins with this phase. Some of the activities in this phase are Understanding the requirements Developing the comprehensive list of backlog Risk Assessment Develop system architecture Review architecture and develop high level design Sprints The goal of sprints is to develop shippable, production-quality application. Typically, there would be representation of all the activities of the SDLC lifecycle in each sprint. Note: It is not necessary that all sprints should have software as its deliverable. There are instances where a sprint would develop documentation like test scripts, user documentation etc. Closure This phase provides the team with the opportunity to view the complete application and have all sign off activities. A deep dive session into learning is also done during this phase. Overview of Performance Engineering Traditionally every software application has been tested and validated against the functional requirements. As we integrate more systems that are on various platforms, realized in various programming languages to serve a common need, we have built applications that are huge, that communicates across various layers and protocols. Performance Engineering is the discipline of SDLC that focuses on ensuring that applications are architected, designed, built and tested for performance. Performance engineering does not just involve reactive approaches like tuning, but focuses heavily on designing the application to deliver on SLA s. A typical performance engineering activity would include Collecting and validating the non functional requirements Developing the required models for analysis Developing a performance test strategy Reviewing the architecture and application code to ensure compliance to performance best practices Reviewing the deployment of the application, and For pre-existing applications, reviewing the design and code to suggest appropriate tuning activities Why do we need Performance Engineering in SCRUM? Performance Engineering is an important activity for projects executed using any methodology. It
becomes more critical for projects using SCRUM or any other agile methodology because of the following reasons Immediate feedback about system performance in each sprint Delivering production quality software during each sprint continuously is the crux of Scrum. In such a scenario, it is important to be able to receive feedback about the performance of the system as they are incrementally built in each sprint. It is a well known fact that the later issues are found, the costlier it is to fix them. It is easy to develop a false impression about performance Usually functional testing happens in a scaled down version of the environment with scaled down data. Clearly, performance is not the focus here and is normally ignored. This leads to the false impression that the application is performing well or will perform well. Also, in an agile methodology, functionalities and hence application code piles up with each cycle, and this leads to poorly performing code being injected into the system. What is more dangerous is the fact that it goes unnoticed. Refactoring may introduce code that performs poorly Evolving and fine tuning design is an inherent part of scrum. This leads to refactoring both framework components and application code. There might be instances when such refactoring can inject code that performs poorly. The goal is shippable, production quality code It is stressed enough that the every scrum works towards delivering a production quality application. Meeting the Quality of Service requirements is an absolute must for an application to be deemed production ready. Performance engineering ensures that an application is designed, built and validated against the required Quality of Service requirements. Hence the need for performance engineering is inherently built into Scrum. Performance Engineering Processes in Scrum This section will suggest various performance engineering activities that should be planned for and executed in the stages of scrum lifecycle. Planning System Architecture Sprints Closure Understand and Validate NFR s Architecture Assessment The NFR Viewpoint Identify and Plan Performance Tests Performance Test Strategy Identify bottlenecks and fix performance issues Setup Performance Monitoring Systems Identify Performance engineering activities in product backlog Review Design and code Figure 2: Performance Engineering In SCRUM
Planning Stage This phase involves understanding the requirements, and planning for the sprints ahead. The various performance engineering activities that should be planned during this phase are listed and explained below Use cases and Performance Assigning performance requirements to use cases will enable architects to understand the performance needs of various areas of the application from a functional perspective. This approach could also aid in obtaining sign off s on various use cases on Quality of Service requirements. Understand and validate Non Functional Requirements This is an area everyone would agree is important and critical, but is the one often missed or partly done due to various constraints. Efforts spent here will help the team to validate the performance requirements and be realistic about it. Performance Test Strategy It is important to define the test strategy, which will include details like Scope of Performance Testing Infrastructure needs Methodology used for testing Workload characteristics used for testing What are the tool requirements and licensing cost? What are the metrics that are captured? How will the results be shared? Identify Performance engineering activities in product backlog While we are identifying feature lists for the product backlog, it is also important to identify the various performance engineering activities like workload modeling, performance testing, performance assessment etc. Many a times these activities are inherently assumed and not planned, resulting in insufficient planning and poor execution. System Architecture Phase Defining architecture for the system determines the success of the code to meet the requirements. While there has been a lot of focus on validating the architecture for the various functional and business requirements, it is sad that reviewing architecture for NFR s is not planned as a matter of priority. Architecture Assessment The Performance viewpoint Every software development project should undergo an architecture assessment. It is more important in an agile methodology, as the inherent nature of the model is dynamic and having an architecture that meets the requirements will enhance the success of the project. During the architecture definition stage, most often than not, we perform a qualitative assessment of the architecture. Such assessments help understand how the various quality of service requirements like performance are addressed in the architecture. Some of the well known assessment methodologies are Architecture Tradeoff Analysis Method Cost Benefit Analysis Method Active Reviews for Intermediate Design Architecture reviews conducted at this stage will provide us with areas of architecture that have to be modified / updated to achieve better performance. It should be ensured that the architecture is updated and bought to an agreeable shape before the sprints could begin. The above step is critical as any significant changes to the architecture during the sprints will adversely affect the progress.
Sprints Sprints are the building blocks of any scrum methodology. Shippable, production quality software are developed, tested and deployed during each sprint. A strict control on Quality of Service parameters during the sprints will ensure that the application meets the Quality of Service requirements. Code right Every application development uses coding nest practices that talks about naming conventions, usage of proper headers, commenting guidelines etc. It is very unusual to find coding guidelines that talks about coding right to achieve good performance. Normally it is expected and left to the developer to know how to write good performing code. Effort spent during the construction period to emphasize and follow good coding practices will go a long way in saving time and effort in fixing performance issues in later sprints. Create Performance Unit tests Unit tests normally test functionality and business rules. Developers should write unit tests that tests the performance of the application developed in a sprint. Combined with the performance numbers from the use cases, performance unit tests can give a clear indication of the application meeting the performance requirements. It is important to note that architecture and design would evolve during the initial sprints. Hence, there is a minimal benefit in practicing unit level performance testing. As the sprints become longer, which is a sign of the maturity of the application, we should focus on unit level performance testing. Until then performance should be measured at a system level. Review Design and Code There should be a constant endeavor to review the design and code during the sprints for existing and potential performance issues. Automating code review using tools like FxCop, which has rule sets to identify performance issues can be used to expedite the process. With this iterative and consistent approach to identifying and fixing performance issues, there is a continuous improvement to the quality of code. Identify and Automate Performance Tests The team identifies and closes on the list of features that will be implemented during each sprint. During this process, we should also plan for conducting the performance test for the features that have been implemented. As a best practice, we should identify the list of features for performance testing a sprint ahead. This will aid the team in automating the performance tests that can be used in the subsequent sprint. Automation of tests is mandatory as manual effort will lead to delays and errors, which will prove costly in a time-boxed sprint. Figure 3: Performance test planning in SCRUM Identify bottlenecks and fix performance issues Every performance test is executed with the focus of identifying bottlenecks in the application. Fixing
the bottlenecks should be done during the course of each sprint, so as not to pile up the issues and buggy code which leads to performance issues. Performance issues discovered during a sprint should be planned as features that should be implemented during subsequent sprints. Figure 4: Plan on fixing performance issues in subsequent sprints Performance Engineering Sprint Initial sprints will witness an evolution of architecture and design. There will be changes to the functionality and refactoring to the application code. As the sprints become more mature, which are characterizes by their longer duration, we should consider on planning a parallel sprint for performance engineering activities. We can also consider a parallel performance engineering sprint for every 2-3 normal sprints, so as to have a considerable functionality implemented for performance testing and assessment. By being a parallel, independent sprint, a performance engineering sprint does not affect the progress of the application development. Also, it provides an isolated environment for the performance engineering team to analyze the application performance. Activities like performance testing, performance assessment, capacity projection can be carried out in this sprint. Closure This phase involves deploying the complete application in the production environment, and obtaining sign off s from the stakeholders. Setup Performance Monitoring Systems The team should plan on using systems and tools to monitor and report performance issues, so that they can be resolved quickly. Challenges in Executing Performance Engineering in SCRUM Implementing performance engineering processes in scrum has its set of challenges. This section explores them and suggests ways to solve them. Engaging the customer Being qualitative rather than quantitative This has been the challenge with any agile methodology and it just gets bigger with integrating performance engineering. It is usual for customers to expect the applications to be extremely quick and respond well. It is a challenge to engage stakeholders to arrive at a qualitative number rather than being quantitative. Project teams use questionnaires, analyze logs, and create workload models to validate performance requirements to arrive at a realistic goal. Automate tests Manual is not on Advantages of automation are well known, and so are the challenges associated with it. Identification of the right tools for automation and having the right skilled team are essential for effectively practicing performance engineering in a scrum model. Estimating for that additional time for automation and procuring licenses for various toolsets are an
important investment which will bear results on the long run. Mindset The toughest nut! While the concept of adapting full fledged performance engineering in a software development lifecycle is still catching on, there might be resistance in following it in agile development. Ideas such as associating performance numbers to use cases and writing performance unit tests are new to the industry and a change in the mindset is needed to ensure that it works. References Agile Software Development - http://en.wikipedia.org/wiki/agile_software_development Performance Engineering - http://en.wikipedia.org/wiki/performance_engineering SCRUM Development Process Ken Schwaber Realizing Continuous Performance - Management Best Practices Documentation Steve Haines