High Level Requirements for Capacity Management Nov 2011 While every IT System needs to be managed year on year for capacity, the traditional methodology is to add capacity on demand or to live within a budget. Quite often this is a result of not being holistic in capacity management. Once the right set of questions are in front of a CIO or an IT Systems manager, it makes the planning that much easier. This document proposes a set of questions to be asked when managing capacity of any production system. Copyright (C) 2011 Rajesh Mansharamani Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License". http://www.gnu.org/licenses/fdl.html
High Level Requirements for Capacity Management Rajesh.Mansharamani@swperfengg.com November 2011 By high level requirements we mean the questions that the relevant stake holders are going to ask when it comes to managing capacity of any production system. Before we get on to the set of questions, we need to be clear about the overall objective of capacity management. We define it as: Overall Objective: For a given choice of infrastructure (hardware and software technologies) what is the best way to meet SLAs of customer business transactions, for an ongoing business workload. Before proceeding let us explain what is meant by the three highlighted terms above. Given choice of infrastructure means a set of hardware technologies available for use and a set of software technologies or platforms available to be deployed on these hardware technologies. Hardware technologies would include servers, storage, and network equipment, running one or more operating systems on them. Software technologies would include web, application, database servers, messaging solutions, document management solutions, ERP solutions, CRM solutions, and any packaged or custom built software solution to meet the customer s IT requirements. Best way could mean one or more of the following: i. At the least cost (without compromising selected SLAs) ii. At any cost (for specific business requirements or for specific points in time) iii. Within a budgeted cost (even if that means compromising some SLAs) Ongoing business workload means today s business workload as well as planning for the near to mid term future workload. Business workload means the volume of business transactions and business data to be managed per unit of time. Now we need to get in to the requirements. But we need to know for whom these requirements are to be stated. In other words who is the stakeholder who is involved with capacity management in production. Who are the stakeholders? Typically there will be five key stakeholders internal to any IT system. These are: a. BIO: Business Information Officer b. CIO: Chief Information Officer, who would be owning the infrastructure c. IT Managers who own budgets for infrastructure d. IT System Administrators 1
e. IT Vendors responsible for subsets of the data centre applications and/or infrastructure Now let us get on to the set of questions that these stakeholders would like to ask related to performance and capacity management. What questions will be asked by the stakeholders? a. BIO: i. What is the peak, average, and maximum business workload (as per key business metrics such as orders) per day, per week, per month? ii. Which business transactions are growing at the fastest rate? iii. What is the growth rate of key business transactions (e.g. account opening) and key business entities (e.g. customers), as seen from data available in the enterprise? iv. What are the correlations across business transactions and between business transactions and business entities? v. Based on data available from the past, what is the estimate for business workload for the next quarter and for the next year? vi. Which business transactions are failing to meet their SLAs? And what is causing the violation? vii. What are the options available to further reduce SLAs of key business transactions? b. CIO: i. What is the average utilization of servers in the data centre? ii. For a given business workload forecast how much capacity needs to be planned for servers, storage, and networks? (At the least cost?) iii. How will the change in business SLAs impact the data centre server utilization? How much more capacity would be required to maintain utilization below X%? iv. How much additional storage will be required for business workload forecast, SLA management, compliance requirements? v. Can application hosting be rebalanced across servers to reduce server counts or to better meet SLAs? vi. How much reduction in utilization and improvement in SLA would be expected by migrating to the latest generation of hardware? vii. What are the alternatives available to speedup a given batch or a given business transaction? How much time would these take? viii. Which applications eat up the maximum capacity in my data centre? Are these applications most critical to the business? ix. Which applications can be hosted on VMs without compromising business SLAs? x. What is the capacity division across security processing, web processing, application processing, data processing, communication, DR processing? xi. Which applications are highly correlated in terms of workload and capacity utilization? 2
c. IT Managers owning one or more applications: i. What is the workload for my set of applications and how does it relate to the overall business workload? ii. What percentage of response time do my applications contribute towards overall business SLAs? iii. Will moving my applications to a dedicated set of servers help and if so by what percentage in terms of SLAs? iv. If I had to rehost some application components on to shared infrastructure how much would it benefit my application and the other applications running on the same infrastructure? v. Which components of my applications are getting hit the most in terms of workload, SLAs, utilization and what can I do to take care of this? vi. A new application (component) is being released next week. How can it be deployed optimally and what impact would it have on the business SLAs? d. IT System Administrators: i. In the mass of servers/technologies how do we pinpoint which are the ones that we need to focus on? What SLAs/thresholds should we keep for alerts? ii. A given device keeps running out of capacity? What is the root cause of the problem and how do we fix it? iii. We need to look at hardware utilization over time and figure out when abnormal trends can occur? iv. We need to know which set of server and software components are likely to trip whenever there is a surge in workload? We need to be able to model surges of workload and generate pseudo alerts of components that are most likely to cause issues including failures. v. We need to know when will we run out of capacity (CPU, storage, memory) at the current rate of consumption? We need to do a what if analysis of bringing in more capacity and seeing its impact. vi. We would like recommendations on the tuning options available to reduce response times/utilizations on a given set of servers/software components. e. IT Vendors: i. We need an unbiased view of whether the infrastructure/applications we are managing is responsible for violation of business SLAs ii. We need to be alerted whenever there is a problem or a problem likely to happen in the infrastructure/applications that we are managing iii. Another vendor has released new infrastructure/applications in the data centre today. And today itself problems have started arising in IT operations. We need to be able to check whether this change is impacting any of our infrastructure/applications and whether we need to do anything about it. 3
iv. We need to model the impact of higher workload/more stringent business SLAs on our infrastructure/application performance Approach to Capacity Management Given these diverse requirements across various stakeholders it is usually best to have a capacity management tool that analyzes past data and makes predictions about the future. Third party tools are available but unless one goes through their own requirements in detail it is futile to expect a third party to tool to suit your purposes from day one. The minimum data that should be available is resource utilizations, response times both end to end and component level, and business and technology workload, across several months. Information about events such as version releases of applications and technologies, will help in bringing out various correlations. The tool should have the ability to correlate data, validate data, predict performance as a function of workload, apart from highlighting current bottlenecks, and future bottlenecks. The tool should be able to highlight which resources are lying underutilized and what should be done to best utilize them. The tool should be able to provide for what if analysis such as what would happen if extra capacity was added at the bottleneck, or what would happen if the workload grew by 30%. Technically the tool should be able to provide for charting, should have basic performance modeling built in, and should support a lot of analysis. One should think of the tool like a data warehouse of performance data. 4