FICS Infrastructure Management Architecture

Transcription

1 FICS Infrastructure Management Architecture Establishing a Control System for the Enterprise IT Infrastructure Published: April 2007 Copyright 2007 OSIsoft, Inc. All rights reserved. WiredCity and the WiredCity logo (and IT Monitor) are trademarks of OSIsoft, Inc.

2 Contents Introduction...3 Executive Overview...3 Challenges in Managing the Enterprise Infrastructure...4 Today's Infrastructure Management Environment...5 What Is Required to Manage the Enterprise Infrastructure...6 Architectural Approach...6 Feedback-Based Control Systems...7 Open-Loop Control Systems...7 Closed-Loop Control Systems FICS...7 Modeling...11 Creating a FICS-Based Management Infrastructure...12 Getting Started...12 Assessing Systems and Components...14 Establishing Goals...15 Establishing the Architecture and Models...15 Defining a FICS Architecture...15 Developing a Baseline Model...16 Defining the Desired State...16 Applying FICS...16 Locking Down the Process...17 Use Case Cornell University Weill Medical College, Institute for Computational Biomedicine...17 Introduction...17 ii

3 Introduction IT infrastructures have become an integral part of the business of almost every enterprise. The services that comprise these infrastructures are no longer technical adjuncts to primary business objectives, but now represent the foundations on which business offerings are built. The success of most enterprises is dependent on the ability of an organization to successfully manage the infrastructure in a way that ensures that services are available and meet the demands of the business, today and as the business grows. Making this happen requires the right management architecture one that can dependably and efficiently deliver IT services in a controlled manner based on decisions driven by what is right for the business. Infrastructure modeling has become an effective tool for designing and managing an infrastructure. Modeling itself is not enough, though managing and evolving the infrastructure in a manner that meets current and future business requirements, minimizes problems, and minimizes the time required to make everything work right together requires an effective control system. Feedback IT Control System (FICS) provides a highly controllable, structured, efficient approach to managing the infrastructure that maximizes the predictability and responsiveness of the infrastructure. Executive Overview At its best, management of an enterprise IT infrastructure can be the cornerstone of a growing and successful business. At its worst, infrastructure management can be a quagmire of data, processes, and tools so complex and voluminous that even understanding it all can be a significant challenge. The Feedback IT Control System (FICS) makes it possible to go from challenging to cornerstone in a way that streamlines infrastructure management and establishes a framework that helps ensure effective alignment with the needs of the business. It does this by overlaying the management infrastructure with a set of control actions that make it possible to understand what is happening in the environment and quickly respond to requirements as they emerge. As a result the infrastructure management goes from this: Diverse and complex management systems and components, many which may be in unknown states, that complicate infrastructure management more than they make the infrastructure manageable. Ad hoc tools and processes that are resource intensive to manage and difficult to coordinate. Enormous amounts of management data that can be almost impossible to understand and extract the information that is really important. To this: Streamlined management systems and components that are well understood and make it easy to manage the infrastructure effectively and efficiently. A holistic approach to managing tools and processes that simplifies management. Concise and targeted management data that makes it easy to understand and respond to requirements, in real time and over time. FICS does not require specific management tools or processes it comprises architectural principles that can be applied to any management infrastructure. It is based on application of feedback-based controls similar to those of Six Sigma, but focusing more on details than statistics. It is easy to start implementing FICS-based architecture, regardless of the current condition of the management infrastructure because it can be applied to a single system or component at a time. As it is applied to additional systems and components, especially those that are critical to the business, it can provide a very predictable, highly available, and extremely responsive infrastructure. Page 3

4 Synchronization of the IT management infrastructure with management of the core business of an enterprise requires a robust and adaptive infrastructure that is driven by business initiatives, not infrastructure technologies. Traditional approaches to infrastructure management are generally based on cause-and-effect responses to emerging needs, incorporating new processes and technologies to meet specific needs. In this environment, technical requirements often dictate the modifications made to the infrastructure. These modifications are generally made across a broad group of systems, with wide ranging impacts on the infrastructure, the services that run there, and sometimes business itself. Such wholesale modifications are not only time-consuming to implement, but can also be extremely difficult to manage because of the number of variables introduced simultaneously. Too often the result is a complex interweaving of diverse components and complicated processes that make it difficult or impossible to know how everything works typically there are parts that work well and others that don t work well, and it is difficult to distinguish between them because results are not readily traceable and measurable. The parts that don t work well together often require manual intervention to reconcile differences and make results usable, which can be very time consuming and error prone. Enterprises today need more transparent, effective, and robust infrastructure management that minimizes any problems that might affect the business at the same time that it provides consistently predictable and effective results that can be measured and improved based on business value. This requires the use of sound architecture design principles focused on operational excellence and representing an effective control system that both protects and fosters the stability and agility of the infrastructure in a way that supports the growth of the business itself. FICS is a set operational excellence principles for infrastructure management architecture that can be applied to the management of any IT infrastructure. This control system uses a structured model-based approach to managing the infrastructure in a way that makes it possible to: Determine what works and what doesn t work, including establishing measurements that enable effective tracking and control and provide the basis for determining the business value of making modifications. Assess the impacts of modifications early in the planning and design process to ensure that results are predictable and effective. Keep the infrastructure in a known state that is both stable and agile, supporting proactive and responsive management. The result is infrastructure management that provides the highest levels of availability, predictability, and adaptability for all infrastructure systems and components. The basic FICS components include: A feedback-based control system that addresses the functionality and flow of information required to manage the infrastructure A modeling methodology that enables evaluation and refinement of proposed modifications early in the design and planning processes. A defined process for implementing and using FICS to manage the infrastructure. This document describes FICS and how to use it to manage an enterprise IT infrastructure. It also provides a use case that demonstrates how FICS can be implemented and how well it can work. Challenges in Managing the Enterprise Infrastructure While demands on enterprise systems are growing, so are business requirements, including the requirement to reduce total cost of ownership (TCO) and align the IT infrastructure more closely with business goals and requirements. The primary challenges of managing the IT infrastructure in a way that satisfies demands and meets business requirements grow out of two underlying causes: Page 4

5 Today s infrastructure management environment is complex, presenting complicated issues that must be addressed daily this environment often evolves in ways that introduce many unknown factors, further complicating management. Requirements for effective infrastructure management are not well understood because they have generally not reflected a holistic view of the infrastructure and what is really required to manage it. This is especially true in environments that have a multiple and disparate management systems. These root causes present a multitude of challenges that must be understood and addressed to effectively and efficiently manage an enterprise IT infrastructure. Today's Infrastructure Management Environment An infrastructure management environment includes a multitude of service-based systems and other components, as well as the components required to manage them. The nature of the infrastructure management environment makes it challenging to manage systems and components because: The infrastructure is large and complex. As users demand support for increasingly sophisticated technologies (such as wireless products) and business leaders undertake additional business opportunities (including mergers and acquisitions), the infrastructure continues to grow, becomes more heterogeneous, expands across political and other boundaries, and grows more complicated. Many infrastructure components are changing rapidly, both to keep up with technology (including security enhancements) and to meet the needs of the business. This makes it difficult to maintain a cohesive view of the infrastructure, including the components used to manage it. Many individual systems are not well documented and some are not in known (reproducible) states. Identifying and documenting all system components can require a considerable amount of time and effort, which is not always justifiable from a business perspective. Unfortunately, management processes and technologies tend to evolve in an ad hoc manner in response to specific needs. Environment systems and other components in the infrastructure evolve in their own distinct ways, based on the environmental requirements and demands of the business. In general, the current state of infrastructure management in an enterprise is in one of the following stages: Basic. A very reactive approach focused on maintaining a status quo and responding to problems as they occur. Standardized. A more stable approach focused on incorporating standards and building on defined, repeatable processes that reduce problems and improve predictability. Real-Time. A streamlined and proactive approach that builds on standardization and automated tools to provide real-time support for systems that efficiently and effectively meet current business demands, quality initiatives, and in-depth security requirements. Dynamic. A sophisticated and forward-looking approach that integrates IT management with business management, providing agile and improvement-oriented support for emerging business needs. Progression from an initial, more reactive stage to a more advanced and proactive stage of infrastructure management is important for meeting the needs of today s enterprises, but determining how to make infrastructure management work as required is a challenge. The sheer number of processes and technologies available to address specific infrastructure management requirements can be overwhelming, as can effectively integrating them in a cohesive manner. The result of the ongoing drive to meet the requirements for infrastructure management is generally that the enterprise implements a variety of management tools and processes to address specific needs some work, Page 5

6 but many fall short of expectations and requirements. Even tools and processes that work often do not work well together. Tools and processes may include a large numbers of features and functionality that are complicated to manage (many of which can be extraneous bells and whistles that are more confusing than useful). Working with such ad hoc tools and processes can make it difficult, seemingly impossible in some instances, to provide the degree of control, cohesion, and support needed to ensure the predictability, efficiency, and transparency necessary for infrastructure management if an enterprise is to effectively meet current and emerging requirements. It can be a significant challenge even to determine how effectively the needs of the business are being met by infrastructure systems and components, including the tools and processes used to manage the infrastructure. What Is Required to Manage the Enterprise Infrastructure One of the most important challenges facing enterprises today is determining what is required to effectively and efficiently managing the infrastructure. If you ask this question of IT professionals responsible for infrastructure management in enterprises today, you would probably notice a lot of common responses, including the need for: Having an accurate inventory of what environments are available, what physical systems make up those environments, and what business roles the applications that run on those systems perform. More automation and command-line support to enable more centralized and efficient management. More self-healing components and systems that keep the infrastructure running and provide indicators of system state and status, as well as identification of any trends that may be significant. More effective notification and reporting that can be customized to the needs of the business. Simpler interfaces and better integration of components, preferably with a single dashboard for monitoring and managing the infrastructure. Better integration of components, even heterogeneous components from multiple vendors. All of these can significantly improve infrastructure management, but none is guaranteed to provide what is really required for successful and ongoing management of the infrastructure this is because introduction of any of these into an infrastructure, each of which represent a potentially significant change to infrastructure management, can cause unpredictability and instability of individual systems and components. And, if it is not clear what is working well (and not working) in the current environment, changes may or may not provide the required business value. What is needed to effectively address infrastructure management requirements today and in the future is an architecture that provides a comprehensive view of the infrastructure and a more controlled method of managing and evolving it. This architecture must enable a clear understanding of the infrastructure management tools and processes, support measurement of results, and meet both day-to-day management requirements and strategic business directions. Architectural Approach The traditional view of infrastructure management focuses more on components than architecture these individual components, including tools and processes, together can make up a complete entity but generally have few shared characteristics or properties, even inherent defects in individual components or systems are generally component-specific. The number of disparate components and the even greater number of characteristics and properties make it difficult to assess individual characteristics and properties, understand the impact that they have on the infrastructure as a whole, and determine how well they support infrastructure management. However, the individual properties and characteristics of tools and processes can significantly affect the ability of an enterprise to successfully manage the infrastructure. An architecture-based approach to infrastructure management provides a more holistic view of the Page 6

7 infrastructure and its management, facilitating an understanding of the individual properties and characteristics that are essential to good management. FICS is a model-based architectural approach to meeting the infrastructure management challenges and requirements of today s enterprises. FICS makes it possible to streamline the management process, identifying and implementing tools and processes that work and identifying and eliminating components that don t work or are not appropriate, based on individual properties and characteristics. The FICS approach uses methodologies that are more common to scientific endeavors than enterprise infrastructures methodologies that address critical needs for effective control and evolution of the infrastructure. Critical requirements of the FICS architectural approach are: A feedback-based control system that uses continuous analysis along with specific control actions to manage the infrastructure, starting with infrastructure management tools and processes. A modeling method that builds on the control system, providing a way to assess potential impacts of specific control actions, including validation of results early in the planning and deployment process. Together, establishment of a feedback control system and the related modeling method can help eliminate elements that could introduce instability, at the same time that they provide significant increases in the manageability, responsiveness, and functionality of the infrastructure. Note: FICS can be applied to more than just infrastructure management it can be extremely effective at controlling a broad spectrum of systems and services. FICS-based infrastructure management should have highest priority, though, and should be the starting point, so this document covers only the application of FICSbased architecture to infrastructure management. As covered in the Creating a FICS-Based Management Architecture section later in this document, FICS should be implemented incrementally based on specific priorities that ensure appropriate controls and business justification. Feedback-Based Control Systems To be effective, a management control system must provide a method of constantly assessing and responding to the requirements of the infrastructure and the business. Control systems generally fall into one of two categories: Open-loop control systems Closed-loop control systems Open-Loop Control Systems A control system is open-looped if the output (result of an action or activity) is not fed back to or does not directly impact the input (including the start state and forces that affect it). Traditional infrastructure management systems are generally based on an open-loop control system although results can positively affect system operation, controls do not ensure a direct mapping of output to input. This type of control system can introduce system instability or other problems because it does not require monitoring or use of the output, which makes it almost impossible to effectively validate and act upon results. For example, a traditional braking system for an automobile is a static system that applies braking pressure to stop a car. However, if conditions are less than optimal (such on icy roads), no feedback is provided to make the braking action predictable. The result can be the inability to stop or skidding out of control. At best, an open-loop system offers unpredictable results. Closed-Loop Control Systems FICS A control system is close-looped if the output, including results of all actions applied to any part of a system or component, is applied back to the input in a way that either conforms to the input or directly Page 7

8 impacts the input, indicating specific and measurable differences between the input (initial state) and output (results). Successful application of a closed-loop control system can provide reliable management of system and component properties and characteristics, as well as actions related to those properties and characteristics, on an ongoing basis. For example, the antilock braking system (ABS) uses a closed loop feedback system when the brakes are applied, the system makes discrete and continuous measurements of each wheel and feeds them back to a control device, which uses the information to automatically apply brake force metering proportional to what is needed at each wheel to provide steering stability even in unpredictable conditions. Bosch, the firm that developed ABS, is recognized as a pioneer in the development of closed-loop control systems. For more information, see ABS a success story at The bottom line is that a closed-loop control system offers more predictable and dynamic management capabilities than an open-loop control system. It does this by: Providing more complete information and a more tightly controlled methodology, both of which are important for making good decisions about how to manage infrastructure systems and components. Providing continual and usable feedback that can be used to improve the design and operation of the infrastructure systems and components. Facilitating automation of response mechanisms. The following figure shows the basic feedback mechanism of a closed-loop control system: Figure 1. Model for a Basic Closed-Loop Control System As this high-level model shows, a closed-loop control system is dependent on comparison of an initial state to feedback, which represents the difference (delta) between the input and feedback. This delta is used to determine an appropriate control action to address a requirement. The Feedback IT Control System (FICS) is a closed-loop control system. It uses feedback from the existing infrastructure as the basis for effectively managing the infrastructure and ensuring that all control actions support the needs of the infrastructure and the business. FICS provides operational excellence principles that are essential for effectively and efficiently managing the infrastructure. FICS Architecture Essentials Critical to FICS is architecture control at its simplest, architecture control represents the actions in a closed-loop feedback system that must be performed to respond to specific feedback, including all problems or requirements that might require potential modifications to systems and components. This includes actions required for infrastructure management systems and components, such as those arising from identification of new business requirements, behavior of infrastructure components, receipt of consumer input, and response to requirements of other systems. These actions must be carried out in a systematic and controlled manner. The actions required to do this include: Page 8

9 Compare and evaluate effects relative to the desired state. Comparison of actual results to expected results provides an indicator of the successful operations of the system. The result of the comparison is either a decision about whether or not a control action is appropriate. Control action. Control actions represent the point at which a requirement or condition (such as a problem indicator) is converted into a specific action to be taken. A control action must be specific to what needs to happen and identify all potential impacts of carrying out the action. Architecture control encompasses the most complex and important elements of FICS, and it drives the type of data that needs to be provided to manage the infrastructure. Definition and implementation of architecture control elements almost always require participation of a diverse team consisting of both internal and external resources that are essential to effective comparison and analysis, as well as definition of specific control actions required for ongoing management of the infrastructure in a way that is compliant with the architecture. FICS Components and Functionality As covered in the previous sub-section, architecture control is critical to FICS and provides the basis for using feedback to determine modifications to be made to the infrastructure. However, effective comparison and evaluation, as well as implementing appropriate control actions, are dependent on other architectural elements and the functionality they provide, including the following: Business requirements. Infrastructure management architecture must be driven by the needs of the business and its priorities, as defined in business requirements. Business requirements are driven both by internal actions and conditions and by external systems and trends. Business requirements are determined by the business and technical decision makers responsible for establish strategic objectives for the enterprise and determining how best to meet them. Decision makers provide input to the current infrastructure model to indicate modifications needed in the model to address business requirements. IT requirements. Essential to the implementation of infrastructure management architecture is a standards-based approach to managing services in the infrastructure. The IT Infrastructure Library (ITIL) provides the framework for such an approach.. This helps ensure the alignment to the IT department and business users. Infrastructure systems and components. Infrastructure systems and components include servers and the services that run on them, as well as client devices, networking devices, and other components required to provide required IT-based functionality and service to the enterprise. These systems and components must be available and capable of meeting business requirements. Various factors can affect the ability of these systems and components to do what is required factors such as outside disturbances (such as security threats), usage levels, and the service life of components. Users, customers, and external support vendors. Consumers of services make demands on infrastructure systems and components and expect specific levels of support. These consumers can provide input in a variety of ways, such as calls to a service desk, requests for new support or services, or feedback on existing services. Other systems. Functionality provided by other systems, including external systems, can affect the infrastructure systems and components, both in terms of availability of services and in the functionality provided by those systems. Monitoring and notification. The state and status of the infrastructure systems and components must be constantly tracked and identified in a way that provides appropriate indicators if any system or component is not meeting specific needs or has a potential problem. Indicators can be based on business requirements, service level agreements (SLAs), consumer expectations, and other defined conditions. The purpose of monitoring and notification is to provide input that enables appropriate responses to address a specific situation responses include comparison and analysis, as well as control actions), as well as identification of significant trends: Page 9

10 Instrumentation logs and events. Sensors provide the operational information on how things are functioning that are generally recorded in a log or catalog. These can be automated status updates, errors, security alerts, end user calls, or other sources of information about the state and status of your systems. Instrumentation logs and events are source information for monitoring and notification. Aggregate and report. The volume of raw data, especially system-generated data that is available for infrastructure management makes it essential to have an effective method or culling extraneous or non-actionable items, allowing only data that is appropriate to effective management to be presented for action. Even filtered data and defined control actions can be of limited value for strategic management purposes for specific organizations and functions. Effective aggregation of data in ways that illustrate patterns, problems, and potential trends at a level that can be easily consumed by personnel and systems requiring the information is essential to enabling effective management of the infrastructure. Aggregation and reporting is closely related to monitoring and notification, but represents a more sophisticated and approach one that can be tailored to specific needs and audiences, and used is a wide variety of ways to facilitate infrastructure management. FICS Architecture The following figure shows the architectural elements that comprise FICS architecture and the flow of data through the model. External System Dependent System Outside Disturbance Desired State (Business/Technical Decision Maker) Compare & Effect Analysis (Bus/Ops) Control Action (Ops/Dev/Bus) System Customers realtime feedback state feedback Instrumentation & Monitoring Catalog (Ops/Dev) Instrumentation Logs & events Customer Feedback Publish Data Aggregate & Reporting System (Ops) Receive Data Other Systems Unactionable Events (SPAM) Figure 2. FICS Architecture Page 10

11 Control in this architecture is based on two primary types of feedback: Real-time feedback, such as might be required by ongoing tracking of the status of a system (based on logging, notification, and reporting). State feedback, such as might be required by business-based requirements or changes to other infrastructures that affect system design and functionality. Regardless of the type of feedback used in the compare and analyze action and, if appropriate, to initiate a control action, the process is basically the same: 1. A desired state is the starting point and ending point of the process. This desired state defines both the baseline for the system and any business or IT requirements related to the infrastructure. 2. Ongoing operations and forces have the potential to continually generate feedback that must be acted upon, including Requirements, such as those identified to meet strategic initiatives, as identified by business decision makers, or respond to external system requirements, such as those defined by business partners. Daily and other routine input, such as outside disturbances, customer interactions and feedback, data received from other systems, instrumentation logs and system events, and aggregation and reporting of data. 3. Comparison and effect analysis, (based on all input, including requirements and feedback), results in either: Discarding the input, if it represents an unactionable event or other input that cannot be acted upon or does not justify initiating a control action. Definition of a control action (based on the delta between the start state and the goal state). The control action can directly impact the infrastructure, as well as a dependent system. 4. Results of a control action must be fed back to the baseline, and any results that impact other organizations or systems (outside of the infrastructure) must be published to the appropriate organizations. This process can provide a consistent framework for incrementally identifying the need for and implementing control actions, which can effectively minimize any disruption to the business at the same time that it ensures that business requirements are addressed as needed. Even if FICS is implemented across multiple systems in a complex environment, it can significantly reduce the amount of time that it takes to progress from the current state of a management infrastructure to the goal state. However, using this architecture to determine control actions is only part of an effective control system as covered earlier, effective infrastructure control also requires a method of defining the complete infrastructure in a way that supports the FICS architecture (especially the compare and analysis actions) and assessing impacts of potential control actions early in the planning and deployment process. Modeling is an effective method of doing this. Modeling At the highest level, modeling requires identification of all infrastructure systems and components and their relationship to each other, as well as using a what if approach to assess the impacts of potential control actions required to manage the infrastructure, including making changes to systems and components. Modeling is a multi-discipline approach that requires: Identifying all areas that might be affected. Page 11

12 Communicating the intent and nature of the control action to affected areas. Creating plans, specifications, and test strategies that cover how impacts will be assessed. Completing the assessment and feeding results back to the initiator of the proposed control action. In addition to creation of initial models for the infrastructure, modeling requires creating individual models as needed to evaluate the need for and implement specific control actions. Modeling at this level should be done early enough in the process to enable assessment of the impacts of implementing a control action before it is actually implemented in the production environment. The compare and effect analysis assessment based on modeling should provide measurable results that can be used to make a business case for implementing the control action. Modeling can be applied to more than one scenario, effectively comparing and analyzing multiple potential solutions to determine which one is best for the infrastructure and the business. Although effective modeling can be complex and resource intensive, especially in the beginning, it can significantly reduce costs and the use of resources at the same time that it improves the manageability of the infrastructure. Creating a FICS-Based Management Infrastructure Creating and applying a FICS-based management architecture that is appropriate for the enterprise requires: Deciding where to start the implementation, which should be focused on a single system or component. Doing an initial assessment of how well the selected system or component works or doesn t work. Establishing goals for addressing the needs of the system or component. Establishing the FICS architectural principles and initial models. Applying FICS. Locking down and validating the results. Initially, creating a FICS-based management infrastructure should focus the initial definition of the right FICS processes and application of them to a specific area of the infrastructure, as covered in the following subsections, but successive iterations should result in the refinement of the processes and comprehensive streamlining and improvement of the management infrastructure. Getting Started The first step to implementing FICS is determining where to start. Effective infrastructure management should start with application of FICS to a single management component or the management of a single infrastructure system, subsystem, or component. Although the initial application of FICS may be very limited in scope, it provides the basis for understanding how feedback-based control and modeling can best be applied to other systems and components. Ideally, FICS would be applied to all systems and components in the infrastructure, not just the management systems and components. Realistically, the decision to apply it to a specific system or component should be based on the business value of doing so. Accordingly, the deciding the first system or component to which to apply FICS is generally based on one of two approaches: Apply FICS to the smallest or least complicated infrastructure management system or component, which would generally be one with a specific purpose, in a known state, with a moderate to low Page 12

13 amount of output, and with well defined management processes. This approach facilitates initial application, validation, and customization of the FICS architectural principles to align with business needs, and such a limited scope project requires minimal resources and facilitates assessment of feedback and definition of control actions. Apply FICS to the highest priority management system or component or simply represents a clear benefit to the business. If a current tool or process is critical to ensuring that infrastructure services meet the needs of the business, it can be beneficial to start with it. Although, if it is not a fairly simple system or component, it can require more resources and be more complicated to complete the application, validation, and customization efforts. Apply FICS to the most troublesome system or component. For instance, it could be a system or component that requires excessive resources to maintain or operate, does not provide the right level of support for the business, poses a liability, or has already been identified as a candidate for replacement or modification (because of known problems, outdated technologies, business restructuring, or other issues). Regardless of which approach you use, it is a good idea to keep the initial effort as limited in scope as possible to enable appropriate attention to the understanding and applying FICS architecture principles in a way that is most appropriate to your environment. Before starting, ensure that the following requirements are met: Make the appropriate teams and personnel aware of the requirements of the project and ensure their commitment to participating from beginning to completion. Even testing personnel, traditionally viewed as essential for later stages of an implementation effort, need to be involved from the very beginning. And planning personnel, traditionally participating more in initial stages, need to stay involved throughout the process in order to enable effective use of feedback. Ensure that the architecture and models are well defined and communicated. Everyone to be involved should be on board from the beginning. This should not be a problem if the requirement in the previous bullet is met, because all appropriate personnel will be involved in the definition of the architecture and models, so the result of their efforts should be comprehensive and well understood. Controls are strictly enforced to ensure no ad hoc adjustments (i.e. that no modifications are made without assessing impact and modeling the changes before actually applying them). This requires a multi-discipline approach to ensure that each team and its personnel adhere to the requirements of the architecture and do not introduce variables or perform random actions to which FICS has not been applied. Results are recorded and measurements made that clarify how the process worked (and anything that didn t work), and can be used as effective feedback for determining potential adjustments that need to be made. The results of the initial application of FICS should include not only a more effective and efficient system or component, but also a proof of concept for the FICS architecture and models. As covered later in this section, FICS should actually be applied not only to the first system or component, but also to the definition and refinement of the architecture and models themselves. So at the end of the initial effort, both the system or component and the architecture and its related models have all been defined and validated, providing a new baseline for each. Later subsections of this section cover how to use FICS and customize it for the specific needs of your enterprise, so you do not need to address that now. To start with, you only need to determine the best starting point for the effort and understand what is required to start it. Page 13

14 Assessing Systems and Components Each system and component to which FICS is applied needs to be assessed and well understood. This is not an assessment of the features and functionality that it provides, but an understanding of the results it provides and how they are used (or not used). This is required to establish the initial goals for the project, as well as to create the baseline, as covered later in this section. Assessing a system or component is basically a matter of evaluating what works and what doesn t work about the system or component. This includes everything related to the system or component including: The tool or technology The processes used to implement the tool or technology. The roles of the personnel required to manage the tool or technology, as well as those required to define the processes used to implement the tool or technology. The assessment of the infrastructure management system or component should include two key measurements: How well it addresses the needs of the business. This assessment should be stated in terms that are specific to the business and that capture what is really important to the business, as identified by business decision makers. Some of the more typical characteristics to be assessed include: Quality Throughput Availability Performance These characteristics are key determinants for the instrumentation required to manage the infrastructure to effectively meet the needs of the business, instrumentation must address all key characteristics. How well it works in terms of manageability. Although it is essential that a management system or component meet business requirements, as covered above, it is also important that it enable effective and efficient operations and management. This includes assessing easy it is to manage and operate and how effectively it supports the use and communication of data, including activities such as the analysis, filtering, and reporting of data. As much as possible, the results of the both the business assessment and the manageability assessment need to be quantified. Quantifiable measurements related to meeting business requirements are common for instance, the mean time between failures, average throughput, percent of downtime, and response times. The quantification of manageability measurements may not be as obvious, but can include the time required to perform specific tasks, how much of the functionality is automated, the percentage of items (such as events) that are actionable, or other measurements specific to the system or component. FICS cannot be effectively used without good measurements because measurements are the cornerstone for subsequent stages, including: Establishing goals that can be quantified and later determining if the goals have been met. Applying FICS architectural principles, especially the comparison and analysis activities, the definition of well defined control actions, and the assessment of the impact of implementing a control action. This includes determining potential modifications to instrumentation that might be required to manage the infrastructure in a way that meets business needs and manageability requirements. Page 14

15 Assessment can be challenging because a system or component generally consists of many functions and features, many of which may be unnecessary or unused. A starting point for the assessment is clearly defining the features and functionality that are used and how well they work. The assessment should also include features and functionality that are not used, as well the rationale for not using them. Part of the assessment may include recommendations for implementing or removing specific features and functionality (if the system or component allows it). The bottom line of the assessment should be a determination of how well the system or component does what it is supposed to do at the business level and from a manageability perspective. This not only establishes the basis for the subsequent stages, but also results in the initial awareness of what might modifications might be required in the management infrastructure. It is possible that the result of going through the assessment stage, or the next stage where goals are established, may be the determination that the system or component is not the right one, in which case the result will be identification of a need for a new system or component FICS can still be applied to the identification and implementation of a new system or component. If this is the case, measurements of the existing system or component are still required because any potential new system or component should be compared to the previous one to ensure that it provides an appropriate level of improvement. Establishing measurements is the first step toward achieving the ideal state, which is one that enables fast responses to requirements and that can provide predictable and measurable results and supports effective and proactive management of the entire infrastructure. Establishing Goals The overall goal of implementing FICS is generally to streamline infrastructure management, making it more effective and efficient. FICS is much more about process than technologies, but properly applied, FICS will result in improvements to both the infrastructure management process and the tools and technologies used to manage the infrastructure. Goals should address strategic directions, as well as immediate expectations. For instance, a long-term goal may be full automation of day-to-day operations and maintenance such a long-term goal should be included in the goal statement, but a goal for a specific modification to a system or component should quantify how it will facilitate accomplishment of the longterm goal. These goals should also reflect the underlying functionality to be provided and the expected behavior, including the input and output for the system or component. Additional goals should be specific to the system or component to which FICS is applied. The goals should be stated in terms of the business impact of the system or component, reflecting what is required for it to be effective and efficient, including objectives that support each goal. Objectives cover the measurements to be applied to determine if the goal is met. Establishing the Architecture and Models FICS architecture and the related models provide the framework for introducing the appropriate disciplines and controls into infrastructure management. Defining a FICS Architecture Figure 2 of the FICS Architecture section, earlier in this document, is a Visio diagram of a FICS architecture. You can copy this diagram and modify as appropriate to create an appropriate architecture for your environment. In general, the architecture shown there can be applied with little or not modifications. However, the architecture requires supporting information to clarify the elements of the architecture and their relationships to each other. For instance, if an external system has the potential to provide input to the desired state, document what this system is, the type of input that might arise, and the role of the external Page 15

16 system (why your infrastructure management efforts should address input from the external system). As much as possible, clarify all of the elements of the architecture so that it is comprehensive and can be clearly communicated. The architecture should not, however, identify individual systems and components used for infrastructure management the baseline model that you develop, as covered in the next subsection, will cover the infrastructure management specifics. Also, in the initial architecture, the desired state should only be a reflection of the state of infrastructure management as it is at the time the FICS architecture is defined additional requirements that might need to be addressed should not be incorporated until after the baseline model has been developed. Developing a Baseline Model Establishing a baseline for infrastructure management is a matter of creating a model that shows the functionality provided by the system or component, the tasks related to invoking the functionality, and the results of using the functionality. As covered earlier in Modeling section of this document, models consist of diverse components and can be very complex. It is recommended that your initial model be as simple as possible, focusing on key characteristics you can refine and evolve the model over time as needed to address other characteristics or improve specific functionality. The baseline model should incorporate the measurements (including specific metrics) made during the assessment stage, as covered previously. The result of each modification to the model, which is made only after application of the FICS architectural principles, is a new baseline model that is then used to assess the next change that might be required. The baseline model should always match your current management infrastructure, but should never be modified unless application of FICS architectural principles confirm that it is the modification should be made. Defining the Desired State The desired state is a model of the projected state of the system or component to be modified (or replaced). It is a combination of the baseline model, as covered in the previous subsection, with specific requirements that are being considered for implementation. The first requirements to be addressed should be based on business and manageability priorities, with additional consideration give to the ease of implementing the modification required to address the requirements. The requirements should clearly state the anticipated value of the modification to be made and any potential impacts it might have, and should include the measurements to be used to validate the results of the modification. For instance, if it is estimated that a 20% performance improvement is needed to justify making the modification, you should validate that this is achieved with the modification. Applying FICS Applying FICS is basically a matter of identifying a issue to be addressed or a potential modification to be made. This can be in response to specific input, such as a change in the desired state, the result of troubleshooting efforts, requirements from external systems, or outside disturbances. In general, the application is done for one of two reasons: To make a modification to the state of the infrastructure management system or component based on business requirements. To provide real-time response to a management issue, as part of ongoing operations and maintenance. The essential actions for using FICS to improve a specific system or component in the management infrastructure include: Comparing and assessing the input to determine if a modification (control action) is required. Page 16

17 Defining and implementing the control action, if appropriate. It is important that the control action cover not only what needs to be modified and how it will be done, but also incorporate a rollback strategy, in it is necessary to return the system or component to its previous state Feeding the results back into the model and assessing the impact of performing the control action. Completing these actions may require additional actions, such as updating the catalog, publishing results to other systems, and creating reports summarizing the change. In either case, use the FICS architecture to identify affected components and ensure that everything is appropriately applied. As covered earlier in this document, before actually implementing a control action, you should model it on paper and, as much as possible, validate the control action before implementing it in the production environment. To validate the modification, you need to determine: Did the modification to the system or component (and, if appropriate, deployment of a new system or component) work as expected? This should be evaluated in terms of the value of the business. Is the result providing the required level of improvement? This should cover the efficiency and effectiveness of the results compared to the baseline. Locking Down the Process After completing the initial implementation of FICS, validate that it works the way that it should, and make adjustments as required, including adjustments to the FICS architecture. This means capturing results of the implementation, comparing this information (feedback) to the architecture and models you defined at the beginning, and defining control actions for fine-tuning the architecture and models as appropriate (based on modeling potential impacts of any proposed control action). As covered earlier, this means ensue that: The appropriate teams and personnel are aware of project requirements and are committed to the project. The architecture and models are well defined and understood by everyone involved. Controls are strictly enforced to ensure no ad hoc adjustments (no modifications are made without assessing impact and modeling the changes before actually applying them. Results are recorded and measurements made that clarify how well the process worked (and anything that didn t work) and used as feedback to evaluate impacts and potential adjustments that need to be made. The refined process can then be used to respond to additional requirements, each of which may identify additional tweaks to the FICS architecture and models that can improve their usefulness. Use Case Cornell University Weill Medical College, Institute for Computational Biomedicine Introduction A group of a 100 users run computer models to describe and explore the properties of biological systems at Cornell University Weill College, Institute for Computational Biomedicine. To service this staff there is a computer cluster composed of 100 compute nodes with 2 processors per node. Some models are compute intensive; exploring the way proteins in cell membranes control the signaling that goes on between the complex world inside the cell and the entities outside. Other models are data intensive: post-genome Project identifying the functions of the proteins expressed by DNA. Page 17

18 The compute cluster is an essential tool to allow the staff to complete their work. How they use the compute cluster can affect the performance and efficiency of the compute nodes, yet the users had no data on the results of their actions. Providing information on how well the compute nodes are operating and how the queues are progressing would enable the staff to see how their actions affect the compute nodes. The Director, Institute for Computational Biomedicine Weill Medical College of Cornell University, Dr. Harel Weinstein set the vision to build a first-class computational resource for his staff, enabling Jason Banfelder, System Administration manager and Associate Technology Engineer, and his team to think about ways they can create a compute resource, providing a high level of service. Ideally, what the Institute needed was a method to track service levels in relation to computer operations, connecting IT operations to the users actions. How do you optimize operations of a complex IT system like a compute cluster while maintaining a high service level? Given many users were used to having their own compute resources before the shared compute cluster, users had been able to run jobs whenever they wanted, and prioritize their own work. To meet the expectation of high level of service, there is sufficient compute capacity to handle the most important jobs during the highest demand times. As part of the agreement in running the compute cluster, groups are allocated a % of the system. It is important to show the allocation is being met during the peak demand times when compute resources cannot meet total demand, and queues build up. In order to manage the IT Service it was necessary to measure the allocation, prioritize the jobs, and report on the status of the queues. To solve the job management problem, PBS Professional by Altair Engineering is used. PBS Professional is simple enough users were not required to learn the details of how the compute cluster works. PBS Professional has its own reporting, but the IT team wanted the job information combined with other operational data. This is where a management tool was needed. In searching for a tool to support IT Service Management, Jason needed something easy to use, can give information relevant to the Director, his IT staff, and users on how the compute cluster is functioning. This information can then be used to make decisions on optimizing operations. In looking for a management tool, there are process solutions like ITIL (IT Infrastructure Library) for service management, but ITIL can be a lengthy process to go through to identify the specific actions to take. The process framework from ITIL made sense, but given the size of the staff, ITIL is a lot of overhead. Jason needed end-to-end management tools, but many of these tools are developed for large IT departments, assuming a large IT budget for tools and staff. Ideally, there would be a tool which could be used in modular fashion to phase in service management. The tool Jason eventually found to meet his needs is WiredCity s IT Monitor, a tool proven to work in data intensive operations like process control data in manufacturing. A lesson learned from Edward Deming, The key is to practice continual improvement and think of manufacturing as a system, not its bits and pieces can be applied to IT thinking of IT as a system, and not the components. How many changes in an IT environment are measured after put in place? Usually people are so busy fixing problems; they are not measuring how well their fixes work, and how the change affected the total system. Jason has taken this to heart and integrated all the pieces of the compute cluster in one complete system, including a real time and historical data monitoring system to measure how well the system works compared to business and technical metrics. Using control theory and closed loop feedback as a best practice, a Feedback IT Control System (FICS) is in place to measure, report, take action, and measure deltas of real vs. expected. A closed loop system like cruise control, has a feedback controller measuring speed, report results, compare the results, and makes Page 18

19 adjustment to the throttle to maintain the speed. Most service management systems are not specifically designed for IT process control; WiredCity s IT Monitor is. Figure 3: Feedback IT Control System Diagram Connecting the business requirements, IT operations, and the users in one easy-to-understand system where each group sees how the pieces fit together. Components of Feedback IT Control system: 1) The Desired State of the system is defined by the business and technical decision makers (Harel Weinstein and Jason Banfelder) 2) Are the results of the system meeting the business and technical goals? 3) Changes are implemented in Control Actions to the compute cluster. 4) The compute cluster system is updated with changes. 5) Data is collected from the computer cluster components and users. 6) Data is aggregated into reports and filtered, providing real time feedback and state information; both of these are archived for historical trending analysis. 7) Data with no actionable events (equivalent of SPAM) are separated for historical analysis to mine for potential ways to use. 8) Unlike a car s cruise control which is sealed at the factory, the closed loop system is dynamic and needs changes to instrumentation and analysis to optimize operations. This requires service management component to track the instrumentation and monitoring changes. The monitoring solution has both real time data and historical trending analysis capabilities. The real time data is useful if you know how to interpret the # s and baselines are set. The historical data provides a Page 19

20 comprehensive way to measure change, comparing current operations to the past, and establish new baselines. Both of these data types enable the IT team to close the loop in measuring operations, creating feedback, taking action, and measuring effects of change. The closed loop systems are ideal for automation; enabling quicker decisions to be made with less man power. One of the bold efforts Jason chose to implement is to expose the issues in running a compute cluster to users. Most people would try to create rules and procedures to follow in order to improve their compute jobs. The problem with rules and procedures are they are implemented in open loops; you are told what to do and whether you follow the rules or not, you don t know the results. The effects of sharing the data to the users are more immediate and in a closed loop system, the users see the results of their action. This is in much the same spirit as Toyota s Production System where continuous process flow brings problems to the surface. With visibility of the queues and compute cluster components users were able to see when the compute cluster was busy, how the components were working. In the diagram below is a complete view of the queues, server room temperature, compute cluster nodes, file servers, web, and backup functions. An interesting discovery early on was how much traffic would hit their web servers at seemingly random times. Concerned about security attacks, the team researched the history of web traffic, and discovered the spikes which lasted 1 ½ days were caused by internal web crawlers which indexed the web server when a new wiki was created. Other results from implementing IT service management using FICS are: 1) Job Processing, Compute Nodes, storage, network, web, and back up are all measured in one system. 2) Users have visibility of the queues for all users and how their particular jobs are progressing. Page 20