The power of automation: how HP Enterprise Services (formerly EDS) is achieving transformational ROI Table of contents A nightmare of IT complexity.................................. 2 HP Server Automation: UNIX sendmail and the patch from hell........................................... 3 Improved provisioning: going for the gold image standard.............. 3 Application release management: an army of two.................... 4 HP Network Automation: saving time and effort with improved visibility.............................. 4 An automatic advantage: tangible benefits and the numbers to back them up.............................. 5 Moving forward: ongoing momentum for additional benefits............ 6 Getting started: lessons from the EDS experience..................... 6 How HP can help........................................... 8 Find out more............................................. 8
Challenges led the organisation to consider automation as a way to ensure business efficiency, drive down costs, and deliver greater value to its customer base. A nightmare of IT complexity HP Enterprise Services is big. Formerly known as EDS, the IT service provider operates the second largest commercial network in the world. Founded in 1962 and recently acquired by HP, the group virtually invented the IT outsourcing industry and has managed to stay ahead of the competitive curve ever since. Today it uses automation to help manage 29 data centres around the world, 233 satellite locations, more than 68,000 servers, and 35,000 network devices with plans to grow to more than 250,000. It also has more than 5,000 employees using automation to execute nearly 900,000 server management jobs a month on behalf of more than 900 clients. These include some of the world s most well-known organisations such as Rolls- Royce, General Motors, Bank of America, Kraft, and the United States Navy. All of this makes HP Enterprise Services the most prolific user of automation on the planet. To manage this volume effectively, HP Enterprise Services uses HP Server Automation software for server management, HP Network Automation software for managing the network, and HP Operations Orchestration software to automate manual processes. But it wasn t always this way. Long before it was acquired by HP, EDS faced a number of daunting challenges that stemmed from the sheer size and complexity of the globally dispersed IT environment it manages for itself and its customers. One challenge was a near crippling lack of control and consistency regarding its daily IT processes. Each team within its various data centres around the world had its own way of executing its responsibilities. Among other things, this made patch management a virtual nightmare. Teams would download different patch bundles for the same software issue and apply them in an inconsistent manner following manual, ad hoc change processes. Even though these processes were repeatable, each data centre operated as independent silos within the larger organisation. On a case-by-case basis, system administrators developed scripts and other local automation tools to do their jobs more effectively but this approach lacked the global standardisation that would have benefited the organisation as a whole. The complexity and lack of standardisation for the EDS environment also led to extremely costly compliancerelated processes. For every audit, teams of highly paid IT experts would have to spend hours compiling documentation to meet audit requirements. This often required teams to manually pull data from assets across all of its 29 data centres around the world data that would need to be compiled at the global level by upper level management. The effort associated with compliance took these managers and IT experts away from the core mission of managing their client environments in the most effective manner possible. One of the main activities for executing on this mission included server provisioning a common but also extremely important task for any IT service provider. But here, too, the experience for IT experts within the organisation was extremely taxing. Lacking a single organisation-wide gold standard for operating system images, teams would provision new servers with inconsistent configurations. This caused tremendous conflicts from a landscape perspective resulting in arduous processes when it came to identifying the root cause of problems and applying fixes. 2
These challenges led the organisation to consider automation as a way to ensure business efficiency, drive down costs, and deliver greater value to its customer base. Rather than taking a big-bang approach, EDS approached automation on an incremental basis focusing on its most prominent pain points first and building its capabilities over time. HP Server Automation: UNIX sendmail and the patch from hell For any large IT organisation, applying patches to servers over a large, distributed IT infrastructure is a time-consuming, error-prone task. This was no different for EDS. One story in particular helps illustrate this point. A number of years ago, IT managers at EDS identified a security vulnerability for its UNIX servers regarding the sendmail functionality. Discovered right before the Christmas holiday, the issue needed to be addressed because it posed a serious threat to EDS and its customers. The upshot was that EDS needed to identify all of the vulnerable servers around the globe. On a Friday afternoon, IT managers started the process by waking up other managers on the other side of the world and asking them to go into the office to run a series of commands on each server in their data centre. Each manager then needed to tabulate the results, populate a spreadsheet, and send it back to headquarters where that data was consolidated and sent back up the chain of command. All in all, this process took about six weeks simply to identify the vulnerable servers. Later, EDS developed a fix for the server vulnerability and deployed it around the globe for all impacted servers and geographies. This required further effort, working with multiple teams in multiple languages around the globe and coordinating activities so that the patch was applied in a consistent manner. As a result of experiences such as this, EDS decided to implement HP Server Automation software. Since then, a similar crisis occurred that dramatically illustrates the advantages of automation. In this instance, the organisation identified another vulnerability that required the team IT to again identify all affected servers around the globe. But this time around, it took a single IT expert with no prior experience using HP Server Automation just one hour to run a script across all data centres, identify all vulnerable machines, and report back to upper level management. By the time this task was completed, other IT staff had downloaded the patch and bundled it for deployment across all affected servers. In all, the process took less than eight hours. Improved provisioning: going for the gold image standard When it comes to time-consuming IT tasks, few jobs rank with provisioning a new server. In the past, EDS lacked a single enterprise-wide gold image to standardise the way in which it provisioned a new server. Every data centre was on its own. Typically, IT experts would install the base operating system from CDs and DVDs organised into cabinets a lengthy, hands-on, one-by-one process. At one point, an important customer in the travel industry asked EDS to provision 350 new Linux servers. This took 18 system administrators a full three weeks to complete. Today, the organisation uses HP Server Automation software to manage the server provisioning process. The productivity and cost reduction improvements have been tremendous. Now it takes just two IT staff members a day and a half to provision 350 servers. What s more, through automated, repeatable processes, EDS has been able to introduce consistency throughout the organisation. While the organisation will still customise server configurations to meet specific customer needs, a single base operating system serves as the gold image standard across all IT operations. In addition to reducing cost through lower overhead, this has also helped to improve compliance while dramatically increasing efficiency. 3
Application release management: an army of two One EDS customer in the automobile services industry used a small army of IT experts to make daily changes to a business-critical application with limited controls at best. Changes were often made less than five minutes prior to deployment. Oftentimes these changes were incomplete meaning that the same application that worked in the test environment broke in production. Because of the speed at which the teams had to work, they often left no instructions regarding what changes were made. This helped introduce errors into the system on a regular basis. With no instructions available to document the change, response teams took an exorbitant amount of time trying to fix the problem. By introducing HP Server Automation software and HP Operations Orchestration software into the customer s environment, EDS was able to achieve dramatic results. For example, the customer reduced its application management staff from a small army numbering in the mid-teens to a team of just two. This alone has resulted in more than one million dollars U.S. in annual savings. Equally important from a business risk perspective, EDS was able to enforce discipline and consistency at the company. Today, internal IT staff members are required to test earlier and follow policies about when they can make changes. Application changes are released as a batch on a consistent basis every Friday afternoon. All instructions are included and associated with the change as part of a predetermined process workflow. This allows the company to make successful changes almost 100 per cent of time. HP Network Automation: saving time and effort with improved visibility EDS manages one of the most complex networks on the planet. This is partially due to the fact that the network is not at all flat it integrates with the networks managed by a long list of customers. This poses extreme challenges for the network management team at EDS many of which have been addressed by HP Network Automation software. Network auditing provides a good example. In the past, an audit would take nearly seven weeks to complete with IT experts gathering information and generating reports to show network compliance. Today, the team runs a script from a central location to gather the same information. This takes less than one hour. Change management at the network level is vastly improved as well. Before automation, the EDS network team had to follow lengthy manual processes to provide pre- and postchange information to clients to document changes and verify success. Today, programs automatically check router and switch status before and after changes and the results are automatically sent to the client. This saves approximately 45 minutes per change. With more than 1,000 changes per month for some clients, this saves EDS more than 750 hours on a monthly basis. HP Network Automation also pays dividends by simply increasing network visibility for overworked network administrators. In the past, it took an administrator an inordinate amount of time to track down information such as port types and MAC addresses for various network devices. In addition, administrators would spend considerable time searching for devices that failed snapshots. This was an important risk mitigation task because a failed snapshot often means that unauthorised changes have been made to the configuration settings of a network device, potentially resulting in an expensive outage. For failed devices, the administrator would then have to ping the device and run Simple Network Management Protocol (SNMP) and telnet checks to perform triage duties for the network operations centre. Today, this same port and failed device information is gathered automatically. This replaces hours and hours of manual effort with reports that can be generated in just minutes. 4
An automatic advantage: tangible benefits and the numbers to back them up Automation at EDS has literally transformed the way the organisation carries out its business and today, as HP Enterprise Services, the organisation sees automation as a strategic asset that enables it to outstrip the competition. Benefits include: Improved consistency: Automation enforces standards across global operations and enables HP Enterprise Services to minimise landscape complexity, reduce the risk of conflicts, and improve productivity. Improved service levels and uptime: Automation minimises the potential for errors that come with manual processes and enables HP Enterprise Services to respond to vulnerabilities with greater speed, helping to enable higher service levels and improved uptime. Improved mean time to repair: Automation improves visibility and consistency across the global landscape, helping HP Enterprise Services to respond to critical performance problems with greater speed, accuracy, and consistency. Improved productivity: Automation frees up critical IT experts to focus on higher level strategic concerns and work uninterrupted by menial tasks that are now managed by tools that do the job faster and more effectively. For HP Enterprise Services, automation speaks directly to the bottom line as well. As in any business, time is money in the IT services industry. The table below shows how the organisation has been able to dramatically reduce time spent on otherwise manual IT tasks. This has helped to save money and drive down IT spend across all of its global data centres. IT process Changing passwords: 200 servers every 30 days Oracle deployment: 5 thousand times per year Patching: 1,800 UNIX systems IT staff productivity: server to admin ratio Before HP Automation After HP Automation 13 hours 30 seconds 16 hours 6 hours 400 hours 2 hours 30 to 1 200 to 1 5
Moving forward: ongoing momentum for additional benefits In the IT services industry, the competition is always close behind and HP Enterprise Services has no plans to rest easy with the benefits EDS has gained. Already, the organisation has plans to build off its automation foundation to improve ticketing and incident management. This will involve using HP Operations Orchestration to integrate more than 50 ticketing systems used globally. Plans include creating a globally consistent process flow for creating incident and problem tickets and managing the associated change through automated means. This will help alleviate the otherwise complex and time-consuming tasks of coordinating with different groups throughout the process which inevitably leads to time delays and greater potential for errors being made. HP Enterprise Services also wants to expand the use of automation to new teams and groups that have emerged through the acquisition of EDS. Most importantly for the strategic direction of the company, top-level management at HP Enterprise Services now fully realises the powerful benefits of automation and has committed to continuously consider new tools and approaches to automation that will help the organisation compete more effectively on the global stage. Getting started: lessons from the EDS experience For EDS, automation didn t happen overnight. In fact, it started with an initial recognition that without automation to help out in a specific area of the business, the organisation would face more problems than it could reasonably handle. Along the way, the organisation has learned some important lessons. These include the following: Identify your biggest pain point first. It only makes sense to focus on the low hanging fruit that will pay off the most in the shortest amount of time. For EDS, the starting point focused on patching. To address its patching challenges, the organisation chose HP Server Automation. From there, EDS expanded its use of automation to address audit and compliance, provisioning, configuration management, and application release management. Don t boil the ocean. Avoid the temptation to implement too much at once in the hope that you ll be able to realise outsized benefits right away. A phased approach that demonstrates ROI for each project helps to generate buy-in throughout the organisation over time without causing undue disruptions to the business. 6
Figure 1. Service-centric, integrated automation suite HP Operations Orchestration HP Client Automation HP Network Automation HP Server Automation HP Storage Essentials BSM Monitoring events and alerts HP BSA Essentials HP Service Automation Visualizer HP Service Automation Reporter HP Community HP Subscription Services CMDB ITSM Help desk/ change request Show, don t tell. Creating a culture of automation within an organisation takes time, but the pay-offs can be tremendous. It is well worth the while to take the time to communicate the power of automation through demos of planned or live implementations and to take seriously the job of evangelising the benefits of automation throughout the organisation. Build an automation team. EDS built a formal team to take responsibility for automation throughout the organisation. This team included governance at the technical and management levels to ensure process compliance. The team also included an infrastructure group to design and deploy products, an architecture group to create process and product improvements, a content group to create and manage all automation material used throughout the organisation, and a support group to provide end-user and product management support. Many organisations may choose to implement a full-blown automation centre of excellence (ACE) that centralises all automation activities according to a shared service model and empowers the ACE to take full responsibility for automation at the enterprise level. Bring in other organisations over time. Over years of experience with automation, EDS realised that it made little sense to keep automation responsibilities with system administrators alone. Realising the strategic importance of automation, upper level executives made the decision to bring in a wide range of IT domain groups, including teams that managed storage, backup and restore, system monitoring, security, network administration, hardware, databases, and more. The advantage has been standardisation and consistency across the global team. Thus, when one team develops an automated process flow for managing a specific task, it can share the same flow with another team across the globe. This helps to spread the benefits of automation throughout the enterprise. Choose a vendor that allows you to grow. While a phased approach focused on concrete problems is the most advisable path to take when it comes to automation projects, organisations should also keep an eye on the future. If the immediate project under consideration pays off as anticipated, will you want to build off this success or extend it to a wider user base? If so, consider an automation vendor that offers a flexible yet comprehensive suite of automation software capable of scaling to your needs as your business evolves and grows. 7
How HP can help As the world s leading provider of automation software, HP can help your organisation increase efficiency, cut costs, and realise your automation objectives. HP Business Service Automation includes a wide range of service-centric, integrated software and tools that can be implemented in phases to help you move along the automation maturity path while delivering significant ROI each step of the way. This enables you to: Find out more. To find out more about how HP can help your business perform more effectively and efficiently through automation, call your HP representative today or visit us online at www.hp.com/go/bsa. Create a common view of business services across the IT environment Automate change across all of the devices making up the business service for cost efficiency Connect IT processes and coordinate siloed teams through a common workflow Integrate with monitoring and ticketing tools for a holistic service management solution Get connected www.hp.com/go/getconnected Get the insider view on tech trends, alerts and HP solutions for better business outcomes Technology for better business outcomes To learn more, visit www.hp.com/go/bsa Copyright 2010 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein. UNIX is a registered trademark of The Open Group. Linux is a U.S. trademark of Linus Torvalds. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. 4AA1-6721EEW, January 2010