I D C E X E C U T I V E B R I E F Workload Automation Challenges and Opportunities May 2011 Sponsored by BMC Executive Summary Enterprise IT workload environments are becoming more complex, dynamic, and unpredictable. Traditional static batch processing workloads are being run side by side with dynamic businesscritical composite applications and process flows that can have highly variable resource consumption requirements and peak hours. Organizations are also aggressively implementing private and hybrid cloud strategies that rely heavily on virtualization, self-serve workload management, and automated orchestration technologies. This Executive Brief discusses IDC's perspective on how enterprise workload management requirements are changing and highlights the ways that workload automation solutions can address these emerging requirements. Forces Driving Increased Workload Complexity IT organizations are tasked with ensuring that mission-critical business workloads are available, reliable, cost-effective, and secure. Achieving these objectives can be challenging in highly controlled traditional IT environments where processing windows and computing resource requirements are predictable and change occurs slowly. In today's highly complex and dynamic datacenters, it can be almost impossible unless IT organizations make extensive use of policy-driven automated workload management systems. The major forces shaping the increasingly unpredictable behavior of today's enterprise workload environments include:! Increasing heterogeneity of hardware and software environments! Increasing variability and unpredictability of workload processing peak hours and volumes! Growing use of self-service and cloud architectures shifting workloads from centralized schedules to user-driven schedules! Ongoing compliance, data protection, and security concerns Most enterprise-class IT environments incorporate a wide range of systems, storage, and network hardware; operating systems; and hypervisors, as well as an array of applications based on a wide range of architectures including legacy mainframe applications and modern applications written using J2EE,.NET, SOA, and emerging "built for cloud" platforms. Each application has its own specific system, database, and middleware configuration and processing requirements. Multiple applications may be aggregated in composite applications and business services, which we refer to here as workloads. IDC 1121
Beyond the physical and virtual heterogeneity and diverse application architectures found in many organizations, workload peak hours and processing requirements are becoming less predictable and more variable over time. In traditional environments, many business-critical workloads were executed at predictable times and consumed predictable amounts of resources. IT invested significant resources in optimizing the performance and resource consumption of these processes. The longlasting success of calendar-driven job scheduling software tools demonstrates the value associated with optimizing workload processing in predictable environments. While some workloads such as payroll continue to be tightly scheduled, many others are becoming more event or user driven. Distributed, Web-based systems allow many more processes and workloads to be activated on demand, based on user-initiated events that are outside the control of IT organizations. In addition, the increasing availability of self-service portals for provisioning and orchestrating IT resources and business processes across public, private, and hybrid cloud environments creates even more uncertainty and results in more variability across workload environments. Self-Service and Cloud Drive Dynamic Workload Environments IDC's research indicates that the impact of self-service provisioning will increase in the coming years. A recent North American survey of 250 IT decision makers shows that more than half (54%) expect to implement self-serve provisioning technologies in production workload environments by 2013. Self-service is an important part of cloud computing strategies because it enables business users and developers to more quickly access IT resources but it also leads to more dynamic and less predictable workload processing requirements. By 2013, 61% of North American IT organizations with more than 500 employees expect at least some of their production workloads will be supported by private cloud environments that make use of virtualization, self-serve provisioning, end-to-end performance monitoring, and policy-driven workload automation (see Figure 1). Figure 1 Expected Use of Private Cloud by 2013 Planning stages/in lab (16.0%) Not in use (22.8%) In production (61.2%) n = 250 Base = North American IT decision makers from organizations with 500+ employees Source: IDC's Private Cloud Management Survey, 4Q10 2 2011 IDC
By definition, cloud architectures are designed to share computing resources dynamically across multiple workloads and user groups. As more and more mission-critical production workloads move into these types of environment, IT organizations not only have to make sure that computing resources are available to support the workloads when needed but also have to ensure that workloads perform as expected. IT is also tasked with enforcing corporate and regulatory requirements about information and data protection, transborder data portability, and compliance and change monitoring. In addition, IT needs to address all these concerns simultaneously, even as workloads continue to increase in complexity, peak hours become less predictable, and business stakeholders demand more and more business agility, cost transparency, and operational flexibility. Strategies for Using Workload Automation to Address These Challenges Given the rate and pace of change and the increasing level of interdependencies across today's complex workload environments, most IT organizations can no longer keep up with business requirements using manual processes and static, calendar-driven job scheduling tools. Rather, IT organizations need integrated processes, policies, and automated software solutions to help optimize workload performance, operational costs, and resource consumption in real time while ensuring security and compliance as required. IDC's research indicates that to effectively automate and orchestrate the mapping of workloads to resources, IT organizations need to integrate processes, policies, and tools that were formerly separate disciplines. Specifically:! Job scheduling! Automated server and application provisioning and patching! Workload orchestration/migration! Virtualization management! Runbook automation To optimize the way workloads consume dynamic, virtualized, and pooled infrastructure resources going forward, IT teams will need to deploy integrated infrastructure and workload automation tools that can:! Support traditional batch- and calendar-driven job scheduling workflows and today's dynamic workload management requirements using a single, common interface and policy engine! Proactively monitor and forecast workload resource consumption and provision/deprovision resources and migrate workloads as required to maintain SLAs! Automatically monitor configurations and compliance with gold images and patch workloads to maintain consistency with corporate standards and compliance requirements on an ongoing basis! Balance the needs of calendar-based job scheduling, application patching, and event-driven workloads by automatically coordinating processing and patching windows and conflicts! Dynamically allocate workloads to the appropriate physical and virtual as well as public or private cloud computing resources, based on business service management policies regarding SLAs, cost, and security! Integrate smoothly with end user facing service catalogs and self-serve portals to support faster, more seamless user-driven workload activation and processing 2011 IDC 3
IDC expects that most enterprise IT organizations will continue to need to optimize workload performance, costs, and resource consumption across heterogeneous public and private infrastructure and application resources for the long term. Increasingly, workload automation solutions will become part of the fabric of complex cloud datacenter environments. Benefits of Workload Automation In the past, many enterprise organizations have relied on multiple, separate tools and custom integration via scripts, or even manual intervention, to tie together workload and infrastructure automation. This approach can often be slow to implement and costly to maintain. Integrated platforms that can orchestrate, monitor, and optimize workloads using a consistent set of interfaces and a common policy engine enable IT organizations to control costs and deliver consistent service levels while ensuring compliance with corporate and regulatory information management policies (see Figure 2). Figure 2 Integrated, Automated Workload Management Environment IT Administrator Console Self-Serve Portal & Service Catalog Workload Automation Policy Engine Traditional calendardriven job scheduling Dynamic workload allocation, patching, and migration Infrastructure provisioning and patch Public cloud resources Source: IDC, 2011 This policy-driven approach to workload management is required to protect the business from inappropriate information disclosure or transborder data violations. By setting workload SLAs and monitoring actual performance and availability across peak hours and unexpected demand spikes, IT organizations can proactively detect changes to usage patterns and anticipate the need to spin up additional internal resources or burst workloads onto public clouds. 4 2011 IDC
A more proactive, automated, and integrated workload management environment enables IT organizations to work more efficiently, improve end-to-end business service availability and performance, and optimize both internal capex and external public cloud spending. It also ensures compliance with business rules about data protection, thereby reducing business risk and improving business/it alignment. Conclusion Integrated, automated workload management solutions will be critical to maintaining service levels and controlling datacenter and IT costs in the future as application, infrastructure, and cloud environments become more complex and workload requirements become less and less predictable. Workload automation software has evolved considerably from the days of calendar-driven job scheduling solutions. Today, when combined with a commitment to standardized configurations, SLAs, and security policies, workload automation solutions can drive major improvements in the ability of IT organizations to respond to business priorities while keeping complex workload environments up and running in a cost-effective and reliable manner. This capability provides direct benefits to both IT and the business by enabling IT organizations to respond to business requests more quickly and enabling them to execute change rapidly and with confidence. A B O U T T H I S P U B L I C A T I O N This publication was produced by IDC Go-to-Market Services. The opinion, analysis, and research results presented herein are drawn from more detailed research and analysis independently conducted and published by IDC, unless specific vendor sponsorship is noted. IDC Go-to-Market Services makes IDC content available in a wide range of formats for distribution by various companies. A license to distribute IDC content does not imply endorsement of or opinion about the licensee. C O P Y R I G H T A N D R E S T R I C T I O N S Any IDC information or reference to IDC that is to be used in advertising, press releases, or promotional materials requires prior written approval from IDC. For permission requests, contact the GMS information line at 508-988-7610 or gms@idc.com. Translation and/or localization of this document requires an additional license from IDC. For more information on IDC, visit www.idc.com. For more information on IDC GMS, visit www.idc.com/gms. Global Headquarters: 5 Speen Street Framingham, MA 01701 USA P.508.872.8200 F.508.935.4015 www.idc.com 2011 IDC 5