WHITE PAPER The Evolution of Job Scheduling: CA's Approach to Workload Automation Sponsored by: CA Tim Grieser November 2007 IN THIS WHITE PAPER Global Headquarters: 5 Speen Street Framingham, MA 01701 USA P.508.872.8200 F.508.935.4015 www.idc.com This IDC White Paper presents an overview of key concerns driving IT management and how they translate into needs that can be addressed through modern approaches to job scheduling and workload automation. The paper traces evolutionary trends in job scheduling software, which is growing from traditional batch job schedulers to cross-platform and event-driven capabilities. It also explores the role of job schedulers in providing automation capabilities. The White Paper examines CA's workload automation strategy, which is based on the evolutionary development of CA's portfolio of job scheduling and management software tools into a comprehensive workload automation solution. The objective of CA Workload Automation is to help IT organizations optimize real-time performance of complex business workloads across diverse IT infrastructures, including both mainframes and distributed systems. Key capabilities include dynamic event-driven control of production workloads based on event triggers and defined policies to achieve IT and business service objectives. SITUATION OVERVIEW Key Concerns Driving IT Management IDC tracks key IT concerns on an ongoing basis through a combination of structured surveys, end-user interviews, case studies, and other in-depth research. The following key concerns were recently expressed by IT managers and professional staff:! Pressures to contain costs and achieve efficiencies. Controlling IT costs continues to be a key requirement for IT organizations. IT budgets have been under pressure for a number of years, with cost containment a principal objective. Operational costs are significantly related to the number of IT staff required to perform specific functions. Automation is key to containing or reducing operational costs.! Need to align with the business and show business relevance. Increasingly, IT is viewed as a service provider to the business, providing and managing services that deliver and support key business processes and applications. Alignment includes such activities as relating infrastructure events to business service and end-user impact.
! Need to achieve service objectives for key workloads. Service objectives for key workloads must be achieved. Operational service levels such as transaction volumes, throughputs, and completion times must be compared with service objectives, with provisions for adjusting workload priorities and resource allocations to meet business requirements.! Need to improve infrastructure utilization. IT is under pressure to improve the efficiency and ROI of the IT hardware and software assets deployed in the datacenter and throughout the organization. A critical responsibility is improving server utilization for mainframes as well as for distributed Unix, Linux, and Windows systems used for such functions as ERP applications, databases, transaction processing, and middleware.! Need for workflow automation and process standardization. While automation continues to be a key strategy for improving IT efficiency, it is increasingly viewed as a means to ensure that required workflows and processes are executed correctly, according to standards set by the IT organization. Indeed, ensuring that standardized processes are executed correctly is a primary condition for satisfying a number of regulatory compliance requirements. Evolving Requirements for Job and Workload Scheduling Fundamentally, job scheduling software is used to manage the flow of work on systems. This category includes batch job schedulers and workload balancing software working at the system and application levels. Traditional job scheduling is based on managing the execution sequence of a set of batch jobs. Job execution sequences can be based on a number of factors, including time-of-day and calendardriven requirements, resource availability, external priorities, and completion of other jobs and other types of "triggers." Enterprise Job Scheduling Many business processes are based on the completion of complex job sequences that are controlled by job scheduling and workload management software. Enterprise job scheduling brings automation and centralized control to the execution of the steps needed to execute business workloads and processes. Enterprise job scheduling helps to support key IT and business priorities, including efficient operations, cost savings, delivery of required service levels, application integration, and communication of status to IT operations staff and users. System-Level and Application-Level Scheduling System-level job scheduling generally invokes applications and does not include detailed knowledge of application-specific details. Many enterprise applications include heavy batch processing requirements and contain an internal job scheduler. Examples include SAP, PeopleSoft, and Oracle E-Business Suite. An external enterprise job scheduler can support wider levels of application automation than an internal scheduler alone. This allows integration of other custom or packaged applications with the enterprise application. 2 #209302 2007 IDC
Multiple Platform Support Job scheduling was initially designed to process workloads running in mainframe environments. Classic examples include the use of job schedulers to schedule the execution of batches of financial transactions such as those accumulated during the day to be processed during an overnight or third shift "batch window." Today, job schedulers also exist for distributed platforms, especially Unix and Windows. Workloads can include jobs that run on multiple platforms, requiring cross-platform job scheduling. These kinds of jobs must be scheduled in a coordinated fashion, such as using "triggers" that signal when a job on one platform has completed and a job on another platform can be started. Integrated job scheduling software can allow a single group of IT staff to manage the schedules across platforms without requiring specialized platform or application skills. Event-Driven Job Scheduling Traditional job schedulers are driven by time-of-day and calendar processing requirements. The large growth in online and distributed applications has raised the need for event-driven scheduling, which can initiate a job in response to an external event, such as the arrival of a financial transaction from an end user, over the Web. Event-driven job scheduling extends automation to real-time and online applications, driven by the competitive need for Web-based access to IT services. Automating Job Scheduling Processes IT departments are employing a variety of tools and methods to automate the operational management of jobs and workloads. The need for real-time access and operations, the complex infrastructure environments, and the need to combine job scheduling with other IT management functions have led to a patchwork of point-topoint connections and specific product integrations, often achieved by the use of hard-coded scripts. However, such techniques are typically tactical in nature and often lack a higher-level focus on workflows, IT processes, and business services. Workloads A workload may be defined as a related set of computer-based jobs or tasks that need to be executed on an IT facility. Workloads are often characterized by the type of application or utility function being performed, such as online financial transaction updates, ERP applications, or file backups. Traditionally, workloads have been managed from an IT operational perspective in terms of computer resource requirements and required service deadlines, such as calendar-based date and time or specific IT-related events, as reflected in traditional job scheduling software. This focus stems from a classic job-shop view of IT processing where trade-offs between IT resources, processing efficiency, and required deadlines can be made to optimize workload processing. 2007 IDC #209302 3
Workflows A workflow is a larger, procedural view of work that includes a sequence of steps that must be performed to accomplish a business or IT process. Workflows include the idea that each step involves rules or policies, tasks, and data that must be processed and passed on to the next step. Workflows include both manual tasks and computerbased processing steps needed to accomplish defined business or IT processes. Managing and Automating Processes Job and workload scheduling software tools have evolved to the point where they can typically be used to automate calendar- and event-driven operations. However, merging such functions into higher-level process-focused management and automation requires additional software and integration with other management tools One approach is to use scripts to define specific processes and integrations a "hard-coded" approach that can work in specific cases but lacks generality and flexibility. Another approach is to integrate job scheduling software tools with higherlevel control software, such as workflow engines and orchestrators. CA WORKLOAD AUTOMATION CA Workload Automation is CA's approach to developing the next generation of workload management solutions. It represents an evolution from classic job and task scheduling to consolidated job management of pooled resources and to dynamic, automated, policy-driven on-demand workload management supporting business service objectives. CA Workload Automation is based on extending and integrating capabilities contained in the current CA product portfolio. The CA portfolio includes CA-7, AutoSys, Workload Control Center (WCC), as well as ESP and dseries from the Cybermation acquisition. Figure 1 shows the CA Workload Automation road map. 4 #209302 2007 IDC
FIGURE 1 CA Workload Automation Road Map Phase 1: Foundation Workload Automation Capabilities/Characteristics Phase 1 Phase 2 Phase 3 Phase 4 Phase 5 Phase 2: Product Characteristics Phase 3: ESP Integrations Phase 4: Single WA Engines MF/Dist Phase 5: Full WA Capabilities Web Services Enablement Dynamic WP J2EE Job Types Web Service Job Simplified Types Management Dynamic CPM Role Based Process Orientation AutoSys r11 ESP 5.5 WCC r11 CA-7 r11 UEJM r1 AutoSys 4.5 Cybermation Acquisition Job Scheduling Job Management Extended Web Services Enablement Advanced Dynamic WP Resource Optimization Upgrade Paths ESP Agents UWCC/ESP CA-7 r11.1/11.3 AutoSys r11.3 ESP 11.3 WCC r11.3 dseries 11.3 SLA-Based Automation Workflow Management Policy Management Automated Monitoring Mainframe WA Engine r12 Distributed WA Engine r12 CA-7 r11.5 AutoSys r11.5 ESP 11.5 WCC r11.5 dseries 11.5 Grid Enabled Common Scheduling Language Workload Service CA Workload Automation Source: CA, 2007 CA's road map projects the path and time line for developing the full CA Workload Automation solution. Key product phases yet to be delivered include Phase 3: CA/Cybermation integrations; Phase 4: single Workload Automation engines for mainframe and distributed systems (one for each); and Phase 5: the full Workload Automation solution. CA Workload Automation: Solution Characteristics The full CA Workload Automation solution is being built to deliver broad-based multiplatform facilities for dynamic service management, event- and policy-driven workload scheduling, resource allocation, automation, and business process optimization. CA Workload Automation is designed to provide real-time or "on-demand" responses to events, in accordance with business priorities or other policy-based service requirements. CA Workload Automation will incorporate CA's "service-oriented automation" approach to connect infrastructure components and events to the execution of business and IT processes. Key characteristics of the CA Workload Automation solution are noted in the following sections. 2007 IDC #209302 5
Simplified Workload Automation CA's workload automation strategy is based on simplifying the work needed to implement automation by integrating CA job scheduling software with software for higher-level management and control functions. The software functions in this strategy include workflow engine, business rules, IT event and calendar, as well as IT and business workflow integration. Service-Oriented Automation One of the major foundations of CA's workload automation strategy is Web Services enablement of key components. This permits exposing workload management as a service, as well as providing a strong basis for integration of both CA and third-party scheduling and automation software, through Web Services interfaces. Serviceoriented automation also enables a number of capability extensions, including an increased number of job types and a wider range of business events. Workload Resource Optimization Another key aspect of CA's workload automation strategy is optimization of IT infrastructure resource allocation and utilization. This includes workload placement to achieve resource utilization objectives, dynamic monitoring of resource usage, and exploitation of virtualized workload resources such as virtualized infrastructure pools. Dynamic Workload Orchestration Higher-level control is to be implemented through dynamic orchestration, based on automated real-time monitoring, expanded event sensing, and dynamic critical path analysis. This is key to providing a dynamic "on-demand" or "sense-and-respond" capability so that business service objectives can be achieved, with the capability to make adjustments to workload resource assignments under varying operational conditions. Key benefits expected from implementing CA Workload Automation solutions include increased IT efficiency and cost savings through automation, centralized workload management and control, improved infrastructure utilization, and improved ability to meet IT and business service objectives. CA Workload Automation: Solution Components Today, CA Workload Automation encompasses five major solution components, as follows. CA Workload Control Center The CA Workload Control Center is a portal-based management application and graphical user interface that serves as a central point of reporting and control for CA mainframe and distributed workload automation software. WCC integrates the scheduling engines and agent technology derived from CA automation products, including AutoSys, CA-7, and ESP, to deliver cross-platform and distributed scheduling capabilities. WCC connects to subcomponents using Web Services interfaces. 6 #209302 2007 IDC
WCC supports extensive monitoring facilities that can be used to track and identify problems in business processes. It includes graphical flow views, portlets for command line management, a 3270 emulator, and executive dashboards. WCC implements security functions including role-based control of access to WCC functions and workload automation features. Figure 2 shows examples of WCC displays. FIGURE 2 CA Workload Control Center Source: CA, 2007 2007 IDC #209302 7
Enterprise Reporting The current release of CA Workload Automation includes enterprise reporting enabled through OEM technology from Business Objects. Enterprise reporting provides hundreds of predefined reports for Workload Automation; facilities for report customization; and an embedded report server from Business Objects. Figure 3 shows an example of CA Workload Automation enterprise reporting. FIGURE 3 Enterprise Reporting Source: CA, 2007 8 #209302 2007 IDC
CA Workflow CA Workflow is a workflow engine software component, a part of CA's common technology product portfolio. The CA Workflow engine is embedded as part of the most recent WCC product distribution. CA Workflow provides facilities for automation of IT processes and procedures, including process orchestration. CA Workflow supports form-based automation of manual inputs as well as the ability to integrate with Web Services, J2EE, and command line based services to name a few. Individual workflows can be accessed through Web Services interfaces. Workflows can access a range of job managers to invoke job executions, and job managers can access workflows through Web Services interfaces. Selected third-party workflow engines are also supported by WCC. Figure 4 shows an example of CA Workflow use in WCC. FIGURE 4 CA Workflow Source: CA, 2007 This example shows how a number of predefined processes built with CA Workflow can be initiated from WCC by an operator. 2007 IDC #209302 9
Web Services API Web Services enablement provides the common interface for integrating CA Workload Automation solution components. It also facilitates integration with third-party components such as application-specific workflow engines. CA Workload Automation uses Web Services to expose product capabilities as services to other applications. CA has implemented a "Web Service job type" that provides the ability to integrate Web Service exposed application logic into a job stream as a job. CA MDB The embedded CA management database (MDB) enables monitoring and data-level integration with other CA Enterprise IT Management (EITM) software solutions. The MDB database can be used to discover and pool resources available for executing workloads. CA Workload Automation can pass information to the MDB for use by other CA products such as products for reporting and service-level management. CHALLENGES/OPPORTUNITIES Software for job scheduling and workload automation has been shown to provide tangible benefits for IT organizations in terms of such factors as cost savings, operational efficiency, infrastructure optimization, reduction in errors, meeting business service objectives, and even reduction in downtime. CA has assembled a substantial portfolio of job scheduling and automation software products through a series of acquisitions, such as Cybermation, combined with CA's organic software development. CA product capabilities include job scheduling for mainframes and distributed systems, cross-platform scheduling, as well as calendar- and event-based scheduling facilities. CA's workload automation solution strategy is an ambitious move not only to integrate many of the existing products but also to create an overall automation platform that provides higher-level facilities including workflows, orchestration, and crosscomponent communication through Web Services enablement. While the benefits of such a strategy are readily apparent, a large number of moving parts need to come together in a coordinated fashion to achieve successful implementation. CA's ambitious Workload Automation road map must be carefully evaluated by current users of CA's scheduling products to see how the existing products will play and be supported in the future. As is the case with any major development project, successful execution is an absolute requirement for market success, including strong emphasis on product quality given the high impact on IT operations. Also, automation is an area that frequently arouses initial user skepticism especially when new automation projects are being put in place. 10 #209302 2007 IDC
SUMMARY AND CONCLUSION CA's workload automation strategy is based on the evolutionary development of CA's portfolio of job scheduling and management software tools into a comprehensive workload automation solution. CA needs to leverage its impressive customer base for evaluations, tests, and feedback to provide proof points and gain positive momentum. CA's support for migration and compatibility as the various solution phases roll out will be a key determinant of user acceptance. Copyright Notice External Publication of IDC Information and Data Any IDC information that is to be used in advertising, press releases, or promotional materials requires prior written approval from the appropriate IDC Vice President or Country Manager. A draft of the proposed document should accompany any such request. IDC reserves the right to deny approval of external usage for any reason. Copyright 2007 IDC. Reproduction without written permission is completely forbidden. 2007 IDC #209302 11