State of Wisconsin Enterprise Distributed Batch Scheduling Service Offering Definition (SOD)

State of Wisconsin Enterprise Distributed Batch Scheduling Service Offering Definition (SOD)

Document Revision History Date Version Creator Notes Sep 19, 2008 - Sep 22, 2008 1.0 Tom Ober Initial draft Oct 6, 2008 1.0 Tom Ober Updates from Lee Henn, Steve Moschkau Oct. 22, 2008 1.0 Tom Ober Nov. 4, 2008 1.0 Tom Ober Updates from Steve Moschkau, Lee Henn, Lisa Onken Updates from Steve Moschkau, Lee Henn, and Lisa Onken Nov. 21, 2008 1.1 Lisa Onken Batch Server/Drawing updated Dec. 4, 2008 1.2 James Sylla Minor changes to formatting Dec. 29, 2008 1.3 Lisa Onken Updates to document due to comments by KA and AD. Jan. 05, 2009 1.4 Nan Linde Accept all changes clean copy for review Jan. 20, 2009 1.5 Lisa Onken Updates to document based on discussions with Kevin A. and Lee H. July 28, 2009 1.6 Lisa Onken Updates/edits October 30, 2009 1.6 Dan Proud Editing

Table of Contents Introduction...4 What Is Included...5 What Is Not Included...7 Benefits...7 Service Description...7 Service Period...9 Roles and Responsibilities...9 Business Continuity...9 Performance Metrics (Monitoring/Alerting/Reporting)...10 Configuration Diagram...10 How Services Are Charged...11

Introduction The Enterprise Distributed Batch Scheduling (EDBS) service offering allows customers to schedule, execute, monitor, and manage distributed environments batch jobs and workflow activities. This service is composed of one or more of the following pieces of the BMC Control-M distributed environment products: Control-M Enterprise Manager, Control-M/EM GUI Server, Control-M/EM GUI, Control-M/EM Desktop, Control-M/EM Global Condition Server (GCS), Control-M/EM Global Alert Server (GAS), Control-M/Server, and Control-M Agents. Additional components to this service are: hosted Dedicated Agency Batch Servers (DABS), hosted Enterprise Batch Servers (EBS) shared, and Hosted Dedicated Agency Application Servers. The access to Control-M/Enterprise Manager is available via a Citrix connection. This Citrix connection allows a pass through to a Control-M/EM GUI Server component on a DET server. The primary components used by the customers are Control-M/EM GUI and Control-M/EM Desktop software. Control-M/Server is the scheduling engine and is accessed by users through the Control-M/Enterprise Manager. The Control-M Agent components can be installed on each server where the customer batch job is to be executed or they can be installed on a batch server that will connect to other servers to run the jobs. There is an additional component to the Control-M distributed environment called Control-M Configuration Manager that is reserved for use by DOA/DET Technical staff. When subscribing to this service you are automatically authorizing DET to charge the consulting rate for troubleshooting issues your agency may experience with those items that are the agency s responsibility. There are four options within the service for EDBS implementation. The options an agency uses will be determined through discussions with DET, as some business requirements do not lend themselves to certain options. Option 1: Agency Managed Physical Services (AMPS/Co-Location) agencies When Applicable: This option is for those agencies that have elected only the AMVS or AMPS services provided by DET. These agencies will either own or need to purchase the identified licensed pieces of Control-M that an agency needs to provide. DET will manage: Control-M E/M, Control-M GUI Server, Control-M Desktop, Control-M GCS, and Control-M GAS. Co-located agency will provide: Control-M Server, Control-M Server Agent, request/maintain Citrix access, create/promote/maintain their batch jobs, job schedules, network connectivity, desktop license, job tasks licenses and optional agency batch server. EDBS SOD v1.6.doc 4 Last Updated: 10/30/2009 4:24:19 PM

Option 2: Agency Application environment managed by DET When Applicable: This option is selected when the batch jobs need to run directly on the application servers. The application servers must be managed by DET. In a consolidated environment the EDBS service offering by DET is as follows: DET will manage: Control-M E/M, Control-M GUI Server, Control-M Desktop, Control-M GCS, Control-M Server, Control-M Agent on dedicated application servers, and job tasks. Consolidated agency will need to request/maintain Citrix access through the DET Citrix Service, create/promote/maintain their batch jobs, desktop license, job schedules and network connectivity. Option 3: Consolidated agency using Dedicated Agency Batch Server (DABS) When Applicable: DABS will host dynamic agency batch jobs. This is a similar to option 2; however, batch jobs will reside on an agency dedicated batch server rather than directly on every application server. This option is used when the jobs are dynamic in nature (require changes to the jobs daily/weekly), and/or there are many batch jobs required by the agency, and/or additional applications must be loaded on the server for the batch jobs to work (e.g., Crystal Reports). DET will manage: Control-M E/M, Control-M GUI Server, Control-M Desktop, Control-M GCS, Control-M Server, dedicated Agency Batch Server. Consolidated agency will need to request/maintain Citrix access through the DET Citrix service, create/promote/maintain their batch jobs, desktop license, job schedules and network connectivity. Option 4: Consolidated Agency using Enterprise Batch Server (EBS): When Applicable: EBS servers will host static batch jobs (jobs that do not change for weeks/months) for agencies that do not require any additional application software to be hosted on the EBS servers. Number of jobs per agency should not exceed 100 different jobs on the EBS servers else they should be moved to a dedicated Agency Batch Server. DET will manage: Control-M E/M, Control-M GUI Server, Control-M Desktop, Control-M GCS, Control-M Server, and Enterprise Batch Servers (EBS). Consolidated agency will need to request/maintain Citrix access through the DET Citrix Service, create/promote/maintain their batch jobs, desktop license, job schedules and network connectivity. What Is Included Technical Support of the DET Control-M Server, Agent, Enterprise Manager, GUI Server, Control-M Desktop, Hosted Batch Servers, and Citrix connection Backup and recovery of the servers OS, Application and data through DET standard backup/recovery software; scheduled backups of the database through Oracle tools that is then backed up by DET s standard backup/recovery software EDBS SOD v1.6.doc 5 Last Updated: 10/30/2009 4:24:19 PM

Redundant installation of the applications in passive mode with mirroring and failover of the database Maintenance and installation of infrastructure components depending on the option selected; includes the operating system, software patches, product upgrades, and security fixes Monitoring of the related DET-managed server s UP/DOWN status and specific system services and events Systems performance monitoring performed by current monitoring tool. (Depending on the service and configuration selected, a monitoring agent can be installed on the server to monitor CPU, space utilization, memory, etc.) Planning and design options, system setup, and other customer specific requirements based on the option chosen. DET Firewall Access point between agency managed Control-M Server and DET Control-M Enterprise Manager Agencies can elect to run batch jobs on dedicated batch servers or agency dedicated application servers that are performing tasks other than running batch jobs. Control M Server agents will be installed where ever the batch is ran. If the Control-M Agent is deployed to a Windows server, the agent can run under a specific Active Directory domain account or it can run under the Local System account. The agency deploying the batch job(s) will be responsible for the creation and management of an Active Directory domain account if one is utilized for a Control-M agent running on a server in an agency domain. If the Control-M Agent is on a server in the Enterprise domain then DET will be responsible for the creation and management of the Active Directory domain account if one is utilized. Each batch job deployed to a Control-M Agent server can be configured to use the account the Control-M agent is running under for security purposes or it can be configured to use a specific account for that particular batch job. The agency deploying the batch job(s) will be responsible for the creation and management of a specific account if one is utilized by a batch job or jobs. Multiple Control-M agents can be run on a server but the DET standard is to run one Control-M agent per server. DET will be responsible for setting up multiple agents on a server for options 2-4, if necessary. Non-production servers with a Control-M agent installed on them would utilize the production implementation of Control-M for job scheduling purposes. The 'test' environment for Control-M falls into the Isolated Test Environment and will be used for DET testing purposes. Option 4 only: Staging of batch job code to Hosted Batch Server Management and support of access paths for batch jobs run on the Hosted Batch Server to connect to appropriate application or data servers EDBS SOD v1.6.doc 6 Last Updated: 10/30/2009 4:24:19 PM

What Is Not Included Citrix licenses for customers who use it for access to the EDBS system Batch job development or troubleshooting Management and support of agency maintained batch servers (Options 1 and 3) Production control work related to agency batch job development or troubleshooting Scheduler or Production Control staff training Option 1 only: Management or support of Control-M Servers that are owned, operated, and maintained by customer agency staff Management or support of agency Control-M Agents on agency-managed servers that are connected to an agency Control-M/Server that are installed on servers owned, operated, and maintained by customer agency staff (option 1 only) Database or Database Support on the agency Control-M Server Agency Firewall Access point between the Agency Control-M Server and DET Control-M Enterprise Manager Administrator training for installation, implementation, configuration, and issue resolution Benefits This EDBS system will provide for the communication of job conditions between the batch scheduling systems on distributed servers and the mainframe system. The ability to automate the execution of batch jobs on multiple types of distributed platforms Sharing of infrastructure and software, which can lead to reduction in hardware, software, and support costs Improved product usability, including more control of scheduled jobs, improved tracking functionally for audits, additional view enhancements, and more functionality using Control-M/Enterprise Manager Service Description The Enterprise Distributed Batch Scheduling service offering provides the State Enterprise with access to a leading edge distributed batch product that functions with cross-scheduling platform communications. This distributed batch system consists of: Control-M/Enterprise Manager, Control-M/EM GUI Server, Control-M/EM GUI, Control-M/EM Desktop, Control-M/EM Global Condition Server, Control-M/EM Global Alerts Server, Control-M/Server, Control-M/Agent, Control-M/Configuration Manager, and Hosted Batch Servers. EDBS SOD v1.6.doc 7 Last Updated: 10/30/2009 4:24:19 PM

Control-M/Enterprise Manager is the main component which all customers will have access to via the Control-M/EM GUI server. The Control-M/Enterprise Manager GUI is used by customers to manage all scheduling and workflow monitoring activities on the Control-M/Server by accessing it via a Citrix connection. The Control-M/EM Desktop component works in tandem with Control-M/Enterprise Manager to provide customers access to a robust GUI that they can use for job creation, editing, and defining automated scheduling choices. The job form allows for setup of scheduling parameters including job flow dependencies, run-time conditions, notifications, and logical system resources. The created or edited jobs are then uploaded to the database of Control-M/EM and then to the Control-M/Server database. Control-M Desktop also allows for use of Templates, Calendar creation for reusable job scheduling, and mass job creation and updates. Control-M Server is also a component of the distributed batch environment that is considered the engine used to drive scheduling in a data center. Depending on the configuration selected as part of the EDBS system, this piece might or might not be available to the customer for direct access. Control-M/Server submits jobs for execution via a Control-M/Agent installed on a server. The completion status of jobs can be viewed via Control-M/Enterprise Manager for ease of managing and viewing. Control-M/Agent is a part of the distributed batch environment that is connected to a Control-M/Server. This is the software responsible for the job submission to the server OS for execution, and gathering job execution information. The completion of jobs and execution info is transmitted back to Control-M/Server for post-processing handling. Control-M/EM Configuration Manager is a GUI console that allows for administrative tasks and component configuration work to be completed from one central location. This is only available to the DOA/DET Technical Support staff. Control-M/EM Global Alerts Server is part of the distributed batch environment that allows for monitoring and handling of alerts. You must sign on to the Global Alerts server to view any of the production or test system alerts. These alerts can come from shout messages, jobs ending not okay (a status defined by the application) and communication link breakdowns. Control-M/EM Global Conditions Server allows for prerequisite conditions creating dependencies between jobs running on different Control-M Servers. It is recommended that a naming standard be used that allows for easy identification of the global conditions. There are 4 options within the service for EDBS implementation. The options each agency uses will be determined through discussions with DET since some business requirements do not lend themselves to certain options. DMAS agencies can elect to subscribe to all options or just one of options 2-4, depending on business requirements. Option 1: For agencies that have selected AMVS or AMPS service and either have or will obtain their own Control-M Server that they will manage and support. The use of this service is to manage the scheduling of batch jobs across platforms between the mainframe and the distributive systems. Option 2: Hosted Dedicated Agency Application Servers may be used when batch jobs need to be executed directly on the application server(s). DET will provide guidance through discussions with the Agency to determine which option(s) best fit what the agency needs for their batch environment. EDBS SOD v1.6.doc 8 Last Updated: 10/30/2009 4:24:19 PM

Option 3: Hosted Dedicated Agency Batch Servers (DABS) provides Windows servers to host in a centralized location batch jobs for numerous application and data servers for a specific agency. DABS servers will host dynamic agency batch jobs (require changes to the jobs daily/weekly) and/or there is a large number of batch jobs required by the agency and/or additional applications must be loaded on the server for the batch jobs to work (e.g., Crystal Reports). Option 4: Hosted Enterprise Batch Servers (EBS) provides Windows servers to host in a centralized location batch jobs for numerous application and data servers. EBS servers will host static batch jobs (jobs that don t change for weeks or months) for agencies that do not require any additional application software to be hosted on the EBS servers. When the hosted batch servers are used, an agency-created batch job is submitted via a Service Request to DET and is then staged to the hosted batch environment by DET technical staff who set up appropriate access paths for the batch file to run correctly against the identified servers. If the servers the batch file runs against are test, development, acceptance, or production, then the corresponding batch file is considered to be test, development, acceptance, or production. It is recommended that a naming standard be used for batch files and schedules that allows for easy identification of the server environment the schedule is set up for. The actual process of code promotion from one environment to the next is the responsibility of the agency that creates the actual executable batch job. There should be a new schedule created for each batch job as it is promoted to different environments. Service Period July 1 through June 30 reviewed and renewed annually. Roles and Responsibilities Click here to see the Roles and Responsibilities for the Enterprise Distributed Batch Scheduling service. Business Continuity Business Continuity considerations are covered by the EDBS servers running Linux VMs with data stored on the SAN. The Control-M server can failover to the Control-M Enterprise Manager server because of the database being replicated/mirrored between them. The Control-M Enterprise Manager server cannot failover to the Control-M server. Data replication for the database is done at the Secondary Data Center location. Servers in the VM environment are highly fault tolerant. Complete system recovery can occur within 8 hours. EDBS SOD v1.6.doc 9 Last Updated: 10/30/2009 4:24:19 PM

Performance Metrics (Monitoring/Alerting/Reporting) Monitoring: The EDBS servers are monitored by the DET Enterprise Monitoring product for items such as server disk space, CPU, and memory usage. There is also monitoring for the UP/DOWN status and selected services/processes on these Control-M servers. Alerting: For problems with the EDBS system or servers, the DET Enterprise Monitoring product will be sending alerts when a monitored event s condition or threshold is met. This alert will be in the form of an e-mail and/or an automated incident ticket and will be assigned to the BIS OPS Control Automation queue. Alerting and Performance Metrics for individual batch jobs are dependent on the systems where they are run, are external to the scheduling system, and the responsibility of the agency. Reporting Metrics for the Enterprise Distributed Batch Scheduling service will be determined at a later date. Configuration Diagram Agency existing Control-M Server Agency existing Control-M Server EDBS Environment Architecture Control-M 6.3.01 Control-M Configuration Manager on Windows (Virtual) Mainframe Control-M Hosted Dedicated Agency Batch Server(s) with Control-M agent on Windows SUSE LINUX 9.0 Operating System on VM using BMC supplied Oracle DB Dedicated install BMC Supported Citrix Servers (Control-M/EM GUI & Desktop clients used by agencies) connected to EM GUI Server ` Web access to interface for Citrix MetaFrame Server at DET Hosted Shared Enterprise Batch Server(s) with Control-M agent on Windows Hosted Dedicated Application Server(s) with Control-M agent Primary Centralized Control-M Server (active) Control-M Enterprise Manager (active) & Secondary Centralized Control-M Server (inactive) used for failover. Secondary server DB is mirrored and constantly updated to remain identical to Primary DB. Storage Area Network https://metaframe.wisconsin.gov EDBS SOD v1.6.doc 10 Last Updated: 10/30/2009 4:24:19 PM

How Services Are Charged Charges will occur monthly through the Enterprise Billing System (EBS) and will be based on a total count of jobs assigned to or being managed by a specific customer for that month. A complete listing of billable jobs will be attached to the monthly billing subscription within EBS for the customer s reference. Rates for jobs vary based on a customer s specific choice of implementation options noted in the Introduction of this document. Please see the DOA IT Services Rate Sheet for rate information. EDBS SOD v1.6.doc 11 Last Updated: 10/30/2009 4:24:19 PM