CA AutoSys Workload Automation r11 Troubleshooting Presenters: Dan Shannon, Elizabeth Dexter
Terms of This Presentation This presentation was based on current information and resource allocations as of October 2009 and is subject to change or withdrawal by CA at any time without notice. Notwithstanding anything in this presentation to the contrary, this presentation shall not serve to (i) affect the rights and/or obligations of CA or its licensees under any existing or future written license agreement or services agreement relating to any CA software product; or (ii) amend any product documentation or specifications for any CA software product. The development, release and timing of any features or functionality described in this presentation remain at CA s sole discretion. Notwithstanding anything in this presentation to the contrary, upon the general availability of any future CA product release referenced in this presentation, CA will make such release available (i) for sale to new licensees of such product; and (ii) to existing licensees of such product on a when and if-available basis as part of CA maintenance and support, and in the form of a regularly scheduled major product release. Such releases may be made available to current licensees of such product who are current subscribers to CA maintenance and support on a when and if-available basis. In the event of a conflict between the terms of this paragraph and any other information contained in this presentation, the terms of this paragraph shall govern. 2 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
For Informational Purposes Only Certain information in this presentation may outline CA s general product direction. All information in this presentation is for your informational purposes only and may not be incorporated into any contract. CA assumes no responsibility for the accuracy or completeness of the information. To the extent permitted by applicable law, CA provides this document as is without warranty of any kind, including without limitation, any implied warranties or merchantability, fitness for a particular purpose, or non-infringement. In no event will CA be liable for any loss or damage, direct or indirect, from the use of this document, including, without limitation, lost profits, lost investment, business interruption, goodwill, or lost data, even if CA is expressly advised of the possibility of such damages. 3 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
Agenda: >Preparing for the Installation >Installation and Initial Configuration >Customisation and Tuning >Common Issues 4 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
Preparing for Installation >Architecture What are you going to install, and where? What are you planning to support? >System Machine sizing i > OS requirements Patches, libraries > Networking Can the machines communicate? 5 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
Installation > Verify that the common component installs were successful > Verify that you can communicate between machines 6 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
Architecture 7 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
WCC Ports Services Related Applications Start Stop SSL Launcher Server: WCC Main WCC Main Login Web Services Web Services Samples Application Credential Application 8080 8005 8443 Configuration Server: Configuration 10132 10133 10134 Event Server: Event Console Quick Edit 10138 10139 10140 Job Status Console Servant: 10143 10144 10145 Job Status Console Server: Job Status Console 10122 Job Status Views Monitoring Server: Job Flow Design Job Flow Monitoring Critical Path Monitoring 10135 10136 10137 Scheduling Server: Job Editor Job Editor Plus (BLOB editor) Application Editor 10129 10130 10131 UI Framework Server: Quick View Quick Start Enterprise Command Line 10146 10147 10148 HA Server: High Availability Reporting Collectors 10149 10150 10151 8 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
Installation and Initial Configuration > Installing the Common Components SSA EEM CCS/Event Management > Connectivity it EEM AutoSys/WCC/SSA / > Multicast (WCC) 9 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
Common Issues > Collecting Information for Support Release, maintenance level, logs, system information > SSA Issues > EEM Issues > AutoSys Issues > WCC Issues > Multiple NIC cards > Command Sponsor is not installed 10 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
Logging and Debugging > CA Workload Automation AutoSys Logging > EEM Logging g > WCC Logging > SSA Logging 11 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
CA Workload Automation AutoSys Logging > ISDBGACTIV >Log Files univagent.out as_server.ins event_demon.ins auto_rem.joid.runnum 12 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
EEM Logging (SDK) > New With SDK 8.4 SR01 >EIAMCONFIG Environment > Configuration Files eiam.config logger.config 13 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
WCC Logging > WCC Logging > Location: <WCC Install Root> Logs folder Uses log4j > Tomcat Logs Access (not currently used) Application Admin, config, event, haserver, jobcontrolservant, jobcontrolserver, launcher, monitoring, reporting, uiframework Log4j.properties Install Services <WCC Install Root>/xxx/logs 14 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
Determining the Version >AutoSys >WCC autoflags uejmver.bat (Windows), uejmver.sh (*NIX) 15 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
SSA Logging > Environment Variables CSAM_SSL_DEBUG CSAM_DEBUG CSAM_LOG 16 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
Directory Layout /opt/c A /U nicentera utos ysjm $ A U T O U S E R $AU TO SYS o u t a r c h iv e b in a g e n t o u t lib IN G tx jo b s t d b o b j snm p O R A S Y B d o c m a n 8 r e q u e s t r e s p o n s e in s ta ll d a ta m e s s a g e s b in C o n fig u ra tio n F ile s : * c o n f ig. $ A U T O S E R V * autosys.sh.host * a u t o s y s. csh. h o s t * a u t o s y s. k s h. h o s t * a u t o s y s. b a s h. h o s t e t c. te s t b in_b a k... lib_b a k.. jil d e m o n ite ly O utthed oor 17 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
SSA Related Files 18 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
SSA Configuration Global 19 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
SSA Configuration Port (Cont) 20 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
SSA Application Logic 21 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
SSA Configuration Port 22 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
AutoSys Communication Problems > CAUAJM_E_10029 Communication attempt with the Unicenter AutoSys JM Application Server has failed! [host:port]. Can you ping the App Server host? Is the App Server running? Are AutoServer/AutoServerPort values correct? Is port 7163 (SSA) blocked by firewall? Are SSA port configurations the same on client & App Server machines the same? EnablePmux EnableSSL 23 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
AutoSys Communication Problems > CAUAJM_E_10527 Timed out waiting for response from the Unicenter AutoSys JM Application Server [host:port] Is the App Server running? Are there messages in the App Server log file? ISDBGACTIV=LIGHT,LOG Is the SSA PortRange for 49152-50176 configured correctly? EnablePmux EnableSSL Do the App Server and RDBMS machines have adequate resources? Does increasing DB_CONNECTIONS reduce problem? 24 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
Troubleshooting the Scheduler Job stuck in STARTING > Did Scheduler complete the Agent handshake? CAUAJM_I_10082 [POLAL02-GX280 connected for job1 626.225.1] > What does corresponding Job Agent log file show? CAUAJM_E_10527 Timed out waiting for response from the Unicenter AutoSys JM Application Server. CAUAJM_E_10221 Exhausted list of application servers. Failing request. 25 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
Troubleshooting the Agent CAUAJM_E_50026 ERROR: AutoPing WAS NOT SUCCESSFUL! > Does univagent log file show activity under ISDBGACTIV=LIGHT,JOB traces? > Do OS network tools reach Agent machine? (ping, nslookup, tracert/traceroute) > Is firewall blocking SSA physical port 7163? > Do SSA settings for AutoServerPort match (autoping D)? > Do SSA settings for PortRange=49152-50176 match? EnablePmux EnableSSL 26 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
EEM Troubleshooting > Performance Issues Check logging levels Active Directory connectivity Linux kernel Cache settings > Unexpected permissions errors Check for mixed case directory entries > Active Directory Group Retrieval Filters 27 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
EEM Troubleshooting CAUAJM_E_10448 FATAL SECURITY ERROR: Tampering or corruption of the security-related database keys detected > Did someone install a new instance using an MDB from a previous instance installation? > Did someone directly modify the ujo_alamode or ujo_keymaster keys? > What is the state of the database tables? Did some unknown corruption of keys take place? 28 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
EEM Troubleshooting CAUAJM_E_10436 Security server unreachable or invalid authentication certificate file > Is the EEM backend running? > Do OS network tools reach EEM backend machine? (ping, nslookup, tracert/traceroute) > Is firewall blocking EEM physical port 5250? > Was the proper EEM backend host name entered? > Are you able to connect to the EEM backend via the as_safetool? safetool? > Does the AUTOUSER folder contain a valid certificate file? 29 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
EEM Troubleshooting Policy problems > Does the EEM web UI Permission Check work as expected? > Does autosec_test test work as expected? > Does as_safetool work? 30 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
WCC - Other > Job Flow Monitoring Monitor ID Effective limit on the number of jobs in a flow > Job Status Console Global l Session/Global l ID Prior to WCC r11.1 SP1, check alarms/alerts filters > Unexpected login prompts IAM Security setting in server definition 31 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
Command Sponsor Troubleshooting > Required for Quick View and ECLI in WCC >Requires a credential user that is authorised on the OS to execute the command > Restricted to the commands in the AutoSysCommandISponsorFilters.txt txt file > Uses igateway 32 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA
Questions? 33 October 2009 [CA AutoSys Workload Automation r11 Troubleshooting] Copyright 2009 CA