Safe Harbor Statement

Transcription

1

2 Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle.

3 Best Practices In E-Business Suite Performance Tuning Kandasamy Nainamalai Senior Principal Software Engineer - Applications Technology Group

4 Program Agenda 1 Applications Architecture 2 Defining & Isolating issue 3 How to approach a performance issue 4 SQL Trace and TKPROF output 5 AWR/Statspack Report 6 Best Practices Maximizing the performance 4

5 <Insert Picture Here> Oracle Applications Architecture

6 Oracle Applications Architecture 3-tier architecture Database Tier Application/Middle Tier Desktop Tier

7 3-Tier Architecture

8 <Insert Picture Here> Defining and Isolating the Issue

9 Defining and Isolating the Issue What is slow? 1. The entire system is slow. 2. A subsystem is slow: - Forms-Based Applications - Web-Based Applications - Concurrent Manager 1. A particular process is slow.

10 Defining and Isolating the Issue When is it slow? Month end/year end Peak hours Concurrency, due to Load (or) H/W is not sized. All the time Was it working fine earlier? All of a sudden List out recent changes Patch, Gather Schema, New RAC-node/modules/users, Data load, OS, Network/DNS Deteriorated over a period Who all are affected? All users Specific to some module

11 Defining and Isolating the Issue By this time, we should be in a position to isolate which component is the source of the performance issue. Once we identify the component, we need to investigate what could be the problem. How to go about doing that?

12 <Insert Picture Here> How to Approach a Performance Issue

13 <Insert Picture Here> Approaching Performance Issue Gathering Required Information

14 Approaching Performance Issue Desktop Although it s a thin client; browser uses Java for the presentation Layer. It requires some resource at the Desktop. Check if there s enough resource CPU/RAM Ensure that no heavy applications are running when accessing applications.

15 Approaching Performance Issue Concurrent Manager Concurrent jobs are CPU intensive. Hence ensure enough resource CPU/RAM is available on the DB tier. Check RAC & PCP configuration. Check concurrent manager definition for Sleep time & Cache are set correctly Check resource-intensive batch requests are scheduled during peak hours. ( if yes, Separate manager for long running process with less number of managers) Check there are short running requests waiting for a longer time to process (Resolution: create Specialized Managers) Check Fnd_concurrent_requests & Processes are purged & defragmented regularly. Avoid enabling an excessive number of standard or specialized managers. Note Best Practices for Performance for Concurrent Managers

16 Approaching Performance Issue Forms Server Memory Intensive. Ensure sufficient RAM,Swap & CPU. Check Sar & top (look for spinning f60webmx) Check FORMS60_CATCHTERM FORMS_RECORD_GROUP_MAX Check if Debug/FRD enabled. Check load balancing enabled and works effectively. Latest Forms Patch set level with IO patches. Servlet Vs Socket Mode

17 Approaching Performance Issue Applications How many concurrent users are there? What are all the modules used? Any resource-intensive module is used. When was gather schema statistics request run last? Diagnostics profiles are enabled ; this will consume lots of resource. Sign on Audit is disabled if not needed. Audit on other tables have been enabled (additional overhead) Purging is done periodically.

18 Approaching Performance Issue Database DB processes are basically CPU Intensive. Check adequate resource (CPU/RAM) is available. Monitor Sar/vmstat output (first thing to do) If Wio is high then need to check I/O setup & SQL queries If CPU is high then check for CPU-intensive queries. Monitor top OS process at OS level Top (HP) / prstat (Sun) / ps aux (AIX) init.ora setup (Refer note ) Version of the Database & performance patch applied. AWR / Statspack

19 Approaching Performance Issue Web Tier (OC4J, Apache & Jserv) Check if JVM is hanging (due to Out of Memory issue). Check Heap size and how many JVM threads are defined. Disable statement/debug level logs when not needed. AOL/J Database Connection Pool Status (Doc ID ) Review DBC file configuration Review Apache/Jserv configuration Network It plays vital role if Applications accessed through VPN/WAN Check for sufficient Bandwidth (ping host l 1024 t) Latency should be as less as possible. Network Test ((Doc ID ) Socket Vs Servlet

20 <Insert Picture Here> SQL Trace and TKPROF Output When specific process takes time

21 Different Trace Levels Four possible levels: Level 1 : Regular Trace / SQL Trace Level 4 : Trace with Binds Level 8 : Trace with Waits Level 12 : Trace with Binds and Waits Level 8 is the preferred for analyzing performance issues.

22 Create Trace File Form Trace menu availability: or higher Trace file name contains application User Name Trace in Pre Identify the Session/Process Id then enable trace as below. ORADEBUG SETOSPID &&pid ORADEBUG EVENT TRACE NAME CONTEXT FOREVER, LEVEL 8 rem ORADEBUG EVENT TRACE NAME CONTEXT off

23

24 Concurrent Report Tracing Report Level Vs Program level 1. Set Enable Trace flag in the Concurrent Program Definition Form. (Program level) 2. Enable trace for a specific request (SRS form > Debug Options) (Request level) The profile Concurrent: Allow Debugging should be set to Yes. Recommended way to enable trace.

25 Enabling Trace at the Program Level

26 Enabling Trace at the Request Level

27 Create a Trace File Self Service To generate DB trace if web pages are slow. There are two options. 1. Set Profile Option FND: Diagnostics to Yes 2. Alternative: Profile Option called Initialization SQL Statement Custom What to choose from the above two options? Choose option 2, if the web page hangs and prevents you from enabling trace.

28

29 Create a Trace File Anywhere Profile Option : Initialization SQL Statement - Custom Can always be used, e.g.: Concurrent Programs Background processes like Workflow... Set on User level only SQL code will execute for each new DB session which it initialized for an application user SQL Statement for a level 12 trace: begin fnd_ctl.fnd_sess_ctl ('','','TRUE','TRUE','LOG','ALTER SESSION SET tracefile_identifier=''mytrace'' EVENTS=''10046 TRACE NAME CONTEXT FOREVER, LEVEL 12'''); end;

30

31 What Is TKPROF? Formatted report of SQL Trace file Includes all SQL statements with statistical info Allows to identify the performance problem by using different sort methods Invoke tkprof from Database Tier (10g/11g)

32 Getting TKPROF Output tkprof tracefile.trc output.txt sort=fchela,exeela,prsela Invoke tkprof from Database Tier Upload raw trace and tkprof output i.e. output.txt and tracefile.trc

33 total time # rows

34 Identify Culprit SQL Statement Identify Culprit SQL statement is within TKPROF files Most relevant is TKPROF Sorted by elapsed time In general the first SQL statement is the culprit Tip: Always look for large numbers

35 Gather Enhanced Explain Plan Isolate the culprit SQL statement in a text file Gather Enhanced Explain Plan using SQLTXPLAIN.SQL: Note Isolate the expensive DML. Collect statistics for the tables involved in the query. exec fnd_stats.gather_table_stats ('APPLSYS','FND_CONCURRENT_REQUESTS',PERCENT=>99); Any full table scan is used. See any index can be used Create new custom indexes on custom tables. Its suggested not to create indexes on seeded tables unless recommended by us. Re-write the custom query to speed up the query Ensure Temp, interface tables are purged and defragmented. Unselective index is being used. Rebuild the indexes associated only if necessary. Patch (for Higher version of code ) which resolves it. Log an SR with Oracle Support

36 <Insert Picture Here> AWR/Statspack Report

37 DEMO Demo AWR Report If you see any DML is consuming more elapsed time, get the explain plan using sql_id as below. You can run the following commands to obtain the run-time statistics, as well as past plans recorded in AWR: SQL> SELECT * FROM table(dbms_xplan.display_cursor('&sql_id',null,'allstats')) ; SQL> SELECT * FROM table(dbms_xplan.display_awr('&sql_id',null,null,'all'));

38 AWR/ Statspack Report Top 5 Events section This section shows the Top 5 timed events that must be considered to focus the tuning efforts. Top 5 Wait Events ~~~~~~~~~~~~~~~~~ Event Wait Waits Time (cs) Wt Time db file sequential read % Total ,797,963 11,207, db file scattered read 792,958 9,651, buffer busy waits 158,725 1,554, , , ,489, , log file sync SQL*Net break/reset to client

39 AWR / Statspack Report Handling db file scattered read This wait happens when a session is waiting for a multi block IO to complete. This typically occurs during full table scans or index fast full scans. Check if there are any full table scans. In this case query tuning should be used to optimize the SQL. See if partitioning can be used to reduce the amount of data you need to scan. Review SGA Database Buffer Cache

40 Best Practices for Maximizing the Performance Techstack upgrade (for Performance & High-Availability) Upgrade Database to (Note & Doc ID ) OracleAS 10g Release 3 Patch Set 5 ( ) (Note ) Forms bundle patch on (Doc ID ) Upgrade EBS to Latest version (or) (Note 269.1, ) Apply Recommended Performance Patches (Doc ID )

41 Best Practices for Maximizing the Performance Regular Health check to be performed Database Sufficient resource CPU/RAM available (20-30% free RAM/CPU is healthy) Sar & Top process has to be examined (Store it ; can be used later to compare) Ensure init.ora parameters are as per standards. AWR/Statspack report examined often. Forms Server Sufficient resource RAM & SWAP available Spinning of forms process set FORMS_RECORD_GROUP_MAX=10000 (Doc ID ) Cancel Query FND: Enable Cancel Query to No Tweak FORMS60_CATCHTERM & FORMS_CATCHTERM Servlet Vs Socket Mode Note Using Forms Socket Mode with Oracle E-Business Suite Release 12

42 Best Practices for Maximizing the Performance Concurrent Manager Tune using Best Practice Concurrent Manager (Note ) Sufficient resource CPU/RAM available Sleep time (30 secs) & Cache size (equal to number of process) Specialized Manger are used if needed. Purge / de-fragment Concurrent tables Applications Gather schema statistics collected at regular interval with % 11g Database, Apply Apps Patch (12.1.x) (12.0.x) ( ) to speed to Gather schema Stats. Leave % as null (Doc ID ) Diagnostics profiles/frd/traces are disabled if not needed. Hardware Sizing. Sign-on Audit & Audit tables are disabled if not needed. Purge & De-fragment interface/transaction tables periodically. (Note ) Purge portal is available in OAM that can be used to purge Apps.

43 Best Practices for Maximizing the Performance Network Audit Network usage. Tool to help identify the source of the problem Oracle E-Business Suite Network Utilities: Best Practices [ID ] Note Symmetrical Network Acceleration with Oracle E-Business Suite Release 12 Note Performance Degradation in Forms in R12 When not Using Swan Color Scheme Web Tier (OC4J, Apache & Jserv) Ensure heapsize is set properly (Doc ID ) Tune s_oacore_jvm_start_option (R12) (Doc ID ) $ grep s_oacore_jvm_start_option $CONTEXT_FILE (OS command to check it out) Tune oacore_nprocs( R12) Tune ApJServGroup OACoreGroup in jserv.properties (R11i )

44 Best Practices for maximizing performance (12.2 ) Oacore JVM Configuration Configure 2 GB JVM Heap Space for oacore. This can roughly support 150 to 200 users depending on usage. Default is 512 MB (roughly supports 50 users) Add more oacore instance to support more users. use multiple managed instances instead increasing heap size. Note: You should always size your systems based on tests using above representative data and workloads for your own environment. 44

45 Best Practices for maximizing performance Oacore JVM Configuration 45

46 Summary Performance Tuning Monitor Resource (CPU/ I/O /RAM) Utilization. Ensure Init.ora standards are met (Note ) AWR Report for Database wide performance issue Best Practice Concurrent Manager (Note ) Purge & Defragment interface/transaction tables. (Note ) Gather Schema Statistics at regular Interval. Apply FND_STATS patch (12.1.x) (12.0.x) ( ) to speed to Gather schema Stats (relevant for 11g Database). Upgrade Applications & Techstack.

47 References A Holistic Approach to Performance Tuning Oracle Applications Systems (Doc ID ) Best Practices for Performance for Concurrent Managers in EBS (Doc ID ) Reducing Your EBS Data Footprint using Archiving, Purging, and ILM (Doc ID ) bde_chk_cbo.sql - EBS initialization parameters - Healthcheck (Doc ID ) Collecting Diagnostic Data for Performance Issues in Oracle E-Business Suite (Doc ID ) Oracle E-Business Suite Recommended Performance Patches (Doc ID )

48 References JVM: Guidelines to setup the Java Virtual Machine in Apps Ebusiness Suite 11i and R12 (Doc ID ). Forms Process (FRMWEB) Consumes 100% of CPU in Oracle Applications R12 (Doc ID ) Best Practices for Gathering Statistics with Oracle E-Business Suite (Doc ID )

49 References Upgrading E-Business Suite. Which Oracle E-Business Suite Release Should You Target: 12.1 or 12.2? (Doc ID ) Upgrade Advisor: E-Business Suite (EBS) Upgrade from to (Doc ID 269.1) Interoperability Notes EBS R12 with Database 11gR2 (Doc ID ) Interoperability Notes Oracle EBS 11i with Oracle Database 11gR2 (11.2.0) (Doc ID ) Upgrading to the Latest OracleAS 10g x Patch Set in Oracle E-Business Suite Release 12 (Doc ID ) Upgrading OracleAS 10g Forms and Reports in Oracle E-Business Suite Release 12 (Doc ID )

50 References Upgrading E-Business Suite. These documents provide a starting point for customers moving to Oracle E-Business Suite Release Which Oracle E-Business Suite Release Should You Target: 12.1 or 12.2? (Doc ID ) Scenarios for Getting Started With Oracle E-Business Suite Release 12.2 (Doc ID ) Oracle E-Business Suite Technical Planning Guide, First Edition, Release 12.2 (Doc ID ) Useful 12.2 Documents For Customers And ATG Support Engineers (Doc ID ) Database Preparation Guidelines for an E-Business Suite Release 12.2 Upgrade (Doc ID ) 12.2: Consolidated List of Patches and Technology Bug Fixes (Doc ID ) R12.1 and 12.2 Oracle E-Business Suite Preinstall Patches Report [Video] (Doc ID ) Oracle E-Business Suite Recommended Performance Patches (Doc ID ) Best Practices for Minimizing Oracle E-Business Suite Release 12 Upgrade Downtime (Doc ID ) Express Diagnosis of Oracle E-Business Suite Release 12 Upgrade Performance Issues (Doc ID ) Oracle E-Business Suite Performance Guide (Doc ID ) Oracle E-Business Suite Release 12.2: Upgrade Sizing and Best Practices (Doc ID )

51 You can visit us at

52 Q&A 52

53 53

54