2013 OTM SIG CONFERENCE Performance Tuning/Monitoring Alex Chang alex.chang@inspirage.com July 30, 2013
Agenda General guidelines Effective tuning goal Tuning session Tuning life cycle Common tools Case studies & Common tuning problems Q&A Page 2
Effective Tuning Goal Specific Measurable Achievable Cost effective 3
Tuning Session Define the problem and state the goal Collect current performance statistics Consider common performance errors Build a trial solution Implement and measure the change Decide: Did the solution meet the goal? No. Then go back to step 3 and repeat Yes. Then create a new baseline 4
Tuning Life Cycle An OTM life cycle can be divided into different phases: Functional and technical design and implementation Testing Deployment Production: Troubleshooting and tuning Use a test instance to determine whether the solution eliminated the bottleneck Migration, upgrade, and environment changes Any change of OS, database and OTM requires thorough testing to avoid performance issues. 5
Tuning Life Cycle Functional and technical design and implementation Proactive (make it better, so it will not break) Test scenarios Measure results Find the problem areas Resolve the problem Whenever possible, you should start tuning at this level. With a good design, many tuning problems do not occur It is less expensive to correct bugs and performance problems early in the life cycle than later 6
Tuning Life Cycle Testing: More realistic test data The test case should exercise the application functions, expected loads, and stress tests of improbable loads Check physical layout Monitor for resource contention Test for resource exhaustion Incorporate the business requirements for recovery time and availability as much as possible 7
Tuning During the Life Cycle Deployment: When OTM is initially deployed, the performance expectations are often different than reality Generate regular performance reports and save them as baselines Monitor growth and performance Compare new performance reports to the previous reports Tune before the performance degrades to an unacceptable level 8
Design and Implementation Production You need to know: What has changed? Were is the baseline? 9
Design and Implementation Migration, Upgrade, and Environment Changes Create a test system. Capture a representative workload Execute the workload against the test system. Collect statistics. Reset the test system, make a change. Execute the workload. Capture statistics and compare. 10
Common Tools OTM diagnostics servlets Database monitoring tools AWR ADDM OS monitoring tools SAR/TOP JVM tools Console logs Thread dumps Heap dumps Network monitoring 11
OTM Diagnostic Servlets Event diagnostics http://localhost/gc3/glog.webserver.event.eventdiagservlet Cache diagnostics http://localhost/gc3/glog.webserver.cache.cachediagservlet Object lock diagnostics http://localhost/gc3/glog.webserver.synch.object.objectlockdiagservlet Mediator diagnostics http://localhost/gc3/glog.webserver.mediator.mediatordiagservlet Threads http://localhost/gc3/glog.webserver.thread.threaddiagservlet 12
Event Diagnostics 13
Event Diagnostics 14
Cache Diagnostics 15
Object Lock Diagnostics 16
Oracle Enterprise Manager 17
Oracle Enterprise Manager 18
Oracle Enterprise Manager 19
Case Study 1: Design Phase Business requirement: The client wants to manage its fleet resources for its operational units in OTM as follows. Driver: Available, Intransit, Unavailable Power unit: Available, Intransit, Unavailable Equipment: Intransit Empty, Intransit Loaded, Total intransit, Assigned, Out of Service, Loaded Available, Empty Available etc. 20
Case Study 1 : Design Phase Potential approach Location Asset Bucket Inventory Standard built-in inventory control and use SKU tables Buckets are setup by resource type (i.e. 48 Trailer,48 Flatbed etc.) 41 buckets proposed 21
Case Study 1: Design Phase 22
Case Study 1: Design Phase Location asset bucket inventory Pre-setup buckets for all operational units would result in the number of bucket records equal to # locations x bucket counts ( i.e. 100,000 locations x 41 buckets = 4.1 Million records) Daily shipments : 500,000 shipments 1,500,000 insert and update per day Serious performance bottleneck 23
Case Study 1 : Design Phase Pros, Cons and Data Use Analysis 24
Common Tuning Problems Logging The most frequent cause of performance problems Single thread on the log file each thread must wait its turn to write to the log file Create user logs for debugging and disable these when not in use 25
Bulk Plan No Logging Bulk plan 500 orders without logging Start time : 23:38:58 End time: 23:39:45 Duration: 47 seconds 26
Bulk Plan Logging Bulk plan 500 orders with logging Start time : 00:11:04 End time: 00:12:22 Duration: 78 seconds 27
Bulk Plan No Logging Performance improved by 65%!!! These are simple test scenarios with very small data volume in my database. In reality, performance improve much more. Minimize logging!! NO Detail logs!! Disable logs for all domains!! 28
Bulk Plan No Logging Use Logging Overview to check 29
Common Tuning Problems Business Number Generator (BNG) contention Business number generator assigns serialized business numbers to relevant business objects. Single thread Eliminate unnecessary use of BNG 30
BNG Impact Created 500 order releases without BNG Used Integration layer to control order release keys 31
BNG Impact Created 500 order releases without BNG Start time: 01:17:40 End time: 01:18:19 Duration: 29 seconds 32
BNG Impact Created 500 order releases with BNG BNG assigned unique keys to each order 33
BNG Impact Created 500 order releases with BNG Start time: 01:15:44 End time: 01:16:27 Duration: 43 seconds Spent 48% more time!! 34
Common Tuning Problems Inefficient or high-load SQL statements Long parse time Excessive I/O (physical reads and writes) Excessive buffer gets Excessive CPU time Excessive waits 35
Top SQL Reports 36
Explain Plan 37
Common Tuning Problems Scalability Minimize the size of the object_lock table Clear as frequently as possible No need to keep data for more than one day even shorter 38
Common Tuning Problems Scalability Object Lock Cleanup 39
Common Tuning Problems Bulk Plan Query phase Spend extra time to fine tune SQL queries Keep data volume in mind Planning phase Itinerary, rate offering and rate record need to be as specific as possible Design them with zero or minimal overlap with each other Try not use global accessorial codes 40
Common Tuning Problems Shipment Created Phase Bulk Plan 500 orders/2 shipment built No agent listens to shipment created event Duration: 58 seconds 41
Common Tuning Problems Bulk Plan 500 orders/2 shipment built An agent listens to shipment created event Duration: 74 seconds 27% more time 42
Common Tuning Problems Bulk Plan 500 orders/2 shipment built Created a new process Duration: 65 seconds 12% more time 43
Common Tuning Problems Bulk Plan Shipment created phase Agents that are triggered on shipment creation can prolong the bulk plan significantly!! Create new process if possible Potential data lock or contention issue if an order are in multiple shipments or multi-leg shipments 44
Case Study 2 - Migration Bulk plan performance issue after OTM upgraded to 6.3 Before upgrade 10 minutes for a couple hundred orders After upgrade 3 hrs 45
Case Study 2 - Migration Cause: high volume of gets and low hit ratios glog.cache.reaccessorialcache glog.cache.rerategeocache glog.cache.tpackageditemcache glog.cache.trategeoseqcache Cache size issue 46
Common Tuning Problems Agent Avoid one event triggers multiple agents and each agent checks saved conditions Data contention: For example, one agent set status and another agent sends integration. Use parent-child agent approach 47
Common Tuning Problems Agent Avoid sequential agents Agent A triggers agent b and agent b triggers agent c. Consolidate into one agent Avoid overuse direct sql update, assign variable and saved query Consider using PL/SQL to consolidate them 48
Q&A 49