2013 OTM SIG CONFERENCE Performance Tuning/Monitoring

Transcription

1 2013 OTM SIG CONFERENCE Performance Tuning/Monitoring Alex Chang July 30, 2013

2 Agenda General guidelines Effective tuning goal Tuning session Tuning life cycle Common tools Case studies & Common tuning problems Q&A Page 2

3 Effective Tuning Goal Specific Measurable Achievable Cost effective 3

4 Tuning Session Define the problem and state the goal Collect current performance statistics Consider common performance errors Build a trial solution Implement and measure the change Decide: Did the solution meet the goal? No. Then go back to step 3 and repeat Yes. Then create a new baseline 4

5 Tuning Life Cycle An OTM life cycle can be divided into different phases: Functional and technical design and implementation Testing Deployment Production: Troubleshooting and tuning Use a test instance to determine whether the solution eliminated the bottleneck Migration, upgrade, and environment changes Any change of OS, database and OTM requires thorough testing to avoid performance issues. 5

6 Tuning Life Cycle Functional and technical design and implementation Proactive (make it better, so it will not break) Test scenarios Measure results Find the problem areas Resolve the problem Whenever possible, you should start tuning at this level. With a good design, many tuning problems do not occur It is less expensive to correct bugs and performance problems early in the life cycle than later 6

7 Tuning Life Cycle Testing: More realistic test data The test case should exercise the application functions, expected loads, and stress tests of improbable loads Check physical layout Monitor for resource contention Test for resource exhaustion Incorporate the business requirements for recovery time and availability as much as possible 7

8 Tuning During the Life Cycle Deployment: When OTM is initially deployed, the performance expectations are often different than reality Generate regular performance reports and save them as baselines Monitor growth and performance Compare new performance reports to the previous reports Tune before the performance degrades to an unacceptable level 8

9 Design and Implementation Production You need to know: What has changed? Were is the baseline? 9

10 Design and Implementation Migration, Upgrade, and Environment Changes Create a test system. Capture a representative workload Execute the workload against the test system. Collect statistics. Reset the test system, make a change. Execute the workload. Capture statistics and compare. 10

11 Common Tools OTM diagnostics servlets Database monitoring tools AWR ADDM OS monitoring tools SAR/TOP JVM tools Console logs Thread dumps Heap dumps Network monitoring 11

12 OTM Diagnostic Servlets Event diagnostics Cache diagnostics Object lock diagnostics Mediator diagnostics Threads 12

13 Event Diagnostics 13

14 Event Diagnostics 14

15 Cache Diagnostics 15

16 Object Lock Diagnostics 16

17 Oracle Enterprise Manager 17

18 Oracle Enterprise Manager 18

19 Oracle Enterprise Manager 19

20 Case Study 1: Design Phase Business requirement: The client wants to manage its fleet resources for its operational units in OTM as follows. Driver: Available, Intransit, Unavailable Power unit: Available, Intransit, Unavailable Equipment: Intransit Empty, Intransit Loaded, Total intransit, Assigned, Out of Service, Loaded Available, Empty Available etc. 20

21 Case Study 1 : Design Phase Potential approach Location Asset Bucket Inventory Standard built-in inventory control and use SKU tables Buckets are setup by resource type (i.e. 48 Trailer,48 Flatbed etc.) 41 buckets proposed 21

22 Case Study 1: Design Phase 22

23 Case Study 1: Design Phase Location asset bucket inventory Pre-setup buckets for all operational units would result in the number of bucket records equal to # locations x bucket counts ( i.e. 100,000 locations x 41 buckets = 4.1 Million records) Daily shipments : 500,000 shipments 1,500,000 insert and update per day Serious performance bottleneck 23

24 Case Study 1 : Design Phase Pros, Cons and Data Use Analysis 24

25 Common Tuning Problems Logging The most frequent cause of performance problems Single thread on the log file each thread must wait its turn to write to the log file Create user logs for debugging and disable these when not in use 25

26 Bulk Plan No Logging Bulk plan 500 orders without logging Start time : 23:38:58 End time: 23:39:45 Duration: 47 seconds 26

27 Bulk Plan Logging Bulk plan 500 orders with logging Start time : 00:11:04 End time: 00:12:22 Duration: 78 seconds 27

28 Bulk Plan No Logging Performance improved by 65%!!! These are simple test scenarios with very small data volume in my database. In reality, performance improve much more. Minimize logging!! NO Detail logs!! Disable logs for all domains!! 28

29 Bulk Plan No Logging Use Logging Overview to check 29

30 Common Tuning Problems Business Number Generator (BNG) contention Business number generator assigns serialized business numbers to relevant business objects. Single thread Eliminate unnecessary use of BNG 30

31 BNG Impact Created 500 order releases without BNG Used Integration layer to control order release keys 31

32 BNG Impact Created 500 order releases without BNG Start time: 01:17:40 End time: 01:18:19 Duration: 29 seconds 32

33 BNG Impact Created 500 order releases with BNG BNG assigned unique keys to each order 33

34 BNG Impact Created 500 order releases with BNG Start time: 01:15:44 End time: 01:16:27 Duration: 43 seconds Spent 48% more time!! 34

35 Common Tuning Problems Inefficient or high-load SQL statements Long parse time Excessive I/O (physical reads and writes) Excessive buffer gets Excessive CPU time Excessive waits 35

36 Top SQL Reports 36

37 Explain Plan 37

38 Common Tuning Problems Scalability Minimize the size of the object_lock table Clear as frequently as possible No need to keep data for more than one day even shorter 38

39 Common Tuning Problems Scalability Object Lock Cleanup 39

40 Common Tuning Problems Bulk Plan Query phase Spend extra time to fine tune SQL queries Keep data volume in mind Planning phase Itinerary, rate offering and rate record need to be as specific as possible Design them with zero or minimal overlap with each other Try not use global accessorial codes 40

41 Common Tuning Problems Shipment Created Phase Bulk Plan 500 orders/2 shipment built No agent listens to shipment created event Duration: 58 seconds 41

42 Common Tuning Problems Bulk Plan 500 orders/2 shipment built An agent listens to shipment created event Duration: 74 seconds 27% more time 42

43 Common Tuning Problems Bulk Plan 500 orders/2 shipment built Created a new process Duration: 65 seconds 12% more time 43

44 Common Tuning Problems Bulk Plan Shipment created phase Agents that are triggered on shipment creation can prolong the bulk plan significantly!! Create new process if possible Potential data lock or contention issue if an order are in multiple shipments or multi-leg shipments 44

45 Case Study 2 - Migration Bulk plan performance issue after OTM upgraded to 6.3 Before upgrade 10 minutes for a couple hundred orders After upgrade 3 hrs 45

46 Case Study 2 - Migration Cause: high volume of gets and low hit ratios glog.cache.reaccessorialcache glog.cache.rerategeocache glog.cache.tpackageditemcache glog.cache.trategeoseqcache Cache size issue 46

47 Common Tuning Problems Agent Avoid one event triggers multiple agents and each agent checks saved conditions Data contention: For example, one agent set status and another agent sends integration. Use parent-child agent approach 47

48 Common Tuning Problems Agent Avoid sequential agents Agent A triggers agent b and agent b triggers agent c. Consolidate into one agent Avoid overuse direct sql update, assign variable and saved query Consider using PL/SQL to consolidate them 48

49 Q&A 49