<Insert Picture Here> Adventures in Middleware Database Abuse

Transcription

1

2 <Insert Picture Here> Adventures in Middleware Database Abuse Graham Wood Architect, Real World Performance, Server Technologies

3 Real World Performance

4 Real-World Performance Who We Are Part of the Database Development Organization Global Team: USA, France, Germany, UK, Denmark, Norway, China, India 300+ Man years of Oracle database experience Our goal: Increase the performance of the Oracle Database in the Real- World Our methods: Use the product as it was designed to be used Numerical and logical debugging techniques Educate others about the best performance methods and techniques Avoid and eliminate tuning by hacking/guessing/luck

5 What is Real World Performance? Bridging the Chasm from Today s Performance to What is Possible

6 What is Real World Performance? Common Mistakes That Stop Customers Achieving The Best Performance Incorrect Use of Tools/Products Not understanding the big/little data design challenge Moving to a solution before understanding the problem

7 Real-World Performance

8 Agenda Connection pools Threads vs Arrays Load balancing

9 Connection Management

10 Connections and Connection Pools The primary reason for escalation of OLTP systems into the Real-World Performance Group is poor connection pooling strategies. Symptoms of a poor connection strategy: A high number of connections to the database ( 1,000s ) A dynamic connection pool with a large number of logon/off to the database ( > 1/Sec ) Periods of acceptable performance and then unexplainable/undebuggable periods of poor performance/availability The inability to determine what is happening in real time

11 It is mostly about simple math Proc/Core 10 Proc/Core Avg 50 Proc/Core Avg 10 Proc/Core Max 50 Proc/Core Max 10 Proc/Core Min 50 Proc/Core Min

12 Connection Management Scenario Application must be able to handle high loads. Workload is highly dynamic such as retail on Black Friday We need thousands of connections to handle our load

13 Connection Management Scenario Small number of connections gives stable throughput and response time Small amount of time queueing in the middle tier What if we let it self-tune by increasing maximum pool size without changing workload?

14 Connection Management Analysis Connection count increases rapidly Throughput drops and is unstable Response time increases Host CPU utilization rises

15 Connection Management Scenario Reduce pool size back to original level Response time and stability restored Host CPU load drops

16 Threads vs Arrays Already have a service that processes a single message To process more messages concurrently using multiple threads change the service to process a set of messages.

17 Threads vs Arrays Scenario Already have a service that processes a single message Process more messages concurrently by using multiple threads or change the service to process a set of messages. Which is better?

18 Threads vs Arrays Analysis Not much of a contest Array takes 1minute Threads takes 22 minutes With higher load on server But still resource available Larger pool size for threads?

19 Threads vs Arrays Analysis 4x threads improved performance Array takes 1minute Threads takes 10 minutes Much higher load on server This is ONE dedicated process vs 64 parallel threads

20 Threads vs Arrays Scenario What happens when we add an index to both systems?

21 Threads vs Arrays Analysis Both systems slowed by index Multiple threads suffers from contention, buffer busy wait, TX index contention

22 Common Architecture and Load Balancing Errors

23 Is Load Balancing Slowing Your System? Scenario Devices ship files. Files read and processed by multiple application servers Each application server uses multiple threads that connect to database through a connection pool which is distributed by a scan listener over two instances.

24 Is Load Balancing Slowing Your System? Problem It s too slow It s a problem with the database Look at all those waits Need to be able to process an order of magnitude more data Obviously need to move to Hadoop

25 Is Load Balancing Slowing Your System? Analysis Only small amount of data being processed. Both instances essentially idle with most processes waiting in RAC and concurrency waits.

26 Is Load Balancing Slowing Your System? Solution Remove all of those RAC waits by running against a single database instance.

27 Is Load Balancing Slowing Your System? Analysis Throughput up by factor of 10x RAC waits gone CPU time actually visible High concurrency waits Buffer busy Tx index contention

28 Is Load Balancing Slowing Your System? Solution Reduce contention waits by processing a file entirely within a single application server

29 Is Load Balancing Slowing Your System? Analysis Throughput improved again Concurrency events reduced but still present

30 Is Load Balancing Slowing Your System? Solution Introduce affinity for a related set of records to a single thread by hashing All records for the same primary key processed by single thread so no contention in index for same primary key vale

31 Is Load Balancing Slowing Your System? Analysis More throughput Log file sync predominant event Latch events CPU usage close to core count

32 Is Load Balancing Slowing Your System? Solution Reintroduce RAC to add more CPU resource Implement separate service for each instance Connect application server to one instance

33 Conclusions Poor connection strategy Increases contention in the database Increases resource usage Increased potential for outage Good connection strategy reduces resource usage on the system reduces user response time allows increased throughput Use arrays rather than more threads for performance Load balancing!= Good performance Be Well Connected!