Self-Tuning Memory Management of A Database System

Transcription

1 Self-Tuning Memory Management of A Database System Yixin Diao diao@us.ibm.com IM 2009 Tutorial: Recent Advances in the Application of Control Theory to Network and Service Management

2 DB2 Self-Tuning Memory Management DB2 UDB Server Memory pools DB2 Clients Technical problems Large systems with varying workloads and many configuration parameters Autonomic computing: systems self-management Disks Agents Memory pools Challenges from systems aspects Heterogeneous memory pools Dissimilar usage characteristics Challenges from control aspects Adaptation and self-design Reliability and robustness 2 IM 2009: Recent Advances in the Applica6on of Control Theory to Network and Service Management

3 Load Balancing for Database Memory Load Balancer Measured Output 1 Resource Allocation 1 Resource Allocation N Resource Consumer 1 Resource Consumer N Measured Output N Resource Load Balancing Fairness optimal? Common measured output? Saved System Time (x i ) BenefitPerPage (y i ) Benefit (sec/page) OLTP simpages savedtime Entry size (Page) Memory Pool Size (u i ) 3 IM 2009: Recent Advances in the Applica6on of Control Theory to Network and Service Management

4 Constrained Optimization and Regulatory Control Saved Disk System Time Time ( x(x i ) i ) Overall Regulatory Control Mem pool 1 (x 1 ) BenefitPerPage (y 1 ) Mem pool 2 (x 2 ) MemoryPool1 size 1 (u 1 ) Mem size 2 (u 2 ) Optimal memory allocation Constrained Optimization Karush-Kuhn-Tucker conditions 4 IM 2009: Recent Advances in the Applica6on of Control Theory to Network and Service Management

5 Dynamic State Feedback Controller State space model Control error Integral control error Feedback control law 5 IM 2009: Recent Advances in the Applica6on of Control Theory to Network and Service Management

6 Incorporating Const of Control into Controller Design before Memory Pool A Remove these pages Pool Size Benefit Ts=12449 write dirty pages to disk Disk after Ts=15703 Memory Pool B before allocate extra memory OS Ts=24827 Major cost: write dirty, move memory, victimize hot Linear quadratic regulation (LQR) J = apple [e T (k) e T I (k)] Q [et (k) e T I (k)]t + u T (k) R u(k) Define Q and R regarding to performance Cost of transient load imbalances Cost of changing resource allocations 6 IM 2009: Recent Advances in the Applica6on of Control Theory to Network and Service Management

7 Adaptive Controller Design Local linear model Decentralized integral control Step Tuner Interval Tuner Response Time Benefit DB2 Clients Model Builder Y Accurate MIMO Control Algorithm Entry Size Greedy (Constraint) Memory Statistics Collector N Fixed Step 4-Bit (Oscillation) Entry Size DB2 Memory Pool Response Time Benefit 7 IM 2009: Recent Advances in the Applica6on of Control Theory to Network and Service Management

8 Experimental Assessment OLTP workload: multiple (20) buffer pools Response time benefits Memory sizes Throughput Increase TP from ~100 to ~250 DSS workload: various query lengths ConfigAdvisor settings Ts = 26342s STMM tuning Ts = 10680s > 2x improvement Time in seconds DSS workload: index drop Execution time for Query 21 (10 stream avg) Some indexes dropped avg = 959 avg = 6206 Reduce 63% avg = Order of execution 8 IM 2009: Recent Advances in the Applica6on of Control Theory to Network and Service Management

9 Comparing Control and Optimization Techniques Control-based approach Local linear model Optimization-based approach Gradient method Decentralized integral control Projected gradient (quasi-newton) Constraint enforcement (projection method) Step length (modified Armijo rule) Less dependence on the model Strictly applies constrained optimization Similarity in a simplified scenario Differences in design considerations Pure average vs. convex sum Pole location vs. Armijo rule Steady-state gain vs. Hessian matrix 9 IM 2009: Recent Advances in the Applica6on of Control Theory to Network and Service Management

10 Simulation Study: Comparison with Optimization Approach Control-based approach Optimization-based approach Memory size WL change Total saved time Without noise (single run) Control intervals Effect of noise (multiple runs) More robust and better uncertainty management Faster convergence, but more sensitive to noise 10 IM 2009: Recent Advances in the Applica6on of Control Theory to Network and Service Management

11 Summary DB2 self-tuning memory management Interconnection, heterogeneity, adaptation and robustness, cost of control Constrained optimization with a linear feedback controller Experimental assessment for OLTP and DSS workloads 11 IM 2009: Recent Advances in the Applica6on of Control Theory to Network and Service Management