COMPUTER MEASUREMENT GROUP INDIA. Large Centralized Tax Processing Solution Benchmark and Performance Analysis

Transcription

1 COMPUTER MEASUREMENT GROUP INDIA Large Centralized Tax Processing Solution Benchmark and Performance Analysis Vijay Jain Computer Measurement Group, India 1

2 Background (Existing Architecture) Client Server Architecture Zonal Centre A Zonal Centre B Zonal Centre C Replace With One Single Centralized System For The Country Computer Measurement Group, India 2

3 Background (Proposed Architecture) Web server Application Server Database Server Zonal Centre A WAN Zonal Centre B Zonal Centre C Zonal Centre D Benchmark to Determine if This is Technically Feasible Computer Measurement Group, India 3

4 RFP Objectives & Ground Rules Application Benchmarking a Pre-Condition To Evaluate Scalability Targets and Recommend Hardware Configuration for the Application Deterministic Think Time Fixed at 5 Seconds Only Configuration Parameter Tuning of Web, App, DB Servers Permitted No Application Code Optimization Permitted 4000 Concurrent Users (To test in steps of 1000, 2000, 4000 users) Average Server Side Response Time/Screen < 1 sec Server CPU & Memory Utilization < 50% Computer Measurement Group, India 4

5 Main Transaction to be Benchmarked Login Step 1: Main Screen Steps 2 and 3: Compute 12.5% Logout Steps 6,7, and 8: Print Result Steps 4 and 5: Refund Average Response Time Per Screen < 1 sec Think Time for entire transaction = 5 secs Observed Cycle Time was 2 minutes!!! Computer Measurement Group, India 5

6 Test Configuration (1000 Users) Web Servers 2 CPU 2 CPU Application Servers 8 CPU 8 CPU Database Servers 16 CPU Computer Measurement Group, India 6

7 Test Result & Outcome With 5 sec think time response time target surpassed at 500 users Tuning and adding resources were of little help Raised the issue with customer Technical Committee revised think time to 40 Secs and left to the vendors bidding for the RFP to do proper capacity planning Computer Measurement Group, India 7

8 Test Configuration (2000 Users) Web Servers 2 CPU 2 CPU 2 CPU 2 CPU Application Servers 8 CPU 8 CPU 8 CPU 8 CPU Database Servers 32 CPU Computer Measurement Group, India 8

9 Test Results: 1000 & 2000 Users Test Type CPU Used (DB) Business TPS Completion Time (Hrs) Returns processed User Login Main Screen Compute Refund Print Result User Exit DB CPU % Apps CPU % Web CPU % 1000 Users :10 1 M Avg ~45% ~40% ~15% 95 th pct Users Throughput Utilization :45 1 M Response Times (sec) ~50% ~40% < 10% Throughput scales almost linearly no apparent bottleneck Response Time Meets RFP Criteria Computer Measurement Group, India 9

10 Database Tuning Cached highly used sequences Increased transaction slots at block level (initrans) to manage high concurrency Pinned packages/functions in memory to avoid reload Raised the priority of Oracle Processes Introduced session cursor caching to reduce latch requirement Created reverse key indexes for surrogated key columns Different instances inserting 2005, 2006, 2007, Contention Normal B*Tree Reverse B*Tree Please note internal representation of bytes will be reversed Computer Measurement Group, India 10

11 4000 User Test Results on App/DB configuration used for 2000 users Results on higher number of CPUs Scalability limiting analysis Further analysis of response time Web Servers Base Configuration 2 CPU 2 CPU 2 CPU 2 CPU 2 CPU 2 CPU Application Servers 8 CPU 8 CPU 8 CPU 8 CPU Database Servers 32 CPU Computer Measurement Group, India 11

12 4000 User Test Results Test Type 4000 Users 2000 Users CPU Used (DB) Throughput Business TPS 47 X* 48 Completion Time (Hrs) 06:00 05:45 Returns processed 1 M Response Time Avg User Login th pct M 0.4 Main Screen Compute Refund Print Result User Exit Utilization DB CPU % ~50% ~50% Apps CPU % ~45% ~40% Web CPU % ~5% < 10% Computer Measurement Group, India 12

13 4000 User Test: Impact of Adding DB CPUs No. of Users No. of CPU (DB) No. of App Server Business Throughput 48 tps 47 tps 47 tps 47 tps Response Times (seconds) User Login Main Screen Compute Refund Print Result User Exit 1 1 Same as for 32 CPUs 32 CPUs Same as for 32 CPUs Utilization DB CPU % ~50% ~50% ~45% ~40% Apps CPU % ~40% ~45% ~40% ~40% Further Database Tuning Also Did Not Help Computer Measurement Group, India 13

14 4000 User Test: Network, Disk, Memory on DB server < 192Kbps on 1 Gbps lan Nothing to be alarmed about. No Disk, Memory, Network bottlenecks < 45% Database does not scale with CPUs Constant at 45% Are there any bad SQLs, Contention problems, Wait Events? Computer Measurement Group, India 14

15 SQL Processing Contribution Extract of Oracle Statspack report for 4000 user test SQL ordered by Gets for DB: ABC Instance: ABC CPU Elapsd Buffer Gets Executions Gets per Exec %Total Time (s) Time (s) Hash Value ,871,006 87,688 5, Module: (TNS V1-V3) SELECT ROWID,SEQ_NO,IND_STAT,BNDL_AREA_CD,BNDL_AO_TYP,BNDL_RANGE _CD,BNDL_AO_NO,BNDL_FIN_YR,BNDL_CNTR_NO,BNDL_SEQ_NO,ACK_NO,AST_Y R,PAN,DT_FILED,NAME,RET_INC FROM AA_RETURN WHERE (SEQ_NO IN (SEL ECT a.seq_no FROM aa_return a WHERE A.RANGE_CD = :1 AND A.AO_NO = :2 AND A.AO_TYP = :3 AND A.area_cd = :4)) and (AST_YR=:5) and SQL ordered by Reads for DB: ABC Instance: ABC CPU Elapsd Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value ,416,210 87, Module: f90runm@server1 (TNS V1-V3) SELECT ROWID,SEQ_NO,IND_STAT,BNDL_AREA_CD,BNDL_AO_TYP,BNDL_RANGE _CD,BNDL_AO_NO,BNDL_FIN_YR,BNDL_CNTR_NO,BNDL_SEQ_NO,ACK_NO,AST_Y R,PAN,DT_FILED,NAME,RET_INC FROM AA_RETURN WHERE (SEQ_NO IN (SEL ECT a.seq_no FROM aa_return a WHERE A.RANGE_CD = :1 AND A.AO_NO = :2 AND A.AO_TYP = :3 AND A.area_cd = :4)) and (AST_YR=:5) and Avg. Response time/execution= /87688 =0.23 sec High physical and logical reads Computer Measurement Group, India 15

16 SQL Processing Contribution Extract of Oracle Statspack report for 4000 user test SQL ordered by Gets for DB: ABC Instance: ABC CPU Elapsd Buffer Gets Executions Gets per Exec %Total Time (s) Time (s) Hash Value ,871,006 87,688 5, Module: (TNS V1-V3) SELECT ROWID,SEQ_NO,IND_STAT,BNDL_AREA_CD,BNDL_AO_TYP,BNDL_RANGE _CD,BNDL_AO_NO,BNDL_FIN_YR,BNDL_CNTR_NO,BNDL_SEQ_NO,ACK_NO,AST_Y R,PAN,DT_FILED,NAME,RET_INC FROM AA_RETURN WHERE (SEQ_NO IN (SEL ECT a.seq_no FROM aa_return a WHERE A.RANGE_CD = :1 AND A.AO_NO = :2 AND A.AO_TYP = :3 AND A.area_cd = :4)) and (AST_YR=:5) and Avg. Response time/execution= /87688 =0.23 sec SQL ordered by Reads for DB: ABC Instance: ABC CPU Elapsd Physical Reads Executions Reads per Exec Avg. %Total Time (s) Time (s) Hash Value Main 14,416,210 Screen 87, sec due to this SQL Module: Compute f90runm@server1 (TNS V1-V3) 13.7 Is it worth tuning? SELECT ROWID,SEQ_NO,IND_STAT,BNDL_AREA_CD,BNDL_AO_TYP,BNDL_RANGE _CD,BNDL_AO_NO,BNDL_FIN_YR,BNDL_CNTR_NO,BNDL_SEQ_NO,ACK_NO,AST_Y Refund Details 2.1 R,PAN,DT_FILED,NAME,RET_INC Print Result FROM AA_RETURN 24.0 WHERE (SEQ_NO IN (SEL ECT a.seq_no FROM aa_return a WHERE A.RANGE_CD = :1 AND A.AO_NO = :2 AND A.AO_TYP = :3 AND A.area_cd = :4)) and (AST_YR=:5) and Observed Excessively High Latch Contention High physical and logical reads Avg. Rsp Time for bus txn = 40.5 sec Computer Measurement Group, India 16

17 What is a Latch in Oracle DB Simple, low-level serialization mechanism to protect shared data structures in the System Global Area (SGA) Provide only exclusive access to protected data structures needed briefly Syntax Check Semantic Analysis Soft Parse Yes Was the statement already parsed by another session? Latches are applied in these stages No Hard Parse Computer Measurement Group, India 17

18 Database Wait Event Analysis Extract of Oracle Statspack report for 4000 user test: 30 min STATSPACK report for Snap Id Snap Time Sessions Curs/Sess Comment Begin Snap: 2 31-Oct-07 18:36:38 4, End Snap: 3 31-Oct-07 19:06:39 4, Elapsed: (mins) Top 5 Timed Events ~~~~~~~~~~~~~~~~~~~~~ % Total Event Waits Time (s) Ela Time latch free 1,539,663 3,587, CPU time 28, db file sequential read 17,221,454 7, log file sync 46, enqueue 5, Average Wait / DB txn = 2.3 * 5.1 = sec Excessively high Latch contention (99% of total wait time) Wait Events for DB: ABC Instance: ABC Avg. Total Wait wait Waits Event Waits Timeouts Time (s) (ms) /txn latch free 1,539, ,587, db file sequential read 17,221, , log file sync 246, enqueue 5, Latch Free wait during 2000 user test was 8.31% and Avg. wait time was 2 ms Computer Measurement Group, India 18

19 Latch Wait s Contribution to Response Time X* DB txns/sec for 47 business txns/sec 157.7/47 = 3.36 db txn/bus txn Snap Id Snap Time Sessions Curs/Sess Comment Begin Snap: 2 31-Oct-07 18:36:38 4, End Snap: 3 31-Oct-07 19:06:39 4, Elapsed: (mins) Load Profile ~~~~~~~~~~~~ Per Second Per Transaction Redo size: 1,537, , Logical reads: 539, , Avg. Main Screen 0.7 Compute 13.7 Refund Details 2.1 Print Result 24.0 Block changes: 8, Physical reads: 9, Physical writes: User calls: 22, Parses: 14, Hard parses: Sorts: 3, Logons: Executes: 38, Transactions: Contribution of Latch Wait Time to Response Time = Avg Wait Time/DB txn * No. of DB txn/business Txn = * 3.36 = seconds Average Response Time for 4000 Users = = 40.5 seconds Computer Measurement Group, India 19

20 Parsing Analysis Extract of Oracle Statspack report for 4000 user test SQL ordered by Parse Calls for DB: ABC Instance: ABC % Total Parse Calls Executions Parses Hash Value ,285,993 14,286, Module: f90runm@server1 (TNS V1-V3) declare p varchar2(32767); begin p := GLOBAL_POLICY(:sn, :on); :v1 := substr(p,1,4000); :v2 := substr(p,4001,4000); :v3 := substr(p,8001,4000); :v4 := substr(p,12001,4000); :v5 := substr(p,16001,4000); :v6 := substr(p,20001,4000); 2,939,514 2,939, Module: f90runm@server1 (TNS V1-V3) declare p varchar2(32767); begin p := MASTER_POLICY(:sn, :on); :v1 := substr(p,1,4000); :v2 := substr(p,4001,4000); :v3 := substr(p,8001,4000); :v4 := substr(p,12001,4000); :v5 := substr(p,16001,4000); :v6 := substr(p,20001,4000); 349,887 2,988, Module: f90runm@server1 (TNS V1-V3) SELECT ABC_NUM From EMP_ABC WHERE ORAUSER = USER Nearly 66% of total parse calls are for two functions : GLOBAL_POLICY & MASTER_POLICY Computer Measurement Group, India 20

21 Function Call Analysis Extract of Oracle TKPROF report of test declare p varchar2(32767); begin p := GLOBAL_POLICY(:sn, :on); substr(p,4001,4000); :v1 := substr(p,1,4000); :v2 := :v3 := substr(p,8001,4000); call count cpu elapsed disk query current rows Parse Execute Fetch total X* 247 function calls per business transaction = 247 * 47 = 11,609 per sec = 41.8 million per hour!!!! declare p varchar2(32767); begin p := MASTER_POLICY(:sn, :on); :v1 := substr(p,1,4000); :v2 := substr(p,4001,4000); :v3 := substr(p,8001,4000); call count cpu elapsed disk query current rows Parse Execute Fetch total Number of calls cannot be reduced by parameter tuning or adding hardware. Only way out is to change the application and hence reduce number of parses / latches / sec. Computer Measurement Group, India 21

22 2000 to 4000 User Analysis: (40 Secs Think Time) Average Rsp Time of Entire Business Txn 2000 user 4000 user 1.1 sec 40.5 sec Business Throughput 48/sec 47/sec Average Latch Wait Time 2 ms 2,330 ms If business throughput is not changing why does response time and wait time increase so drastically by doubling the number of users? N = 2000, R = 1.1, Z = 40 Queueing Theory : Closed System under saturation D max = 20 ms R = N * D max Z where R = average response time of business txn N = 4000, Z = 40, D max = 20 ms N = number of users R = 4000 * 20ms 40 Z = think time = 40 sec D max = demand at bottleneck resource = visit count at bottleneck resource/business txn x average service time at bottleneck resource Computer Measurement Group, India 22

23 Determinants of Performance No changes allowed during benchmark. Main bottlenecks identified Application Tuning Time Spent in Components Is 40 sec Think Time meaningful? Config parameters well tuned Adding more CPUs in one box does not help Workload Think Time in Transactions Infrastructure Resource Consumption Computer Measurement Group, India 23

24 Think Time Analysis With 40 sec think time : 50 Million returns can be processed in ~ 1.5 months Though the target is 4 months So what could be more reasonable think time? Assuming : No. of hours/day = 5 Throughput required to process 50 Million returns = 35 TXNs/Sec Applying Little s Law : N = (R+Z) * X ? 35 Avg. Think Time = (4000/35) 4 = 110 Sec Computer Measurement Group, India 24

25 Test Results: 100 Secs Think Time Test Type 1000 Users 2000 Users 4000 Users 4000 Users (Think Time =100 S) Throughput Business TPS Completion Time (Hrs) 11:10 5:45 6:00 3:35 Returns processed 1 M 1 M 1 M ½ M User Login Main Screen Compute Refund Print Result User Exit DB CPU % Apps CPU % Web CPU % ~45% ~40% ~15% Response Times sec 95 th percentile ~50% ~45% ~10% ~50% ~45% ~5% 1.0 ~45% ~35% ~5% Computer Measurement Group, India Utilization

26 Conclusions Benchmark has proven that application scales linearly till 2000 users with the specified think time Recommendations given for further scalability Analysis of the results and workload has proved that the workload defined for the benchmark is unrealistic System can meet the target of processing 50 million returns in 4 months, with reasonable think time and workload IMPACT The RFP was scrapped and new requirements were laid down with more realistic workload and flexibility Computer Measurement Group, India 26

27 Learnings Gauge the problem - Is it feasible to achieve target with resolution? Whenever doing Benchmark, work on Performance Engineering Model to derive whether it will work or not Evaluate contribution of Wait Events to response time while doing Database Analysis Every Application has certain scalability limits, no matter what hardware you deploy upon Adding hardware is not the solution for all the problems Choosing realistic workload is imperative to ensure its achievability and applicability of the results Unrealistic workload may cause cost and schedule overrun exponentially Computer Measurement Group, India 27

28 Q & A Computer Measurement Group, India 28