Introduction to Software Performance Engineering

1-1 Overview 1. Overview of SPE 1. Software Performance Engineering 2. Distributed System s 3. Interoperability 4. SPE Research and Education Connie U. Smith, Ph.D. PO Box 2640 Santa Fe, New Mexico 87504-2640 (505) 988-3811 http://www.perfeng.com/ Copyright 2002-7, Performance Engineering Services. All rights 2 1. Overview of SPE Performance Balance Software Performance Engineering (SPE) Introduction SPE Process Workload New Software Hardware Configuration CAPACITY Data Required s Case Study RESOURCE REQUIREMENTS Quantitative Assessment Begins early, frequency matches system criticality Often find architecture & design alternatives with lower resource requirements Select cost-effective performance solutions early 3 4 SPE Focus Goal Performance: a measure of how well a software system or component meets its requirements for timeliness responsiveness scalability Software-oriented approach focuses on architecture, design and implementation choices Early assessment of software decisions to determine their impact on quality attributes such as performance reliability reusability maintainability/modifiability Early assessment of the architecture is important because architecture has the most significant influence on quality attributes architectural decisions are the most difficult to change 5 6

1-2 SPE Process Includes Part 2: Review of SPE -based Approach General Principles for Performance-oriented Design, Design Patterns and Antipatterns Data Gathering Strategy Performance Walkthroughs Construction and Evaluation Strategy Techniques to Compensate for Uncertainties Data Presentation & Tracking Verification & Validation 7 8 SPE ing Steps assess performance risk identify critical use cases SPE ing Steps 1. Assess performance risk select key performance scenarios 2. Identify critical use cases establish performance requirements 3. Select key performance scenarios construct performance model(s) 4. Establish performance requirements verify and validate models add software resource requirements add computer resource requirements modify/ create scenarios 5. Construct performance models 6. Determine software resource requirements evaluate performance model(s) [performance acceptable] [feasible] [infeasible] modify product concept revise performance requirements 7. Add computer resource requirements 8. Evaluate the models 9. Verify and validate the models 10 Performance Scenarios Gather Data Workload identification and characterization Benchmark scenarios Performance Goals Precise performance requirements Response time Throughput Resource budgets Resource Usage Estimates Execution Environment Workload Software/ Database 11 12

1-3 Performance s SPE Requirements Conventional s Existing Existing Work Work System Execution Performance Metrics Software Prediction s Existing Work System Execution Performance Metrics Software Execution Low overhead use the simplest possible model that identifies problems Accommodate: incomplete definitions imprecise performance specifications changes and evolution Goals: initially distinguish between "good" and "bad" later, increase precision of predictions provide decision support 13 14 SPE ing Strategies Cost / Benefit Analysis Simple- Strategy start with the simplest possible model that identifies problems with the system architecture, design or implementation plans. Schedule impact Modification Effort Side Effects Predicted Improvement Best- and Worst-Case Strategy use best- and worst-case estimates of resource requirements to establish bounds on expected performance and manage uncertainty in estimates Adapt-to-Precision Strategy match the details represented in the models to your knowledge of the software processing details 15 Risk If alternatives are unacceptable, revise the performance requirements Best alternative may not have the best performance 16 V & V SPE -Based Approach parameters verify workload? System results products validate Verify the model data, validate the results Begin early in the life cycle and continue throughout life cycle Check for model omissions!?? 17 Reduces the risk of performance failures Provides: Refinement and clarification of performance requirements Predictions of performance Estimates of sensitivity to precision of resource usage estimates and workload intensity Quantitative impact of software alternatives Scalability of the architecture and design Identification of critical parts of the system Identification of assumptions that may change results Assistance for budgeting resource demands Assistance for designing performance tests 18

1-4 Workload Data Part 2: Data Required Pareto principle ( 80-20 rule ) More than 80% of the software requests will be for less than 20% of the functions of the system First: scenarios of typical activity Number of concurrent users Request arrival rates Performance requirements Later, add large scenarios, critical scenarios, etc. 19 20 Example: ATM System Software Specifications System Functions: Execution paths for scenarios of interest Get balance Checking Savings Make payment From checking From savings In envelope Withdrawal Checking Savings Credit card Deposit Savings Checking Objects / methods to be executed probability of execution number of repetitions protocol Database accesses, messages sent Level of detail increases as development progresses Scenario? 21 22 Sequence Diagrams ATM Sequence Diagram Sequence diagrams provide a tabular notation for objects (those of interest in the scenario) messages between objects events operation invocation Messages are temporally ordered : User : ATM : HostBank cardinserted requestpin apin *[until done] requesttransaction response alt [type] processdeposit Often created by object-oriented developers during design processwithdrawal processbalanceinquiry terminatesession 23 24

1-5 Example (continued) Environment Processing scenario: Request withdrawal 1. Initiate 2. Get and interpret request {response = withdrawal, checking acct} 3. Trans Authorize (Withdrawal) 4. Dispense cash 5. Print receipt 6. Terminate Performance requirement: Response time secs. Workload intensity, e.g., number of arrivals per hr. per ATM Platform and network characteristics: configuration device service rates Software overhead, e.g., database path lengths, communication overhead, etc. External factors, e.g., peak hours, bulk arrivals, batch windows, scheduling policies Case study Environment Specifications: time for ATM communication CPU speed system configuration (devices, service times, etc.) 25 26 Resource Usage Example Specifications CPU Work Units or # of instructions executed or measurements of similar software I/O Database calls Communication Other devices Software overhead calls & characteristics Component K Instr Disk I/O s ATM Screens initiate get & interpret request withdrawal print balance terminate 200 150 250 dispense cash 350 1 1 400 100 1 0 1 0 0 2 2 0 1 1 27 28 System Versus Software ing Tools Part 3: SPE -based Approach Existing Work New Work Software Performance Execution Graphs (may be spreadsheet, pseudo-code, etc. views of EGs) Performance Metrics System Execution Queueing Networks 29 System Requires more modeling expertise Device usage, overall response time and throughput Useful to evaluate hardware changes Software Requires less modeling expertise Time and resource requirements of processing steps and overall Useful to evaluate software alternatives A combination is best. 30

1-6 1. Start with Sequence Diagram 2. Software Performance s : User : ATM : HostBank cardinserted requestpin apin *[until done] requesttransaction response alt [type] processdeposit processwithdrawal processbalanceinquiry terminatesession 31 Create from sequence diagrams Specify software resource requirements Specify computer resource requirements (e.g., path lengths) Results point to problem areas getaccount getamount request Authorization dispensecash waitfor Customer confirm Transaction Screen 1 Host 0 Log 0 Delay 0 Screen 1 Host 0 Log 0 Delay 0 Screen 0 Host 1 Log 1 Delay 0 Screen 1 Host 0 Log 1 Delay 5 Screen 0 Host 0 Log 0 Delay 10 Screen 0 Host 1 Log 1 Delay 0 32 3. Computer Resource Requirements Overhead Matrix: Computer Resource Requirements summary WorkUnit 12 DB 2 Msgs 2 CPU Disk Delay Network Devices Devices Service Units CPU Disk Display Delay Net 1 1 1 1 1 Sec. Phys. I/O Screens Units Msgs. Ties software resource requirements to devices and service required in computer environment Screen Host Log Delay Service Time 0.001 1 0.005 3 2 0.001 1 1 1 0.02 1 1 0.05 33 34 Software s: Purpose Execution Graphs Derive software execution characteristics Initial check of performance requirements Provide workload parameters for the system execution model Visual representation of software confirmation Presentation of results Formal basis for analysis Aids in comprehension Specification for tool development 35 36

1993 Performance Engineering Services Introduction to Software Performance Engineering 1-7 Notation Typical Structures Basic Node Processing step at current level of detail. Initiate Expanded Node The step has more details. Details are in the associated subgraph. 2 Process request Get request.7 Withdrawal Repetition Node n The node in the is repeated n times. It may be an expanded node. Terminate Interpret request.3 Get balance Case Node Attached nodes are conditionally executed. A probability is associated with each. 37 38 Example: Specific scenario Example: Generic Scenario Withdrawal scenario: Initiate Read ID# from card Initiate Get and interpret request Read PIN 2 Withdraw Dispense cash,print receipt Validate User get request Trans type?..29.7.01 Get checking balance Withdraw from checking Terminate Terminate Make payment 39 40 Graph Analysis 4. Software Results Best case Worst case Average Variance Distribution SPE ED TM Results screen Solution: View: No contention Residence time Contention Rsrc demand System model Utilization Adv sys model SWmod specs Simulation data Pie chart Solve OK Display: Specify: Software mod Names ATM example Resource utilization: CPU 0.00 Get cust ID ATM 0.03 from card DEV1 0.00 CPU 0.00 Get PIN ATM 0.03 from DEV1 0.00 customer Each transaction CPU 0.15 Get type ATM 0.48 and DEV1 1.60 process < 0.250 < 0.500 < 0.750 < 1.000 1.000 totals 0.15 CPU 0.88 ATM 1.00 DEV1 Results Values CPU 0.00 Overhead System model Overhead Service level Terminate ATM 0.35 DEV1 0.00 SPE database Sysmod globals Save Scenario Open scenario 41 42

ATM example Residence Time: 21.5388 Get cust ID Get PIN from Get type Termina te 0.6046 0.6107 Each 11.8527 8.4708 < 1.250 < 2.500 < 3.750 < 5.000 5.000 Resource Usage 0.0198 CPU 21.3131 ATM 0.2060 DEV1 ATM example Resource utilization: CP 0.00 Get cust ID Get PIN from Get type Termina te DE 0.00 Each CP 0.12 AT 0.41 DE 0.43 CP 0.00 AT 0.30 DE 0.00 < 0.250 < 0.500 < 0.750 < 1.000 1.000 < 0.250 < 0.500 < 0.750 < 1.000 1.000 totals 0.12 CPU 0.75 ATM 0.43 DEV1 Introduction to Software Performance Engineering 1-8 5. System Performance 6. System Performance Results System execution model quantifies contention effects Workload intensity Device visits Device service rates OOD Example ATM exampl e 0.122 0.754 0.431 0.000 Utilization: Utilization: 0.43 0.12 ATM Host Bank AT 0.02 DE 0.00 CP 0.00 AT 0.02 Response times Throughput Utilizations Residence times Queue lengths 43 44 ICAD Part 3: Case Study Computer-Aided Design (CAD) application Features interactively construct, view, and analyze drawings that model structures (e.g., aircraft wings) model stored in database several versions of a model may exist in the database 45 46 ICAD DrawMod Scenario Each models consists of nodes defined by position (x, y, z) contain additional information needed for model solution elements (connect two nodes) triangles (connect three nodes) plates (connect four or more nodes) Typical model contains only nodes and consists of 2050 Focus on the scenario in which a typical model is drawn on the user s screen (DrawMod) : ICAD : : Database open() find(modelid) retrieve(, nodes) Performance requirement draw typical model in 10 sec or less close() 47 48

1993 Performance Engineering Services Introduction to Software Performance Engineering 1-9 Architecture 1 Architecture 1 Class Diagram Uses an object for each node and beam modelid : int 1..n Element elementno : int Node 2 Beam Triangle Plate nodeno : int x : int y : int z : int 3 node1 : int node2 : int node1 : int node2 : int node3 : int nodes[] : int 4..n 49 50 Architecture 1 Drawmod Scenario Process and Physical Views : ICAD : Database : open() find(modelid) find(modelid, ) «process» : GUI «process» «processor» Workstation «database» DBMS sort() retrieve(beam) *[each beam] : DBMS GUI find(modelid, node1, node2) : Beam «process» : ICAD ICAD retrieve(node1) retrieve(node2) : Node : Node Process View Physical View close() 51 52 Architecture 1 Execution Graph Architecture 1 Performance Results SPE ED TM Drawmod Software model Drawmod Initialize Add Find Beams Drawmod Time, no contention: 992.329 Initializ 0.272 e < 2.70 < 5.14 < 7.57 Resource Usage Find 0.214 Beams 0.198 CPU 990.078 DEVs Link Expand Sort 2.052 Display Insert node Sort 1.270 Solve Display: OK Specify: Software mod Names Results Values Overhead Overhead Draw System model Service level SPE database Sysmod globals Draw beam 990.512 Save Scenario Open scenario Finish Finish 0.060 53 54

Introduction to Software Performance Engineering 1-10 Performance Results Architecture 2 Architecture Architecture 1 Overall Response Time 992.33 sec Uses Flyweight pattern uses a shared object for and nodes rather than individual objects reduces run-time overhead due to constructor invocations 55 56 Architecture 2 Class Diagram Architecture 2 Drawmod Scenario : ICAD : Database modelid : int 1..n Element elementno : int : open() : Beam : Node : Node find(modelid) find(modelid, ) 1..n sort() Node Beam Triangle Plate retrieve(beam) *[each beam] nodeno : int x : int y : int z : int node1 : int node2 : int node1 : int node2 : int node3 : int nodes[] : int find(modelid, node1, node2) retrieve(node1) retrieve(node2) close() 57 58 Architecture 2 Performance Results Performance Results Drawmod rev1 Time, no contention: 992.267 < 2.70 Initialize 0.272 < 5.14 < 7.57 Architecture Overall Response Time Find Beams 0.214 Resource Usage 0.137 CPU 990.078 DEVs 2.052 Display Architecture 1 Architecture 2 992.33 sec 992.27 sec Sort 1.270 Draw beam 990.450 Finish 0.060 59 60

Drawmod Initializ e Find Beams Sort Draw beam Finish Drawmod Initializ e Find Beams Sort Draw beam Finish DE 0.270 Dis 0.002 CP 0.002 DE 0.212 CP 0.002 DE 1.268 CP 0.194 DE 988.2 Dis 2.050 DE 0.060 < 2.70 < 5.14 < 7.57 Resource Usage 0.198 CPU 990.078 DEVs 2.052 Display < 2.50 < 5.00 < 7.50 totals 0.198 CPU 990.078 DEVs 2.052 Display Draw beam Retriev e beam New beam Find nodes Set up node Draw Draw beam Retriev e beam New beam Find nodes Set up node Draw Each node DE 0.121 DE 0.000 DE 0.120 Each node DE 0.241 DE 0.000 Dis 0.001 < 2.59 < 5.06 < 7.53 < 2.50 < 5.00 < 7.50 Submodel totals 0.000 CPU 0.482 DEVs 0.001 Display 1993 Performance Engineering Services Introduction to Software Performance Engineering 1-11 Architecture 1 Performance Results Architecture 3 Time, no contention: 992.329 0.272 0.214 1.270 990.512 0.060 Resource demand: Time, no contention: 0.483 0.121 0.000 0.120 0.241 0.001 Resource demand: Retains use of Flyweight pattern Modifies database management system new operation retrieveblock() retrieves block of data rather than individual nodes and single block retrieve get 20k 64, or 170 nodes reduces database accesses to 33 to retrieve, plus 9 to retrieve nodes 61 62 Architecture 3 Drawmod Scenario Architecture 3 Execution Graph : ICAD : Database : open() find(modelid) : Beam : Node SPE ED Results screen TM Drawmod design 3 Time, no contention: 7.964 Initialize 0.412 Solution: View: No contention Residence time Get data 5.440 Contention Rsrc demand < 2.81 < 5.21 < 7.60 Resource Usage 0.003 CPU retrieveblock() System model Adv sys model Utilization SWmod specs Each beam 5.908 DEVs 2.052 Display find(nodes) Simulation data Pie chart retrieveblock(nodes) match() Solve Display: Software mod OK Specify: Names Match and draw 2.051 Results Values draw(point1) draw(point2) Overhead System model Overhead Service level Close DB 0.060 draw(point1, point2) SPE database Sysmod globals Save Scenario Open scenario close() 63 64 Performance Results Beyond Simple s Architecture Architecture 1 Architecture 2 Architecture 3 Overall Response Time 992.33 sec 992.27 sec ~8 sec SPE: guidance for conducting steps in process e.g. performance walkthroughs, techniques for conducting studies, etc. Advanced modeling techniques Distributed and web-based system models Real-time, embedded computer systems Principles, Patterns, and Antipatterns Techniques for performance-oriented design Tools for SPE 65 66

1-12 Summary Significance of early assessment of software performance Software Performance Engineering (SPE) Review Case Study: Use of SPE to perform early assessments of software architectures Additional facets of SPE 67 Questions? cusmith@spe-ed.com http://www.spe-ed.com/ 68