Introduction to Software Performance Engineering

Transcription

1 1-1 Overview 1. Overview of SPE 1. Software Performance Engineering 2. Distributed System s 3. Interoperability 4. SPE Research and Education Connie U. Smith, Ph.D. PO Box 2640 Santa Fe, New Mexico (505) Copyright , Performance Engineering Services. All rights 2 1. Overview of SPE Performance Balance Software Performance Engineering (SPE) Introduction SPE Process Workload New Software Hardware Configuration CAPACITY Data Required s Case Study RESOURCE REQUIREMENTS Quantitative Assessment Begins early, frequency matches system criticality Often find architecture & design alternatives with lower resource requirements Select cost-effective performance solutions early 3 4 SPE Focus Goal Performance: a measure of how well a software system or component meets its requirements for timeliness responsiveness scalability Software-oriented approach focuses on architecture, design and implementation choices Early assessment of software decisions to determine their impact on quality attributes such as performance reliability reusability maintainability/modifiability Early assessment of the architecture is important because architecture has the most significant influence on quality attributes architectural decisions are the most difficult to change 5 6

2 1-2 SPE Process Includes Part 2: Review of SPE -based Approach General Principles for Performance-oriented Design, Design Patterns and Antipatterns Data Gathering Strategy Performance Walkthroughs Construction and Evaluation Strategy Techniques to Compensate for Uncertainties Data Presentation & Tracking Verification & Validation 7 8 SPE ing Steps assess performance risk identify critical use cases SPE ing Steps 1. Assess performance risk select key performance scenarios 2. Identify critical use cases establish performance requirements 3. Select key performance scenarios construct performance model(s) 4. Establish performance requirements verify and validate models add software resource requirements add computer resource requirements modify/ create scenarios 5. Construct performance models 6. Determine software resource requirements evaluate performance model(s) [performance acceptable] [feasible] [infeasible] modify product concept revise performance requirements 7. Add computer resource requirements 8. Evaluate the models 9. Verify and validate the models 10 Performance Scenarios Gather Data Workload identification and characterization Benchmark scenarios Performance Goals Precise performance requirements Response time Throughput Resource budgets Resource Usage Estimates Execution Environment Workload Software/ Database 11 12

3 1-3 Performance s SPE Requirements Conventional s Existing Existing Work Work System Execution Performance Metrics Software Prediction s Existing Work System Execution Performance Metrics Software Execution Low overhead use the simplest possible model that identifies problems Accommodate: incomplete definitions imprecise performance specifications changes and evolution Goals: initially distinguish between "good" and "bad" later, increase precision of predictions provide decision support SPE ing Strategies Cost / Benefit Analysis Simple- Strategy start with the simplest possible model that identifies problems with the system architecture, design or implementation plans. Schedule impact Modification Effort Side Effects Predicted Improvement Best- and Worst-Case Strategy use best- and worst-case estimates of resource requirements to establish bounds on expected performance and manage uncertainty in estimates Adapt-to-Precision Strategy match the details represented in the models to your knowledge of the software processing details 15 Risk If alternatives are unacceptable, revise the performance requirements Best alternative may not have the best performance 16 V & V SPE -Based Approach parameters verify workload? System results products validate Verify the model data, validate the results Begin early in the life cycle and continue throughout life cycle Check for model omissions!?? 17 Reduces the risk of performance failures Provides: Refinement and clarification of performance requirements Predictions of performance Estimates of sensitivity to precision of resource usage estimates and workload intensity Quantitative impact of software alternatives Scalability of the architecture and design Identification of critical parts of the system Identification of assumptions that may change results Assistance for budgeting resource demands Assistance for designing performance tests 18

4 1-4 Workload Data Part 2: Data Required Pareto principle ( rule ) More than 80% of the software requests will be for less than 20% of the functions of the system First: scenarios of typical activity Number of concurrent users Request arrival rates Performance requirements Later, add large scenarios, critical scenarios, etc Example: ATM System Software Specifications System Functions: Execution paths for scenarios of interest Get balance Checking Savings Make payment From checking From savings In envelope Withdrawal Checking Savings Credit card Deposit Savings Checking Objects / methods to be executed probability of execution number of repetitions protocol Database accesses, messages sent Level of detail increases as development progresses Scenario? Sequence Diagrams ATM Sequence Diagram Sequence diagrams provide a tabular notation for objects (those of interest in the scenario) messages between objects events operation invocation Messages are temporally ordered : User : ATM : HostBank cardinserted requestpin apin *[until done] requesttransaction response alt [type] processdeposit Often created by object-oriented developers during design processwithdrawal processbalanceinquiry terminatesession 23 24

5 1-5 Example (continued) Environment Processing scenario: Request withdrawal 1. Initiate 2. Get and interpret request {response = withdrawal, checking acct} 3. Trans Authorize (Withdrawal) 4. Dispense cash 5. Print receipt 6. Terminate Performance requirement: Response time secs. Workload intensity, e.g., number of arrivals per hr. per ATM Platform and network characteristics: configuration device service rates Software overhead, e.g., database path lengths, communication overhead, etc. External factors, e.g., peak hours, bulk arrivals, batch windows, scheduling policies Case study Environment Specifications: time for ATM communication CPU speed system configuration (devices, service times, etc.) Resource Usage Example Specifications CPU Work Units or # of instructions executed or measurements of similar software I/O Database calls Communication Other devices Software overhead calls & characteristics Component K Instr Disk I/O s ATM Screens initiate get & interpret request withdrawal print balance terminate dispense cash System Versus Software ing Tools Part 3: SPE -based Approach Existing Work New Work Software Performance Execution Graphs (may be spreadsheet, pseudo-code, etc. views of EGs) Performance Metrics System Execution Queueing Networks 29 System Requires more modeling expertise Device usage, overall response time and throughput Useful to evaluate hardware changes Software Requires less modeling expertise Time and resource requirements of processing steps and overall Useful to evaluate software alternatives A combination is best. 30

6 Start with Sequence Diagram 2. Software Performance s : User : ATM : HostBank cardinserted requestpin apin *[until done] requesttransaction response alt [type] processdeposit processwithdrawal processbalanceinquiry terminatesession 31 Create from sequence diagrams Specify software resource requirements Specify computer resource requirements (e.g., path lengths) Results point to problem areas getaccount getamount request Authorization dispensecash waitfor Customer confirm Transaction Screen 1 Host 0 Log 0 Delay 0 Screen 1 Host 0 Log 0 Delay 0 Screen 0 Host 1 Log 1 Delay 0 Screen 1 Host 0 Log 1 Delay 5 Screen 0 Host 0 Log 0 Delay 10 Screen 0 Host 1 Log 1 Delay Computer Resource Requirements Overhead Matrix: Computer Resource Requirements summary WorkUnit 12 DB 2 Msgs 2 CPU Disk Delay Network Devices Devices Service Units CPU Disk Display Delay Net Sec. Phys. I/O Screens Units Msgs. Ties software resource requirements to devices and service required in computer environment Screen Host Log Delay Service Time Software s: Purpose Execution Graphs Derive software execution characteristics Initial check of performance requirements Provide workload parameters for the system execution model Visual representation of software confirmation Presentation of results Formal basis for analysis Aids in comprehension Specification for tool development 35 36

7 1993 Performance Engineering Services Introduction to Software Performance Engineering 1-7 Notation Typical Structures Basic Node Processing step at current level of detail. Initiate Expanded Node The step has more details. Details are in the associated subgraph. 2 Process request Get request.7 Withdrawal Repetition Node n The node in the is repeated n times. It may be an expanded node. Terminate Interpret request.3 Get balance Case Node Attached nodes are conditionally executed. A probability is associated with each Example: Specific scenario Example: Generic Scenario Withdrawal scenario: Initiate Read ID# from card Initiate Get and interpret request Read PIN 2 Withdraw Dispense cash,print receipt Validate User get request Trans type? Get checking balance Withdraw from checking Terminate Terminate Make payment Graph Analysis 4. Software Results Best case Worst case Average Variance Distribution SPE ED TM Results screen Solution: View: No contention Residence time Contention Rsrc demand System model Utilization Adv sys model SWmod specs Simulation data Pie chart Solve OK Display: Specify: Software mod Names ATM example Resource utilization: CPU 0.00 Get cust ID ATM 0.03 from card DEV CPU 0.00 Get PIN ATM 0.03 from DEV customer Each transaction CPU 0.15 Get type ATM 0.48 and DEV process < < < < totals 0.15 CPU 0.88 ATM 1.00 DEV1 Results Values CPU 0.00 Overhead System model Overhead Service level Terminate ATM 0.35 DEV SPE database Sysmod globals Save Scenario Open scenario 41 42

8 ATM example Residence Time: Get cust ID Get PIN from Get type Termina te Each < < < < Resource Usage CPU ATM DEV1 ATM example Resource utilization: CP 0.00 Get cust ID Get PIN from Get type Termina te DE 0.00 Each CP 0.12 AT 0.41 DE 0.43 CP 0.00 AT 0.30 DE 0.00 < < < < < < < < totals 0.12 CPU 0.75 ATM 0.43 DEV1 Introduction to Software Performance Engineering System Performance 6. System Performance Results System execution model quantifies contention effects Workload intensity Device visits Device service rates OOD Example ATM exampl e Utilization: Utilization: ATM Host Bank AT 0.02 DE 0.00 CP 0.00 AT 0.02 Response times Throughput Utilizations Residence times Queue lengths ICAD Part 3: Case Study Computer-Aided Design (CAD) application Features interactively construct, view, and analyze drawings that model structures (e.g., aircraft wings) model stored in database several versions of a model may exist in the database ICAD DrawMod Scenario Each models consists of nodes defined by position (x, y, z) contain additional information needed for model solution elements (connect two nodes) triangles (connect three nodes) plates (connect four or more nodes) Typical model contains only nodes and consists of 2050 Focus on the scenario in which a typical model is drawn on the user s screen (DrawMod) : ICAD : : Database open() find(modelid) retrieve(, nodes) Performance requirement draw typical model in 10 sec or less close() 47 48

9 1993 Performance Engineering Services Introduction to Software Performance Engineering 1-9 Architecture 1 Architecture 1 Class Diagram Uses an object for each node and beam modelid : int 1..n Element elementno : int Node 2 Beam Triangle Plate nodeno : int x : int y : int z : int 3 node1 : int node2 : int node1 : int node2 : int node3 : int nodes[] : int 4..n Architecture 1 Drawmod Scenario Process and Physical Views : ICAD : Database : open() find(modelid) find(modelid, ) «process» : GUI «process» «processor» Workstation «database» DBMS sort() retrieve(beam) *[each beam] : DBMS GUI find(modelid, node1, node2) : Beam «process» : ICAD ICAD retrieve(node1) retrieve(node2) : Node : Node Process View Physical View close() Architecture 1 Execution Graph Architecture 1 Performance Results SPE ED TM Drawmod Software model Drawmod Initialize Add Find Beams Drawmod Time, no contention: Initializ e < 2.70 < 5.14 < 7.57 Resource Usage Find Beams CPU DEVs Link Expand Sort Display Insert node Sort Solve Display: OK Specify: Software mod Names Results Values Overhead Overhead Draw System model Service level SPE database Sysmod globals Draw beam Save Scenario Open scenario Finish Finish

10 Introduction to Software Performance Engineering 1-10 Performance Results Architecture 2 Architecture Architecture 1 Overall Response Time sec Uses Flyweight pattern uses a shared object for and nodes rather than individual objects reduces run-time overhead due to constructor invocations Architecture 2 Class Diagram Architecture 2 Drawmod Scenario : ICAD : Database modelid : int 1..n Element elementno : int : open() : Beam : Node : Node find(modelid) find(modelid, ) 1..n sort() Node Beam Triangle Plate retrieve(beam) *[each beam] nodeno : int x : int y : int z : int node1 : int node2 : int node1 : int node2 : int node3 : int nodes[] : int find(modelid, node1, node2) retrieve(node1) retrieve(node2) close() Architecture 2 Performance Results Performance Results Drawmod rev1 Time, no contention: < 2.70 Initialize < 5.14 < 7.57 Architecture Overall Response Time Find Beams Resource Usage CPU DEVs Display Architecture 1 Architecture sec sec Sort Draw beam Finish

11 Drawmod Initializ e Find Beams Sort Draw beam Finish Drawmod Initializ e Find Beams Sort Draw beam Finish DE Dis CP DE CP DE CP DE Dis DE < 2.70 < 5.14 < 7.57 Resource Usage CPU DEVs Display < 2.50 < 5.00 < 7.50 totals CPU DEVs Display Draw beam Retriev e beam New beam Find nodes Set up node Draw Draw beam Retriev e beam New beam Find nodes Set up node Draw Each node DE DE DE Each node DE DE Dis < 2.59 < 5.06 < 7.53 < 2.50 < 5.00 < 7.50 Submodel totals CPU DEVs Display 1993 Performance Engineering Services Introduction to Software Performance Engineering 1-11 Architecture 1 Performance Results Architecture 3 Time, no contention: Resource demand: Time, no contention: Resource demand: Retains use of Flyweight pattern Modifies database management system new operation retrieveblock() retrieves block of data rather than individual nodes and single block retrieve get 20k 64, or 170 nodes reduces database accesses to 33 to retrieve, plus 9 to retrieve nodes Architecture 3 Drawmod Scenario Architecture 3 Execution Graph : ICAD : Database : open() find(modelid) : Beam : Node SPE ED Results screen TM Drawmod design 3 Time, no contention: Initialize Solution: View: No contention Residence time Get data Contention Rsrc demand < 2.81 < 5.21 < 7.60 Resource Usage CPU retrieveblock() System model Adv sys model Utilization SWmod specs Each beam DEVs Display find(nodes) Simulation data Pie chart retrieveblock(nodes) match() Solve Display: Software mod OK Specify: Names Match and draw Results Values draw(point1) draw(point2) Overhead System model Overhead Service level Close DB draw(point1, point2) SPE database Sysmod globals Save Scenario Open scenario close() Performance Results Beyond Simple s Architecture Architecture 1 Architecture 2 Architecture 3 Overall Response Time sec sec ~8 sec SPE: guidance for conducting steps in process e.g. performance walkthroughs, techniques for conducting studies, etc. Advanced modeling techniques Distributed and web-based system models Real-time, embedded computer systems Principles, Patterns, and Antipatterns Techniques for performance-oriented design Tools for SPE 65 66

12 1-12 Summary Significance of early assessment of software performance Software Performance Engineering (SPE) Review Case Study: Use of SPE to perform early assessments of software architectures Additional facets of SPE 67 Questions? 68