Outline. Determine Confidence Interval. EEC 686/785 Modeling & Performance Evaluation of Computer Systems. Confidence Interval for The Mean.

Transcription

1 EEC 686/785 Modelig & Performace Evaluatio of Computer Systems Lecture 9 Departmet of Electrical ad Computer Egieerig Clevelad State Uiversity webig@ieee.org (based o Dr. Raj jai s lecture otes) Outlie Review of lecture 8 Simple liear regressio models Review for midterm # Cofidece Iterval for The Mea 3 Determie Cofidece Iterval 4 k samples => k sample meas => ca t get a perfect estimate of μ from ay fiite umber of fiite size samples => use probabilistic bouds c ad c : Probability{c μ c } = -α Cofidece iterval for the populatio mea: (c, c ) Sigificace level: α Cofidece level: 00(-α) Cofidece coefficiet: -α 00(-α)% cofidece iterval for the populatio mea for large sample ( >= 30): ( x z α/s/, x + z α/s/ ) z = ( α / )-quatile of N(0,) α / 00(-α)% cofidece iterval for < 30: ( x t[ α/ ; ] s/, x + t[ α/ ; ] s/ ) t[ α / ; ] = ( α / )-quatile of a t-variate with degree of freedom

2 Comparig Two Alteratives: Paired vs. Upaired 5 Approximate Visual Test 6 Paired: oe-to-oe correspodece betwee the ith test o system B Example: performace o ith workload => use cofidece iterval of the differece Upaired: o correspodece Example: people o system A, o system B => eed more sophisticated method To compare two upaired samples, simply compute the cofidece iterval for each alterative separately Cofidece Itervals for Proportios 7 Sample Size for Determiig Mea 8 Proportio = probability of various categories E.g., P(error)=0.0, P(o error)=0.99 for observatios are of type => Sample proportio = p = / Cofidece iterval for the proportio Assumes ormal approximatio of Biomial distributio => Valid oly if p 0 Need to use biomial tables if p < 0, ca t use t-values p z α / p( p) Larger sample => Narrower cofidece iterval => Higher cofidece Questio: How may observatios to get a accuracy of ±r% ad a cofidece level of 00(-α)%? The 00(-α)% cofidece iterval of the populatio mea is s x z r% accuracy => CI = ( x( r /00), x( + r /00)) => s r s r 00zs x z = x z = x = rx

3 Regressio Models 9 Defiitio of a Good Model 0 A regressio model allows oe to estimate or predict a radom variable as a fuctio of several other variables Respose variable: the estimated variable Predictor variables, predictors, or factors: the variables used to predict the respose Regressio aalysis assumes that all predictors variables are quatitative so that arithmetic operatios are meaigful Simple liear regressio mode: liear regressio model with a sigle predictor variable Regressio models should miimize the distace measured vertically betwee the observatio poit ad the model lie The differece betwee the observed respose ad the predicted respose is called residual, modelig error, or simply error Defiitio of a Good Model Defiitio of a Good Model Two requiremets Zero overall error, i.e., egative ad positive errors cacel out To choose the lie that miimizes the sum of squares of the errors Liear model: ŷ is the predicted respose whe the predictor variable is x b 0 ad b are fixed regressio parameters to be determied from the data Give observatio pairs {(x,y ),,(x,y )}, the estimated respose ŷ i for the ith observatio is The error is: ŷ = b0 + bx y = b + bx ˆi 0 e = y yˆ i i i i 3

4 Defiitio of a Good Model The best liear model is give by the regressio parameter values, which miimizes the Sum of Squared Errors (SSE): e = ( y b bx ) i i 0 i i= i= Subject to the costrait that the mea error is zero i i 0 i i= i= This costraied miimizatio problem is equivalet to miimizig the variace of errors e = ( y b bx ) = 0 3 Estimatio of Model Parameters The regressio parameters that give miimum error variace are xy x y b = b 0 = y bx x x where x = mea of the values of precitor variables = y= mea respose = i i= i i i i= i= xy = x y x = x y xi i= 4 Derivatio 5 Derivatio 6 Error i the ith observatio: Mea error: e = e i = { y i ( b 0 + bx i)} = y b 0 bx i i Settig mea error to 0 => b0 = y bx Substitutig b 0 => e = y yˆ = y ( b + bx ) i i i i 0 i e = y y+ bx bx = ( y y) b( x x) i i i i i Sum of squared errors: SSE = e = [( y y) b ( y y)( x x) + b ( x x) ] i i i i i i= i= SSE = + Fid the value of b that gives mi SSE: => [( yi y) ] b ( yi y)( xi x) b ( xi x) i= i= i= = s bs + b s y xy x dsse ( ) = sxy + bs x = 0 db s xy xy xy = = sx x ( x) b 4

5 Example 4. 7 Example 4. 8 Number of disk I/O s ad processor times of 7 programs were measured as {(4,), (6,5), (7,7), (4,9), (39, 0), (50, 3), (83, 0)} Liear model to predict CPU time as a fuctio of disk I/O s, give = 7, xy = 3375, x = 7, x =3,855, y = 66, y = 88, =38.7, ad = 9.43 xy x y b = = = x x 3,855 7 (38.7) b = y bx = = The liear model: CPU time = (umber of disk I/O s) Example 4. 9 Allocatio of Variatio 0 The purpose of a model is to be able to predict the respose with miimum variability Without a regressio model, oe ca use the mea respose as the predicted value for all values of the predictor variables 5

6 Allocatio of Variatio The errors i this case would be larger tha those with the regressio model, i.e., the error variace = variace of the respose Error = ε i = observed respose predicted respose = yi y Variace of errors without regressio = εi i= = ( yi y) i= = variace of y Allocatio of Variatio yi Total sum of squares (SST): It is a measure of y s variability ad it is called the variatio of y SST = ( y y) = ( y ) y = SSY SS0 i i= i= SSR: sum of squares explaied by the regressio, it is the differece betwee SST ad SSE: SSR = SST SSE or SST = SSR + SSE i i= ( y) Allocatio of Variatio 3 Allocatio of Variatio 4 Coefficiet of determiatio R : the fractio of the variatio that is explaied determies the goodess of the regressio SSR SST SSE Coefficiet of determiatio = R = = SST SST The higher the value of R, the better the regressio R is also the square of the sample correlatio R xy betwee the two variables sxy Sample correlatio( xy, ) = Rxy = sxsy Coefficiet of determiatio = {correlatio coefficiet( xy, )} Shortcut formula for SSE: SSE = y b y b xy 0 Example 4.: for the disk I/O-CPU time data of ex 4.. Calculate the coefficiet of determiatio SSE = y b0 y b xy = = 5.97 SST = SSY SS0 = y ( y) = 88 7 (9.43) = 05.7 SSR = SST SSE = = SSR R = = = SST

7 Stadard Deviatio of Errors Mea squared error (MSE): a estimate of the variace of errors = SSE divided by its degree of freedom: SSE se = Degree of freedom for SSE is because errors are obtaied after calculatig two regressio parameters from the data Stad deviatio of errors: square root of MSE 5 Stadard Deviatio of Errors Degree of freedom for SSE for SSY sice it is obtaied from idepedet observatios without estimatig ay parameters for SS0 sice it ca be computed simply from y for SST sice oe parameter y must be calculated from the data before SST ca be computed for SSR Thus the degrees of freedom add just the way the sums of squares do: SST = SSY SS0 = SSR + SSE = = + ( ) 6 Example Cofidece Itervals for Regressio Parameters 8 For the disk I/O-CPU data of example 4., the degrees of freedom of the sums are: Sums of squares: SST = SSY SS0 = SSR + SSE 05.7 = = Degrees of freedom: 6 = 7 = + 5 The mea squared error is: SSE 5.87 MSE = = =.7 degree of freedom for errors 5 The stadard deviatio of errors is: s = MSE =.7 =.08 e The regressio coefficiets are radom i the same maer as the sample mea or ay other parameter computed from a sample Usig a sigle sample, oly probabilistic statemets ca be made about true parameters β 0 ad β of the populatio => true model is y = β0 + βx b 0 ad b are statistics (mea values) correspodig to the parameters β 0 ad β, respectively 7

8 Cofidece Itervals for Regressio Parameters 9 Cofidece Itervals for Regressio Parameters 30 The stadard deviatio of b 0 ad b s b0 s b x = se + x x s e = / x x / The 00(-α)% cofidece itervals for b 0 ad b ca be computed usig the -α/ quatile of a t variate with - degrees of freedom, i.e., t [-α/;-] The cofidece itervals are: b ts ad b ts 0 b 0 If cofidece iterval icludes 0, the the regressio parameter caot be cosidered differet from zero at the 00(-α)% cofidece level b Example Example s For the disk I/O-CPU data of example 4., we ca calculate the stadard deviatios of b 0 ad b : b0 s b / / x (38.7) = se + = = 0.83 x x 7 3,855 7 (38.7) se.0834 = = = / / x x 3,855 7 (38.7) From table A.4, the 0.95-quatile of a t-variate with 5 degrees of freedom is.05 => 90% cofidece iterval for b 0 is (.05)(0.83) = = (.6830,.6663) Cofidece iterval icludes 0 => b 0 is essetially 0 Similarly, 90% cofidece iterval for b is (.05)(0.087) = = (0.06,0.84) Cofidece iterval does ot iclude 0 => the slope b is sigificatly differet from 0 at this cofidece level 8

9 Case Study Case Study Performace compariso of RPC mechaism o UNIX ad ARGUS UNIX data ARGUS data The liear models are: Time o UNIX = (data size i bytes) + 4 Time o ARGUS = (data size i bytes) + 30 Case Study Cofidece Itervals for Predicatios 36 Cofidece itervals for itercepts overlap while those of the slopes do ot Setup times are ot sigificatly differet Per-byte times (slopes) are differet Developig regressio helps predict the value of the respose variable for those values of predictor variables beyod the curretly measured Give the regressio equatio, we ca predict the respose for ay give value of predictor variable: yˆ p = b0 + bx p This formula gives oly the mea value of the predicted respose based upo the sample 9

10 Cofidece Itervals for Predicatios Stadard deviatio of the mea of a future sample of m observatios is / ( xp x) syˆ = s mp e + + m x x Stadard deviatio of a sigle future observatio (m=) / ( xp x) syˆ = s mp e + + x x Stadard deviatio of the mea of a large umber of future observatios at x p (m= ): / ( xp x) syˆ = s mp e + x x 37 Cofidece Itervals for Predicatios Cofidece itervals for predicatios from regressio models Stadard deviatio of the predictio is miimal at the ceter of the measured rage 38 Example Example Agai, usig the disk I/O-CPU time data of example 4., estimate the CPU time for a program with 00 disk I/O s Regressio equatio: CPU time = (umber of disk I/O's) Therefore, for a program with 00 disk I/O s, the mea CPU time is: CPU time = (00)= Stadard deviatio of errors =.0834 s e Stadard deviatio of the predicted mea of a large umber of observatios is: ( ) = = ,855 7(38.7) s yp ˆ 90% cofidece iterval for predicted mea = ±(.05)(.59)=4.3674±.4500=(.974,6.874) / 0

11 Example Visual Tests for Verifyig the Regressio Assumptios 4 Stadard deviatio of the predicted CPU time of a sigle future program with 00 disk I/O s: / ( ) s y ˆ p = = ,855 7(38.7) 90% cofidece iterval for predicted mea = ±(.05)(.686)=4.3674±3.86=(.0858,7.6489) => cofidece iterval is wider tha that for the mea of a large umber of observatios The true relatioship betwee the respose variable y ad the predictor variable x is liear The predictor variable x is ostochastic ad it is measured without ay error The model errors are statistically idepedet The errors are ormally distributed with 0 mea ad a costat stadard deviatio Visual Tests for Verifyig the Regressio Assumptios 43 Visual Tests for Verifyig the Regressio Assumptios 44 Liear relatioship: prepare a scatter plot of y versus x Ay oliear relatioship ca be easily see from this plot Idepedet errors: after the regressio, compute errors ad prepare a scatter plot of ε i vs. the predicted respose ŷ i Ay visible treds i the plot would idicate a depedece of errors o the predictor variable

12 Visual Tests for Verifyig the Regressio Assumptios 45 Visual Tests for Verifyig the Regressio Assumptios 46 Idepedet errors: additioally, you ca plot the residuals as a fuctio of the experimet umber Ay tred up or dow idicate the presece of other factors, evirometal coditios, or side effects that varied i differet experimets Normally distributed errors: prepare a ormal quatilequatile plot of errors If the plot is approximately liear, the assumptio is satisfied Visual Tests for Verifyig the Regressio Assumptios 47 Example Costat stadard deviatio of errors: also kow as homoscedasticity. Observe the scatter plot of errors vs. predicted respose prepared for the idepedece test If the spread i oe part of the graph seems sigificatly differet tha that i other parts, the the assumptio is ot valid For the same data i example 4., check assumptios: Idepedece check: ok

13 49 50 Example 4.6 Example 4.6 Check for ormality assumptio: ok Check homoscedasticity: some tred towards lower values of ŷ. However, the magitude is small 5 5 Example 4.7 Example 4.7 For the RPC performace study i case study 4. The residual-vs-ŷ plot for the ARGUS data shows a higher spread o the rhs of the graph tha that o the left side => ca be a cocer The ormal quatile-quatile plot for the same residuals => departure from ormality is higher tha that i previous example 3

14 A Systematic Approach to Performace Evaluatio 53 Commoly Used Performace Metrics 54. State goals ad defie the system. List services ad outcomes 3. Select metrics 4. List parameters 5. Select factors to study 6. Select evaluatio techique 7. Select workload 8. Desig experimets 9. Aalyze ad iterpret data 0. Preset results Respose time, Reactio time Turaroud time Stretch factor Throughput Capacity (omial, typical, kee) Efficiecy Utilizatio Reliability, availability Selectig Evaluatio Techiques 55 Selectig Metrics 56 The life-cycle stage is the key Other cosideratios are: Time available, tools available, Accuracy required, Trade-offs to be evaluated, Cost, Salability of results Iclude: Performace: Time: resposiveess Rate: throughput or productivity Resource: utilizatio Error rate, probability Time to failure ad duratio Cosider icludig: Mea ad variace Idividual ad global Selectio criteria: Low-variability No-redudacy completeess 4

15 Types of Work Load 57 The Art of Workload Selectio 58 Test workload Real workload Sythetic workload Test workload used Additio istructio Istructio mixes Kerels Sythetic programs Applicatio bechmarks Services exercised SUT, CUT Metrics ad workload deped o the system The requests at the service-iterface level of the SUT should be used to specify or measure the workload Level of detail Most frequet request Frequecy of request types Time-stamped sequece of requests Average resource demad Distributio of resource demads Timeliess; Loadig level; Impact of other compoets; Repeatability Moitors 59 Moitor Termiology 60 Moitor: a tool used to observe the activities o a system Usage: To improve software performace. Fid frequetly used segmets of the software To measure resource utilizatios ad to fid the performace bottleeck To tue the system To characterize the workload To fid model parameters, to validate models, ad to develop iputs for models Evet Trace Overhead Domai Iput rate Resolutio Iput width 5

16 Moitor Classificatio By implemetatio Software moitor Hardware moitor Firmware moitor Hybrid moitor By trigger mechaism Evet-drive: good for rare evets Timer-drive (samplig moitor): good for frequet evets By result display ability Olie moitors: display the system state cotiuously Batch moitors: collect the data that ca be aalyzed later 6 Two Special Software Moitors Program-Executio Moitor: Software moitors desiged to observe applicatio software Accoutig Logs: resource usage Per activatio: average resource cosumptio per activatio of the program Percetage of total: resource cosumed by all activatios of a particular program, expressed as a percetage of the resources cosumed by all activatios of all programs Per-secod resource-cosumptio rates: divide the resource cosumptio by the elapsed time Per-CPU-secod resource cosumptio: divide the resource cosumptio by the CPU time cosumed 6 Termiology Capacity plaig: esurig that adequate computer resources will be available to meet the future workload demads i a cost-effective maer while meetig the performace objectives Capacity maagemet: esurig that the curretly available computig resources are used to provide the highest performace Capacity maagemet is cocered with the preset while capacity plaig is cocered with the future Bechmarkig: to compare the performace of two competig systems i a objective maer, bechmarks are ru o these systems usig automatic load drivers 63 Guidelies for Preparig Good Graphic Charts Require miimum effort from the reader Maximize iformatio: there should be sufficiet iformatio o the graph to make it self-sufficiet Miimize ik: preset as much iformatio as possible with as little ik as possible. Too much uecessary ifo o a chart makes it cluttered ad uiterestig Use commoly accepted practices: preset what people expect Avoid ambiguity: show coordiate axes, scale divisio, ad origi. Idetify idividual curves ad bars. Do ot plot multiple variables i the same chart 64 6

17 65 66 Gatt Charts Kiviat Graphs Gatt chart ca be used to show the relative duratio of ay umber of Boolea coditios, i.e., coditios that are either true or false A resource beig used or beig idle is a example of a Boolea coditio Each coditio is show to be a set of horizotal lie segmets The total legth of the lie segmets represets the relative duratio of the coditio The positio of various segmets is arraged such that the overlap betwee differet lies represets the overlap betwee the coditios Kiviat graph: a circular graph i which several differet performace metrics are plotted alog radial lies I the most popular versio of the graph, a eve umber of metrics are used. Half of these metrics are HB metrics so that a higher value of the metrics is cosidered better. The other half of the metrics measure are LB metrics, ad a lower value is cosidered better Kiviat graph for a ideal system is star Ratio Games 67 Strategies for Wiig a Ratio Game 68 Ratios provide good opportuities for playig performace games with competitors Ratios have a umerator ad deomiator (base) Two ratios with differet bases are ot comparable However, may examples i published literature where computer scietists have kowigly or ukowigly compared ratios with differet bases Ratio game: the techique of usig ratios with icomparable bases ad combiig them to oe s advatage It is better to use your oppoet s system as base for HB metric 7

18 Basic Probability ad Statistics Cocepts 69 Basic Probability ad Statistics Cocepts 70 Idepedet evets Radom variables Cumulative Distributed Fuctio (CDF): F x (a) = P(x a) Probability Desity Fuctio: f(x) = df(x)/dx Probability Mass Fuctio: f(x i ) = p i + Mea or Expected Value μ = E( x) = p x = xf ( x dx Variace (σ ): Stadard Deviatio: σ Var x E x μ ( ) = [( ) ] i= i i ) Coefficiet of Variatio (C.O.V.): σ/μ Covariace: Cov(x,y) = σ xy = E[(x-μ x )(y-μ y )] = E(xy) E(x)E(y) Correlatio Coefficiet: ρ xy = σ xy/σ x σ y Quatile: P(x x α ) = F(x α ) = α Medium: the 50-percetile Mode ( x μ ) Normal distributio: f ( x) = e / σ, x + σ π Selectig amog the Mea, Media ad Mode 7 Idices of Dispersio 7 Idices of dispersio: variability is specified usig oe of the followig measures Rage miimum ad maximum of the values observed Variace or stadard deviatio 0- ad 90- percetiles Semi-iterquatile rage (SIQR) Mea absolute deviatio 8

19 Selectig the Idex of Dispersio 73 9