COMPARISON OF SAMPLING TECHNIQUES ON THE PERFORMANCE OF MONTE- CARLO BASED SENSITIVITY ANALYSIS. Iain A Macdonald

Transcription

1 COMPARSO OF SAMPLG TECHQUES O THE PERFORMACE OF MOTE- CARLO BASED SESTVTY AALYSS in A Mcdonld tionl Reserch Council Cnd nstitute for Reserch in Construction Ottw, Cnd irc.nrc-cnrc.gc.c Contct: iin.mcdonld@nrc-cnrc.gc.c Eleventh nterntionl BPSA Conference Glsgow, Scotlnd July 27-30, 2009 ABSTRACT Sensitivity nlysis is key prt of comprehensive energy simultion study. Monte-Crlo techniques hve been successfully pplied to mny simultion tools. Severl smpling techniques hve been proposed in the literture; however to dte there hs been no comprison of their performnce for typicl building simultion pplictions. This pper exmines the performnce of simple rndom, strtified nd Ltin Hypercube smpling when pplied to typicl building simultion problem. An integrted nturl ventiltion problem ws selected s it hs n inexpensive clcultion time thus llowing multiple sensitivity nlyses to be undertken, while being relistic s wind nd temperture effects re both modeled. The reserch shows tht compred to simple rndom smpling: LHS nd strtified smpling produce results tht re not significntly different (t 5% level) with incresed robustness (less vrince in the men prediction). However, it should not be inferred from this tht fewer simultion runs re required for LHS nd strtified smpling. Given the results presented here nd in previous work it would indicte tht for prcticl purposes Monte-Crlo uncertinty nlysis in typicl building simultion pplictions should use bout 00 runs nd simple rndom smpling. TRODUCTO The field of sensitivity nlysis is becoming more commonplce in building simultion. Erly work by Loms nd Eppel (992) compred the performnce of three techniques (differentil, Monte-Crlo nd stochstic sensitivity nlysis). Following on from this Mcdonld (2002) embedded sensitivity nlysis in ESP-r nd de Wit (200) pplied the Monte-Crlo technique to the nlysis of nturl ventiltion nd therml comfort. Since then severl uthors hve used the Monte-Crlo technique on diverse rnge of building simultion pplictions (for exmple: Hyun et l 2007 nd Kotek et l 2007). To dte the effect of smpling technique on the results hs not been nlysed in the bove publictions. t is known tht strtified smpling cn introduce n unknown bis into the results of the nlysis (discussed briefly by Mcdonld (2002) nd de Wit (200)) nd tht there cn be vrying degrees of success with Ltin Hypercube (Sltelli et l 2000). This pper will exmine three smpling techniques (simple rndom, strtified nd Ltin hypercube) nd compre their performnce. Smpling techniques t is stndrd sttisticl procedure to use smpling techniques to improve the coverge of the smple, especilly when the function being nlysed is expensive. The im of smpling strtegy is to reduce the vrince in the estimte of the men. When pplied to building simultion this is ttrctive s the evlution of the simultion results (e.g. building energy consumption) cn be costly, therefore ny reduction in the number of simultions required for Monte-Crlo nlysis will result in reduction in computtionl effort. Three smpling techniques will be described: simple rndom smpling. Strtified smpling nd Ltin Hypercube smpling (LHS). Simple rndom smpling This is the most bsic smpling technique described here nd will be used s the bsis for comprisons. The method works by generting rndom number nd scling this to the trget vrible vi its probbility distribution. The method conforms to the lws of sttistics. The men of the estimte is: simple i i () where there re smples nd i is the model output. The smple vrince is: 2 Vr ( simple ) ( i simple ) (2) i nd is n unbised estimte of the popultion vrince. Finlly, the vrince in the estimte of the men is: Vr ( simple ) Vr( simple ) (3)

2 Air speed (m/s) Air speed (m/s) Figure. Simple rndom smpling 00 input vlues. Strtified smpling This method represents n improvement over simple rndom smpling by forcing the smple to conform to the whole distribution being nlysed. To chieve this, the probbility distribution of the trget vrible is typiclly divided into severl strt of equl probbility, one vlue is then chosen t rndom within ech strtum. From Helton nd Dvies (2003) the men of the estimte is: strtified L i i i i i (4) where there re L strt ech with one output (i.e. =L smples in totl), w i is the frctionl weight of the totl popultion covered by strtum L nd i re model outputs. The estimte of the vrince is: 2 Vr ( strtified ) ( i strtified ) (5) i Finlly, the vrince in the estimte of the men is: Vr Simple rndom smpling - 00 input vlues ( strtified ) Vr( simple ) ( ) 2 i strtified i Temperture (degc) Figure 2. Strtified smpling 00 input vlues. Ltin Hypercube smpling Strtified rndom numbers - 00 input vlues (6) Temperture (degc) This method is n evolution of strtified smpling tht cn outperform simple rndom smpling (Slteilli et l 2000). The method works by dividing the input into strt nd then generting smples so tht the vlue generted for ech prmeter comes from different strtum (Helton nd Dvies 2003). For monotonic function the method hs been shown to be better thn simple rndom nd strtified smpling (McKy et l 979), however for non-monotonic functions bis of unknown size is introduced into the results. The men nd vrince of the estimte re the sme s for simple rndom smpling (Sltelli et l 2000, Helton nd Dvies 2003): (7) LHS i i 2 Vr ( LHS ) ( i LHS ) (8) i However, the vrince in the estimte of the men is (mn 999): Vr( LHS ) Vr( LHS ) Cov ( A, A2 ) (9) where term Cov() is the covrince between the input vribles. Thus for LHS to reduce the vrince in the men the covrince should be negtive. mn (999) quotes results showing tht s the smple size tends to infinity this term is nonpositive. Comprison of techniques Equtions through 9 refer only to the results of ech simultion nd the number of simultions, thus it cn be concluded tht the robustness of the Monte-Crlo method is independent of the number of input prmeters (this is discussed elsewhere, for exmple Loms nd Eppel 992 nd Mcdonld 2002). n the context of this pper the robustness of the method cn be mesured vi the vrince in the men output sttistic (equtions 3, 6 nd 9). This vrince ws estimted from repeted Monte-Crlo nlyses s will be described lter. Employing strtifiction (strtified smpling or LHS) forces the input vrible coverge to be better thn for simple rndom smpling. For instnce compre Figure nd Figure 2; the gridlines represent the 0 strt for ech of the two vribles being smpled. Figure hs severl strt without smple (e.g. the top right corner) nd severl with more thn one smple (e.g. lower right corner). This difference becomes less significnt with lrger smple, Figure 3 (mn 999). t is expected tht the curve for strtified smpling flls between the two curves in Figure 3. The difference fter 00 simultions is 2%. Therefore, it should be expected tht the results should converge s the smple size increses in terms of the men nd vrince in the output. Severl sources, for exmple Mxvl (2009), stte tht LHS is fr superior to simple rndom smpling. However, Slteilli et l (2000) stte tht it is not possible to mke firm conclusions. Additionlly, Hyun et l 2007 stte tht the minimum number of simultions required is function of the number of uncertin prmeters this contrdicts the theory

3 presented bove nd ppers to come from discussion of correltion control by Wyss nd Jorgensen (998). A systemtic test of smpling techniques is therefore justified in the context of building simultion. Figure 3. Smple coverge for simple rndom smpling nd LHS. HYPOTHESES Three hypotheses will be tested:. LHS is more robust thn strtified smpling which in turn is more robust thn simple rndom smpling. Robustness is defined s significntly reduced vrince between repeted nlyses (i.e. Vr( ) will be compred). 2. The bis introduced by strtified nd LHS will be insignificnt (using simple rndom smpling s the reference method). 3. The men nd vrince will not be significntly different between the methods fter 00 simultions. TEST MODEL Severl lgorithms exist to clculte infiltrtion rtes from bsic qulittive nd quntittive descriptive dt for houses (ASHRAE 2005). Recent work (Rerdon 2007) compred the vilbility of dt for Cndin houses nd tht required for vilble models. t concluded tht the single-cell Shw model (Shw 987) ws suitble for most nlyses. For the purpose of this study the model is quick to run nd suitbly complex to be representtive of rnge of building simultion problems. The essence of the Shw model is tht the infiltrtion rtes due to stck pressure nd wind effect re clculted individully nd then combined ccording to superposition of their driving pressures to produce totl infiltrtion rte. The key input prmeters re: wind speed, mbient temperture, internl temperture, neutrl pressure rtio nd dt from blower door test (coefficients C nd n from the curve fit). The model is now elborted. Stck-effect driven infiltrtion is modeled by the eqution below: C h S ) V H n 0.5 ( Tin Tout (0) 0.5 fctor hs the units [m 3 s P n )/(L hr K n )], S = infiltrtion ir chnge rte due to stck effect [c/hr], C = house flow coefficient from curve fit of the lekge test dt [L/(s P n )], V = internl volume of the house including bsement [m 3 ], h = height bove grde of the neutrl pressure level [m], H = height bove grde of the upper ceiling of the house [m], T in = indoor ir bsolute temperture [K], T out = outdoor ir bsolute temperture [K], nd n = house flow exponent from curve fit of the lekge test dt. Shw suggests tht for house without flue h/h = 0.64, nd for house with single 27 mm di. flue h/h = 0.86, bsed on the dt set used to develop this model. A lter study (Rerdon 989) with mesured PLs in lrger number of houses hs provided guide for PL=0.6 for houses without n open flue nd 0.7 with n open flue. The form of the curve fit to the lekge test dt (from fn depressuriztion mesurement of the envelope irtightness of the house, following CGSB 986), tht is used to determine the flow coefficient nd flow exponent is the power lw curve: Q C ) m n ( P () Q m = mesured flow rtes [L/s], nd DP = mesured pressure difference cross envelope [P]. Wind driven infiltrtion is modeled by the eqution below: W C V 2n 0.4 U Exposed (2) C W 7 V n 0. U Shielded (3) 0.4 fctor hs units [(m 3 P n s 2n+ )/(L hr m 2n )], 0.7 fctor hs units [m 3 P n s n+ )/(L hr m n )], W = infiltrtion ir chnge rte due to wind [c/hr], nd U = windspeed mesured t height of 20m on-site [m/s]. The combined infiltrtion due to both stck-effect nd wind is modeled by combining these two component infiltrtion rtes using n-qudrture to effectively superpose the pressures creted by these two physicl

4 Count phenomen, since the two component infiltrtion rtes do not simply dd, due to the non-liner reltionship between driving pressure nd driven flow rte. The combined model eqution is: (4) WS = totl combined infiltrtion ir chnge rte [c/hr], F = n empiricl fctor defined by the following: sml F for 0 0. lrg sml sml F for 0. lrg lrg (5) (6) sml = the smller of the two components S nd W, nd lrg = the lrger of the two components S nd W. METHOD Dt from single rel building in Quebec City, Cnd ws used s the input dt for the test model. The sensitivity nlysis ws undertken for rnge of mbient tempertures (8-22 m/s) nd wind speeds (4-6 m/s) using the simple rndom, strtified nd ltin hypercube smpling strtegies. Figure 4 shows typicl cumultive distribution for the input vribles for the simple nd strtified smpling compred to the idel. The results re from 00 point smple nd totlled in 0 bins. As expected the strtified smpling technique produces better distribution compred to the simple rndom smple (i.e. the line is strighter). For ll cses the Monte-Crlo nlysis used smple of runs simultions to generte the output distribution. This nlysis ws then repeted M repetitions times nd the rnge of results clculted Cumultive distribution Bin Simple Strtified del To exmine the differences in predictions resulting from the different smpling procedures the process depicted in Figure 4 ws dopted. For ech smpling option the men nd vrince in the predicted infiltrtion rte were predicted. Four smpling options were used:. Simple rndom smpling. To gurntee suitble coverge 00 smples were generted, thus runs = Strtified smpling. The input distributions for both vribles were divided into 0 equl probbility strt. A smple ws drwn t rndom from ech strtum, to generte 00 input smples, thus runs = Ltin Hypercube Smpling (LHS(0)). Using the sme strt s in cse 2 ten input smples were generted, thus runs =0. 4. Replicted Ltin Hypercube Smpling (LHS(0x0)). The procedure for LHS(0) ws repeted 0 times, ech time new rndom piring ws generted for the two input vribles, thus runs =00. This process ws initilly repeted 0 times (i.e. M repetitions =0) for ech of the four cses nd then 00 times to compre the robustness of the methods. n ddition the simple rndom smpling method ws repeted 00 times for 00, 000, 0000 nd smples (i.e. M repetitions =00 for runs =00, 000, 0000 nd 50000). These results were used s benchmrk ginst which the remining cses could be compred. RESULTS Summry sttistics were generted from the simultions. For ech of the four smpling scenrios the men result nd its vrince ws clculted (from the M repetitions nlyses), in ddition the men of the vrince ws clculted nd its vrince. With these sttistics the hypotheses cn be tested. Simple Rndom Smpling Figure 5 shows the vrince in the men of the four nlyses using different smple sizes for simple rndom smpling only. As cn be seen the vrince is reduced by incresed smpling (i.e. incresing runs ). The significnce of this reduction ws tested ginst n F-distribution (Kreyszig 993). n ll three differences the reduction is significnt t the 5% level. Therefore, the incresed smpling produces significntly more robust result. Figure 4. Typicl cumultive distribution for the input vribles

5 Figure 5. Vrince in men results by runs. Exmining the men nd vrince in the results (Figure 6 nd Figure 7 respectively) it cn be seen tht there is less vrition due to the dditionl runs. Agin the significnce of these differences ws tested t the 5% level (the vrince ws compred s bove nd the mens ginst t-distribution). n ll cses the difference ws not significnt. Overll, this would indicte tht there is little point in running more thn 00 simultions if using simple rndom smpling s the estimtes of men nd vrince will not vry significntly up to simultions. 00 Smples Compring the three smpling methods using 00 smples/strt s described bove:. Figure 8 shows the results for men ir chnge rte. Focussing on the first three columns there is no significnt difference t the 5% level between the results in the first three columns. Therefore, there is no significnt bis introduced. 2. Figure 9 shows the men vrince in the results. Agin focussing on the first three columns only there is no significnt difference t the 5% level between the results in the first three columns. 3. Finlly, Figure 0 shows the vrince in the men ir chnge rte (robustness). Agin focussing on the first three columns only there is significnt difference t the 5% level between the simple nd strtified results nd between the simple nd LHS(0x0) results, i.e. the use of strtifiction genertes more robust result compred to simple rndom smpling. The results in the finl columns of Figure 8, Figure 9 nd Figure 0 show tht by only using 0 smples s opposed to 0 replictions of the LHS method there is no significnt difference t the 5% level for the men nd vrince results, however the robustness is significntly improved by repliction (Figure 0). Figure 6. Men ir chnge rte by runs. Figure 8. Men ir chnge rte. Figure 7. Men vrince in ir chnge rte by runs. Figure 9. Men vrince in ir chnge rte

6 Figure 0. Vrince in men ir chnge rte. Figure 2. Men vrince in ir chnge rte. 0 Smples Compring the three smpling methods using 0 smples for the simple nd LHS methods nd 9 for strtified (in this cse three strt were used for ech of the two vribles resulting in 9 runs):. Figure shows the results for men ir chnge rte. There is no significnt difference between the results t the 5% level. 2. Figure 2 shows the men vrince in the results. Agin there is no significnt difference in these results t the 5% level. 3. Finlly, Figure 3 shows the vrince in the men ir chnge rte (robustness). There is significnt difference t the 5% level between ll methods, i.e. the use of strtifiction genertes more robust result compred to simple rndom smpling. Smple size nd method comprison Compring the results from the 00 run simple smpling cse with the 0 run LHS cse, t the 5% level, there is no significnt difference in the estimtion of the men nd vrince. However, there is significnt difference in the vrince of the men results (robustness). n this cse the lrger smples (simple) is more robust thn the LHS method. The sme results re evident when compring the strtified nd LHS methods. Figure. Men ir chnge rte. Figure 3. Vrince in men ir chnge rte. DSCUSSO This pper set out to test three hypotheses relted to the smpling method used for Monte-Crlo nlysis: Method robustness The results presented hve shown tht for equl smple sizes the robustness of the smpling methods ws s expected. This ws true for the following cses: 00 simple < 00 strtified < 0x0 LHS 0 simple < 9 strtified < 0 LHS where the number is the number of smples (nd 0x0 represents the replicted LHS runs). There is lso significnt chnge in the robustness between: nd 0 LHS < 00 simple 0 LHS < 00 strtified 00 simple <,000 simple < 0,000 simple < 50,000 simple Exmining these results s whole would indicte tht the non-simple smpling techniques should be used to produce more robust nlysis, but not t the expense of fewer simultion runs. Bis n ll cses there ws no significnt difference between the men results for equl numbers of simultions. This ws lso true in the comprison between: 00 simple nd 0 LHS 00 strtified nd 0 LHS

7 This would indicte tht there is no significnt bis introduced by the smpling methods. Men nd vrince The results presented were not significntly different for the following pirs: 00 simple &,000 simple 000 simple & 0,000 simple 0,000 simple & 50,000 simple 00 simple & 00 strtified 00 simple & 0x0 LHS 00 strtified & 0x0 LHS 0x0 LHS & 0 LHS 0 LHS & 9 strtified 0 simple & 0 LHS 0 simple & 9 strtified This would indicte tht with very smll smples (9 or 0 simultions) the results from Monte-Crlo nlysis re ccurte in terms of the estimte of the men nd vrince of the output. Prcticl concerns Given these results it would indicte tht for prcticl purposes there is no requirement for LHS or strtified smpling in typicl building simultion problems. The results presented here re for smples drwn from strt with equl probbilities for multi-vrite problems with multiple probbility distributions mngement of the strt weights could become complex (formule exist for ccounting for the weights but this dds to the complexity of the nlysis). Given the simplicity of simple rndom smpling nd tht fter 00 simultions the coverge of the input rnge is within 2% of tht with LHS, for ese of implementtion, the simple smpling method would be preferred. COCLUSOS This pper set out to exmine three spects of the smpling method used in Monte-Crlo uncertinty nlysis. t hs shown tht:. There is no significnt bis introduced by the strtified nd LHS methods compred to simple smpling. 2. For the nlysed cses there is no significnt difference in prediction of men nd vrince between the methods in terms of smpling method nd number of simultions ( runs ). 3. For the sme number of simultions the LHS method produces more robust result compred to the strtified method, which in turn produces more robust result compred to the simple method. 4. The strtified nd LHS methods cnnot necessrily be used s mechnism to sve simultions runs. Given these results nd the nlysis presented by Loms nd Eppel (992) it would indicte tht for prcticl purposes Monte-Crlo uncertinty nlysis in typicl building simultion pplictions should use bout 00 runs nd simple rndom smpling. REFERECES ASHRAE Hndbook of Fundmentls: Chpter 27 Ventiltion nd nfiltrtion, Atlnt: ASHRAE. Helton J.C. nd Dvis F.J Ltin hypercube smpling nd the propgtion of uncertinty in nlyses of complex systems, Relibility Engineering nd System Sfety (8), pges 23-69, Elsevier. Hyun S, Prk C, Augenbroe G Uncertinty nd Sensitivity Anlysis of turl Ventiltion in High-rise Aprtment Buildings, Proc Building Simultion 2007, Beijing, Chin. mn R.L Appendix A: Ltin Hypercube Smpling Encyclopedi of Sttisticl Sciences, Updte Volume 3, Wiley Y. Kotek P, Jordán F, Kbele K, Hensen J Technique of Uncertinty Anlysis for Sustinble Building Energy Systems Performnce Clcultions, Proc Building Simultion 2007, Beijing, Chin. Kreyszig E Advnved Engineering Mthemtics, John Wiley nd Sons, 7th Edition. Loms K J, Eppel H Sensitivity nlysis techniques for building therml simultion progrms, Energy nd Buildings, no 9. Mcdonld A, 2002, Quntifying the effects of uncertinty in building simultion, PhD thesis, ESRU, University of Strthclyde. McKy M.D., Beckmn R.J. nd Conover W.J A Comprison of Three Methods for Selecting Vlues of nput Vribles in the Anlysis of Output from Computer Code, Technometrics (Vol 2, o 2), pges , Americn Society for Qulity Control nd The Americn Sttisticl Assocition. Mdrs Lectures on Monte Crlo Methods, Fields nstitute Monogrphs, Americn Mthemticl Society. Mxvlue Rerdon, J.T "Assessment of nturl ventiltion for Cndin residentil buildings" CMHC Reserch Report, CHC Librry Ct.o. C MH0 07A75, Cnd Mortgge nd Housing Corportion, Ottw. (ftp://ftp.cmhc- schl.gc.c/chic-ccdh/reserch_reports- Rpports_de_recherche/eng_unilingul/CHC_Ass essment_turl(w).pdf) Sltelli A, Chn K, Scott E M, Sensitivity nlysis, J Wiley nd Sons, Shw, C.Y. (987), Methods for estimting ir chnge rtes nd sizing mechnicl ventiltion systems for

8 houses, ASHRAE Trnsctions, 93 (2), pp , (RCC-30422) (RC-P-597) de Wit S, 200. Uncertinty in predictions of therml comfort in buildings, PhD thesis, Delft University. Wyss, G.D. nd K. H. Jorgensen A User's Guide to LHS: Sndi's Ltin Hypercube Smpling softwre, Albuquerque, M, Sndi tionl Lbortories. Strt Simple Rndom Smpling o Strtified Smpling o Generte 00 rndom vlues for mbient temperture nd wind speed Generte 00 rndom vlues in 0 strt for mbient temperture nd wind speed Generte 0 rndom vlues for mbient temperture nd wind speed in LHS with 0 strt Repet 0 times (i.e. crete 0 replictes) Clculte infiltrtion rte for ech smple (i.e. 00 clcultions) Clculte infiltrtion rte for ech smple (i.e. 00 clcultions) Clculte infiltrtion rte for ech smple (i.e. 0 clcultions) Clculte infiltrtion rte for ech smple (i.e. 00 clcultions) Clculte men infiltrtion rte nd vrince Clculte men infiltrtion rte nd vrince Clculte men infiltrtion rte nd vrince Clculte men infiltrtion rte nd vrince Repeted 00 times? Repeted 00 times? Repeted 00 times? Repeted 00 times? Clculte men response (nd vrince), men vrince (nd vrince) Clculte men response (nd vrince), men vrince (nd vrince) Clculte men response (nd vrince), men vrince (nd vrince) Clculte men response (nd vrince), men vrince (nd vrince) Stop Figure 4. Anlysis process for 00 smple runs