

 Clementine Cox
 1 years ago
 Views:
Transcription
1 Software Reliability via RuTime ResultCheckig Hal Wasserma Uiversity of Califoria, Berkeley ad Mauel Blum City Uiversity of Hog Kog ad Uiversity of Califoria, Berkeley We review the eld of resultcheckig, discussig simple checkers ad selfcorrectors. We argue that such checkers could protably be icorporated i software as a aid to eciet debuggig ad ehaced reliability. We cosider how to modify traditioal checkig methodologies to make them more appropriate for use i realtime, realumber computer systems. I particular, we suggest that checkers should be allowed to use stored radomess: i.e., that they should be allowed to geerate, preprocess, ad store radom bits prior to rutime, ad the to use this iformatio repeatedly i a series of rutime checks. I a case study of checkig ageeral realumber liear trasformatio (for example, a Fourier Trasform), we preset a simple checker which uses stored radomess, ad a selfcorrector which is particularly eciet if stored radomess is employed. Categories ad Subject Descriptors: D..5 [Software Egieerig]: Testig ad Debuggig F.. [Aalysis of Algorithms]: Numerical Algorithms Computatio of trasforms F.3. [Logics ad Meaigs of Programs]: Specifyig ad Verifyig ad Reasoig about Programs Geeral Terms: Reliability, Algorithms, Vericatio Additioal Key Words ad Phrases: Builti testig, cocurret error detectio, debuggig, fault tolerace, Fourier Trasform, resultcheckig, selfcorrectig This work was supported i part by a MICRO grat from Hughes Aircraft Compay ad the State of Califoria, by NSF Grat ccr9009, ad by NDSEG Fellowship daah0493g067. A prelimiary versio of this paper appears as: \Program resultcheckig: a theory of testig meets a test of theory," Proc. 35th IEEE Symp. Foudatios of Computer Sciece (994), pp. 38{39. Name: Hal Wasserma Aliatio: Computer Sciece Divisio, Uiversity of Califoria at Berkeley Address: Berkeley, CA 9470, http.cs.berkeley.edu/halw Name: Mauel Blum Aliatio: Computer Sciece Departmet, City Uiversity of Hog Kog Address: 83 Tat Chee Ave., Kowloo Tog, Kowloo, Hog Kog, Aliatio: Computer Sciece Divisio, Uiversity of Califoria at Berkeley Address: Berkeley, CA 9470, http.cs.berkeley.edu/blum Permissio to make digital or hard copies of part or all of this work for persoal or classroom use is grated without fee provided that copies are ot made or distributed for prot or direct commercial advatage ad that copies show this otice o the rst page or iitial scree of a display alog with the full citatio. Copyrights for compoets of this work owed by others tha ACM must be hoored. Abstractig with credit is permitted. To copy otherwise, to republish, to post o servers, to redistribute to lists, or to use ay compoet of this work i other works, requires prior specic permissio ad/or a fee. Permissios may be requested from Publicatios Dept, ACM Ic., 55 Broadway, New York, NY 0036 USA, fax + () , or
2 H. Wasserma ad M. Blum. RESULTCHECKING AND ITS APPLICATIONS. Assurig Software Reliability Methodologies for assurig software reliability form a importat part of the techology of programmig. Yet the problem of ecietly idetifyig software bugs remais a dicult oe, ad oe to which o perfect solutio is likely to be foud. Software is geerally debugged via testig suites: oe rus a program o a variety of carefully selected iputs, idetifyig a bug wheever the program fails to perform correctly. This approach leaves two importat questios icompletely aswered. First, how do we kow whether or ot the program's performace is correct? Geerally, some sort of a oracle is used here: our program's output may be compared to the output of a older, slower program believed to be more reliable, or the programmers may subject the output to a paistakig (ad likely subjective ad icomplete) examiatio by eye. Secod, give that a test suite feeds a program oly selected iputs outofthe ofte eormous space of all possibilities, how do we assure that every bug i the code will be evideced? Ideed, particular combiatios of circumstaces leadig to a failure maywellgoutested. Furthermore, if the testig suite fails to accurately simulate the iput distributio which the program will ecouter i its lifetime, a supposedly debugged program may i fact fail quite frequetly. Oe alterative to testig is formal vericatio, a methodology i which mathematical claims about the behavior of a program are stated ad proved. Thus it is possible to demostrate oce ad for all that the program must behave correctly o all possible iputs. Ufortuately, costructig such proofs for eve simple programs has proved uexpectedly dicult. Moreover, may programmers would likely d it upleasat tohave to formalize their expectatios of program behavior ito mathematical theorems. Aother alterative isfault tolerace. Accordig to this methodology, the reliability of critical software may beehacedbyhavig several groups of programmers create separate versios. At rutime, all of the versios are executed, ad their outputs are compared. The gross ieciecy of this approach is evidet, i terms of programmig mapower as well as either icreased rutime or additioal parallelhardware requiremets. Moreover, the method fails if commo miscoceptios amog the several developmet groups result i correspodig errors [Butler ad Fielli 993]. We d more covicig ispiratio i the eld of commuicatios, where errordetectig ad errorcorrectig codes allow for the ideticatio of arbitrary rutime trasmissio errors. Such codes are pragmatic ad are backed by mathematical guaratees of reliability. We wish to provide such rutime errorideticatio i a more geeral computatioal cotext. This motivates us to review the eld of resultcheckig.. Simple Checkers It is a matter of theoretical curiosity that, for certai computatios, the time required to carry out the computatio is asymptotically greater tha the time required, give a tetative aswer, to determie whether or ot the aswer is correct.
3 Software Reliability via RuTime ResultCheckig 3 As a easy example, cosider the followig task: give as iput a composite iteger c, output ay otrivial factor d of c. Carryig out this computatio is curretly believed to be dicult, ad yet, give I/O pair hc di, ittakes just oe divisio to determie whether or ot d is a correct output o iput c. These ideas have bee formalized ito the cocept of a simple checker [Blum 988]. Let f be a fuctio with smallest possible computatio time T (), where is iput legth (or, if a strog lower boud caot be determied, we iformally set T () equal to the smallest kow computatio time for f). The a simple checker for f is a algorithm (geerally radomized) with I/O specicatios as follows: Iput: I/O pair hx yi. Correct output: If y =f(x), accept otherwise, reject. Reliability: For all hx yi: o iput hx yi the checker must retur correct output with probability (over iteral radomizatio) p c, for p c a costat close to. \Littleo rule": The checker is limited to time o(t ()). The \littleo rule" is importat i that it requires a simple checker to be eciet, ad also i that it forces the checker to determie whether or ot y =f(x) bymeas other tha simply recomputig f(x) hece the checker must be i some maer dieret from the program which it checks. We will see i Sectio.5 that this gives rise to certai hopes that checkers are ideed \idepedet" of the programs they check,adsomay more reliably idetify program errors. For a example of simple checkig, cosider a sortig task: iput ~x =(x ::: x ) is a array ofitegers output ~y =(y ::: y ) should be ~x sorted i icreasig order. Completig this task at least via a geeral comparisobased sort is kow to require time ( log ). Thus, if we limit our checker to time O(), this will suce to satisfy the littleo rule. Give h~x ~y i, to check correctess we must rst verify that the elemets of ~y are i icreasig order. This may easily be doe i time O(). But we must also check that ~y is a permutatio of ~x. It might be coveiet here to modify our sorter's I/O specicatios, requirig that each elemet of ~y have attached a poiter to its origial positio i ~x. Ay atural sortig program could easily be altered to maitai such poiters, ad oce they are available we ca readily complete our check itimeo(). But what if ~y caot be augmeted with such poiters? Similarly, what if ~x ad ~y are oly available olie from sequetial storage, so that O()time poiterdereferecig is ot possible? The we may still employ a radomized method due to [Wegma ad Carter 98 Blum 988], which requires oly oe pass through each of ~x ad ~y. Variats of the littleo rule have also bee cosidered: for example, requirig checkers to use less space tha the programs they check, or to be implemetable with smallerdepth circuits. For example, a rather trivial result is that, if the successive coguratios of a arbitrary Turig Machie computatio are available, the the computatio may be checked by a very shallow circuit. Ufortuately, it is oly the mechaical operatio of the Turig Machie which isbeig checked here, ot the correctess of the Turig Machie's program. Hece we retur to the traditioal littleo rule, which has geerally proved to be the best route to creative, otrivial checkers.
4 4 H. Wasserma ad M. Blum We radomly select a determiistic hashfuctio h from a suitably deed set of possibilities, ad we compare h(x )++ h(x )withh(y )++ h(y ). If ~y is ideed a permutatio of ~x, the two values must be equal coversely, it is readily prove that, if ~y is ot a permutatio of ~x, thetwovalues will dier with probability. Thus, if we accept oly h~x ~y i pairs which pass t such trials, the our checker ; t,whichmay has probability oferror be made arbitrarily small. A al ote: our deitio of simple checkig is ot ew, but has perhaps ot bee clearly expressed i the checkig literature. I particular, simple checkig has bee deed oly implicitly, as the most eciet case of complex checkig (Sectio.3). Here our particular iterest i very fast checkig has led us to brig out this distictio..3 SelfCorrectors Agai let f be a fuctio with smallest (kow) computatio time T (). Let D be a welldeed probability distributio o iputs to f, ad let P be a program such that, if x is chose Dradomly, P(x) equals f(x) with probability of error limited to a small value p. I the curret paper, D will always be the uiformradom distributio so we are requirig simply that P computes f correctly o most iputs. That P ideed has this property may be determied (with high probability) by testig P o p Dradom iputs ad usig some sort of a oracle or a simple checker to determie the correctess of each output. We evisio this testig stage as beig completed prior to rutime. Aother possibility istheuseofaself tester [Blum et al. 993], which ca give such assuraces ecietly, atrutime, ad with little reliace o ay outside oracle. A selfcorrector for f [Blum et al. 993 Lipto 99] is the a algorithm (geerally radomized) with I/O specicatios as follows: Iput: x (a iput to f), alog with P, a program kow to compute f with probability of error o Dradom iputs limited to a small value p. The corrector is allowed to call P repeatedly, usig it as a subroutie. Correct output: f(x). Reliability: For all hx Pi: o iput hx Pi the corrector must retur correct output with probability (over iteral radomizatio) p c, for p c a costat close to. Timeboud: The corrector's timeboud, icludig subroutie calls to P, must be limited to a costat multiple of P's timeboud the corrector's timeboud, coutig each subroutie call to P as just oe step, must be o(t ()). As a example [Blum et al. 993], cosider a task of multiplicatio over a ite eld: iput is w x F output is product wx F. We assume additio over F to be quicker ad more reliable tha multiplicatio. Assume we kow from testig that program P computes multiplicatio correctly for all but a fractio of possible hw xi iputs. (Note that this kowledge is 00 A rather uusual set of hash fuctios is required moreover, the time required to calculate h, ad hece to complete a check, is debatable. Refer to [Blum 988] for details.
5 Software Reliability via RuTime ResultCheckig 5 i itself oly a very weak assurace of the reliability ofp, asthat fractio 00 of \dicult iputs" might well appear far more tha of the time i the life of 00 the program.) The a selfcorrector may be specied as follows: Algorithm. O iput hw xi, Geerate uiformradom r r F. Call P four times to calculate P(w ; r x; r ), P(w ; r r ), P(r x; r ), ad P(r r ). Retur, as our corrected value for wx, y c := P(w ; r x; r ) + P(w ; r r ) + P(r x; r ) + P(r r ): Why doesthiswork? Note that each of the four calls to P is o a pair of iputs uiformradomly distributed over F, ad so will retur the correct aswer with probability of error at most. Thus, all four retur values are likely to be correct: 00 4 the probabilityoferrorisatmost. Ad if all the retur values are correct, the 00 y c = P(w ; r x; r ) + P(w ; r r ) + P(r x; r ) + P(r r ) = (w ; r )(x ; r )+(w ; r )r + r (x ; r )+r r = [(w ; r )+r ][(x ; r )+r ] = wx: Note that corrected output y c is superior to umodied output P(w x) i that there are o \dicult iputs": for ay hw xi, each timewe compute a corrected value y c for wx, the chace of error (over the choice of r r )is 4. Moreover 00 (usig ew radom values of r r for each y c (i) )adpick the majority aswer as our al output, we thereby make the chace of error arbitrarily small. The price we payforthis ehaced reliability isamultiplicatio of P's usual rutime by a factor of 4t. This performace loss will be further cosidered below. A al ote o selfcorrectig. Above we speak of eedig a assurace that P is correct o all but a small fractio of iputs. Such laguage implies a assumptio that P's behavior is xed (except for possible radomizatio) prior to the selfcorrectig process: i.e., that P is ot a adversary which ca adapt i respose to the corrector. This assumptio is ecessary, as a adaptable adversary could easily fool a corrector such as the above. 3 Achecker, i cotrast, is required to be secure agaist eve a adaptable adversary. Oe may also dee a complex checker [Blum 988], which outputs a accept/reject correctesscheck similar to that of a simple checker, but which, like a selfcorrector, is allowed to poll P at several locatios (ad so, like a selfcorrector, teds to be less eciet tha a simple checker). I geeral, a complex checker seeks to determie the correctess of P(x) by comparig P(x) to several of P's other outputs. More formally, acomplex checker for f is deed as follows: observe that, if we compute several corrected outputs y c () ::: y c (t) 3 [Gemmell et al. 99] however demostrates that selftestig/correctig may be possible eve whe a program has limited adversarial power.
6 6 H. Wasserma ad M. Blum Iput: x (a iput to f), alog with P, a program which may compute f o all, some, or o iputs. The checker is allowed to call P repeatedly, usig it as a subroutie. Correct output: If P is correct both o x ad o all the other outputs sampled by the checker, must accept. If P(x) is icorrect, must reject. Reliability: For all hx Pi: o iput hx Pi the checker must retur correct output with probability (over iteral radomizatio) p c, for p c a costat close to. Timeboud: The checker's timeboud, icludig subroutie calls to P, must be limited to a costat multiple of P's timeboud the checker's timeboud, coutig each subroutie call to P as just oe step, must be o(t ()) (where T () is agai the smallest possible time to compute f). Observe a ecessary weakess of this deitio: i the case that P(x) is correct but at least oe of P's other outputs sampled by the checker is icorrect, the checker may either accept or reject. This weakess is ecessary (if we are to allow algorithms which go beyod simple checkig) because, i the case that P returs oly juk aswers, we caot expect the checker to gure out whether or ot P(x) is correct. I spite of this weakess, the complex checker tells us what we eed to kow: if the checker accepts, wekowthatp(x) is correct if the checker rejects, wekowthatp failed o at least oe of a small, kow set of iputs. A example complex checker (for graph isomorphism) is described i Sectio.7. For more iformatio o resultcheckig, refer to the aotated bibliography..4 Debuggig via Checkers We will argue here that embeddig checkers i software products may be a substatial aid i debuggig those products ad assurig their reliability. It is our hope that resultcheckig may form the basis for a debuggig methodology more rigorous tha the testig suite ad more pragmatic tha vericatio. Throughout the process of software testig, embedded checkers may be used to idetify icorrect output. They will thereby serve as a eciet alterative to a covetioal testig oracle. Furthermore, eve after software is put ito use, leavig the checkers i the code will allow for the eective detectio of ligerig bugs. We evisio that, each time a checker ds a bug, a iformative output will be writte to a le this le will the be periodically reviewed by softwaremaiteace egieers. Aother possibility isa immediate warig message, so that the user of a critical system will ot be duped ito acceptig erroeous output. While it could be argued that buggy output would be oticed by the user i ay case, a automatic system for idetifyig errors should i fact reduce the probability that bugs will be igored or forgotte. Moreover, a checker ca catch a error i a program compoet whose eects may ot be readily apparet to the user. Thus achecker might well idetify a bug i a critical system before it goes o to cause a catastrophic failure. Moreover, checkers may trigger automatic selfcorrectio: potetially a method for the creatio of extremely reliable systems. By writig a checker, oe group of programmers may assure themselves that aother group's code is ot udermiig theirs by passig alog erroeous output. Similarly, the software egieer who creates the specicatios for a program compo
7 Software Reliability via RuTime ResultCheckig 7 et ca write a checker to verify that the programmers have truly uderstood ad provided the fuctioality he required. Sice checkers deped oly o the I/O specicatios of a computatioal task, oe may check eve a program compoet whose iterals are ot available for examiatio, such as a library of utilities ruig o a remote machie. Thus checkers facilitate the discovery of those misuderstadigs at the \seams" of programs which so ofte uderlie a software failure. Evidetly, all of this applies well to objectorieted paradigms. Ulike vericatio, checkig reveals icorrect output origiatig i ay cause whether due to software bugs, hardware bugs, or trasiet rutime errors. Moreover, sice a checker cocers itself oly with the I/O specicatios of a computatioal task, the itroductio of a potetially ureliable ew algorithm for solvig a old task may ot require substatial chage to the associated checker. Ad so, as a program is modied throughout its lifetime ofte without a adequate repeat of the testig process its checkers will cotiue to guard agaist errors..5 CheckerBased Debuggig Cocers Performace loss: Simple checkers are iheretly eciet: due to the littleo rule,a simple checker should ot sigicatly slow the program P that it checks. Selfcorrectors, o the other had, require multiple calls to P, ad so multiply executiotime by a small costat (or, alteratively, require additioal parallel hardware). I a realtime computer system, such a slowdow may ot be acceptable. Hece it is desirable to check a give program by implemetig both a simple checker ad a selfcorrector. The simple checker may the be ru o each output, while oly i the (hopefully rare) case that the output is icorrect eed the selfcorrector be ru as well hece the selfcorrector is kept o the commocase path. Alteratively, i Sectio.6 we will cosider a chage to our deitios which will allow for certai selfcorrectig algorithms to ru at realtime speeds. Realumber issues (see [Gemmell et al. 99 Ar et al. 993]): Traditioal resultcheckig algorithms, such as the example selfcorrector from Sectio.3, ofte rely o the orderly properties of ite elds. I may programmig tasks, we are more likely to ecouter real umbers i.e., approximate reals represeted by a xed umber of bits. Such umbers will be limited to a legal subdomai of <. I Sectio.4, we will see how to modify a traditioal selfcorrectig methodology to accout for this. Moreover, limited precisio will likely result i roudo errors withi P. We must the determie how much arithmetic error may allowably occur withi P, ad must check correctess oly up to the limit of this errordelta. I Sectio, we will see that limited precisio calculatios, while a substatial challege, are by o meas fatal to our checkig project. Testig checkers: A checker, like ay software compoet, must be carefully debugged. Otherwise, a degeerate checker C may idiscrimiately report program P to be correct, thereby givig a false sese of assurace. The problem of costructig a suitable testig suite for a checker is otrivial. Evidetly testig a checker o correct program output is isuciet: we must also test the checker's ability to recogize bugs ad ot just agrat, radom bugs, but various ad subtle oes. Oe possibility would be to employ a mutatio model of software error: i.e., to test C o the output from mutated variats of P.
8 8 H. Wasserma ad M. Blum Buggy checkers: What if we evertheless fail to debug our checker? Ideed, it may be objected that resultcheckig begs the questio of software reliability: for checkig may fail due to the checker's code beig as buggy as the program's. To this objectio we respod with three argumets: argumets which are, however, heuristic rather tha rigorous. First, checkers ca ofte be simpler tha the programs they check, ad so, presumably, less likely to havebugs. For example, moder computers may use itricate, arithmetically ustable algorithms to compute matrix multiplicatio i time, say, O( :38 ), rather tha the stadard O( 3 ). Ad yet Freivalds's O( )time checker for matrix multiplicatio [Freivalds 979] uses oly the simplest of multiplicatio algorithms, ad so is seemigly likely to be bugfree. Secod, observe that oe of the followig four coditios must hold each time a (possibly buggy) simple checker C checks the output of a program P: (I). P(x) is correct C correctly accepts it. (II). P(x) is icorrect C correctly rejects it. (III). \False alarm": P(x) is correct C icorrectly rejects it. (IV). \Missed bug": P(x) is icorrect C icorrectly accepts it. Oly a \missed bug" is a truly bad outcome: for a \false alarm," while aoyig, at least draws our attetio to a bug i C. To reduce the likelihood of missed bugs, it suces to rule out a strog correlatio betwee x values at which P fails ad x values at which C(x P(x)) fails. It is our heuristic cotetio that such a correlatio is ulikely. Ideed, recall that oe eect of the \littleo rule" is that C does't have suciet time to merely reduplicate the computatio performed by P. Thus we claim (heuristically) that C must be doig somethig essetially dieret from what P does, ad so, if buggy, may reasoably be expected to make dieret errors. But the we would expect few correlated errors moreover, we would expect more ucorrelated tha correlated errors, so that a give bug could well be idetied ad xed before it goes o to geerate ay correlated errors. 4 Fially, say that correlated errors oetheless occur. This might i particular be expected due to faults i hardware or software compoets o which both P ad C deped. We argue that, eve i this case, a \missed bug" may ot result. Ideed, cosider our checker for a sortig program (Sectio.), which determies whether or ot ~y is a permutatio of ~x by comparig h(x )+ + h(x ) with h(y )++h(y ): assumig that the sortig program has failed, h(x )++h(x ) ad h(y )+ + h(y ) will ot match (with probability ). If the checker 4 It could be argued that the littleo rule oly prohibits C from duplicatig the most timecosumig of P's compoets P's simpler compoets may be trivially duplicated i C, ad so, if buggy, might produce correlated errors. To this argumet we respod that C's primary fuctio is to check the cetral algorithm of P smaller compoets eeded as subrouties by both P ad C should, if ecessary, be checked separately. As a illustratio of a related issue, cosider a stadard maxflow program, which proceeds by successively adjustig ows util it ds a coguratio which achieves optimal ow across a particular cut. A checker for this program eed do little more tha verify that optimal ow is ideed achieved across the cut which is what the program itself does i its al stage. Hece checkig this sort of program seems trivial ad ieective. Our respose is simply that such a program ideed requires o separate checker, because it is aturally selfcheckig.
9 Software Reliability via RuTime ResultCheckig 9 simultaeously fails, it may miscalculate h(x )++ h(x ) ad/or h(y )++ h(y ) but, we argue, it seemigly is ot likely to fail i exactly such a maer as to get values for the two sumswhich happe to match each other. May checkers proceed i this way, calculatig two quatities ad declarig a error if they do ot match. Degeerate checker behavior seems ulikely to yield matchig quatities ad so we argue that checkers, eve whe buggy themselves, have a atural resistace to the bad case of missig a program bug..6 Stored Radomess We here itroduce a extesio to traditioal checker deitios: we suggest that radomized checkers, rather tha havig to geerate fresh radom bits for each check, should be allowed to use preprocessed stored radomess. That is, prior to rutime while the program is idle, or durig bootup, or eve durig software developmet we geerate radom bits ad do some (perhaps legthy) preprocessig these radom bits, together with preprocessed values depedet o them, form oe or more radom packages. These packages are stored the, at rutime, each check requires such a package as a additioal iput. 5 Stored radomess has several potetial advatages. First, it allows for much or all of a checker's radomess to be geerated before rutime. This could be useful particularly i doig very quick checks of lowlevel fuctioalities such as microprocessor arithmetic (as i [Blum ad Wasserma 996]): i this cotext, havig to geerate radom bits at rutime could be a icoveiece. Secod, we shall see that stored radomess allows for the use of checkig algorithms which would otherwise require too much computatio at rutime: by payig a timecost of (T (N)) i preprocessig, we \cheat" some work out of the way, so that each rutime check ca the be completed i time o(t (N)). For a trivial example of stored radomess, we ca recosider the multiplicatio selfcorrector of Sectio.3. If i a preprocessig stage r r are geerated, P(r r ) is calculated, ad package R = hr r P(r r )i is stored, this leaves oly three calls to P at rutime, rather tha four. However, this improvemet is ot particularly impressive. Stored radomess will be further explicated through more substatial examples: i Sectios. ad.5, we will preset a simple checker which is possible oly if oe allows stored radomess, ad a selfcorrector which, if oe allows stored radomess, ca ru at realtime speeds. How much radomess must be stored? Let N be our xed iput legth, ad let ` be the umber of checks we expect to carry out at rutime. Oe approach would be to preprocess ` packages. The each check would have its ow \fresh" radomess, which would certaily be good from the poit of view of reliability. However, we might expect ` to be N or eve ukow durig preprocessig so this trivial approach could have too high a cost i preprocessig time ad storage space. A opposite approach would be to geerate a sigle radom package ad use it for every check. But would the repeated use of a \stale" radom package reduce 5 This method requires iput legth N to be xed prior to rutime equivaletly, ew preprocessig stages will be required as iput legth icreases. This use of stored iformatio is aalogous to that i the P/poly model of computatio.
10 0 H. Wasserma ad M. Blum the reliability ofourchecker? Sectios. ad.5 iitially choose this approach ad Lemmas 4 ad 0 argue that the cosequet reductio i reliability is ot fatal. Moreover, i Sectios. ad.5 we go o to suggest a itermediate approach, which requires a umber of stored packages that is O(N) ad idepedet of `. Lemmas 5 ad argue that the resultig storedradom checkers are the highly reliable..7 Variatios o the Checkig Paradigm It may still be objected that most programmig tasks are less ameable to checkig tha are the clea mathematical problems with which theoreticias usually deal. Nevertheless, it is our experiece that may such tasks may be subjected to iterestig checks. The followig is alistof checkig ideas which may suggestto the reader how deitios ca be geeralized to make checkig more suitable as a tool for software debuggig. Partial checks: Oe may d it suciet to check oly certai aspects of a computatioal output. Programmers may focus o properties they thik particularly likely to reveal bugs or particularly critical to the success of the system. Certai checkers might oly be capable of idetifyig outputs icorrect by avery large errordelta. Eve just checkig iputs ad outputs separately to verify that they fall i a legal domai may prove useful. Timig checks: Oe might wish to moitor the executiotime of a software compoet a uexpectedly large (or small) time could the reveal a bug. For istace, a programmer's expectatio that a particular subroutie should take time 0 o iput of variable legth could be checked agaist actual performace. Checkig via iteractive proofs: There exist iteractive proofs of correctess for certai mathematical computatios ad may suchiteractive proofs may equivaletly be regarded as complex checkers. For example, it follows from the IP method of [Shamir 99] that, if program P claims to solve a PSPACEcomplete problem, the there is a checker for P which requires polyomial time plus a polyomial umber of calls to P. Similarly, the followig iteractiveproof method, due to [Goldreich et al. 99], may equivaletly be regarded as a complex checker. Say that P claims to decide graph isomorphism. If P says that A = B, we ca check this by ruig P o successively reduced versios of A ad B, thereby forcig P to reveal a particular isomorphism. Less trivially, ifp claims that A 6 = B, we repeat several times the followig check: geerate C, a radom isomorphism either of A (with probability ) or of B (with probability ) the ask P to tell us which oe of A or B is isomorphic to C. If A = B, P ca oly guess which ofthetwo we permuted. For more o iteractive proof, refer to the bibliography. Chagig I/O specicatios: I Sectio., we sawhow a easy augmetatio to the output of a sortig program could make the program easier to check. Similarly, cosider the problem of checkig a (log )time biarysearch task. As traditioally stated (see [Betley 986, p. 35], where it features i a relevat discussio of the diculty of writig correct programs), biary search has as iput a key k adaumerical array a[] ::: a[], where a[] a[], ad as output i such thata[i] =k, or0ifk is ot i the array. Note that the problem as stated is ucheckable: for, if the output is 0, it will
HOW MANY TIMES SHOULD YOU SHUFFLE A DECK OF CARDS? 1
1 HOW MANY TIMES SHOULD YOU SHUFFLE A DECK OF CARDS? 1 Brad Ma Departmet of Mathematics Harvard Uiversity ABSTRACT I this paper a mathematical model of card shufflig is costructed, ad used to determie
More informationCrowds: Anonymity for Web Transactions
Crowds: Aoymity for Web Trasactios Michael K. Reiter ad Aviel D. Rubi AT&T Labs Research I this paper we itroduce a system called Crowds for protectig users aoymity o the worldwideweb. Crowds, amed for
More informationType Less, Find More: Fast Autocompletion Search with a Succinct Index
Type Less, Fid More: Fast Autocompletio Search with a Succict Idex Holger Bast MaxPlackIstitut für Iformatik Saarbrücke, Germay bast@mpiif.mpg.de Igmar Weber MaxPlackIstitut für Iformatik Saarbrücke,
More informationSOME GEOMETRY IN HIGHDIMENSIONAL SPACES
SOME GEOMETRY IN HIGHDIMENSIONAL SPACES MATH 57A. Itroductio Our geometric ituitio is derived from threedimesioal space. Three coordiates suffice. May objects of iterest i aalysis, however, require far
More informationDryad: Distributed DataParallel Programs from Sequential Building Blocks
Dryad: Distributed DataParallel Programs from Sequetial uildig locks Michael Isard Microsoft esearch, Silico Valley drew irrell Microsoft esearch, Silico Valley Mihai udiu Microsoft esearch, Silico Valley
More informationConsistency of Random Forests and Other Averaging Classifiers
Joural of Machie Learig Research 9 (2008) 20152033 Submitted 1/08; Revised 5/08; Published 9/08 Cosistecy of Radom Forests ad Other Averagig Classifiers Gérard Biau LSTA & LPMA Uiversité Pierre et Marie
More informationStéphane Boucheron 1, Olivier Bousquet 2 and Gábor Lugosi 3
ESAIM: Probability ad Statistics URL: http://wwwemathfr/ps/ Will be set by the publisher THEORY OF CLASSIFICATION: A SURVEY OF SOME RECENT ADVANCES Stéphae Bouchero 1, Olivier Bousquet 2 ad Gábor Lugosi
More informationEverything You Always Wanted to Know about Copula Modeling but Were Afraid to Ask
Everythig You Always Wated to Kow about Copula Modelig but Were Afraid to Ask Christia Geest ad AeCatherie Favre 2 Abstract: This paper presets a itroductio to iferece for copula models, based o rak methods.
More informationTeaching Bayesian Reasoning in Less Than Two Hours
Joural of Experimetal Psychology: Geeral 21, Vol., No. 3, 4 Copyright 21 by the America Psychological Associatio, Ic. 963445/1/S5. DOI: 1.7//963445..3. Teachig Bayesia Reasoig i Less Tha Two Hours Peter
More informationSpinout Companies. A Researcher s Guide
Spiout Compaies A Researcher s Guide Cotets Itroductio 2 Sectio 1 Why create a spiout compay? 4 Sectio 2 Itellectual Property 10 Sectio 3 Compay Structure 15 Sectio 4 Shareholders ad Directors 19 Sectio
More informationThe Unicorn, The Normal Curve, and Other Improbable Creatures
Psychological Bulleti 1989, Vol. 105. No.1, 156166 The Uicor, The Normal Curve, ad Other Improbable Creatures Theodore Micceri 1 Departmet of Educatioal Leadership Uiversity of South Florida A ivestigatio
More informationHow Has the Literature on Gini s Index Evolved in the Past 80 Years?
How Has the Literature o Gii s Idex Evolved i the Past 80 Years? Kua Xu Departmet of Ecoomics Dalhousie Uiversity Halifax, Nova Scotia Caada B3H 3J5 Jauary 2004 The author started this survey paper whe
More informationBOUNDED GAPS BETWEEN PRIMES
BOUNDED GAPS BETWEEN PRIMES ANDREW GRANVILLE Abstract. Recetly, Yitag Zhag proved the existece of a fiite boud B such that there are ifiitely may pairs p, p of cosecutive primes for which p p B. This ca
More informationSystemic Risk and Stability in Financial Networks
America Ecoomic Review 2015, 105(2): 564 608 http://dx.doi.org/10.1257/aer.20130456 Systemic Risk ad Stability i Fiacial Networks By Daro Acemoglu, Asuma Ozdaglar, ad Alireza TahbazSalehi * This paper
More informationLeadership Can Be Learned, But How Is It Measured?
Maagemet Scieces for Health NO. 8 (2008) O C C A S I O N A L PA P E R S Leadership Ca Be Leared, But How Is It Measured? How does leadership developmet cotribute to measurable chages i orgaizatioal performace,
More informationAdverse Health Care Events Reporting System: What have we learned?
Adverse Health Care Evets Reportig System: What have we leared? 5year REVIEW Jauary 2009 For More Iformatio: Miesota Departmet of Health Divisio of Health Policy P.O. Box 64882 85 East Seveth Place, Suite
More informationare new doctors safe to practise?
Be prepared: are ew doctors safe to practise? Cotets What we foud 02 Why we ve writte this report 04 What is preparedess ad how ca it be measured? 06 How well prepared are medical graduates? 08 How has
More informationWhich Extreme Values Are Really Extreme?
Which Extreme Values Are Really Extreme? JESÚS GONZALO Uiversidad Carlos III de Madrid JOSÉ OLMO Uiversidad Carlos III de Madrid abstract We defie the extreme values of ay radom sample of size from a distributio
More informationJ. J. Kennedy, 1 N. A. Rayner, 1 R. O. Smith, 2 D. E. Parker, 1 and M. Saunby 1. 1. Introduction
Reassessig biases ad other ucertaities i seasurface temperature observatios measured i situ sice 85, part : measuremet ad samplig ucertaities J. J. Keedy, N. A. Rayer, R. O. Smith, D. E. Parker, ad M.
More informationTesting for Welfare Comparisons when Populations Differ in Size
Cahier de recherche/workig Paper 039 Testig for Welfare Comparisos whe Populatios Differ i Size JeaYves Duclos Agès Zabsoré Septembre/September 200 Duclos: Départemet d écoomique, PEP ad CIRPÉE, Uiversité
More informationTurning Brownfields into Greenspaces: Examining Incentives and Barriers to Revitalization
Turig Browfields ito Greespaces: Examiig Icetives ad Barriers to Revitalizatio Juha Siikamäki Resources for the Future Kris Werstedt Virgiia Tech Uiversity Abstract This study employs iterviews, documet
More informationCatalogue no. 62557XPB Your Guide to the Consumer Price Index
Catalogue o. 62557XPB Your Guide to the Cosumer Price Idex (Texte fraçais au verso) Statistics Caada Statistique Caada Data i may forms Statistics Caada dissemiates data i a variety of forms. I additio
More informationThe Arithmetic of Investment Expenses
Fiacial Aalysts Joural Volume 69 Number 2 2013 CFA Istitute The Arithmetic of Ivestmet Expeses William F. Sharpe Recet regulatory chages have brought a reewed focus o the impact of ivestmet expeses o ivestors
More informationThings Your Next Firewall Must Do
10 Thigs Your Next Firewall Must Do Itroductio: 10 Thigs Your Next Firewall Must Do Much has bee made about brigig applicatio visibility ad cotrol ito etwork security. The reaso is obvious: applicatios
More informationHOMEBUYING STEP BY STEP. A Consumer Guide and Workbook
HOMEBUYING STEP BY STEP A Cosumer Guide ad Workbook CMHC HOME TO CANADIANS Caada Mortgage ad Housig Corporatio (CMHC) has bee Caada s atioal housig agecy for more tha 65 years. Together with other housig
More informationON THE EVOLUTION OF RANDOM GRAPHS by P. ERDŐS and A. RÉNYI. Introduction
ON THE EVOLUTION OF RANDOM GRAPHS by P. ERDŐS ad A. RÉNYI Itroductio Dedicated to Professor P. Turá at his 50th birthday. Our aim is to study the probable structure of a radom graph r N which has give
More information4. Trees. 4.1 Basics. Definition: A graph having no cycles is said to be acyclic. A forest is an acyclic graph.
4. Trees Oe of the importat classes of graphs is the trees. The importace of trees is evidet from their applicatios i various areas, especially theoretical computer sciece ad molecular evolutio. 4.1 Basics
More informationWhat is IT Governance?
30 Caada What is IT Goverace? ad why is it importat for the IS auditor By Richard Brisebois, pricipal of IT Audit Services, Greg Boyd, Director ad Ziad Shadid, Auditor. from the Office of the Auditor Geeral
More informationSupporting medical students with mental health conditions
Supportig medical studets with metal health coditios Supportig medical studets with metal health coditios Cotets 02 Geeral Medical Coucil Supportig medical studets with metal health coditios Published
More informationTrying Juveniles as Adults: An Analysis of State Transfer Laws and Reporting Patrick Griffin, Sean Addie, Benjamin Adams, and Kathy Firestine
U.S. Departmet of Justice Office of Justice Programs Office of Juveile Justice ad Deliquecy Prevetio Natioal Report Series September 2011 This bulleti is part of the Juveile Offeders ad Victims Natioal
More information