Security. Jerey Voas

Transcription

1 Testing Software for Characteristics Other than Correctness: Safety, Failure tolerance, and Security Jerey Voas Reliable Software Technologies (URL: Ridgetop Circle Sterling, VA phone: (703) fax: (703) Abstract Software testing suers from a variety of theoretical and pragmatic limitations. These problems have made software development organizations cynical as to the value of testing, particularly at system testing levels, yet most organizations do not dispute the cost eectiveness of unit and integration testing. This paper introduces a variety of behavioral software quality characteristics, other than traditional \black or white" correctness, that can be tested for via fault-injection methods. These methods in no way replace traditional testing for defects or reliability testing, but test for other characteristics that are important for demonstrating trustworthiness. The fault-injection methods that we will describe are empirical and do not require an oracle (human nor automated), which immediately avoids one big problem of testing. And the fault-injection techniques that we will employ do not use syntactic mutation methods. 1 Introduction Software testing suers from a host of dierent limitations, some of which are pragmatic, while others are theoretical. Yet despite these limitations, software testing continues to be employed universally as the software quality assessment method of choice. The reasons for this are numerous, mainly that testing provides a demonstration that the computations embedded in the syntax provide the intended semantics. It is very hard to see that in any other manner than via dynamic execution even a formal proof of correctness leaves some degree of doubt in the mind of the developer until he or she observes satisfactory output from the code. 1

2 Developing correct software should not always be assumed as the ultimate goal for all of today's systems. There are cases where absolute correctness is not what reduces the chances of a catastrophe, but rather software systems that produce \acceptable", safe output states. For example, in a digital control system of a manufacturing plant, all that may besought is for the software to close a valve in the system given certain input vectors. Potentially, such code might be trivial enough to demonstrate the correctness thereof, yet the overall manufacturing plant could be unsafe, due to the possibility that the software will actually produce a correct result that could enable a system catastrophe. Although this might seem counterintuitive, it is true that external events to the code are potential contributors to undesirable outputs, and said outputs could be produced by correct code. Such plaguing circumstances have lead to the upsurge of interest and research in software quality approaches to characteristics such as \safe software", failure-tolerant systems, and fail-safe systems. Software testing is conditioned on the assumption that the inputs that you test it on are data values for which the software should produce output, i.e., the test cases are in the domain of the function that the software implements. So for instance if the program is not expected to input negative integers or handle them very infrequently, then it is foolish to test the code on them (unless there are arguments for doing so related to enhanced reusability). This assumption that the code only need to work for \expected" input data must be dropped when we start testing for characteristics other than correctness. This paper is intended to briey introduce the reader to the benets that software faultinjection techniques provide for software characteristics other than correctness. Software fault-injection is an extremely tricky business to implement \right," where right is a combination of eciency, plausible fault classes, and error recovery from the injected anomalies that may besointolerable to the code that it \chokes." But if you can get it right, the benets rival those of any other software quality assessment technique, including dynamic testing and formal verication. To begin, we acknowledge that the techniques that we are about to review have been published elsewhere (and in far greater detail with anecdotal results) [5, 7], and several have been transitioned into commercial products. Admittedly, these techniques are computational, but ultimately, you get what you pay for! These techniques do not require an oracle, simply source code, inputs that the code executes on, and denitions of fault (or what we more frequently refer to as anomaly) classes. These techniques haveevolved over 8 years the software fault-injection capabilities have become more rened, the commercial tools more ecient, and the results from various applications more convincing. This work began in 1988 from a crude understanding about software mutation, and from that, became a new theory for why programs can't fail, providing a new software testability paradigm [6, 1]. Then we developed a failure tolerance model that is similar to the testability model, but with the key distinction being that the failure tolerance model considers the potential of the software receiving bad data from external sources, such ashuman operator errors or failed hardware devices. And nally from there, we developed a metric to assess information system vulnerability. 2

3 2 Basic Terminology Our techniques apply fault-injection to source code. To use our techniques then, we must uniquely identify specic syntactic program constructs as well as the internal data states created during execution. To uniquely identify syntactic constructs, we dene a location to be either an assignment statement, an input statement, an output statement, or the <condition> part of an if or while statement. A program state is a set of mappings between all variables (declared and dynamically allocated) and their values at a particular pointinthe program's execution. Note that in a program state we include both the input to the program used during its execution and the value of the program counter. 1 We identify program states only between two dynamically consecutive locations in the code. The execution of a location is considered to be atomic. Hence data states can only be viewed between particular locations. For example, the data state: f(input, 3), (a, 5), (b, 5), (c, undened), (pc, 10)g tells us that the program input that started this execution was a 3, the variables a and b have the value 5, the variable c is undened, and the next instruction to be executed occurs at line 10. Before program execution on an input begins, all variables are undened. In our approach, the input value that starts a program execution is always contained as a member of every program state created during that execution. A program state error is an incorrect variable/value pairing in a program state where correctness is determined according to a specication. To develop the model, we assume that an assertion could be developed for each state that determines if the state is or is not correct. Note that in the nal analysis we will not need to do so. 2 We will refer to program state errors as infections, and will use these two terms interchangeably. If a program state error exists, the program state and variable with the incorrect value at that point are deemed infected. An infected program state may have more than one infected variable. Propagation of a program state error occurs when a program state error aects z O, the output state of the program. Cancellation occurs when the existence of a program state error is not discernible in the program output (i.e., after viewing the output of a program, we have no indication that any program state error ever occurred). The importance of studying how problems propagate in programs cannot be understated: when problems do not propagate, testing cannot detect them, and when they do, unsafe or unsecure events can be exacerbated. If location l is an assignment statement, then lhs(l) is the variable that is being assigned at l, and rhs(l) is the expression being calculated at l. 3 If location l is a conditional statement or loop termination expression, then lhs(l) is referenced from the program counter, and rhs(l) is the boolean expression being evaluated at l. Let S denote a specication, P denote an implementation of S, x denote a program input, denote the set of all possible inputs to P, Q denote the probability distribution 1 For our purposes, the terms \program counter" and \source code line counter" mean exactly the same thing. 2 Determining the correctness of any given program state during the execution of a program is hard (if not impossible) because of the vague nature of program specications. Fortunately, our models are developed using the theoretical existence of tests for correctness at each state. This means that determining the actual correctness of every state is not necessary. 3 lhs is short for \left-hand side" and rhs is short for right-hand side. 3

4 of, l denote a program location in P,andi denote a particular execution (or what we term an \iteration") of location l caused by inputx. Furthermore, let A lp ix represent the program state produced after executing location l on the i th execution of P from input x. In our model often nd it useful to group program states into sets with similar properties. For instance, assume that location l is executed m xl times byinputx. Now consider all of the program states that are created by this input immediately after l is executed. The following equation allows us to formalize such a set: A lp x = fa lp ix j 0 i m xl g We can then create a set of these sets for every x 2 : lp = fa lp x j x 2 g In our fault-injection methods, a simulated infection is the result of changing the value ofavariable so that the program state is known to be altered. As we have already stated, A lp ix denotes the program state created after the i th iteration of location l on input x AlP ix denotes this same program state after a simulated infection is injected into A lp ix.simulated infections are our way to perform fault-injection without relying on mutation techniques. 3 Testing for Safety and Failure-Tolerance We will now describe our approach to testing for the safety and failure tolerance of the code with respect to simulated infections. What this provides is an estimated probability for how likely code execution is to result in output states that are unsafe or simply undesirable from these simulated infections. Much like testing requires an oracle, this method requires a description of what is unsafe or undesirable, but this method does not require what is typically considered as an oracle. 4 Note that what we produce (via our fault-injection) approaches is \numbers." There are those persons in software engineering who crusade against placing numbers on software quality, believing that either numbers cannot be believed or simply are too inaccurate. It is our belief that the only thing wrong with numbers is that the interpretation typically ascribed to them is a total oversell as to their true meaning. It is our contention that numbers are the only way to assess anything meaningful about software quality, provided that the numbers are used as relative measures of quality,notabsolute measures of quality. Ifyou ascribe 10 ;5 to a piece of software, and the smaller the metric the higher the quality, then if another piece of software has a score of 10 ;15, it should be true that the software with 10 ;15 is of higher quality. If not, the numbers are meaningless. The problem in software engineering is that we have taken numbers as absolute, guarantors of quality, which is a foolish interpretation. Numbers have worked well in assessing analog system quality, and therefore it is natural to assume that they would work well for digital systems. The numbers that the approaches that we are about to describe produce are simple probabilistic estimates that can serve as 4 Testing for correctness generally involves a determining a correct answer per input, as opposed to safety that determines a range of tolerable answers per input, including the eects that external events might have on the computation. 4

5 relative predictors of possible future behavior if future circumstances are similar, but the likelihood of this cannot be determined, and hence absolute conclusions cannot be drawn. For critical software, there are three classes of output states that can be produced from a program execution: (1) correct, (2) incorrect but acceptable (non-hazardous), and (3) hazardous. Software failure tolerance refers to the abilityof the software to produce \acceptable" outputs regardless of the program states that are encountered during execution what constitutes an acceptable output is dened by the system level requirements. Software safety refers to the ability of the software to produce \non-hazardous" outputs regardless of the program states that are encountered during execution what constitutes an output hazard is dened by the system level safety requirements. By our denitions, software safety is simply a special form of software failure tolerance. Fault-tolerance refers to a class of outputs that can be tolerated, and software safety refers to a class of outputs that cannot be tolerated. For example, for an input value of 1 to the software, suppose that the correct output value is But suppose that the numerical algorithms we use produce an output of If the set of acceptable, non-hazardous values is [99.0, 101.0] and only this range, then this value is acceptable, and hence the software was failure-tolerant with respect to an inaccurate algorithm. Further suppose that the software, for some reason, does not receive the correct input value of 1 when it should due to some external problem, and instead receives a value of 0.0. And nally suppose that along with its low-accuracy numerical algorithm, the software produces an output value of 102.0, which is hazardous since it is out of the range of [99.0, 101.0]. The latter hypothetical example reveals an interesting situation. The external environment to the software is in a true state that should present the software with an input value of 1, but for anomalous circumstances, the software is presented with an input value of 0. For the external environment in which the software resides, the only acceptable outputs from the software are in [99.0, 101.0] for the true external environment state. If might well be the case that when the actual environment's state is 0.0, a result of from the software is acceptable. Hence merely testing the software with a value of 0 would not reveal the potential impact of such a problem it is only when we simulate this anomalous circumstance that we realize that an input value of 0 could produce a hazardous result if the input value should have been 1.0, even though is a correct result for an input of 0. Simulated infections that we will inject can sometimes result in disastrous output states states that can lead to catastrophes from the systems that they control. For example, if an automatic teller machine gives you no cash when you make a withdrawl, yet debits your account with a maximum withdrawl, that might be considered as catastrophic. When a system gets into a \bad state" during execution, the next eventthatwewould like to occur would be a recovery from the bad state back to an acceptable state. In other words, we do not want the bad state to propagate. If propagation does (or even can) occur, we would like to demonstrate that the \bad state" has no damaging consequences. To improve software's failure tolerance, there are two classes of \problems" that should be protected against: software faults and hardware failures. In the past, the EPA failure tolerance assessment method has simulated both of these fault classes, providing observations of the impact to computations when a hardware sensor sends corrupt data or when the software itself computes intermediate corrupt data (with respect to the specication). The EPA technique simulates fault-classes by inserting instrumentation into the code that forces 5

6 changes (termed \simulated infections") into the state of the program as it executes. A simulated infection is simply a modied data value assigned to some variable or variables during execution. 5 The following algorithm exploits simulated infections to allow observation of the impact of dierent classes of \bad states" [5]. Simulated infections mimic the eects of both programmer faults and hardware failures. Simulated infections are created by perturbation functions. The process of injecting a simulated infection into an executing program is called perturbing the program. A perturbation function is the code instrumentation that takes in a program state as one of its inputs, changes the state according to certain rules that are either fed in to the function or hard-wired in, and produces as output a dierent program state. We are concerned with whether undesirable output events can be observed given the simulated infections employed. The following algorithm provides this information. It determines the proportion of outputs that satisfy the logical predicate, PRED, where PRED determines for any particular input if z O is unacceptable: Algorithm 1: Extended Propagation Analysis (EPA) For each l in P : 1. Set count to Randomly select an input x according to Q, and if P halts on x in a xed period of time, nd the corresponding A lp x in lp if such is created. Set Z to A lp 1x. 3. Alter Z, and denote the new data state as Z. Execute the succeeding code on Z. (If by chance l is executed more than once because of the value we placed into Z, alter each state that occurs immediately after l.) 4. If the output satises PRED, increment count. 5. Repeat steps 2{4 n ; 1 times. 6. Divide count by n yielding alp Q. The value 1 ; alp Q is a measure of failure tolerance, and the value alp Q is the proportion of times z O was undesirable. Today, it is becoming widespread for mechanical systems to be under the control of a \software" brain. The types of safety-critical mechanical systems that we envision this rare-event failure tolerance technique being applied to will consist of three key types of components: (1) a software brain, (2) sensors that feed information into the software brain, and (3) the mechanical parts that perform the desired functionality of the mechanical device. Mechanical systems are governed by physical and chemical laws of nature, and because of this, each component of the mechanical system will deviate from absolute correct behavior from time to time. Since the software brain will make controlling decisions that are based on the information received from these imperfect mechanical parts, it is vital that the mechanical 5 Since true infections are incorrect program state values, our injected infections are termed \simulated" since we cannot know if our modications in the state are forcing the program into an incorrect or possible correct state. 6

7 parts be the highest quality possible to ensure accurate information is provided to the brain. High quality mechanical parts may be prohibitively expensive{one way to circumventthis cost might be to replace high quality parts with redundant cheaper parts. Another option is of course to make the software more tolerant of the imprecisions in the incoming information. Our technique can play aninteresting role here: aiding system developers in determining the impact that varying mechanical component qualities will have on the quality ofthe entire system. For example, the technique would be able to determine what impact a less accurate sensor would have on the outputs of the software before any decision is made as to which sensors are chosen. If the impact is deemed harmful according to our technique, then the system developer has two options: (1) insert software mechanisms to \smooth out" the inaccuracies of the sensor data to a level that does not promote harmful outputs, or (2) opt for higher quality mechanical parts. If the impact of inaccurate incoming information to the software is deemed as unharmful, then the system designer has evidence that the tolerances and imperfections of a particular component are acceptable. So although our technique assesses the failure tolerance of the software, for software embedded in mechanical systems, that tolerance is with respect to imperfect mechanical systems upon which the software depends for state information, and in that sense, this technique is truly a system design tool. In summary, when testing to discover faults, high failure tolerance is annoying. Failure tolerance requires that software be robust and able to recover from unexpected problems, as well as not reveal problems if they exist. During testing, we want faults revealed during operational deployment, we want faults masked. If testing is based primarily on a particular input distribution, then inputs of low probability are less likely to trigger failures during execution. The software segments exercised by these inputs are, therefore, more likely to contain undiscovered faults. However, if the software has high failure tolerance when executing high probability input test cases, faults are still likely to remain hidden during operation i.e., the software will tolerate problems that arise during the operational life-time of the code. It is this interesting paradox that has motivated our workinthisarea. 4 Testing for Vulnerability/Security It came to our attention after much discussion with computer security researchers that their main problem is a special case of failure tolerance, where software failure in their domain is simply the failure of a system to enforce the security policies dened for the system. Hence it should be possible to test for how likely softwareistoallow for security policies to not be enforced via modications to the implementation of Algorithm 1. A vulnerability in a software system is simply a weakness in a system that allows, under some set of circumstances, an intrusion or modication of data to occur, i.e., a vulnerability allows a threat to be successful. Weaknesses could include logic aws, particular input events, or combinations of the two. We dene \level of vulnerability" (or conversely \level of security") as the degree to which access is allowed to an operational capabilitythatis supposedly protected against. For example, if access to inner layers of an operating system are supposed to be protected from users without system authorization, and if such users can frequently gain access, then the operating system will be classied as having a high level of 7

8 vulnerability. The vulnerability testing technique can be specialized to dierent kinds of threats. We began researching this technique with the previous algorithm in hand. As mentioned, that algorithm provides a prediction of the \level of safety" for anomalies such as incoming hardware failures and programmer faults. For simulated infections, we considered incoming simulated threats (such as those catalogued by Landwehr et.al. [3], COAST, and CERT). To be able to test for possible security weaknesses in software, our vulnerability testing method must be able to understand what three dierent types of events are with respect to the code: code weaknesses, input sequences to the software, and intrusions or other security violations of the code. 5 Experimenting with Dierent Input Distributions It is useful information to be able to say something about how likely software is to produce undesirable, unsafe, or vulnerable states according to some input prole, Q. But what if the software someday winds up executing in an environment that is radically dierent thanq? We now highlight recent work we have done for testing how likely the software is to produce safe, acceptable, and non-vulnerable states when the software is executing in the rarest of operating modes. We consider an operating mode to be rare if the likelihood of that mode being observed during operation is small. We have developed a testing technique that studies how software will behave when inputs are being sampled from the \ultra-dependable" region of its input space. The ultradependable region is the portion of the input space that contains inputs that are unlikely to be selected for execution under typical testing and operation. When we observeapro- gram executing on inputs that were selected from the ultra-dependable region of the input space, we gain knowledge of how robust, safe, and secure the software will be during unusual operational events. When we observe decit in these characteristics during our analysis, this triggers a warning. Likewise, if no warning is triggered from our analysis, this suggests that the software is suciently hardened against anomalous events. This suggestion is not a guarantee of future acceptable behavior, but is an indicator of that desired behavior. Q is a key parameter used by EPA, and unfortunately it may be dicult to determine what distribution best represents how your software will be used in practice [4]. You would expect that if you had two dierent distributions, q 0 and q 00,thatwere almost identical (although there were subtle dierences), that the impact of these dierences on the quality of the behavior would be small. What would you expect if you knew that q 0 and q 00 were complete opposites, meaning that q 00 were the inverse of q 0? Would you also expect small dierences on the quality ofthebehavior? We willnow summarize a hypothesis (inspired by Hecht [2]) about employing our fault-injection testing methods with the distribution represented by the complete opposite of the operational prole: If Q is your operational prole, n was a relatively small number, and j j is large, then not only are there many test cases that are not being used with the EPA algorithms nor during testing according to Q, but the likelihood that any of the rarest (most unlikely) test cases being selected is extremely small. To 8

9 compensate, you should force yourself to select some of the lowest probability events in Q, to assess how robust your code is in the rarer scenarios. Realize that when you test according to some distribution Q, even if you test an enormous number of times, the majority of the test cases used are from the subset of that are most likely to be selected. As you sample more and more according to your assumed operational prole, you are potentially getting repeat test cases. Sampling more and more does improve your chances of selecting \infrequent test cases" (which we term as the \ultra-dependable" region of the input space). But to select members of the ultra-dependable region may require sampling an intractable number of test cases if the distribution has narrow spikes. This suggests that it may be meaningful to combine the EPA algorithms with those test cases that represent the ultra-dependable region of an input space, i.e., not the operational prole. This predicts how robust the software will be during unusual operational events. When our predictions are small, this suggests that the software will not perform acceptably if unusual events occur during deployment. Likewise, if our predictions are large, this suggests that the software is suciently hardened against anomalous events. This suggestion is not a guarantee of future failure-tolerant behavior, but is an indicator of that desired behavior. The idea of inverting operational input distributions is not without limits. When you invert a prole to sample the \rarer" test cases, their \rareness" is achieved solely because of the infrequency with which they are sampled during operational use, and not because they are necessarily a small subset of. It is possible that the \rarer" test cases represent a substantial portion of. And also recognize that there are operational distributions for which inversion is useless: a uniform distribution will invert back to the original distribution. Hence distributions with greater variance will be more amenable to this process. 6 Summary We have highlighted three software characteristics that can be analyzed for by non-testing methods. Because the empirical methods that we described do not require a traditional oracle, that persistent limitation of traditional testing is avoided. Nor do these methods require formal analysis, complete and unambiguous requirements, nor any of the other usual requirements of formal methods. Our methods are computationally intensive, not manually intensive. The information they provide is unique among software assessment methods, and hence for certain applications, the computational costs of these methods can be justied. Dijsktra's famous adage of: testing cannot reveal the absence of faults is applicable to our techniques as well. If while using our technique, PRED is not satised, that does not mean that PRED can never be satised. But if you observe thatpred is satised, it is left to you to mitigate the potential of that event occurring in the future, i.e., our technique has only warned of future, pending problems, and now that mustbeinvestigated. It is possible that a particular simulated infection (which did ultimately result in a PRED-satised output) is not plausible for P to create during operation. Here, mitigation is automatic if however a PRED-satised output was observed from a plausible simulated infection, it is prudent to invoke appropriate failure-tolerance improvement procedures for thwarting the propagation of such data state corruptions. Such steps include: (1) higher quality external hardware, (2) 9

10 reduction in the possibilityofhuman factor errors, (3) redundant hardware, and (4) software assertions that are capable of warning and potentially correcting state corruptions. References [1] M. FRIEDMAN AND J. VOAS. Software Assessment: Reliability, Safety, Testability. John Wiley and Sons, New York, ISBN X. [2] H. HECHT. Rare Conditions: An Important Cause of Failures. In Proc. of the Eighth Annual Conference on Computer Assurance, pages 81{85, National Institute of Standards, June [3] C. LANDWEHR, A. BULL, J. MCDERMOTT AND W. CHOI. A Taxonomy of Computer Program Security Flaws. ACM Computing Surveys, 26(3):211{254, September [4] J. D. MUSA. Operational Proles in Software Reliability Engineering. IEEE Software, 10(2), March [5] J. VOAS AND K. MILLER. Dynamic Testability Analysis for Assessing Fault Tolerance. High Integrity Systems Journal, 1(2):171{178, [6] J. VOAS AND K. MILLER. Software Testability: The New Verication. IEEE Software, 12(3):17{28, May [7] J. VOAS, G. MCGRAW, A. GHOSH, F. CHARRON, AND K. MILLER. Dening an Adaptive Software Security Metric from a Dynamic Software Fault-tolerance Measure. In Proc. of Ninth Annual Conference on Computer Assurance, National Institute of Standards and Technology, Gaithersburg, MD, June