APPENDIX 1. ANALYZING THE TRIANGLE PROBLEM. Copyright 2004 Collard 1 & Company Appendix 1 1

Transcription

1 APPENDIX 1. ANALYZING THE TRIANGLE PROBLEM Cpyright 2004 Cllard 1 & Cmpany 1

2 N bk n sftware testing is cmplete withut a discussin f the triangle prblem. Ed Kit. (This is meant as a jke, because this prblem has been thrughly wrked and re-wrked ver the years. Nevertheless, this prblem still retains its educatinal value.) At least six self-prclaimed experts have develped answers fr the triangle prblem (including this self-prclaimed expert, the authr), and each answer is different. When experts d nt agree n the answer t a seemingly simple prblem, perhaps this tells us smething abut the state f the art f testing. The triangle prblem was mentined in Chapter III. Using intuitive, ff-the-cuff techniques, systems prfessinals were able t identify nly abut half the test cases fr the triangle prblem. This prblem prvides an pprtunity t see hw the functinal analysis, path analysis, bundary value and risk assessment methds can be applied. Re-Statement f the Prblem Develp a set f test cases t adequately test a prgram, which wrks as fllws: The prgram reads three input numbers that represent the lengths f the three sides f a triangle. Based n these three input values, the prgram determines whether the triangle is scalene (that is, it has three unequal sides), issceles (tw equal sides), r equilateral (three equal sides). The prgram displays the result n the screen. Glen Meyers riginally described this prblem in his bk, The Art f Sftware Testing. Exercise (Allw 15 t 20 minutes fr this exercise.) There is n ne crrect theretical answer t any testing prblem. The answer depends n the assumptins and judgement calls which the test planner makes. Review and critique the fllwing set f test cases fr the triangle prblem. Hw d yu suggest that we imprve them? Cpyright 2004 Cllard 2 & Cmpany 2

3 Suggested Answer (A) Minimal Test Fr the functinality specified in the exercise, sme peple may argue that a ttal f fur test cases like the fllwing are sufficient t test adequately. Fr example: Test Case Input Values Expected Results 1. 3, 3, 3 Equilateral Triangle 2. 3, 3, 2 Issceles Triangle 3. 3, 4, 5 Scalene Triangle 4. 3, 3,? Invalid Input There are pssible defects that wuld nt be detected by this set f fur test cases, hwever, s ther peple may advcate a mre thrugh test. The questin: "which ne is the crrect r mre apprpriate set f test cases?", can nly be answered in terms f the perceived risk in the situatin and the amunt f effrt we are willing t expend t avid that risk. The right amunt f testing depends n the trade-ff between the risk and ecnmics. If the risk is fairly lw, the abve minimal set f fur test cases may be adequate. If the risk is high, a mre extensive test will be needed. Tw mre extensive test plans fllw n the next several pages. Cpyright 2004 Cllard 3 & Cmpany 3

4 Suggested Answer (B) Autmated Test Given that hardware tday is s fast and cheap it is virtually free, an innvative tester has written the fllwing test prgram t exhaustively and autmatically test the triangle prgram. Review this test prgram and frm an pinin whether this autmated testing is a gd idea. If autmatin is wrthwhile, is the test prgram acceptable as written? fr i = minval t maxval by increment d fr j = minval t maxval by increment d fr k = minval t maxval by increment d enter { i, j, k } capture { result } if [ i = j = k ] and result = equilateral r if [ ( i = j ) and ( i nt = k ) and result = issceles ] r if [ ( j = k ) and ( j nt = i ) and result = issceles ] r if [ ( k = i ) and ( k nt = j ) and result = issceles ] r if [ ( i nt = j ) and ( j nt = k ) and ( k nt = i ) and result = scalene ] then k else write errr_lg [ i, j, k, result ] end end end In this test cde, minval and maxval are set t the minimum and maximum valid input data values, and increment is the smallest numerical increment recgnized by the triangle prgram (e.g., 0.01). enter is the name f a functin which drives the triangle sftware by entering values i, j and k. capture is the name f a functin which captures the result f the triangle sftware cmputatin. errr_lg is the accumulated list f defects fund in testing. Cpyright 2004 Cllard 4 & Cmpany 4

5 Is the abve autmated testing prgram effective? N -- fr tw reasns. First, many equivalent (and thus redundant) test cases are mindlessly included. Fr example, if we are willing t assume that the system behaves in the same if the input data is ( 2, 2, 2 ) r is ( 3, 3, 3 ), then we d nt have t test bth f these situatins. Secnd, there are many negative test cases (i.e., nes based n invalid input data) which have nt been identified and included in this autmated test prgram. See the next sectins fr examples f negative test cases. Cpyright 2004 Cllard 5 & Cmpany 5

6 D yu cnsider the fllwing autmated Mnte Carl test r mnkey test t be wrthwhile? d while we_want_t_keep_testing i = randm ( i ) j = randm ( j ) k = randm ( k ) enter { i, j, k } capture { result } if [ i = j = k ] and result = equilateral r if [ ( i = j ) and ( i nt = k ) and result = issceles ] r if [ ( j = k ) and ( j nt = i ) and result = issceles ] r if [ ( k = i ) and ( k nt = j ) and result = issceles ] r if [ ( i nt = j ) and ( j nt = k ) and ( k nt = i ) and result = scalene ] then k else write errr_lg [ i, j, k, result ] end where randm is a randm number generatr which is used in each iteratin t create new randm values f i, j and k, with each value within the range frm minval t maxval. The answer is that this mnkey test may be useful, prvided that we_want_t_keep_testing is nt set t a pintlessly high value, as an insurance plicy in case the assumptin f equivalence is nt crrect. I.e., the system des nt in fact behave in the same if the input data is ( 2, 2, 2 ) r ( 3, 3, 3 ), r des nt behave in the same way if the input data is ( 3, 4, 5 ) r ( 5, 12, 13 ), and s n. In additin, negative test cases are still inadequately represented. This lack f negative test cases culd be partly cvered by extending the range f randmly generated numbers t belw minval and abve maxval. Cpyright 2004 Cllard 6 & Cmpany 6

7 Suggested Answer (C) Extensive Test Validatin f the Specificatins The first step in any black-bx (i.e., specificatin-based) test planning is t validate the specificatin itself. At least 50% f system errrs riginate in the specificatins, and we cannt simply assume that the specs. are cmplete and crrect as they are presented. In this case, there is at least ne missin in the specificatin -- it des nt state whether r nt negative triangles are acceptable. Fr example, shuld the expected valid respnse t a test case f a triangle with sides f -3, -5, and -3 respectively be "issceles" r "errr"? If this system is designed fr Ph.D. mathematicians, it might allw triangles with negative sides r even unreal (cmplex number) sides. If it is designed fr furth graders, thugh, negative triangles culd be banned. This uncertainty needs t be reslved befre we can cmplete ur test case planning. The Need fr Assumptins Assumptins are critical t test planning, since the set f test cases which are needed will vary based n which assumptins are made. Assumptins need t be dcumented. The mst dangerus assumptins are thse which are nt dcumented but which are simply assumed t be true. Simply assuming that "the wrld has t wrk this way" gets testers int truble. Assumptins als need t be reviewed and cnfirmed with the subject matter expert r the final authrity n the applicatin functinality. Undcumented assumptins tend t be embedded invisibly inside test plans and are nt reviewed with the subject matter expert. Definitin f a Triangle The mst imprtant assumptin t be made is the definitin f the wrd triangle. The client fr the sftware may be a PhD mathematician (r a pre-schl child), whse definitin is different frm ur everyday, cmmn sense definitin. Cpyright 2004 Cllard 7 & Cmpany 7

8 Assumptins (cntinued) ANALYZING THE TRIANGLE PROBLEM The Oxfrd English Dictinary (OED) defines a triangle as a figure (especially plane) bunded by three (especially straight) lines. Webster s Dictinary defines a triangle as a plygn having three sides, and it further defines a plygn as a clsed plane figure bund by straight lines. The Webster s definitin is stricter and mre limiting than the OED s, because Webster s requires a plane (flat) surface and straight lines. We will use the Webster s definitin. Other Imprtant Assumptins The ther assumptins that have been made in develping the fllwing list f test cases are: The input numbers may nt have been previusly edited and cannt be assumed t be "clean", i.e., invalid values such as "?" r "A" may ccur in the input lengths f the sides f a triangle. Negative lengths are nt valid fr the sides f a triangle. Scientific ntatin is nt acceptable. (E.g., lengths cannt be entered in the frm 1.234E6, which stands fr 1,234,000 in scientific ntatin, r in the frm 1.234D6, which stands fr the same number in duble-precisin arithmetic.) If three input numbers are valid lengths fr the sides f a triangle, but d nt tgether describe a triangle that can clse, the input is invalid. (By clsing, we mean a triangle where all six ends n the three sides can meet.) Fr example, the input numbers 1, 10 and 1 may each be valid lengths in themselves, but tgether they d nt describe a triangle. Instead, they make up ne lng side (the ne with length 10) and tw shrt flaps. Cpyright 2004 Cllard 8 & Cmpany 8

9 Assumptins (cntinued) ANALYZING THE TRIANGLE PROBLEM The smallest allwable input length is assumed t be , and the largest allwable input length is assumed t be (Ideally, the functinal spec. in the exercise shuld have defined the smallest and largest allwable input numbers, s we wuld nt need t make an assumptin.) Nte: smetimes the size limits are language- and machine dependent, fr example, the limits may be based n the size f a shrt wrd n a particular platfrm. The precisin f measurement is assumed t be In ther wrds, differences between tw numbers which are less than this cannt be recgnized. All the numbers between and , fr example, are assumed t be runded t the same value, Thus a triangle with side lengths f , and will be accepted as equilateral. The three input numbers are expressed in the same units f measurement. Fr example, if the first number was expressed in feet but the last tw were expressed in inches, the input set f numbers 1, 12 and 12 wuld define an equilateral triangle. Fr extra credit: can yu find any hidden assumptins that are buried in the fllwing set f test cases fr the triangle prblem? Are any f these assumptins nt likely t be bvius t the general reader? (If we are cnfident that an assumptin will be bvius t everyne, it is nt t dangerus fr this assumptin t be undcumented. The dangerus nes are thse assumptins which are bth undcumented and nt bvius.) Cpyright 2004 Cllard 9 & Cmpany 9

10 Suggested Test Cases: I. INPUT VALIDATION 1. Verify that invalid inputs are rejected as expected. Cnfirm that nly three valid numbers can be entered as the lengths f the three sides f the triangle. (These three numbers will be referred t here as i, j, and k.) Three series f tests are needed, fr i, j and k respectively: (a) i is nt valid (i) i is nt numeric alpha; upper and lwer case numeric-alpha mix (e.g., "9A9") nn-alphanumeric (e.g., "?", blank) characters which may have special meaning in the system envirnment (e.g., Escape, Enter, Backspace) scientific ntatin (e.g., 1.2E6 r 4.56D7) (ii) (iii) i is negative (i.e., less than zer) i is zer (iv) i is abve the largest allwable input number (e.g., 999,999,999.99) (v) the number f decimal places input fr i exceeds the maximum allwed (e.g., i = when nly 2 places are allwed) Cpyright 2004 Cllard 10 & Cmpany 10

11 (vi) cmmn r likely keybard data entry "flubs" (errrs), such as a duble decimal pint (i.e., "..") (vii) i cntains leading and embedded blanks (e.g., 1 2 7" instead f 127") (Nte: in each f the abve cases, an apprpriate, clear errr message shuld be displayed.) The abve test cases are negative test cases (i.e., test cases with unacceptable input values), fr the first input number, i. Hw many negative tests are enugh? Fr example, many negative data values are nt included in the abve test cases, such as an input number cntaining an embedded blank like "1 0". Alternatively, are the abve set f test cases t many, i.e., verkill? The answer t the questin (f hw many negative test cases are apprpriate fr the input length i), depends n tw factrs: (i) (ii) The degree f risk inherent in the situatin: if the triangle functin carries a high level f averse cnsequences, lgically mre test cases are justified. The perceived hstility f the input envirnment. If the quantity f incming "garbage" is likely t be high, mre tests are justified. Cnversely, if the input lengths are nt likely t cntain many errrs, then less tests are justified. Can we utilize the cncept f equivalence t trim the number f test cases t be executed, withut materially increasing the risk that defects will nt be detected? If there is a frnt-end filter r edit which is reliable, then it is physically impssible fr invalid values f i t be received. If we are willing t assume that the input has been filtered reliably, then the number f negative test cases needed here is zer. If there is nt a reliable frnt-end edit, but we are willing t assume that all inputs fall int nly tw equivalence classes (numeric and nn-numeric), then nly ne negative test is needed. In ther wrds, if we are willing t assume that if a nn-numeric value fr i such as * is detected and rejected, then the system shuld behave in the same way fr any ther nn-numeric input value (e.g., X,? ). In this situatin (where there is ne equivalence class fr all pssible nn-numeric values), nly ne negative test case wuld be needed. Cpyright 2004 Cllard 11 & Cmpany 11

12 Is the rati f negative t psitive test cases apprpriate? Nte that the number f negative test cases listed (in item 1(a) abve, and in the remainder f items (1), (2) and (3) belw), ttal mre than 50% f all the test cases. The test cases listed in items 4 thrugh 9 are primarily psitive test cases, i.e., have acceptable input values. The answer is the right mix f negative and psitive test cases depends n the situatin. Only if the triangle sftware must have high reliability and rbustness is the number f negative test cases likely t exceed the psitive nes. Otherwise the psitive test cases shuld utnumber the negative nes. Cpyright 2004 Cllard 12 & Cmpany 12

13 1. Verify that invalid inputs are rejected as expected (cntinued). (b) j is nt valid The same types f tests are needed as fr j as were needed fr i. (c) k is nt valid The same types f tests are needed as fr k as were needed fr i. Hw much testing f the secnd and third input lengths, j and k, is apprpriate? If we can assume that the sftware engineer wh develped the triangle sftware is reasnably cmpetent, we can als assume with cnfidence that the sftware engineer wuld have prvided ne cmmn subrutine t edit all f i, j and k at the time they are input. In this situatin, a cmprehensive set f test cases fr j and k, fllwing the cmprehensive set f tests fr i, is unnecessary. The first set f test cases fr i will already have extensively exercised the cmmn edit rutine, s re-testing fr several values f j and k wuld be redundant. If the sftware wrks fr i, it shuld als wrk fr j and fr k. Only ne additinal test case is needed fr j, and als ne fr k, t duble-check that the sftware engineer at least remembered t call r access the cmmn edit rutine fr j and k as well as fr i. A better apprach (instead f many negative test cases fr i, fllwed by nly ne each fr j and k), is t spread the set f negative test cases equally amng i, j and k. Hwever, if we are paranid, we will be willing t assume nthing. Since this is a black bx test, we d nt knw fr sure hw the sftware was cnstructed internally. Althugh it may be an incmpetent practice, the sftware engineer may have written three separate input edit rutines, ne each fr i, j and k. In this circumstance, the fact that the first set f tests fr i wrk crrectly will predict nthing abut the behavir f j and k. Cpyright 2004 Cllard 13 & Cmpany 13

14 In ther wrds, the answer t the questin (f hw much testing is enugh fr j and k), depends n what assumptins we are willing t make. 2. Verify that nly and exactly three numbers can be entered successfully. (a) (b). Verify that n mre than three input lengths can be entered, r if they can be, that the system queries as t which three f multiple input numbers represent the actual three sides f the triangle. Verify that a triangle cmputatin cannt be perfrmed if nly tw values are entered (representing nly tw sides f the triangle). The user interface may well be designed in a way that makes it physically impssible t test fr nly tw numbers r fr fur numbers t be entered. In this circumstance, the abve tests are unnecessary. There is n pint in testing fr situatins which we knw cannt ccur. Hwever, in the functinal spec. (i.e., in the exercise narrative), we are nt tld anything abut the user interface. S the prudent apprach is t include these test cases, then delete them later if and when they are fund t be unnecessary. 3. Verify that the triangle must clse. Verify that invalid triangles, where the sum f any tw sides is nt large enugh t exceed the length f the third side, are detected and identified. In ther wrds, check that the input is rejected with an apprpriate errr message if any f the three fllwing cnditins d nt ccur: (a) (b) i + j = > k (and all three are numeric and psitive) (i) i + j = k (ii) i + j > k j + k = > i (and all three are numeric and psitive) (The same tw tests are needed as fr i + j = > k, abve) Cpyright 2004 Cllard 14 & Cmpany 14

15 (c) k + i = > j (and all three are numeric and psitive) (The same tw tests are needed as fr i + j = > k, abve) Of all the test cases in this answer, the mst verlked by peple wh wrk thrugh this triangle prblem is the pssibility that three numbers are entered which are independently valid but which tgether d nt frm a triangle. (Fr example, the numbers ( 1, 10 and 1 ) d nt frm a triangle.) Peple miss this pssibility either because they are mildly math-phbic, they fell asleep in high schl gemetry class, r d nt wrk with triangles as part f their day-t-day wrk. In any situatin, there is a backgrund level f knwledge that is needed in rder t identify the test cases cmpetently. In this particular situatin, peple d nt test fr a triangle that des nt clse simply because they d nt recgnize that the pssibility exists. What abut the situatin where a triangle des clse, but is flat with zer area? This ccurs with sets f input numbers like ( 3, 3 and 6), r ( 1, 2 and 3 ). Are these legitimate triangles r nt? Like the citizens f Lilliput (wh went t war ver which end f their biled eggs t break pen, the pinted end r the rund end), we prbably will have sme peple wh say yes and thers wh say n. In additin, we are assuming traditinal Euclidean gemetry, the kind we learned in high schl, where triangles are drawn n surfaces which are flat. Ph. D. mathematicians als utilize ther types f gemetry, with names like Lbatchevsky and Riemannian gemetries, which can have curved surfaces. With curved surfaces, a triangle culd have lengths f ( 1, 10 (n a wrap-arund surface), and 1 ), and still have all the ends meet s that the triangle clses. Cpyright 2004 Cllard 15 & Cmpany 15

16 If we wish t becme really esteric, there is anther unstated assumptin that we need t challenge. S far, we have assumed, withut really thinking abut it, a tw-dimensinal (flat) r three-dimensinal universe. Physicists say that the physical universe has fur dimensins, including time (and maybe mre than fur). A triangle exists in any tw r three f these fur dimensins, which means we need t ask nt nly if the edges f a triangle meet, but als when they meet. Cpyright 2004 Cllard 16 & Cmpany 16

17 II. TYPE OF TRIANGLE (internal flw lgic) The fllwing tests are based n an assumed internal lgical sftware design, as shwn by the paths n the fllwing graph. While there is n guarantee the internal sftware design actually cnfrms t this graph, the test cases based n the assumed internal flw lgic are as fllws: 4. Verify that equilateral triangles are prcessed crrectly. Verify that the displayed result is "Equilateral" when the three sides are equal (i = j = k). 5. Verify that issceles triangles are prcessed crrectly. Verify that the displayed result is "Issceles" when any tw sides are equal. This requires three tests: (a) (b) (c) i = j, and i nt equal t k (and all three inputs are numeric and psitive) j = k, and j nt equal t i (and all three are numeric and psitive) k = i, and k nt equal t j (and all three are numeric and psitive) D we need t test all three f these cnditins, 5(a), 5(b) and 5(c). If we assume equivalence, in the sense that the rder f data entry des nt matter, then any ne f these three will suffice. Fr example, we are willing t assume that if the input sequence [ 3,3,2 ] is recgnized as issceles, then s will be the sequences [ 3,2,3 ] and [ 2,3,3 ]. Hwever, what abut an input sequence like [ 2,2,10 ]? A paranid persn might wnder whether the sftware first determines that this triangle must be at least an issceles ne, based n i = j = 2, but then simply assumes that the same triangle cannt be equilateral because the third side is nt equal (it has a length f 10). Thus, the final determinatin is that [ 2,2,10 ] is by default issceles, which is an incrrect cnclusin. Cpyright 2004 Cllard 17 & Cmpany 17

18 6. Verify that scalene triangles are prcessed crrectly. Verify that if a triangle has three unequal sides, the displayed result is "Scalene". In additin t the abve set f test cases fr equilateral, issceles and scalene triangles, d we need t test fr any special types r shapes f triangles? Fr example, d we need t test specifically fr a right-angled triangle, r can we just assume that this is already cvered by the earlier, mre general test cases? Similarly, d we need t test fr any special shape (fr example a triangle with angles f 30, 60 and 90 )? The answer t this questin lies in an analysis f cmmnality versus differences. Is there anything distinct r unusual abut a right-angled triangle, fr example, that means the results f any test f a issceles r scalene triangle wuld nt als apply t the rightangled triangle? There is nthing apparently different -- if the sftware wrks fr any samples f scalene and issceles triangles, lgically it shuld als wrk fr the rightangled ne. If we have a hunch, thugh, that smething strange may ccur with the right-angled triangle, it is a gd idea still t test it, n matter what ratinal lgic tells us. III. BOUNDARY VALUES 7. Verify that extreme lengths are prcessed crrectly. Verify that bth very large and very small triangles are prcessed crrectly. Fr example, if the smallest and lngest allwable lengths f the sides are and respectively, then try the fllwing bundary values: (a) (b) (c) i = j = k = (shuld be accepted as equilateral). i = j = k = (shuld be accepted as equilateral). i = ; j = k = (shuld be accepted as issceles). Cpyright 2004 Cllard 18 & Cmpany 18

19 (d) i = , j = and k = (shuld be accepted as equilateral). These test cases may reveal defects caused by internal arithmetic verflws r runding errrs in the cmputatins. Cpyright 2004 Cllard 19 & Cmpany 19

20 IV. ASSUMED LOGIC 8. Verify that the prcess exits and shuts dwn withut any prblems. The functinal specificatin prvided in the exercise says nthing abut the pssibility f prcessing anther triangle after the first ne has been analyzed. It is likely, thugh, that this prcessing is intended t ccur in an iterative cycle. Presumably, the sftware shuld inquire at the end f prcessing ne triangle whether the user wants t d anther ne r t shut dwn. This assumptin can be plitically dangerus, because we are nw testing fr features beynd what the functinal spec. calls fr. This assumptin adds anther test case (number 8 abve). Fr example, if after prcessing ne triangle the system prmpts: "D yu want t analyze anther triangle?" and a respnse f "N" r "Escape" causes the system t freeze, this is a defect. If a respnse f "Y" r "y" des nt initialize the system ready fr the next set f inputs, this is a defect. Als, test the result when the query is answered with an input the is nt any f "Y", "y", "N" and "n". 9. Verify that the result is displayed n the screen in an acceptable manner. The exercise states that the result f the prcess must be displayed "n the screen". If the prgram prints the right result but cannt crrectly display it n the screen, it des nt meet the required functinality. T test the screen display, we need additinal infrmatin which is nt prvided in the exercise itself: What GUI display mechanisms must be supprted? (E.g., Windws, Mac, X- Windws, OS/2, CICS?) What screen technlgies must be supprted? (E.g., VGA, Super VGA, mnchrme vs. clr, different types f screen sizes?) What display frmat is acceptable? Cpyright 2004 Cllard 20 & Cmpany 20

21 If a diagram f the triangle is displayed, calculated frm the three numbers entered n the keybard (which may be suitable fr a children's math tutrial package), is its shape crrect? What if the rapidity f the input data entry exceeds the system s ability t receive and prcess the data? What abut testing fr usability? Fr example, are the instructins and errr messages meaningful t the typical user? If the delimiter amng the input data fields is the Enter key, des it wrk as expected? What if it is hit multiple times? What if the tab is used instead f the enter key? What abut testing fr internatinal users? What if the system messages are in German but mst users speak nly English? What abut the sftware dcumentatin? It is nt gd practice t deliver sftware withut adequate dcumentatin. The dcumentatin test addresses these types f questins: - Des the dcumentatin describe crrectly hw the sftware wrks? - Is the dcumentatin readable and usable? * * * * * * * * * * * * * There are ther tests that may be apprpriate t. Fr example, if the data entry f the three numbers can be perfrmed either by keystrkes r muse clicks, a reasnable test is t check that bth input methds wrk crrectly. What ther test cases can yu find? Cpyright 2004 Cllard 21 & Cmpany 21