How to Answer Essay Questions in the Graduate Level Computer Sciences Exams

Transcription

1 Hw t Answer Essay Questins in the Graduate Level Cmputer Sciences Exams Cem Kaner Flrida Tech Cmputer Sciences February The Prblem Last fall, I reviewed the grading f the Sftware Engineering cmprehensive. The mst striking prblem that I saw in the failing and brderline-passing exams was the weak rganizatin f the answers. Fr example: Several answers missed sectins f a multiple-issue questin, lsing critical pints simply because nthing was said (even thugh the student prbably knew the answer). Several answers missed the pint f the questin, and spent a lt f space n irrelevant material. Several answers were s disrganized that it tk significant wrk t find the relevant material in the mass f irrelevancy. Ultimately, sme students wh might have passed n the basis f their knwledge f the material, failed because f their weak essay-answering skills. Several f the exams in ur Department ask essay-style questins. We expect graduate students t be able t parse an essay-style questin and write an rganized answer. That expectatin is nt ging t change. These ntes suggest sme ways t imprve yur apprach t essay-style questins. Cem Kaner Essay Questins Page 1

2 SUMMARY SUGGESTIONS FOR THE TYPICAL ESSAY EXAM IN COMPUTER SCIENCE 1. STUDYING CEM KANER FEBRUARY 2002 Put tgether a list f surce materials fr the exam. This includes the relevant chapters f the curse textbk and all required readings. Als nte any references that were listed as ptinal but recmmended. These ften prvide additinal insights that will help yu understand the material (including an exam questin n the material). If yu are lucky, the instructr has prepared this fr yu already in a study guide. Get cpies f all f the surce materials (including recmmended readings) Create a list f all the key wrds used in the class. Prepare definitins fr each ne. An exam might nt have any questins that specifically ask fr definitins, but yu will ften find it useful t define a term as part f yur answer t a lng essay questin. Create a set f sample questins that cvers the material yu are studying. If yu can, get questins frm cpies f previus exams, especially thse written by the instructr wh is teaching yur curse. If yur curse text is published with a study guide, take sample questins frm the guide. If yur curse text includes practice exercises, include these in yur set f sample questins. Fr the rest, make a list f every significant tpic in the curse yu are studying, and make up questins fr every tpic nt already well cvered with the sample questins yu have n hand. Nw answer yur sample questins. N ne wuld prepare t run a race primarily by talking abut races with friends, r watching a race n televisin, r reading abut races in bks. T prepare t run a race, yu have t practice running. T prepare t write an exam, yu have t practice utlining and answering questins f the kind yu will face in the exam. Yur first attempt fr each questin shuld be pen with n time limit. Check yur lecture ntes AND any ther readings. AFTER yu have tried yur wn answers, cmpare ntes with friends. The best way t prepare fr these tests is t attempt each questin n yur wn. AFTER yu have tried yur wn answers, cmpare ntes with yur friends. I recmmend that yu study with ne r mre partners. 3-5 peple is a gd sized grup. 8 is t many. 2. STARTING THE EXAM Skim the questins Get a fast verview f the structure and cverage f the exam. Nte whether any questins are related. Smetimes, thinking thrugh the answer t ne questin can help yu plan sme f the details f yur answer t anther. Develp a budget fr yur time Cem Kaner Essay Questins Page 2

3 The exam is time-limited, but sme exams are intended t be easy t finish in the time available; thers will be high-pressure. What s the situatin with this exam? If this is a time-pressured exam, quickly build a time budget fr it. Fr example, if the exam is 100 pints lng, and yu have 60 minutes fr the exam, then yu have t earn 1.67 pints per minute. A 5-pint definitin is nly wrth 3 minutes. A 25-pint essay is wrth 15 minutes. If yu spend mre than yur budget n ne questin, yu will be shrt n anther ne. The ideal time budget shuld allw at least 10 minutes at the end f the exam fr review. Chse the rder f yur answers In almst every exam, yu can answer the questins in any rder that yu chse. If yu are nervus, start by answering the questin yu find easiest. Then the secndeasiest, and s n. By the time yu have reached the harder questins, yu will prbably have relaxed enugh t remember the relevant material. If yu are nt s nervus, answer questins in any rder that seems sensible. Fr example, sme students tend t spend t lng n shrt answer questins. They shuld answer lng essays first and cram in their shrt answers at the end, when time pressure is mre bvius. 3. ANSWERING THE QUESTIONS Lk fr the call f the questin Every well written exam questin asks specifically fr sme infrmatin. Yur task is t prvide the infrmatin that the questin is calling fr, and t nt prvide infrmatin that the questin des nt call fr. Dn t answer what has nt been asked. Fr example, if I ask yu t define ne thing, dn t define that and then give me the definitin f smething related t it. If yu d, (a) I wn t give yu extra credit, (b) I ll think that yu dn t knw the difference between the tw things, and (c) if yu make a mistake, I ll take ff pints. Give the number f items requested. Fr example, if I ask fr tw scenari tests, dn t give ne r three. If yu give ne, yu miss pints. If yu give three, I will either grade the first tw and ignre the third (this is my nrmal apprach) r grade the first tw that I happen t read (whatever their rder n the page) and ignre the third. I will never read the full list and grade what I think are the best tw ut f three. Beware f simply memrizing sme pints ff a slide. If I think yu are giving me a memrized list withut understanding what yu are writing, I will ruthlessly mark yu dwn fr memrizatin errrs. In general, if yu are repeating a set f bullet pints, write enugh detail fr them that I can tell that yu understand them. Be aware f the meaning f the questin s wrds and answer apprpriately. Fr example, these are typical meanings f sme frequently used exam wrds: Cmpare: When yu cmpare tw things, pint ut (and perhaps explain r describe) similarities between them. Cntrast: When yu cntrast tw things, pint ut (and perhaps explain r describe) differences between them. List: When yu prvide a list f items, write briefly, using nly ne r a few wrds per item. N explanatin f the items is needed. Identify: Same as list. Describe: A descriptin f smething attempts t give the reader a picture f it. A descriptin f a prcess is ften chrnlgical, explaining the steps f the prcess in Cem Kaner Essay Questins Page 3

4 rder. A descriptin f an abstract cncept includes details (perhaps examples) that make it easier fr the reader t understand r imagine. If yu are asked t describe the relatinship amng things, yu might find it easiest t wrk frm a chart r a picture. Yu are prbably nt required t use a diagram r chart, but many prfessrs will welcme it if it helps yu get acrss yur answer. If the questin asks yu t describe r define smething that is primarily visual (such as a table r a graph), yur answer will prbably be easier t write and understand if yu draw an example f what yu are defining r describing. Define: Prvide the meaning f a wrd r phrase. Check with yur prfessr abut this. Sme prfessrs expect frmal definitins. Others just want yu t describe the cncept behind the wrd. Sme prfessrs welcme examples if they help yu clarify the definitin. Others dn t. While a few prfessrs might penalize yu fr this style, when I find it hard t define r describe smething, I ften write my answer arund an example. Analyze: Nrmally means, t break a cncept r prcess r set dwn int cmpnent parts and discuss the relatinships amng the cmpnents. Smetimes yu are asked t analyze in a specific way, such as analyze the impact f X n Y. In this example, yu might explain the effects f X (and the cmpnents f X) n Y. Explain: Help the reader understand smething. Explanatins ften fcus n why r hw. Evaluate: Discuss smething in a way that allws yu t reach a judgment (a cnclusin) abut it. In yur answer, explain the ratinale behind yur cnclusin. In a mathematically r frmally riented questin, evaluate might mean t wrk thrugh the frmal material in rder t reach the result. (Think f evaluating an expressin.) Argue: Pick a psitin in a debate r cntrversy, state it, and prvide a series f arguments that supprt yur psitin. Discuss: This is a vague wrd. It is ften intended as an pen questin, inviting yu t chse the best way t address the questin. Fr example, Discuss the effect f X n Y might be best addressed as Describe and explain the effect f X n Y. If the relatinship between X and Y is cntrversial, then discuss invites yu t present multiple viewpints and the data that supprt them. If I ask yu t analyze smething accrding t the methd described in a particular paper r by a particular persn, I expect yu t d it their way. If I ask yu t describe their way, d s. If I ask yu t apply their way, yu dn t have t describe it in detail, but I expect yu t d the things they wuld d in the rder they wuld d them, and t use their vcabulary t describe what yu are ding. Outline yur answer: Unless yur answer will be trivially simple, develp a structure fr the infrmatin that yu will prvide. Often, the best utline explicitly cnnects t the questin, using the key wrds in the questin. The utline shuld explicitly cver every issue raised in the questin. If the questin cntains multiple parts, make a separate sectin fr each part. Structure yur answer accrding t the utline AND SHOW THE STRUCTURE: Fr example, use the structure f the utline as headings. Make yur rganizatin f the answer easily visible t the grader. If the questin cntains multiple parts, make a separate sectin fr each part. Cem Kaner Essay Questins Page 4

5 Prvide apprpriate amunts f infrmatin r detail. If a questin asks abut sme, that means at least tw. I nrmally expect three items in respnse t a sme. Similarly if the questin asks fr a list, I am expecting a list f at least three. If I ask yu fr the result f a calculatin, such as the number f paths thrugh a lp, shw yur calculatins r explain them. Let me understand hw yu arrived at the answer. 4. BE AWARE OF GRADER BIAS Sme factrs, in general, bias markers. These are generalizatins, based n research results. I try, f curse, t be unbiased, but it s a gd idea t keep these in mind with ANY grader fr ANY exam: Exams that are hard t read tend t get lwer grades. Suggestins: Write in high cntrast ink (such as black, medium). Write in fairly large letters. Skip every secnd line. Dn t write n the back f the page. If yur writing is illegible, print. If I can t read smething yu wrte, I will ignre it. Start a new questin n a new page. Mre generally, leave lts f space n the page. This gives yu rm t supplement r crrect yur answer later (when yu reread the exam befre handing it in) and it gives me rm t write cmments n the answer, and it makes the answer easier t read. Answers that are well-rganized are mre credible. Suggestins: If the questin has multiple parts, start each part n a new line, and identify each part at its start. In a list, start each list item n a new line maybe bullet the list. Fr example, cnsider the questin: What is the difference between black bx and white bx testing? Describe the advantages and disadvantages f each. Yu culd rganize this with five headings: Additinal pints t cnsider. 4. EXAM ERGONOMICS Use a gd pen. Lawyers and thers wh d lts f handwriting buy expensive funtain pens fr a reasn. The pen glides acrss the page, requiring minimal pressure t leave ink. If yu use a funtain pen, I suggest a medium pint (write large) t avid clgging. Als try gel pens r rllerballs. Get ne that yu can write with easily, t avid writer s cramp. Basic ballpints are very hard n yu. Lk at hw tightly yu hld it and feel hw hard yu press n the page. Study Guide Suggestins -- Page 2 Difference between black and white Advantages f black bx Disadvantages f black bx Advantages f white bx Disadvantages f white bx Cem Kaner Essay Questins Page 5

6 2. Sme Examples The fllwing examples are frm the Fall 2002 sftware engineering cmprehensive. I was nt invlved in the writing f these and s have n pride f authrship in them. Every well written exam questin asks specifically fr sme infrmatin. Fr example, cnsider this frm the Fall 2002 Sftware Engineering cmprehensive: Cmpare and cntrast tw lifecycle mdels. What advantages r disadvantages des each mdel have ver the ther? Sme students circle r underline keys wrds in a questin like this, and use arrws t highlight relatinships. They check ff circles and arrws as they answer that part f the questins. A marked-up versin f this questin might lk like: [Cmpare and cntrast] tw [lifecycle mdels.] What advantages r disadvantages des [each mdel] have [ver the ther]? This wrks well fr sme peple. I smetimes find it t cncise t wrk with, and s I generally list the elements f a multi-part questin: Cmpare tw lifecycle mdels Cntrast tw lifecycle mdels What advantages des each mdel have ver the ther? What disadvantages des each mdel have ver the ther? This defines the call f the questin. It specifies what infrmatin is requested (called fr). Ntice that yu have NOT been asked t define a lifecycle mdel (r a mdel in general) Ntice that yu have NOT been asked fr empirical research results Ntice that yu have NOT been asked fr a lnger list f mdels -- just tw, please. Ntice that when yu describe an advantage f ne mdel, it shuld be an advantage OVER THE OTHER Fr example, suppse yu cntrast spiral and evlutin. Bth f these have a cmmn advantage ver the waterfall mdel--they bth expect requirements t change ver time and have sensible ways t deal with change, whereas the waterfall des nt. HOWEVER, even thugh the ability t cpe with changing requirements is an advantage fr any incremental prcess ver waterfall, it is nt clear that spiral has this as an advantage ver evlutin (r that evlutin has this as an advantage ver spiral). Yu have t explain what makes spiral better than evlutin and what makes evlutin better than spiral. T me, the mst difficult part f this questin is the distinctin between advantages and disadvantages. Yu COULD ratinalize away this distinctin, saying that an advantage fr ne apprach is a disadvantage fr the ther, s all yu really have t d is list advantages f each. Cem Kaner Essay Questins Page 6

7 Instead, I ask Yu COULD d this, but yu SHOULD NOT. NEVER ASSUME THAT A DISTINCTION MADE IN THE EXAM QUESTION IS MEANINGLESS OR UNIMPORTANT. (Many distinctins are unimprtant -- they are just there because the exam writer is inexperienced r didn t have enugh time t carefully edit the questins. But mst distinctins (fr example between advantages OR disadvantages) are intentinal. Define the characteristics f a gd test and a bad test. Explain the differences, why gd is gd and why bad is bad. Prvide and justify a definitin f a gd tester. In analyzing a questin like this, it is essential t lk fr the call f the questin -- the specific issues yu are being asked t address: DEFINE the characteristics f a gd test DEFINE the characteristics f a bad test EXPLAIN THE DIFFERENCES: why gd is gd and why bad is bad PROVIDE and JUSTIFY a definitin f a gd tester. Many answers failed because they simply mitted sme parts f this questin. Fr example, several answers failed t cntract gd versus bad tests. Others failed t justify the definitin f a gd tester. Here s anther example: What is the mst expensive phase in sftware engineering? Justify yur answer One mre example: Parse the questin: What is THE MOST EXPENSIVE PHASE in sftware engineering? JUSTIFY yur answer. It shuld be bvius that yu are being asked t talk abut ONE phase. It shuld als be bvius that mst f the pints fr the answer t this questin will be in the justificatin f yur chice. Several failing answers wrte abut several phases instead f the ne the student cnsidered cnsider the mst expensive. In these cases, we assigned grades based n the first phase mentined by the student and ignred anything said abut any f the ther phases. Sme f the answers gt s muddled up in utlining csts acrss the lifecycle that they never drew the cnclusin (which is the mst expensive phase), and s nthing in the answer justified the answer that was never prvided. These answers culd justifiably have been given a grade f zer. Describe a prcess fr perfrming a sftware review. Clearly identify each step, and describe the purpse f each step. T answer this: Cem Kaner Essay Questins Page 7

8 Describe a Prcess fr perfrming A sftware review Fr each step in yur prcess: Clearly identify that step Describe the purpse f that step. My answer t a questin like this wuld: Start with an verview f a sftware review Then list, step by step, frm start t end, the main steps invlved in setting up, running, and using the results f the review Then take each f these steps in turn. Make a heading fr each. Discuss the purpse f the step under the apprpriate heading. 3. Exams Mst f the cmprehensives are based n a general study guide, and the first time yu see the exam questins is in the exam. A few f the exams are based n a study guide that lists the pl f questins frm which yur exam will be cnstructed -- yur studying is fcused n these questins and shuld include writing practice answers. The exams that are based n a fixed pl are graded mre harshly because we knw that yu had time t think thrugh the questin, lk up references, and write practice answers as part f yur studying. I write mre extensively abut the fixed-pl apprach in a paper that describes my apprach t testing in the Sftware Testing curse. Yu can find this paper at 4. Cmmn Prblems Many students aren t used t answering essay questins and s they deal with them ineffectively. This prblem is nt unique t cmputer science students. Fr example, teaching first year law students hw t answer essay questins is a critical task, repeated in curse after curse. 1 Many universities publish guidelines fr undergraduates n hw t study fr, and rganize, essay questins. 2 These ntes describe the prblems that I ften see and hw I handle them as a grader. The examples are all taken frm my Sftware Testing curse. 4.1 Weak Structure Cnsider the fllwing questin as an example: Define a scenari test and describe the characteristics f a gd scenari test. Imagine develping a set f scenari tests fr the Outlining feature f the wrd prcessing mdule f OpenOffice. What research wuld yu d in rder t develp a series f scenari tests fr Outlining? Describe tw scenari tests that yu wuld use and explain why each is a gd test. Explain hw these tests wuld relate t yur research. This has several cmpnents: Define a scenari test Describe the characteristics f a gd scenari test What research wuld yu d in rder t develp a series f scenari tests fr Outlining? Describe tw scenari tests yu wuld use. Cem Kaner Essay Questins Page 8

9 Explain why each f the tw scenari tests is a gd test Explain hw these tw scenari tests wuld relate t yur research A well rganized answer will have at least six sectins, ne fr each f the bulleted cmpnents. Yu might have tw additinal sectins, by splitting Describe tw scenari tests yu wuld use and Explain why each f the tw scenari tests is a gd test int tw sectins, ne fr each test. Withut structure, it is easy t miss a sectin and thereby t lse pints. Students must learn t fcus their answer t match the call f the questin (the specific issues raised in the questin). 4.2 Shtgun Answers A student using a shtgun strategy respnds with a cre dump f everything that seems t be relevant t the general tpic. Much f this infrmatin might be crrect, but if it is nn-respnsive t the call f the questin, it is irrelevant and I will ignre it. Hwever, t the extent that irrelevant infrmatin is incrrect, if I ntice an errr, I will deduct pints fr it. Here s an example f a questin that yielded a lt f shtgunning and nt enugh pints: Imagine that yu are an external test lab, and Sun cmes t yu with OpenOffice. They want yu t test the prduct. When yu ask them what test dcumentatin they want, they say that they want smething apprpriate but they are relying n yur expertise. T decide what test dcumentatin t give them, what questins wuld yu ask (up t 7 questins) and fr each answer, hw wuld the answer t that questin guide yu? In the Testing curse, we lked ver a lng list f requirements-eliciting questins. 3 Students were free t use the nes we discussed r t supply their wn. The questin des nt call fr definitin / discussin f IEEE 829 r fr a list f the cmmn test dcumentatin cmpnents. It desn t call fr a descriptin f test matrices r a discussin f hw t create them. I gt these and much mre n a recent exam. Unless this infrmatin was cuched in terms f a questin r the interpretatin f the questin/answer, it was irrelevant--a waste f the students time (and in a time-limited exam, a tax n the student s ability t cmplete the rest f the test in the time available). 4.3 Time Management Many students have time management prblems in essay exams. The cmprehensives are nt intended t be time-pressured exams. Nr are my exams. But many students run ut f time. Budget yur time. If yu have a 100 pint exam and 75 minutes, yu shuld be earning pints per minute. Dn t spend 20 minutes answering a 10-pint questin. Spend between 5 and 10 minutes, preferably 7.5 r fewer. 4.4 Lack f Preparatin If yu are writing a cmprehensive exam, d nt (nt, nt, nt) expect that the questins will be taken frm a curse that yu tk r that they will be marked the same way as yur instructr marked them. Many different peple teach the same curse. Their grading apprach, exam style, and priritizing f material are all different. The study guide fr a cmprehensive exam is the descriptin / list f material that will be n the exam. Anything in the study guide might be n the exam. Tpics nt cvered in the study guide will prbably nt be n the exam. Read the study guide. If the study guide lists a pl f questins, study them in detail and develp answers fr each ne. Cem Kaner Essay Questins Page 9

10 If the study guide lists a set f tpics and sme reference materials, Read the Reference Materials. Fr each tpic, figure ut which references prvide infrmatin n a given tpic and write ut a summary f that infrmatin. 4.5 Weak Grup Preparatin If the study guide lists a pl f questins, the best way t prepare fr these tests is fr each student t attempt each questin n her wn. The first attempt shuld be pen bk with n time limit. After each student has her wn answers, she shuld cmpare ntes with ther students. The diversity f appraches will highlight ambiguities in the questin, hidden assumptins n the part f the student, and muddled, disrganized thinking abut the structure and call f the questin. Independent preparatin by several students is essential. Unfrtunately, many students frm study grups in which they either: Divide up the questins. One r tw students attempt t answer each questin and then reprt back t the grup. The rest f the students then attempt t memrize the answers. Or, attempt t develp the answers in-grup, fur r mre students arguing and tgether. Neither f these appraches wrks well. There are s many questins in the study list that few (r n) students can effectively memrize all the answers. As a result, I ften see answer fragments, relevant material mixed with irrelevant (smething memrized fr a different questin), r answers that have been distrted (such as frgtten wrds, pints made s far ut f sequence that they dn t make sense, etc.) The grup-think apprach wrks better but ften prduces weak answers. The grup tends t latch nt the first answer that appears t make sense. Or it latches nt the answer advcated by the ludest r mst persuasive r mst persistent student in the grup. It is much mre effective t start frm a diverse grup f prepared answers, with the peple wh understand and can explain why they prepared the answers in they way they did. students are picking up cncepts in the testing curse. Appendix A: Exam Study Guide Questins Nte: I dn t include all f the fllwing questins in every list and I change the list frm year t year. I cver different tpics frm year t year and sme f these will be irrelevant in a given year. It s imprtant t keep the wrklad manageable. Depending n yur schl s culture, yu might make yur list lnger r shrter--but dn t underestimate yur students. Many students will rise t a challenge, especially if they believe yu are genuinely interested in their wrk. The exams are clsed bk. Timing, Cverage and Difficulty f the Exam The questins in each sectin belw vary in difficulty and length. In drafting an exam, I answer each questin that is a serius candidate fr inclusin in the exam and clck my answer. T clck the answer, I write the answer ut nce, t get my thinking and structure dwn. Then I write a secnd draft and time that. (Remember, students have been drafting their answers in the curse f studying fr the exam, s n the exam, they are generating the Nth draft answer.) I allw students twice as lng as it tk me t hand write my secnd draft. Fr a 75 minute exam, I cumulate questins t ttal minutes, leaving the extra minutes fr students wh write slwly. Fr example, the exam might include 4 definitins, 4 shrt answers and 2 lng answers. This particular exam ffers 100 pints wrth f questins, but sme f my 75 minute exams are ut f 95 r the ttal cunt is less imprtant than the estimated time and difficulty f the cmplete prduct. I rate questins as Easy, Medium, and Hard and drive the difficulty f the exam by the mix f the ratings. Cem Kaner Essay Questins Page 10

11 Finally, I pick the questins in a way that reasnably represents what we cvered in class. In sme cases, the questins rely n explicitly required readings rather than n material we cvered in the lecture. In sme cases, the questins n the list cver material that I dn t actually reach in time fr the exam. These questins are excluded frm the exam. Ambiguity One f the advantages f circulating the questins in advance is that the students can challenge them befre the exam. Surprisingly, a questin might be perfectly clear t the students in ne semester but ambiguus t the students in the next semester. I encurage students t draw ambiguities t my attentin. I reslve the ambiguities by sending an electrnic mail message t the students. I may exclude the questin frm the exam if the crrectin came t late r the answer t the crrected questin is t cmplex. Part 2: Shrt Answers (10 pints each) Shrt answers shuld take abut 5 minutes t answer. In planning the timing f the exam, I allw abut 6 minutes per shrt answer questin. 1. Give tw examples f defects yu are likely t discver and five examples f defects that yu are unlikely t discver if yu fcus yur testing n line-and-branch cverage. 2. Give three different definitins f sftware errr. Which d yu prefer? Why? 3. Ostrand & Balcer described the categry-partitin methd fr designing tests. Their first three steps are: (a) Analyze (b) Partitin, and (c) Determine cnstraints Describe and explain these steps. 4. A prgram asks yu t enter a passwrd, and then asks yu t enter it again. The prgram cmpares the tw entries and either accepts the passwrd (if they match) r rejects it (if they dn t). Yu can enter letters r digits. Hw many valid entries culd yu test? (Please shw and/r explain yur calculatins.) 5. A prgram is structured as fllws: It starts with a lp, the index variable can run frm 0 t 20. The prgram can exit the lp nrmally at any value f the index. Cming ut f the lp, there is a case statement that will branch t ne f 10 places depending n the value f X. X is a psitive, nn-zer integer. It can have any value frm 1 t MaxInt. In 9 f the 10 cases, the prgram executes X statements and then ges int anther lp. If X is even, the prgram can exit the lp nrmally at any value f its index, frm 1 t X. If X is dd, the prgram ges thrugh the lp 666 times and then exits. In the 10th case, the prgram exits. Ignre the pssibility f invalid values f the index variable r X. Hw many paths are there thrugh this prgram? Please shw and/r explain yur calculatins. 6. Cnsider a prgram with tw lps, cntrlled by index variables. The first variable increments (by 1 each iteratin) frm -3 t 20. The secnd variable increments (by 2 each iteratin) frm 10 Cem Kaner Essay Questins Page 11

12 t 20. The prgram can exit frm either lp nrmally at any value f the lp index. (Ignre the pssibility f invalid values f the lp index.) If these were the nly cntrl structures in the prgram, hw many paths are there thrugh the prgram? If the lps are nested If the lps are in series, ne after the ther If yu culd cntrl the values f the index variables, what test cases wuld yu run if yu were using a dmain testing apprach? Please explain yur answers with enugh detail that I can understand hw yu arrived at the numbers. 7. List and briefly explain three strengths f the waterfall lifecycle. 8. List and briefly explain three strengths f the evlutinary lifecycle. 9. Describe the characteristics f a gd scenari test. 10. List and explain fur claimed strengths f manual scripted tests and fur claimed weaknesses. 11. List (and briefly describe) fur different missins fr a test grup. Hw wuld yur testing strategy differ acrss the fur missins? 12. Distinguish between using cde cverage t highlight what has nt been tested frm using cde cverage t measure what has been tested. Describe sme benefits and sme risks f each type f use. (In ttal, acrss the tw uses, describe three benefits and three risks.) Part 3: Lng Answers (20 pints each) 4. Describe a traceability matrix. Hw wuld yu build a traceability matrix fr the wrd prcessr in OpenOffice? What is the traceability matrix used fr? What are the advantages and risks assciated with driving yur testing using a traceability matrix? Give examples f advantages and risks that yu wuld expect t deal with if yu used a traceability matrix fr the wrd prcessr. Answer this in terms f tw f the main features f the wrd prcessr. Yu can chse which tw features. 5. What is regressin testing? What are sme benefits and sme risks assciated with regressin testing? Under what circumstances wuld yu use regressin tests? 6. Why is it imprtant t design maintainability int autmated regressin tests? Describe sme design (f the test cde) chices that will usually make autmated regressin tests mre maintainable and explain (briefly) why each chice increases maintainability. 7. Imagine testing a file name field. Fr example, g t an Open File dialg, yu can enter smething int the field. D a dmain testing analysis: List a risk, equivalence classes apprpriate t that risk, and best representatives f the equivalence classes. Fr each test case (use a best representative), briefly explain why this is a best representative. Keep ding this until yu have listed 12 best-representative test cases. 8. Cnsider testing the OpenOffice functin that lets yu enter data int a spreadsheet n a Presentatin slide. Cem Kaner Essay Questins Page 12

13 Hw wuld yu develp a list f risks fr this capability? (If yu are talking t peple, wh wuld yu ask and what wuld yu ask them?) (If yu are cnsulting bks r recrds r databases, what are yu cnsulting and what infrmatin are yu lking fr in it?) Why is this a gd apprach fr building a list f risks? List 10 risks assciated with this functin. Fr each risk, briefly (very briefly) describe a test that culd determine whether there was an actual defect. 9. Imagine that yu were testing the feature, Save With Passwrd in the OpenOffice wrd prcessr. Explain hw yu wuld develp a set f scenari tests that test this feature. Describe a scenari test that yu wuld use t test this feature. Explain why this is a particularly gd scenari test. 10. Imagine that yu were testing the feature, Save With Passwrd in the OpenOffice wrd prcessr. Explain hw yu wuld develp a set f sap peras that test this feature. Describe ne test that might qualify as a sap pera. Explain why this is a gd sap pera test. 11. Imagine that yu were testing the feature, Insert Object in the OpenOffice Presentatin mdule. Explain hw yu wuld develp a set f scenari tests fr this feature. Describe a scenari test that yu wuld use t test this feature. Explain why this is a particularly gd scenari test. 12. Define a scenari test and describe the characteristics f a gd scenari test. Imagine develping a set f scenari tests fr the Outlining feature f the wrd prcessing mdule f OpenOffice. What research wuld yu d in rder t develp a series f scenari tests fr Outlining? Describe tw scenari tests that yu wuld use and explain why each is a gd test. Explain hw these tests wuld relate t yur research. 13. The racle prblem is the prblem f finding a methd that lets yu determine whether a prgram passed r failed a test. Suppse that yu were ding autmated testing f spell-checking in the OpenOffice wrd prcessr. Describe three different racles that yu culd use r create t determine whether this feature was wrking. Fr each f these racles, identify a bug that wuld be easy t detect using the racle. Why wuld this bug be easy t detect with this racle? and identify anther bug that yur racle wuld be mre likely t miss. Why wuld this bug be harder t detect with this racle? 14. Yu are using a high-vlume randm testing strategy fr the OpenOffice wrd prcessing prgram. Yu will evaluate results by using an racle. Cnsider testing the spell-checking feature using racles. Hw wuld yu create an racle (r grup f racles)? What wuld the racle(s) d? Nw cnsider the placement f ftntes at the bttm f the page. Hw wuld yu create an racle (r grup f racles) fr this? What wuld the racle(s) d? Which racle wuld be mre challenging t create r use, and why? Cem Kaner Essay Questins Page 13

14 15. Yu are using a high-vlume randm testing strategy fr the OpenOffice Presentatin prgram will evaluate results by using an racle. Cnsider inserting a spreadsheet int a slide. When yu enter values int the spreadsheet, yu can insert functins int the cells f the spreadsheet. Think abut testing thse functins using racles. Hw wuld yu create an racle (r grup f racles)? What wuld the racle(s) d? Cnsider entering a chart int a slide. Once yu have entered data int the chart, OpenOffice draws the chart. Think abut testing the chart as drawn t determine whether it prperly shws the chart data. Hw wuld yu create an racle (r grup f racles)? What wuld the racle(s) d? Which racle wuld be mre challenging t create r use, and why? 16. Imagine that yu were testing the OpenOffice wrd prcessr feature that lets yu save a dcument in HTML frmat. Describe fur examples f each f the fllwing types f attacks that yu culd make n this feature, and fr each ne, explain why yur example is a gd attack f that kind. Input cnstraint attacks Output cnstraint attacks Strage cnstraint attacks Cmputatin cnstraint attacks. (Ntes fr yu while yu study. Refer t Jrgensen / Whittaker s paper n hw t break sftware. Dn t give me tw examples f what is essentially the same attack. In the exam, I will nt ask fr all 16 examples, but I might ask fr 4 examples f ne type r tw examples f tw types, etc.) 17. What is the Defect Arrival Rate? Sme authrs mdel the defect arrival rate using a Weibull prbability distributin. Describe this curve and briefly explain three f the claimed strengths and three f the claimed weaknesses r risks f using this curve. APPENDIX B: The Study Guide that I Give Students Yur test will be sampled frm a list f questins that I give yu in advance. Thse questins include definitins, shrt answer and lng answer questins. Slutins might require shrt essays, mathematical derivatins, r cde fragments. Yu may nt use any reference materials during the test (clsed bk test). I recmmend that yu study with ne r mre partners. 3-5 peple is a gd sized grup. 8 is t many. The best way t prepare fr these tests is t attempt each questin n yur wn. Yur first attempt fr each questin shuld be pen with n time limit. Check the lecture ntes AND the required readings. AFTER yu have tried yur wn answers, cmpare ntes with yur friends. Wrking with thers will help yu discver and wrk thrugh ambiguities befre yu take the test. If a questin is unclear, send me a nte befre the test. If yu tell me early enugh, I can fix it. If a questin takes t lng t answer, send me a nte abut that t. When yu write the test, keep in mind that I am reading yur answer with the gal f finding reasns t give yu pints: If the questin cntains multiple parts, give a separate answer fr each part. If a questin asks abut sme, that means at least tw. I nrmally expect three items in respnse t a sme. Similarly if the questin asks fr a list, I am expecting a list f at least three. Cem Kaner Essay Questins Page 14

15 Be aware that different wrds in questins have different meanings. Fr example, each f the fllwing wrds calls fr a different answer: identify, list, define, describe, explain, cmpare, cntrast. If I ask yu t list and describe sme things, give me a brief identificatin (such as a name) f each and then a descriptin fr each ne. If yu find it hard t define r describe smething, try writing yur answer arund an example. If yu are asked t describe the relatinship amng things, yu might find it easiest t wrk frm a chart r a picture. Yu are nt required t use a diagram r chart (unless I ask fr ne), but feel free t use ne if it will help yu get acrss yur answer. If I ask yu t describe r define smething that is primarily visual (such as a table r a graph), yur answer will prbably be easier t write and understand if yu draw an example f what yu are defining r describing. If I ask yu fr the result f a calculatin, such as the number f paths thrugh a lp, shw yur calculatins r explain them. Let me understand hw yu arrived at the answer. If I ask yu t analyze smething accrding t the methd described in a particular paper r by a particular persn, I expect yu t d it their way. If I ask yu t describe their way, d s. If I ask yu t apply their way, yu dn t have t describe it in detail, but yu have t d the things they wuld d in the rder they wuld d them, and yu have t use their vcabulary t describe what yu are ding. The test is time-limited 75 minutes. Plan t spend n mre than 4 minutes n any definitin, n mre than 10 minutes n any shrt answer, and n mre than 15 minutes n any lng answer. Spend less n mst answers. Suppse the test has 4 definitins (20 pints), 2 shrt answers (20 pints), and 3 lng answers (60 pints). Yu shuld plan t spend, n average, abut 3 minutes per definitin, abut 8 minutes per shrt answer, and abut 12 minutes per lng answer (ttal = 64 minutes). Use the remaining 11 minutes t check yur wrk. Pick the rder f yur answers. If yu spend t lng n definitins, start writing yur lng answers first. If yu are nervus, start with the questins yu find easiest t answer. Study Guide Suggestins -- Page 2 Be aware f sme factrs that, in general, bias markers. These are generalizatins, based n research results. I try, f curse, t be unbiased, but it s a gd idea t keep these in mind with ANY grader fr ANY exam: Exams that are hard t read tend t get lwer grades. Suggestins: Write in high cntrast ink (such as black, medium). Write in fairly large letters. Skip every secnd line. Dn t write n the back f the page. If yur writing is illegible, print. If I can t read smething yu wrte, I will ignre it. Start a new questin n a new page. Mre generally, leave lts f space n the page. This gives yu rm t supplement r crrect yur answer later (when yu reread the exam befre handing it in) and it gives me rm t write cmments n the answer, and it makes the answer easier t read. Answers that are well-rganized are mre credible. Suggestins: If the questin has multiple parts, start each part n a new line, and identify each part at its start. In a list, start each list item n a new line maybe bullet the list. Fr example, cnsider the questin: What is the difference between black bx and white bx testing? Describe the advantages and disadvantages f each. Yu culd rganize this with five headings: Difference between black and white Advantages f black bx Cem Kaner Essay Questins Page 15

16 Disadvantages f black bx Advantages f white bx Disadvantages f white bx Dn t answer what has nt been asked. Fr example, if I ask yu t define ne thing, dn t define that and then give me the definitin f smething related t it. If yu d, (a) I wn t give yu extra credit, (b) I ll think that yu dn t knw the difference between the tw things, and (c) if yu make a mistake, I ll take ff pints. Give the number f items requested. Fr example, if I ask fr tw scenari tests, dn t give ne r three. If yu give ne, yu miss pints. If yu give three, I will either grade the first tw and ignre the third (this is my nrmal apprach) r grade the first tw that I happen t read (whatever their rder n the page) and ignre the third. I will never read the full list and grade what I think are the best tw ut f three. Additinal pints t cnsider. Beware f simply memrizing sme pints ff a slide. If I think yu are giving me a memrized list withut understanding what yu are writing, I will ruthlessly mark yu dwn fr memrizatin errrs. In general, if yu are repeating a set f bullet pints, write enugh detail fr them that I can tell that yu understand them. Use a gd pen. Lawyers and thers wh d lts f handwriting buy expensive funtain pens fr a reasn. The pen glides acrss the page, requiring minimal pressure t leave ink. If yu use a funtain pen, I suggest a medium pint (write large) t avid clgging. Als try gel pens r rllerballs. Get ne that yu can write with easily, t avid writer s cramp. Basic ballpints are very hard n yu. Lk at hw tightly yu hld it and feel hw hard yu press n the page. Appendix C Grading the Exam Here are examples f hw I have graded exam questins. These ntes are a bit cleaned up, t make them understandable t anther reader. Hwever, I ve left in several discussins that I inserted in my grading ntes at the time that I graded the questin. Mst f these are abut hw t grade the questin Others are abut what will have t be taught in the future, r prvided in reference materials, t help students achieve better grades next time. I ften include reference material r lists in my grading ntes. These are ften there simply t jg my memry. Smetimes, hwever, they are there t remind me f what I have t treat as acceptable. The prblem is that students, relying n reputable surces (including ther curses) will give answers that, in my humble pinin, are mistaken r ridiculus. I will accept many f these answers even thugh I disagree with them, but t maintain grading cnsistency, I need a list f what I will accept and what I will nt accept. Otherwise, my tlerance f sme f these answers will vary t much with my md. The critical features f my grading structure (what yu ll always see in my wrking ntes) are: Table frmat, with ne clumn fr each pint-deserving type f infrmatin. (One rw per student.) Fr all but the mst trivial questins, I actually fill in the table fr each questin. This gives me detailed infrmatin abut each student s perfrmance n each questin, which I use t gd advantage when a student cmes t me t questin her grade. Cem Kaner Essay Questins Page 16

17 Fr mst questins, including all cmplex questins, I shw the pints available within the clumn n the clumn itself. An utline f a sufficient answer. On a cmplex questin, I will ften paste in a list r discussin frm lecture ntes r ne f the readings. Suppse that a questin is wrth 20 pints. In the clumns, I might allw up t 30 pints. This is primarily because different peple legitimately apprach the same questin in different ways and s my grading structure has t allw fr this. N matter what the student s ttal pint cunt is, I smetimes reserve the 20 th pint fr style and rganizatin. That is, yu can get 19/20 based n the standard pint cunt, but I wn t give a perfect grade unless I think the answer is well written. This is an especially cmmn decisin when the ttal f available pints is beynd 20, and s a disrganized answer that cvers lts f grund wuld therwise get a perfect grade. Als, sme questins are just t hard. I might allw 1 r 2 pints merely fr attempting the questin. One f my frequent clumns is clue. This is a surce f discretinary pints. I define the discretinary rule n a questin-by-questin basis. Sme examples: Definitins If I allw up t 1 pint fr Clue, the default is 0 pints. If the student s answer shws mre insight than I think is reflected in the pint cunt fr the answer, I will raise the grade by 0.5 r 1 pints. If I allw up t 2 pints fr Clue, the default is prbably 0 pints, but the mst cmmn scre is prbably 1 pint. 5 pints each Pwer f a test Prbability ntin If bug is there Ability t reveal bug Stat cmparisn Grade Pwer f a test. If tw tests are ptentially capable f expsing the same type f defect, ne test is mre pwerful if it is mre likely t expse the defect. likelihd f revealing (r ability t reveal) a defect if the defect is there <I give a max f 3.5/5 if the answer desn t pint ut that this depends n whether the bug is there.> if tw tests can reveal the same defect equally well, the mre pwerful test can als reveal ther defects analgus t the cncept f statistical pwer Explratry testing Test and learn in parallel abut prduct, risks, market, test methds Dn't fllw preexisting detailed plan Whittaker's attacks Other Final / 5 Explratry Testing invlves simultaneus learning, testing, evaluating, planning, and reprting. Cem Kaner Assessment in the Sftware Testing Curse: Appendix C Page 17

18 Alternative answer: simultaneus testing and learning, plus mentin f Whittaker's attacks. We discuss Hw t Break Sftware in class. In a different class, the apprpriate surce f examples might cme frm Hendricksn s Bug Hunting slides, etc. Strage cnstraints [Nte t reader: the cncept f strage cnstraints cmes frm a paper by Whittaker & Jrgensn, that was expanded in Whittaker s bk, Hw t Break Sftware. We lk at fur fundamental types f cnstraints, input cnstraints, utput cnstraints, strage cnstraints, and cmputatinal cnstraints. Many bugs are caused by a prgrammer s failure t cnsider the pssibility f vilatin f ne f these types f cnstraints.] I d like t see smething abut data structures. I expect t see discussin f limitatin f strage in terms f the types f data that are stred r the place f strage, nt input/utput/cmputatin verflws. The attacks n strage are attacks n the data structure, hw the data is stred. Future ntes: We need the additinal details available Why Sftware Fails fr this t be a gd questin. Frm James Whittaker and Alan Jrgensen s (2000) paper, Hw t Break Sftware Data is the lifebld f sftware; if yu manage t crrupt it, the sftware will eventually have t use the bad data and what happens then may nt be pretty. It is wrthwhile t understand hw and where data values are established. Essentially, data is stred either by reading input and then string it internally r by string the result f sme internal cmputatin. By supplying input and frcing cmputatin, we enable data flw thrugh the applicatin under test. The attacks n data fllw this simple fact as utlined in attacks Hwever, withut access t the surce cde, many testers d nt bther t cnsider these attacks. We believe, thugh, that useful testing can be dne even thugh specifics f the data implementatin are hidden. We like t tell ur students t practice lking thrugh the interface. In ther wrds, take nte f what data is being stred while the sftware system is in use. If data is entered n ne screen and visible n anther, then it is being stred. Infrmatin that is available at any time is being stred. Sme data is easy t see. A table structure in a wrd prcessr is ne such example in which nt nly the data but the general strage mechanism is displayed n the screen. Sme data is hidden behind the interface and requires analysis t discver its prperties. Once the nature f the data being stred is understd, try t put yurself in the psitin f the prgrammer and think f the pssible data structures that might be used t stre such data. The mre that prgramming and data structures are understd, the easier it will be t execute the fllwing attacks. The mre cmpletely yu understand the data yu are testing, the mre successful the attacks will be at finding bugs. Twelfth attack: Apply inputs using a variety f initial cnditins Inputs are ften applicable in a variety f circumstances. Saving a file, fr example, can be perfrmed when changes have been made, and it can als be perfrmed when n changes have been made. Testers are wise t apply each input in a number f different circumstances t accunt fr the many such interactins that users will encunter when using the applicatin. Thirteenth attack: Frce a data structure t stre t many/t few values There is an upper limit n the size f all data structures. Sme data structures can grw Cem Kaner Assessment in the Sftware Testing Curse: Appendix C Page 18

19 t fill the capacity f machine memry r hard disk space and thers have a fixed upper limit. Fr example, a running mnthly sales average might be stred in an array bunded at 12 r fewer entries, ne fr each mnth f the year. If yu can detect the limits n a data structure, try t frce t many values int the structure. If the number is particularly large, the develper may have been slppy and nt prgrammed an errr case fr verflw. Special attentin shuld be paid t structures whse limits fall n the bundary f data types 255, 1023, and s n. Such limits are ften impsed simply by declaratin f the structure s size and very ften lack an verflw errr case. Underflw is als a pssibility and shuld be tested as well. This is an easy case, requiring nly that we delete ne mre element than we add. Try deleting when the structure is empty, then try adding an element and deleting tw elements and s n. Give up if the applicatin handles 3 r 4 such attempts. Furteenth attack: Investigate alternate ways t mdify internal data cnstraints The phrase the right hand knweth nt what the left hand deth describes this class f bugs. The idea is simple and develpers leave themselves wide pen t this attack; in mst prgrams there are lts f ways t d almst anything. What this means t testers is that the same functin can be invked frm numerus entry pints, each f which must ensure that the initial cnditins f the functin are met. An excellent example f this is the crashing bug ne f ur students fund in PwerPint, regarding the size f a tabular data structure. The act f creating the table is cnstrained t as the maximum size. Hwever, ne can create such a table, then add rws and clumns t it frm anther lcatin in the prgram crashing the applicatin. The right hand knew better than t allw a table but the left hand wasn t aware f the rule. Equivalence class Tw tests are members f the same equivalence class if yu expect the same results frm each. Tests are equivalent with respect t a thery f errr. Tw tests might be equivalent relative t ne ptential failure and entirely different with respect t a different ptential failure. Shrt Answers 10 pints each Nested analysis Series analysis First variable -3,0,20 Secnd variable 10,20 Used invalid values <Pints> ignre 10 Ttal Cnsider a prgram with tw lps, cntrlled by index variables. The first variable increments (by 1 each iteratin) frm -3 t 20. The secnd variable increments (by 2 each Cem Kaner Assessment in the Sftware Testing Curse: Appendix C Page 19

20 iteratin) frm 10 t 20. The prgram can exit frm either lp nrmally at any value f the lp index. (Ignre the pssibility f invalid values f the lp index.) Analysis If these were the nly cntrl structures in the prgram, hw many paths are there thrugh the prgram? If the lps are nested If the lps are in series, ne after the ther If yu culd cntrl the values f the index variables, what test cases wuld yu run if yu were using a dmain testing apprach? Please explain yur answers with enugh detail that I can understand hw yu arrived at the numbers. The first variable has 24 pssible values (-3, -2, -1, 0, 1,..., 20) Secnd variable has 6 pssible values (10, 12, 14, 16, 18, 20) a) Analysis if lps are nested. Suppse lp 1 was 1, 2, 3, Suppse lp 2 was 4,5 Lp 1 with lp 2 inside it = 2 ne thrugh lp (1,4) (1,5) 4 twice thrugh lp (1,4,2,4), (1,4,2,4,5), (1,4,5,2,4), (1,4,5,2,4,5) 8 three thrugh lp (1,4,2,4,3,4), (1,4,2,4,3,45), (1,4,2,45,3,4), (1,4,2,45,2,45), (1,45,2,4,3,4), (1,45,2,4,3,45), (1,45,2,45,3,4), (1,45,2,45,3,45) Illustrates the general rule: If N1 = number f values f the uter lp and N2 = number f values f the inner lp, Number f paths = sum (i=1 t N1) N2^i Example 2 + 2*2 + 2*2*2 fr ur sample lp The sum is therefre (sum)(i=1-t-24) 6^i E E E E+14 Cem Kaner Assessment in the Sftware Testing Curse: Appendix C Page 20