Generic Assessment Rubrics for Computer Programming Courses

Transcription

1 Gnric Assssmnt Rubrics for Computr Programming Courss Aida MUSTAPHA, Noor Azah SAMSUDIN, Nuriz ARBAIY, Rozlini MOHAMED, Isrdza Rahmi HAMID Faculty of Computr Scinc Information Tchnology, Univrsiti Tun Hussin Onn Malaysia Parit Raja, Batu Pahat, Johor, Malaysia {aidam, azah, nuriz, rozlini, ABSTRACT In ming, on problm can usually b solvd using diffrnt logics constructs but still producing th sam output. Somtims studnts gt markd down inappropriatly if thir solutions do not follow th answr schm. In addition, lab xrciss ming assignmnts ar not ncssary gradd by th instrucrs but most of th tim by th taching assistants or lab dmonstrars. This in grading inconsistncis in trms of th marks awardd whn th sam solution is bing gradd by diffrnt prson. To addrss this issu, a st of assssmnt rubric is ncssary in ordr provid flxibility for critical crativ solutions among studnts as wll as improv grading consistncis among instrucrs taching assistants or dmonstrars. This papr rports th dvlopmnt of assssmnt rubric for ach domain in computr ming courss; cognitiv, psychomor, affctiv. Th rubrics wr thn implmntd for on acadmic smstr consisting of 14 wks. An intrratr rliability analysis basd on Kappa statistic was prformd dtrmin th consistncy in using th rubrics among instrucrs Th wightd kappa is 0.810, thrfor, th strngth of agrmnt or th rliability of th rubric can b considrd b vry good. This indicats that th scoring catgoris in th rubrics ar wll-dfind th diffrncs btwn th scor catgoris ar clar. Kywords: Scoring, assssmnt rubric, computr ming, cognitiv, psychomor, affctiv, Kappa statistics. INTRODUCTION Grading ming assignmnts projcts ar similar grading traditional assignmnts such as writtn ssays. Th primary distinctions btwn thm ar th uniqu kywords or constructs across diffrnt ming languags th divrs possibl solutions associatd with a particular problm solving tchniqus. Traditional assssmnt for computr ming assignmnts projcts usually dpnds on an answr schm that includs th sourc cod as a modl answr with marks allocatd spcific lins of cod. This modl answr is thn usd by th instrucrs allocat marks th studnts s basd on th providd sourc cod in th answr schm. Th problm with th traditional schma-basd approach of awarding marks according a point-pr-statmnt is that studnts ar bing gradd basd similarity of thir solution th answr schm. This lads littl or no considration givn crativity originality in th studnt solutions. In ming, th sam problm can usually b solvd using diffrnt constructs but still producing th sam output. Studnts oftn gt markd down inappropriatly if thir solution is not xactly th sam as th instrucr s solution or altrnativly markd up if thir solution is similar th providd solution. In addition, lab xrciss ming assignmnts ar not ncssary bing gradd by th instrucrs but most of th tim by th taching assistants or lab dmonstrars. This in grading inconsistncis in trms of th marks awardd whn th sam solution is bing gradd by diffrnt prson. Instrucrs, for xampl, may mphasiz on th dsign of th solutions. Dmonstrars, on th othr h, may mphasiz on th ming syntax. To addrss this issu, a st of assssmnt rubric is ncssary in ordr provid flxibility for critical crativ solutions among studnts as wll as improv grading consistncis among instrucrs taching assistants or dmonstrars. Th litratur has rvald that stratgis usd grad ming assssmnts has volvd from grading studnts basd on an answr schm whr marks ar allocatd individual ming statmnts a mor holistic inclusiv mthodology using rubrics. A rubric is a st of ordrd catgoris which a givn pic of work can b compard. Scoring rubrics spcify th qualitis or procsss that must b xhibitd in ordr assign a particular valuativ rating for a prformanc (McDanil, 1993). As a grading ol, rubrics hav succssfully nabl th instrucrs assss th studnt s undrsting crativity a solution in ming courss (Bckr, 2003; Ahonimi Karavirta, 2009; Payn t al., 2012) as wll as valuating rsarch skills in stratgic managmnt (Whitsll Hlms, 2013), thical bhavior (Carlin t al., 2011), critical thinking in nginring (Ralsn Bays, 2010; Loon Lao, 2014), rflctiv writing in mdicin (Wald t al., 2012). 53

2 This study hypothsizs that rubrics provid th ncssary guidanc that nabl instrucrs award marks as a whol for studnts ability in problm solving, crativity, asthtics of any graphical usr intrfac as wll as th us of good ming practic stards. Th cntral focus of this rsarch will b on crating a st of rubrics as a bnchmark masur studnt larning outcoms in introducry computr ming courss offrd by th Faculty of Computr Scinc Information Tchnology (FCSIT) at Univrsiti Tun Hussin Onn Malaysia (UTHM). At prsnt, UTHM has cop with vry larg first yar classs with avrag of 70 studnts pr sction with multipl sctions catr four spcializations of undrgraduat Computr Scinc s: Softwar Enginring, Information Scurity, Wb Tchnology, Multimdia Computing. This ncssitats for mor than on instrucr taching assistants for lab sssions in ach. Du th high numbr of studnt nrollmnt divrs background of th instrucrs or dmonstrars, grading lab assignmnts group projcts is particularly a challng spcially in nsuring fair dlivry all studnts. Th main goal for this study is promot critical crativ thinking skills improv grading consistncis in ming subjcts by introducing a gnralizd ming rubric b usd across all ming languags such as C, C++, Java. Th outcom of this rsarch is abl incras th ffctivnss in taching larning activitis in trms of consistnt assssmnt of th cours larning outcoms. Th rubric dvlopd in this study is prsntd in th sction following th rlatd works. Nxt, th rsarch mthodology is dtaild out xplain th validation procss of th dvlopd rubrics followd by th findings. Finally, th papr is concludd with som indication for futur rsarch. RELATED WORK Th Outcom-basd Education (OBE) systm mphasizs th importanc of a curriculum contnt b drivn by larning outcoms (Spady, 1994). In OBE, th larning outcoms ar xprssd as statmnts of knowldg skills individual studnts should possss at th nd of th cours thy nrolld. An OBE systm offrs a comprhnsiv approach organiz oprats an ducation systm that is focusd on succssful dmonstration of larning sought from studnts at th nd of th larning cycl (Murphy Duncan, 2007). Th OBE systm has bn introducd th Faculty of Computr Scinc Information Tchnology (FCSIT) at Univrsiti Tun Hussin Onn Malaysia (UTHM) sinc Th larning outcoms of a ar st by various lvl of acadmic managmnt tam at FCSIT. Thr ar thr primary componnts of th OBE systm; Program Educational Outcom (PEO), Program Larning Outcom (), Cours Larning Outcom (CLO). Th PEO xprsss statmnts of long trm objctivs that dscrib what a Computr Scinc should b abl dmonstrat as a rsult of attnding its. Clarly, th achivmnt of th PEO at faculty lvl is gard th achivmnt of th vison mission of UTHM. Tabl 1 shows th PEO for on of th Computr Scinc undrgraduat offrd at FCSIT, which is th Bachlor of Computr Scinc (Softwar Enginring). PEO 1 PEO 2 PEO 3 PEO 4 Tabl 1: Program Educational Outcom (PEO). Apply basic knowldg, principls skills in th fild of Computr Scinc mt th job spcification. (Knowldg / Practical Skills) Implmnt th rsponsibility for solving problms analytically, critically, ffctiv, innovativ markt-orintd. (Critical Thinking Problm Solving / Lif-long Larning Information Managmnt / Entrprnurship Skills) Acts ffctivly as an individual or in a group convy information within th organization community. (Tam Working Skills / Communication Skills) Practicing good valus thics in a profssional mannr in th community abl act as a ladr. (Profsional, Social, Ethics, Humanity / Ladrship Skills) Th PEO statmnts ar furthr rfind stablish. Th s highlight individual studnt s abilitis that rflct thir larning xprincs at FCSIT. In addition, th managmnt tam of FCSIT is also considr th gnral larning objctivs st by th Malaysian Qualifications Agncy (MQA, 2008) th Ministry of Highr Education (MOHE) in xprssing th. As a rsult, th ar xprssd satisfy componnts of MQA stards which includ knowldg, practical skills, communication, critical thinking problm solving, tamwork, lif-long larning information managmnt, ntrprnurship, moral, profssional thics finally ladrship. Studnts of th undrgraduat s at FCSIT ar xpctd 54

3 acquir th upon compltion of thir studis. Th implmntation of th is h is thn distributd across individual courss in th undrgraduat s. Tabl 2 shows th for Computr Scinc s at FCSIT Tabl 2: Program Larning Outcom (). Applying knowldg undrsting of ssntial facts, concpts, principls thoris in th fild of Computr Scinc Softwar Enginring. (Knowldg K) Implmnting Softwar Enginring knowldg in analyzing, modling, dsigning, dvloping valuating ffctiv computing solutions. (Practical Skill PS) Communicat in spokn writtn form in ordr convy information, problms solutions th problms ffctivly. (Communication CS) Analyz th appropriat tchniqus in th fild of Softwar Enginring solv problms using analytical skills critical thinking. (Critical Thinking, Problm Solving CTPS) Dmonstrat tamwork skills, intrprsonal social ffctivly confidntly. (Tam Work TS) Using th skills principls of liflong larning in acadmic carr dvlopmnt. (Lif Larning Information Managmnt LL) Fostring ntrprnurship in carr dvlopmnt. (Entrprnurship ES) Adopt valus, attituds rsponsibilitis in a profssional mannr from ths aspcts of sosial, thics humanity. (Moral, Profssional Ethics EM) Effctivly carry out th rsponsibilitis of ladrship. (Ladrship LS) Th s srv as th basis of dtrmining th cours larning outcoms (CLO) for vry cours offrd. Each st of ming CLO in th cours syllabus is mappd th of FCSIT. Th mapping is known as CLO- matrix. Th CLO shall b constructd in such a way accommodat th. Th stablishmnt of th CLO in ming courss applis principls of Bloom s Taxonomy which covrs thr larning domains outlind by MQA stard: cognitiv, affctiv, psychomor (Bloom t al., 1994). Tabl 3 prsnts th complt st of lvls in ach domain. Tabl 3: Lvls in cognitiv, psychomor, affctiv domain basd on Bloom s taxonomy. Lvl Cognitiv Domain Lvl Psychomor Domain Lvl Affctiv Domain C1 Knowldg (KN) P1 Prcption A1 Rciving phnomna C2 Comprhnsion (CO) P2 St A2 Rsponding phnomna C3 Application (AP) P3 Guidd rspons A3 Valuing C4 Analysis (AN) P4 Mchanism A4 Organizing valus C5 Synthsis (SY) P5 Complx ovrt rspons A5 Intrnalizing valus C6 Evaluation (EV) P6 Adaptation P7 Origination Evntually, masur th achivmnt of cognitiv, psychomor, affctiv domain in ach CLO, a studnt is valuatd using on fiv assssmnt ols: quiz, tst, laborary assignmnts, projct, final xam. Each of th assssmnt ol is assignd nsur positiv achivmnt for th courss. Indd, such information has implication on th achivmnt of CLO that ar usually valuatd at th nd of th larning procss. Tabl 4 shows a sampl of spcification tabl valuat th cognitiv domain in an objct-orintd ming cours. Th spcification tabl is dsignd plan th distribution of marks basd on taxonomy lvl mapping. Such constructiv mapping is valuabl valuat how th CLO ar valuatd rlatd finally implis th PEO. 55

4 Tabl 4: A spcification tabl for an objct-orintd ming cours. Qustion Cours Contnt/ Topic Marks Distribution basd on Bloom s Subtal No. Taxonomy KN CO AP AN SY EV Lvl 1 Lvl 2 Lvl 3 Q1 (a) Chaptr 2: Primitiv typs 3 24 Q1 (b) Chaptr 3: Fundamntal of OO 6 Q1 (c) Chaptr 3: Fundamntal of OO 6 Q1 (d) Chaptr 4: Objct classs 9 Q2 (a) Chaptr 3: Fundamntal of OO Q2 b) Chaptr 3: Fundamntal of OO 15 Q3 (a) Chaptr 5: Inhritanc 5 25 polymorphism Q3 (b) Chaptr 5: Inhritanc 20 polymorphism Q4 (a) Chaptr 4: Objct classs 5 24 Q4 (b) Chaptr 4: Objct classs 10 Q4 (c) Chaptr 4: Objct classs 9 Subtal basd on taxonomy (Marks) Subtal for ach lvl (Marks) % Cognitiv lvl (%) 20% 52% 28% 100% Distribution of cognitiv lvl (%) 5% 35% 60% 100% At FCSIT, th spcification tabl is usd assss only th cognitiv domain via quizzs, tsts, final xams. Th assssmnt mthod is still using th answr schm. Howvr, assssmnts for lab assignmnts projcts ar not ncssary bing gradd by th instrucrs but most of th tim by th taching assistants or lab dmonstrars. This calls for th nd of a gnralizd rubric covr all continuous larning assssmnts othr than tsts final xams. RESEARCH METHODOLOGY A rubric is a st of catgoris dvlopd basd on a spcific st of prformanc critria. As an assssmnt ol, a rubric should covr all larning domains offrd in computr ming courss. Th purpos of such classification is catgoriz diffrnt objctivs that ducars st for th studnts bcaus ducars hav focus on all thr domains crat a mor holistic form of dlivry. In ordr dvlop th rubric, th first stp is th larning outcoms at th lvl followd by th cours lvl bfor th typs of assssmnts could b dtrmind. Th rubric can thn b dvlopd for a spcific typ of assssmnt such as lab assignmnts or group projcts. In this study, th rubric dvlopmnt validation procss ar foundd on th principl of continuous fdback improvmnt involving th following stps: Stp 1: Idntify Program Larning Outcoms () Cours Larning Outcoms (CLO) From th curricula, all ming courss ar slctd involving diffrnt languags (i.. C, C++, Java). Th s CLOs for ach cours wr tabulatd compard. At FCSIT, UTHM, ach cours has thr CLOs in avrag. Nxt, th assssmnt typs wr dtrmind across all th courss th prcntag of ach assssmnt typ according th CLO wr distributd. Again, th typs of assssmnt includ tsts, assignmnt, practical/lab, group projct final xamination. Tabl 5 shows th mapping of s CLOs across all ming courss. Th typs of assssmnts ar also indicatd for ach larning objctiv. From th list of assssmnt mthods providd in th tabl, quiz, tst, final xaminations in CLO1 ar gradd basd on traditional schma-basd approach bcaus th ols ar only assssing th cognitiv larning domain in computr ming. Lab assignmnts (CLO2) projcts (CLO2, CLO3), howvr, ar dsignd assss all thr larning domains; cognitiv, psychomor, affctiv. Bcaus ach CLO assss only on larning domain, th rubrics dvlopd will b catgorizd according th CLO. For ach CLO, th lvl of domain for cognitiv, psychomor, affctiv ar also assignd. 56

5 Tabl 5: Mapping of cours larning outcoms larning outcoms across all ming courss. Program Larning Outcom () Knowldg Knowldg & Practical Communication Skills Critical Thinking & Problm Solving Tam Working Skills Lif-long Larning Entrprnurship Skills Profssionalism, Social, Ethics H it Ladrship Skills Cours Larning Outcoms (CLO) CLO 1 CLO 2 CLO 3 Dsign problm solving procss basd on objct orintd concpt. Construct an objct orintd computr application using Java ming languag. Dmonstrat th implmntatio n of objct orintd concpt using any high lvl ming languag. 1 2 P4 3 4 C5 5 6 A Assssmnt Quiz, Tst, Lab, Projct, Final Examinatio n Lab, Projct Projct Prsntatio n Stp 2: Formulat th rubric In formulating th rubric, on or mor dimnsions that srv as th basis for judging th studnt work wr dtrmind. Each CLO was brokn in on or mor objctivly masurabl prformanc critria along with its sub-critria. Th basic dimnsion in th rubric is th assssmnt typ, whthr dlivrd by th studnts in th form of writtn rports or via prsntation. Nxt, for ach dimnsion, a scal of valus from 1 5 on which rat ach dimnsion is assignd; 1 is bing vry poor, 2 is poor, 3 is fair, 4 is good, 5 is xcllnt. Finally, within ach scal, th stards of xcllnc for spcifid prformanc lvls accompanid wr providd. Tabl 6 Tabl 8 show th rubric for CLO1 (cognitiv), CLO2 (psychomor), CLO3 (affctiv), rspctivly. Assssm nt Rport Tabl 6: Rubric for CLO1. Dsign problm solving procss using algorithm/objct-orintd concpts (Cognitiv C5, 4 CTPS). Critria Subcritria Lv l Idntify C2 Unabl analyz problm input/ only on ly ly output any input or som all input input output input rquirmn output output ts output ly all input output provid altrnativ Construct C3 Unabl 57

6 dmonstrat dsign solution flowchart or psudocod construc t construct but mistak on symbol construct ly construct ly us propr lmnts construct ly, us propr lmnts documntation Tabl 7: Rubric for CLO2. Construct a computr application/objct orintd computr application using objct:- orintd concpts (Psychomor P4, 2 Practical Skill) Assssmn t Critria Sub-critria Lv l P3 typ or Rport run/dbug prform input validation Appropriat choic of variabl nams or (i.. array/ linkd list) Corrct choic of squntial, slction or rptition Fr from syntax, logic, runtim rrors Validat input for rrors out-ofrang P4 P3 P3 Unabl typ or structur Unabl structur Unabl run Th s incorrc t typ or ly ly run but hav logic rror Th s not display ly Dos not chck for rrors ouf- rang typ or not not run ly without any logic rror Th s not display ly. Dos littl chck for rrors ouf- rang typ or partially partially run ly without any logic rror display inappropri at output Th works mts all spcifications. Dos som chcking for rrors ouf- rang typ or run ly without any logic rror display appropriat output Th works mts all spcifications. Dos xcption al chcking for rrors ouf- rang Prsntatio Commnt / P1 No Docum Docum Documnt Documnt 58

7 n radabl Dscription Indntation / Naming Convntion P2 docum ntation Unabl organiz th cod ntation is simpl commnt in cod Th cod is poorly organiz d vry difficult rad ntation is simpl commnt s mbdd d in cod with hadr sparatin g th cods Th cod is radabl only by a prson who alrady knows its purpos ation is simpl commnts hadr that usful in undrstan ding th cod Th cod is fairly asy rad ation is wllwrittn clarly xplains what th cod is accomplis hing Th cod is xtrmly wll organizd asy follow Tabl 8: Rubric for CLO3. Dmonstrat th implmntation of problm solving procss/objct-orintd concpts using high-lvl ming languag (Affctiv A3, 6 Liflong Larning) Assssmnt Critria Sub-critria Lv l Dmonstrat A3 Unabl xplain a xplain xplain xplain undrsting xplain littl som ntir on dsign dsign dsign dsign dsign ly dsign ly as it is provid altrnativ Prsntation dmonstrat in group Organizatio n of group prsntatio n Coopratio n from all mmbrs A4 A2 Matrials ar not organiz d with missing information Unabl cooprat in a group Matrials ar partially organiz d with missing information Forcd coopration through intrvntion Matrial s ar partially organiz d with information Dmonstrat coopration aftr intrvntion Matrials ar highly organizd with information Dmonstrat coopration through prsonal dominanc solutions Matrials ar highly organiz d with additiona l information Dmonstrat coopration through group hirarchy Th rubrics hav bn dvlopd as a 2D grid in Microsoft Excl sht, whr ach row dscribs on valuation critria th columns indicat th lvl of achivmnt. Sinc th rubric is alrady in an Excl form, th instrucrs simply fill in th studnt prformanc according th dsird column th form will add up th corrsponding valus a final scor. Stp 3: Tst th rliability of th rubric Rliability rfrs th consistncy of assssmnt scors. On a rliabl tst, a studnt would xpct attain th sam scor rgardlss of whn th studnt compltd th assssmnt, whn th assssmnt was scord, who 59

8 scord th assssmnt. In ordr masur th rliability of th rubrics, th ratr rliability in th form of rliability cofficint is masurd. Ratrs rliability rfrs th consistncy of scors that ar assignd by two indpndnt ratrs (intr-ratr rliability) that ar assignd by th sam ratr at diffrnt points in tim (intraratr rliability) (Moskal Lydns, 2000). According Jonsson Svingby (2007), th consnsus agrmnt among ratrs dpnds on th numbr of lvls in th rubric, whrby fwr lvls lad highr chanc of agrmnt. This study adoptd th masurmnt of intr-ratr rliability basd on Kappa statistics (Cohn, 1960). In Cohn s kappa, valus btwn rprsnt fair agrmnt byond chanc. Valus 0 as indicating no agrmnt as non slight, as fair, as modrat, as substantial, as almost prfct agrmnt (McHugh, 2012). EVALUATIONS Th rubrics dvlopd in this study was implmntd in thr ming courss ar offrd during th First Smstr of 2015/2016. Th courss wr Computr Programming (BIT10303) using C ming languag, Objct-Orintd Programming (BIT20603) using C++ ming languag, Java Programming (BIT33803). Th rubrics wr consistntly usd for grading lab assignmnts group projcts throughout th 14-wk priod of th smstr. All th assignmnts projcts wr gradd indpndntly by two rom instrucr or lab dmonstrar using th sam rubric. Tabl 9 shows th tal numbr of studnts works/artifacts bing compild gradd basd on th rubrics. Tabl 9: Summary of tal writtn artifacts gradd using th rubrics. Th artifacts for lab assignmnts groups projcts ar in th form of sourc cods. Cours No. of Studnts (a) No. of Instrucrs/ Dmonstrars (b) No. of Lab (c) No. of Assignmnts (d) No. of Projcts () Total Artifacts (a * (c + d + )) BIT (S1) + 37 (S2) = ,067 BIT (S1) + 37 (S2) = BIT (S1) = Total 2,513 *Si indicat sction numbr. Basd on Tabl 9, all sts of scors (i.. four sts for BIT10303, two sts ach for BIT20603 BIT33803) ar thn statistically analyzd for intr-ratr rliability using th Cohn s Kappa (Cohn, 1960). According this mtric, a Kappa of 1 indicats a prfct agrmnt, whras a kappa of 0 indicats agrmnt quivalnt chanc. Th analysis was prformd using th Statistical Packag for th Social Scincs (SPSS), vrsion Not that th instrucrs or dmonstrars ar rfrrd as ratrs in calculating th kappa valus. Two ratrs wr romly pickd valuat th ach artifact. Tabl 10 prsnts th for both ratrs on vry artifact. Tabl 10: Assssmnt for 2,513 artifacts by two indpndnt ratrs. Ratr #2 1 (vry 2 (poor) 3 (fair) 4 (good) 5 (xcllnt) poor Ratr #1 ) Total 1 (vry poor) (poor) (fair) (good) (xcllnt) ,513 Basd on Tabl 10, th tal numbr of obsrvd agrmnts is 735, which constituts 69.04% of th obsrvations. Th numbr of agrmnts xpctd by chanc is 509.1, which is 20.26% of th obsrvations. Th kappa valu is with 95% confidnc intrval from Basd on th kappa valu, th rliability of th rubrics is considrd b good basd on th strngth of agrmnt btwn th two ratrs. Howvr, this calculation only considrd xact matchs btwn th two ratrs. Sinc th scal of dimnsions 60

9 (vry poor, poor, fair, good, xcllnt) ar ordrd, clos matchs wr also bing considrd. This mans if th first ratr assssd an artifact as fair th othr as good, this is closr than if th ratr assssd th artifact as poor th othr xcllnt. Th calculation of wightd kappa assums th catgoris ar ordrd accounts for how far apart th two ratrs ar. Th wightd kappa is 0.810, thrfor, using this approach th strngth of agrmnt or th rliability of th rubric can b considrd b vry good. This indicats that th scoring catgoris in th rubrics ar wll-dfind th diffrncs btwn th scor catgoris ar clar. CONCLUSIONS A gnric ming rubric is proposd b usd across all ming courss offrd by FCSIT at UTHM involving a varity of high-lvl ming languags such as C, C++, Java. Th rubrics ar shard with th studnts vry tim a lab xrcis or assignmnt is assignd hlp thm bttr undrst th balanc of th diffrnt activitis in thir final grad. From th rubrics, studnts ar abl stimat th amount of ffort that ar achiv th prfct scor. In this way, studnts ar also playing activ rol of bcoming indpndnt in dtrmining thir own larning objctivs. In th futur, th rubrics will b usd in stablishing bnchmarks for th ming courss analyzing studnt prformanc improv th larning larning procss including making adjustmnts th curriculum. ACKNOWLEDGEMENT This projct is sponsord by th Contract Rsarch Grant from th Cntr for Acadmic Dvlopmnt Training (CAD) at Univrsiti Tun Hussin Onn Malaysia (UTHM). REFERENCES Ahonimi, T. & Karavirta, V. (2009) Analyzing th us of a rubric-basd grading ol. 14 th Annual ACM SIGCSE Confrnc on Innovation Tchnology in CS Education (pp ). Bckr, B. (2003). Grading ming assignmnts using rubrics. 8 th Annual Confrnc on Innovation Tchnology in Computr Scinc Education (pp ). ACM, Nw York, NY, USA. Bloom, B. S., Andrson, L., & Sosniak, L. (1994). Bloom s taxonomy: A forty-yar rtrospctiv. Assssing Scholarly. Chicago: NSSE. Ball, CE (2012). Carlin, N., Rozmus, C., Spik, J., Willcockson, I., Sifrt, W., Chappll, C., Hsih, P.-H., Col, T., Flaitz, C., Engbrtson, J., Lunstroth, R., Amos, C., & Boutwll, B. (2011). Th halth profssional thics rubric: Practical assssmnt in thics ducation for halth profssional schools. Journal of Acadmic Ethics, vol. 9, no. 4 (pp ). Cohn, J. (1960). A cofficint for agrmnt for nominal scals. Education Psychological Masurmnt, vol. 20 (pp.37-46). Hrman, J.L. (1992). A practical guid altrnativ assssmnt. Association for Suprvision Curriculum Dvlopmnt, 1250 N. Pitt Strt, Alxria, VA Jonsson, A. & Svingby, G. (2007) Th us of scoring rubrics: Rliability, validity ducational consquncs. Educational Rsarch Rviw, vol. 2 (2007) (pp ). Loon, J.E.V. & Lai, H.L. (2014). Information litracy skills as a critical thinking framwork in th undrgraduat nginring curriculum ASEE North Cntral Sction Confrnc. McDanil, E. (1993). Undrsting ducational masurmnt. Dubuqu, IA: William C. Brown. McHugh, M.L. (2012) Intrratr rliability: th kappa statistic. Biochm Md, vol. 22, no. 3 (pp ). Moskal, B.M. & Lydns, J.A. (2000). Scoring rubric dvlopmnt: validity rliability. Practical Assssmnt, Rsarch & Evaluation, 7(10). Malaysian Qualification Agncy (MQA). (2008). Cod of Practic for Programm Accrditation. Murphy, J.J. & Duncan, B.L. (2007). Brif intrvntion for school problms (2nd d.): Outcom-informd stratgis. Nw York: Guilford Prss. Payn, N., Kolb, D., & Kotz. G. (2012). Schming optimiz marking in computr ming: From mmos rubrics. ICERI 2012 (pp ). Ralsn, P. & Bays, C. (2010). Rfining a critical thinking rubric for nginring. Amrican Socity for Enginring Education. Spady, W.G. (1994). Outcom-basd ducation: Critical issus answrs. Amrican Association of School Administrars, 1801 North Moor Strt, Arlingn, VA Wald, H.S., Borkan, J.M., Taylor, J.S., Anthony, D., & Ris, S.P. (2012). Fostring valuating rflctiv capacity in mdical ducation: Dvloping th REFLECT rubric for assssing rflctiv writing. Acadmic Mdicin, vol. 87, no. 1 (pp.41-50). Whitsll, M. & Hlms, M.M. (2013). Assssing businss studnts rsarch skills for th capsn projct in th stratgic managmnt cours. Journal of Businss & Financ Librarianship, vol. 18, no