Automated Grading of DFA Constructions

Transcription

1 Automted Grding of DFA Constructions Rjeev Alur nd Loris D Antoni Sumit Gulwni Dileep Kini nd Mhesh Viswnthn Deprtment of Computer Science Microsoft Reserch Deprtment of Computer Science University of Pennsylvni Redmond University of Illinois t Urn-Chmpign Astrct One chllenge in mking online eduction more effective is to develop utomtic grding softwre tht cn provide meningful feedck. This pper provides solution to utomtic grding of the stndrd computtion-theory prolem tht sks student to construct deterministic finite utomton (DFA) from the given description of its lnguge. We focus on how to ssign prtil grdes for incorrect nswers. Ech student s nswer is compred to the correct DFA using hyrid of three techniques devised to cpture different clsses of errors. First, in n ttempt to ctch syntctic mistkes, we compute the edit distnce etween the two DFA descriptions. Second, we consider the entropy of the symmetric difference of the lnguges of the two DFAs, nd compute score tht estimtes the frction of the numer of strings on which the student nswer is wrong. Our third technique is imed t cpturing mistkes in reding of the prolem description. For this purpose, we consider description lnguge MOSEL, which dds syntctic sugr to the clssicl Mondic Second Order Logic, nd llows defining regulr lnguges in concise nd nturl wy. We provide lgorithms, long with optimiztions, for trnsforming MOSEL descriptions into DFAs nd vice-vers. These llow us to compute the syntctic edit distnce of the incorrect nswer from the correct one in terms of their logicl representtions. We report n experimentl study tht evlutes hundreds of nswers sumitted y (rel) students y compring grdes/feedck computed y our tool with humn grders. Our conclusion is tht the tool is le to ssign prtil grdes in meningful wy, nd should e preferred over the humn grders for oth sclility nd consistency. 1 Introduction There hs een lot of interest recently in offering college-level eduction to students world- This reserch is prtilly supported y NSF Expeditions in Computing wrds CCF , nd CCF wide vi informtion technology. Severl wesites such s EdX ( Courser ( nd Udcity ( re incresingly providing online courses on numerous topics, from computer science to psychology. Severl chllenges rise with this new teching prdigm. Since these courses, often referred to s mssive open online courses (MOOCs), re typiclly tken y severl thousnds of students locted round the world, it is prticulrly hrd for the instruction stff to provide useful personlized feedck for prctice prolem sets nd homework ssignments. Our focus in this pper is on the prolem of deterministic finite utomt (DFA) construction. The importnce of DFA in computer science eduction hrdly needs justifiction. Beside eing prt of the stndrdized computer science curriculum, the concept of DFA is rich in structure nd potentil pplictions. It is useful in diverse settings such s control theory, text editors nd lexicl nlyzers, nd models of softwre interfces. We focus on grding ssignments in which student is sked to provide DFA construction corresponding to regulr lnguge description. Our min gol is tht of utomticlly mesuring how fr off the student solution is from the correct nswer. This mesure cn then e used for two purposes: ssigning prtil grde, nd providing feedck on why the nswer is incorrect. Figure 1 shows five solutions from the ones we collected s prt of n experiment involving students t UIUC. The solutions re for the following regulr lnguge description: L = {s s contins the sustring exctly twice} For this prolem the lphet is Σ = {, }. Current technologies for this kind of prolem [Aut, 21] simply check whether the DFA proposed y the student is semnticlly equivlent to the correct one. For this prticulr exmple such technique would only point out tht the first solution A 1 is correct, while ll the other ones re wrong. Such feedck, however, does not tell us how wrong ech solution is. The four DFAs A 2, A 3, A, nd A 5 in Figure 1 re representtive of different mistkes. We first concentrte on A 2. In this ttempt the DFA ccepts the lnguge L 1 = {s s contins the sustring t lest twice} This exmple shows common mistke in this type of ssignments: the student misunderstood the prolem. We need n utomted technique tht is le to recognize this kind of

2 A 1 A 2 DFA Attempt , 6 A 1 ccepts the correct lnguge Grde: 1/1 A 2 ccepts the strings tht contin ` t lest twice insted of exctly twice Grde: 5/1, A A 3 misses the finl stte 5 3 Grde: 9/ A A 5, 1 2 3, , Grde nd Feedck A ehves correctly on most of the strings Grde: 6/1 A 5 ccepts the strings tht contin ` t lest twice insted of exctly twice Grde: 5/1 Figure 1: Exmple of DFA grding. The drk sttes re finl. Column 1 contins the nme of the DFA depicted in column 2. Column 3 shows the grde computed y our tool for the DFA with the corresponding feedck. mistke. The necessry ingredient to ddress this tsk is procedure tht, given DFA A, cn synthesize description of the lnguge L(A) ccepted y A. Here question tht immeditely rises is: wht should the description lnguge for L(A) e? Idelly we would like to descrie L(A) in English, ut such description cnnot e esily sujected to utomted nlysis. A etter option is logicl lnguge tht is not only efficient to reson out, ut one which lso provides rich set of primitives t level of strction tht is close to how lnguge descriptions re normlly stted in English. For this purpose, we extend well-known logic, clled mondic-second order logic (MSO) [Thoms, 1996; Büchi nd Lndweer, 1969], tht cn descrie regulr lnguges, nd we introduce MOSEL, n MSO-equivlent declrtive logic enriched with syntctic sugr. In MOSEL, the lnguges L nd L 1 cn e descried y the formuls indof = 2 nd indof 2 respectively. Thnks to this forml representtion, we cn compute how fr prt two MOSEL descriptions re from ech other nd trnslte such vlue into grde. To compute the distnce etween two descriptions we use n lgorithm for computing the edit distnce etween trees [Bille, 25]. We design two lgorithms: the first one computes the DFA corresponding to MOSEL description, nd conversely the second one computes the MOSEL description of the lnguge ccepted y DFA. Despite the high computtionl complexity of such lgorithms, through severl optimiztions, we were le to mke them work on exmples used to lern utomt. We executed the first lgorithm on ll the DFA ssignments ppering in [Hopcroft et l., 26], chieving running times elow 1 second. On the sme set of ssignments we were le to execute the second lgorithm on 95% of the prolems, chieving running times elow 5 seconds. The pproch presented in the previous prgrph is le to cpture prticulr clss of mistkes. However, severl DFAs, such s A 3 in Figure 1, do not fll in this clss. A 3 hs the full structure of the correct DFA ut stte 5 is not mrked s finl. A possile MOSEL description of A 3 is indof = 2 endwt where the second conjunct indictes tht ll strings must end with. This description is syntcticlly fr from the description of L cusing the corresponding grde to e too low. This exmple shows tht there should e metric tht tells how fr A 2 is from correct DFA. To ddress this clss of mistkes we introduce notion of DFA edit distnce tht given DFA A nd regulr lnguge R computes how mny sttes nd trnsitions of A we need to modify in order to otin DFA A tht ccepts R. Such computtion nturlly trnsltes into grde. The previous techniques cover two rod clsses of mistkes. However, in severl cses they re still not enough. The lnguge ccepted y the DFA A in Figure 1 hs complicted MOSEL description nd the numer of opertions needed to fix A is quite high (more thn 5) ecuse we need to dd new stte nd redirect severl edges. However, this solution is on the right trck nd ehves correctly on most of the strings. The student just did not notice tht in stte the mchine does not necessrily red the symol cusing strings such s to e rejected. Hence, A correctly rejects ll the strings tht re not in L, ut lso rejects few more. Following this intuition we introduce notion of lnguge density nd we use it to pproximte the percentge of strings in Σ on which DFA A misehves. Agin, such quntity nturlly trnsltes into grde. We finlly comine the three techniques to compute unique grde. DFA A 5, despite eing syntcticlly different from A 2, computes exctly the sme lnguge s A 2. This similrity might e hrd to notice for humn. While our tool, using the sme pproch s for A 2, ssigned the sme grde to oth the ttempts, we oserved in our experiments tht the sme humn grder ssigned different grdes. We evluted our tool on DFAs sumitted y students t UIUC nd compred the grdes generted y the tool to those provided y humn grders. First, we identified severl instnces in which two identicl DFAs were grded differently y the sme grder, while this ws not the cse for the tool. Second, we oserved tht the tool produces grdes comprle to those produced y humns. In order to check such properties, we used sttisticl metrics to compre the tool with two humn grders, nd mnully inspected the cses in which there ws discrepncy etween the grdes ssigned y the tool nd y the humn. The resulting dt suggests tht the tool grdes s well s humn, nd we often found tht, in cse of discrepncy, the grde of the humn ws less fir thn tht of the tool. 2 MOSEL: Declrtive Descriptions of Regulr Lnguges This section provides preliminry ckground on DFAs, defines the lnguge MOSEL, nd presents lgorithms for trnsforming MOSEL descriptions into DFAs nd vice-vers. 2.1 Bckground on DFAs A deterministic finite utomton (DFA) over n lphet Σ is tuple A = (Q, q, δ, F ) where Q is finite set of sttes, q Q is the initil stte, δ : Q Σ Q is the trnsition function, nd F Q is the set of ccepting sttes. We define the trnsitive closure of δ s, for ll Σ, s Σ,

3 δ (q, s) = δ (q, s), if δ(q, ) = q, nd δ (q, ε) = q. The lnguge ccepted y A is L(A) = {s δ (q, s) F }. 2.2 The Lnguge MOSEL MOSEL ws designed with the gol to e () expressive enough to descrie prolems tht rise in common ssignments, () simple enough to hve close correspondence to nturl lnguge descriptions so tht the syntctic distnce etween MOSEL descriptions reflects the distnce etween their English lnguge descriptions, nd (c) succinct enough to hve smll descriptions for common DFA ssignments. The syntx nd semntics of MOSEL cn e found in the full version of this pper [Alur et l., ]. We illustrte fetures of this lnguge through some exmples. The 5 lnguges descried in the first column of Tle 1 cn e descried y the following MOSEL formuls: L 1 = egwt indof % 2 = 1: strings tht strt with n nd hve nd odd numer of sustrings; L 2 = indof 2 indof 2: strings tht contin t lest two s or t lest two s; L 3 psle x % 2 = 1}: strings where every odd position is leled with n ; L = egwt ll % 3 : strings tht strt with nd with length not divisile y 3; L 5 = indof = 2: strings tht contin the sustring exctly twice; nd L 6 = indof 1 endwt : strings tht contin the sustring t lest once nd end with. 2.3 From MOSEL to DFAs Next, we descrie how we trnsform MOSEL formul φ over n lphet Σ into the corresponding DFA A φ, such tht A φ descries the sme lnguge s φ. Since MOSEL only dds syntctic sugr to Mondic Second Order Logic (MSO) over strings, MOSEL formuls cn e trnsformed into equivlent MSO formuls. In the first step of our trnsformtion we inductively trnsform φ into n MSO formul φ. Next, we use stndrd techniques to trnsform n MSO formul into the corresponding DFA [Henriksen et l., 1995]. Such techniques inductively generte the DFAs corresponding to ech su-formul of φ nd then comine such DFAs using utomt opertions. In the trnsformtion from MSO to DFA, the lphet is enriched with itvectors tht represent the vlues of the quntified vriles, cusing the lphet to grow exponentilly in the numer of nested quntifiers. We implemented the trnsformtion using the Automt lirry [Venes nd Bjørner, 212]. This lirry relies on BDDs to succinctly represent lrge lphets, mking our trnsformtion efficient in prctice. During the inductive trnsformtion we lwys keep the minimized DFA in order to void low-up in the numer of sttes. For every exercise E ppering in [Hopcroft et l., 26], our tool generted the DFA from the corresponding MOSEL description of E in less thn 1 second. 2. From DFAs to MOSEL While it is well known tht every DFA cn e trnsformed into n equivlent MSO formul, in the stndrd trnsformtion from DFA to MSO, the distnce etween MSO representtions is not meningful s it directly reflects the DFA structure rther thn the ccepted lnguge. Since our gol is to use MOSEL descriptions to cpture the syntctic difference etween two lnguges, we use different pproch. Given DFA A, we use n itertive deepening serch to enumerte ll possile MOSEL formuls nd find the one tht descries L(A). As we showed in the previous susection, MOSEL descriptions of common DFA ssignments re succinct. Thnks to such succinctness the rute force pproch works dequtely for our purpose. Since typicl formuls hve smll size/width, we use n itertive deepening serch on the width of formul. We next descrie some optimiztions tht mke this pproch fesile in prctice. In order to check whether formul φ is equivlent to the trget lnguge, the simplest pproch would e tht of using the lgorithm of Section 2.3 to generte the utomton A φ corresponding to φ nd then run DFA equivlence lgorithm. Such procedure does not scle in prctice. In our implementtion we first check whether φ ehves correctly on some selected inputs nd, only if it psses this test, we compute the DFA for φ nd perform the equivlence test. Next, we descrie the procedure for generting the input test set. Given the input DFA A = (Q, q, δ, F ), we compute two sets P nd N of positive nd negtive exmples respectively, tht is, P L(A), nd N L(A) =. For every two sttes q 1, q 2 Q, we dd the string s q1 p q1 to P, nd the string s q2 n q2 to N, where δ (q, s qi ) = q i nd δ (q i, p qi ) F nd δ (q i, n qi ) F. Similrly, for every two sttes q 1, q 2 Q, nd for every Σ, we dd the string s q1 p δ(q1,) to P nd the string s q2 n δ(q2,) to N. The sets P nd N contin t most Q 2 Σ strings. Finlly, for every DFA B = (Q B, q B,, δ B, F B ) such tht Q B Q A, it is enough to test Q B on the test sets P nd N in order to check whether L(A) L(B). Therefore, if the miniml DFA corresponding to φ hs fewer sttes thn A, we detect the inequivlence of φ nd A y simply testing the formul on P nd N. In our experiments this technique yields 3X speed-up. Next, we optimized the lgorithm for common cses in which the trget formul is disjunction or conjunction. When we come cross formuls φ 1, φ 2 which pss the negtive test set N during the enumertion, we check if φ 1 φ 2 psses the positive test set P nd follow it with n equivlence check if needed. Conjunctions re hndled in similr mnner. This optimiztion permits to identify formuls of ig width erly in the serch. For exmple, the tool ws le to synthesize the formul descriing the lnguge in Figure 2 in less thn second. Without this optimiztion it would hve tken more thn 5 minutes to rech the corresponding formul in the enumertion. Finlly, we implemented severl pruning techniques. First, using syntctic properties of the input DFA we cn void enumerting some formuls. For exmple, if the DFA hs only loops of size 1, nd 2, the formul % 3 = cn e stticlly ruled out. Secondly, we stticlly remove severl terms tht re equivlent to other terms of smller width ( φ). These pruning techniques cused 5X speed-up on our test set of 3 exmples. We tested the lgorithm on the exercises in [Hopcroft et l., 26] for which the lphet only

4 contined two elements.. For ech exercise, given the corresponding DFA solution A, we were le to generte MOSEL description of A in less thn 5 seconds for 95% of the prolems. In few cses, the MOSEL description ws too ig nd the tool ws not le to generte it. 3 An Algorithm for Grding DFA Constructions We next ddress the prolem of grding student ttempt. Given trget lnguge L T, nd student solution A s, we need metric tht tells us how fr A s is from correct solution. Bsed on our experience relted to teching nd grding DFA constructions, we identified three clsses of mistkes: Prolem Syntctic Mistke: the student gives solution for different prolem (see (2) nd (5) in Figure 1); Solution Syntctic Mistke: the student omits trnsition or finl stte (see (3) in Figure 1); nd Prolem Semntic Mistke: the solution is wrong on smll frction of the strings (see () in Figure 1). We investigted three pproches tht try to ddress ech clss. First, we use the clssic notion of tree edit distnce [Go et l., 21] to compute the difference etween two MOSEL formuls. Secondly, we introduce notion of DFA edit distnce to cpture the distnce etween DFAs. Lst, we use the concept of regulr lnguge density to compute the difference etween two lnguges when viewed s sets. 3.1 Prolem Syntctic Distnce The following metric cptures the cse in which the MOSEL description of the lnguge corresponding to A s is close to the MOSEL description of the trget lnguge L T. This metric computes how syntcticlly close two MOSEL descriptions re. We consider MOSEL formuls s the ordered trees induced y their prse trees. Given MOSEL formul φ, we cll T φ its prse tree. Given two ordered trees t 1 nd t 2, their tree edit distnce TED(t 1, t 2 ) is defined s the minimum numer of edits tht cn trnsform t 1 into t 2. Given tree t, n edit is one of the following opertions: relel: chnge the lel of node n; node deletion: given node n with prent n, 1) remove n, 2) plce the children of n s children of n, inserting them in the plce left y n; nd node insertion: given node n, 1) replce consecutive susequence C of children of n with new node n, nd 2) let C e the children of n. We use the lgorithm in [Go et l., 21] to compute TED. Next, we compute the distnce D(φ 1, φ 2 ) etween two formuls φ 1 nd φ 2 s TED(T φ1, T φ2 ). Finlly, we compute WTED(φ 1, φ 2 ) = def D(φ 1, φ 2 )/ T φ2, where T is the numer of nodes in T. In this wy, for the sme numer of edits, less points re deducted for lnguges with igger description. Since we re ultimtely interested in grding DFAs, given DFA A s we use the procedure proposed in 2. to compute the formul φ As corresponding to A s. Exmple 1 Consider the lnguge L corresponding to φ = def indof % 2 = 1 egwt over the lphet Σ = {, }. Let s ssume the student provides the DFA A tht implements the lnguge φ = def indof % 2 = 1 egwt, where hs een replced y. The prolem syntctic distnce will yield the following vlues: TED(φ, φ) = 1, nd WTED(φ, φ) = 1/9. In this cse pplying one node releling is enough to fix T φ. We omit the prse tree of φ which contins 9 nodes. 3.2 Solution Syntctic Difference The following metric cptures the cse in which the student DFA A s is syntcticlly close to correct one, y computing how mny edits re needed to trnsform A s to mke it ccept the correct lnguge L T. We define the notion of DFA edit distnce. Given two DFAs A 1, A 2, we sy tht the difference etween A 1 nd A 2, DFA-D(A 1, A 2 ) is the minimum numer of edits tht cn trnsform A 1 into some DFA A 1 such tht L(A 1) = L(A 2 ). Given DFA A, n edit is one of the following opertions: trnsition redirection: given stte q nd symol Σ, updte δ(q, ) = q to δ(q, ) = q where q q ; stte insertion: insert new disconnected stte q, with δ(q, ) = q for every Σ; nd stte releling: given stte q, dd it or remove it from the set of finl sttes. To tke into considertion the severity of mistke sed on the difficulty of the prolem, we compute the quntity WDFA-D(A 1, A 2 ) = def DFA-D(A 1, A 2 )/k + t, where k nd t re, respectively, the numer of sttes nd trnsitions of A 2. Exmple 2 Consider the DFA A 3 in Figure 1 where stte 5 is mistkenly mrked s non-finl. A 1 is the correct solution for the prolem. In this cse DFA-D(A 3, A 1 ) = 1 WDFA-D(A 3, A 1 ) = 1/12+6 = 1/18 since pplying one stte releling will fix A 3. In the tool we compute this metric y trying ll the possile edits nd checking for equivlence with technique similr to the one presented in Section 2.3. A similr distnce notion grph edit distnce [Bille, 25]. However this metric does not tke into ccount the lnguge ccepted y the DFA. 3.3 Prolem Semntic Difference The following metric cptures the cse in which the DFA A s ehves correctly on most inputs, y computing wht percentge of the input strings is correctly ccepted/rejected y A s. Given two lnguges L 1 nd L 2, we define density difference to e DEN-DIF(L 1, L 2 ) def = lim n + ((L 1 \ L 2 ) (L 2 \ L 1 )) Σ n mx( L 2 Σ n, 1) Σ n denotes the set of strings in Σ of length n. Informlly, for every n, the expression E(n) inside the limit computes the numer of strings of length n tht re misclssified y L 1 divided y the numer of strings of length n in L 2. The mx in the denomintor is used to void divisions y. Unfortuntely, the density difference is not lwys defined, s the limit my not exist. Exmple 3 Consider the lnguges L A corresponding to ll % 2 = nd L B corresponding to true (i.e. Σ ) over

5 the lphet Σ = {, }. The limit DEN-DIF(L A, L B ) is not defined since it keeps oscillting etween nd 1. In prctice we compute the pproximted density A-DEN-DIF(L 1, L 2 ) = def ( 2k n= E(n))/2k + 1, where k is the numer of sttes of the minimum DFA representing L 2. This pproximtion is not precise, ut it is very helpful for cpturing the cses in which the student forgot finite numer of strings in its solution (for exmple only ε). Exmple Consider the DFAs A 1 nd A in Figure 1 nd their respective lnguges L(A 1 ) nd L(A ). In this cse A-DEN-DIF(L(A ), L(A 1 )) =.9. This vlue is the one used to compute the grde shown in Figure 1. Similr notions of density hve een proposed in the literture [Bodirsky et l., 2; Kozik, 25]. These definitions hve good theoreticl foundtions, ut, unlike our metric, they re undefined for most DFAs. 3. Comining the Approches The forementioned pproches need to e comined in order to compute the finl grde. We re wre of the mny mchine lerning techniques tht could e used for comining the three fetures, ut insted, we decide to use simple comintion function for the following reson : 1) in the future we would like to extrct feedck informtion from the computed grde, nd 2) in generl, only one of the three feture succeeds in computing positive grde. Next, we provide the generl schem of the comining function. First, ech deduction v, which rnges etween nd 1, is scled to new vlue v using formul of the form v := (v + c) 2 c 2 where c is constnt. We used trining set of 6 mnully grded ttempts to identify the constnts c for the comining function. Finlly, we pick the metric which wrded the highest score. Experimentl Evlution The im of our experiment is to evlute to wht extent the grdes given y our tool nd those given y humn instructors gree. To do so we collected round 8 ttempts t DFA construction questions y students tking theory of computtion course for the first time. For ech prolem we hd two instructors nd our tool grde ech ttempt seprtely. In order to see how well the tool does we compre sttistics tht revel vrition etween humn grders nd vrition etween humn grder nd our tool. To mesure the extent of greement etween two grders we employ Person s correltion coefficient. The correltion coefficient is numer etween -1 nd 1. A vlue of 1 indictes tht the pired points re linerly relted with positive slope. When this quntity is closer to 1 it indictes tht the two mesurements eing compred tend to vry together in the sme direction. In order to otin sis for compring the correltion coefficients we lso see how nive grder would perform with respect to humn grders. There could e mny wys to define nive grder. A simple one tht we consider uses the following grding scheme: (i) it wrds ner mximum (9 or 1) mrks to the correct solutions, nd (ii) for incorrect solutions it deducts mrks sed on the numer of sttes tht (1) (2) (3) ,, () (5) ,, Figure 2: Selected ttempts for the lnguge L 1 = {s s strts with nd hs odd numer of sustrings}. re lcking or in excess nd dds smll rndom noise. We summrize the resulting clcultions in Tle 1. Detiled Anlysis In the following we only consider the first prolem where the lnguge is L 1 = {s s strts with nd hs odd numer of sustrings}. The first column in the verges reds.99 for H 1 -H 2 mening tht H 1 hs wrded, on verge, 1 point more thn H 2. The next two columns show tht H 1 is on verge closer to the nive grder N nd to the tool T thn it is to H 2. However, the stndrd devition for H 1 -N (2.62) is greter thn tht for H 1 -T (1.99), which mens tht the grdes given y our tool show lot less vrition, nd re in fct closer to H 1 more often thn N. The Person correltion coefficients shows tht the degree of correltion etween the tool T nd H 1 (.83) is clerly etter thn tht etween N nd H 1 (.65), nd t the sme time comes very close to the degree of correltion etween two humn grders H 1 nd H 2 (.87). We sy tht two grders gree on n ttempt with threshold of t if the grdes given y the two grders do not differ y more thn t. The plot on the right shows 3 curves. Ech curve compres two grders 9 1 nd displys how 8 the percentge of 7 prolems on which H1- N 6 they gree increses H1- H2 5 with the threshold H1- T vrying from to 3 1. The three curves Percentge of,empts within X Grde Difference compre T, H 2 nd N ginst H 1, nd it is esy to see tht our tool T comes to n greement much fster thn N. More surprisingly, the tool lso comes to n greement fster thn H 2. Figure 2 shows five cses in which either the humn grders nd the tool hve discrepncy. In cse (1) the computed lnguge L 1 is descried y the formul indof % 2 = 1 egwt. Since the MOSEL descriptions of L 1 nd L 1 re similr, the tool gives high grde for this ttempt. However, L 1 hs n esier construction thn L 1. We ttenute these flse high grdes y deducting extr points when the size of the miniml DFA corresponding to the student solution hs less sttes thn the trget DFA. Cse (2) shows DFA for which the lnguge density is low, cusing the tool to wrd this ttempt with 7 points. One cn rgue tht this grde is too high for such DFA. However, the sme

6 Attempts Averge Stndrd Devition Person Correltion Prolem Tot. Dis. H 1-H 2 H 1-T H 1-N H 1-H 2 H 1-T H 1-N H 1-H 2 H 1-T H 1-N L 1 = {s s strts with nd hs odd numer of sustrings} L 2 = {s s hs more thn 2 s or more thn 2 s} L 3 = {s s where ll odd positions contin the symol } L = {s s egins with nd s is not divisile y 3} L 5 = {s s contins the sustring exctly twice} L 6 = {s s contins the sustring nd ends with } Tle 1: Compring grdes given y humns nd tool. The grdes were etween nd 1. H 1 nd H 2 denote the two humn grders, T the tool, nd N the nive grder. A-B denotes the difference etween the grders A nd B when grding ech individul ttempt. For ech prolem L i the tle shows in the order: the numer of student ttempts, the numer of distinct ttempts, the verge difference, the stndrd devition, nd the Person s correltion etween single ttempt grdes. DFA s (2) ws sumitted y multiple students, nd, while the tool lwys wrded 7 points, oth humn grders were inconsistent: H 1 grded five identicl ttempts with,5,7,7, nd 7 points, while H 2 wrded 1,1,2,3, nd 8 points. Cse (3) shows DFA for which the DFA edit distnce yields too generous grde. For this DFA it is enough to remove the trnsition δ(, ) in order to otin DFA tht ccepts L 1. However, in this cse the mistke is deeper thn simple typo. Cse () shows DFA for which the humn wrded too high score. Even though this DFA hs severl syntctic mistkes, H 1 wrded the sme grde s for ttempt (5), where only one finl stte is missing. For cse (5), where stte 3 ws mistkenly mrked s non-finl, oth H 1 nd H 2 lcked in consistency. H 1 wrded different grdes from 8 to 1 for 7 identicl ttempts. Strengths of the tool Inspecting the dt for the 131 ttempts for the lnguge L 1 we oserved the following: 1) in 6 cses one of the humn grders mistkenly ssigned the mximum score to n incorrect ttempt; 2) in more thn 2 out of 3 cses in which T nd H 1 were disgreeing y t lest 3 points, H 1, fter reviewing the ttempt, greed with the grde computed of T; nd 3) in more thn 2 cses t lest one of the humn grders ws inconsistent: i.e. two syntcticlly equivlent ttempts were grded differently y the sme grder. Limittions The tool suffers two types of limittions: ehviorl nd structurl. The former type concerns the filure of the grding techniques on some prticulr exmples. An exmple is the ttempt (3) of Figure 2 where smll DFA edit distnce did not reflect the severity of the mistke. As for the structurl limittions, our techniques re crfted to perform well on prolems ppering in theory of computtion ooks [Sipser, 1996; Hopcroft et l., 26]. Such techniques do not scle for DFAs with lrge lphets or mny sttes. Moreover the MOSEL edit distnce fils when the lnguge does not dmit succinct description. We do not elieve these to e ctul limittions, ecuse such situtions typiclly do not rise in undergrdute level DFA constructions. However, we pln on extending our tool to del with these cses. 5 Relted Work JFLAP [Rodger nd Finley, 26] is mture system used widely for teching utomt nd forml lnguge theory. To the est of our knowledge JFLAP does not offer n utomtic grding of DFA constructions. Benedikt et.l hve proposed notion of regulr lnguge repiring [Benedikt et l., 211]. Their pproch could lso e used for defining grding metric, however, we elieve tht such metric would not e good fit for our purposes since it cnnot e ssocited with ny nturl feedck. Singh et.l. hve proposed n utomtic grding frmework for progrmming prolems [Singh et l., 213]. Given error model in the form of expression rewrite rules, their system uses SAT sed techniques to find the miniml numer of corrections tht cn fix the student solution. Our syntctic edit distnce pproch is similr to this proposl. There hs een lot of work in the AI community for uilding utomted tutors for helping novice progrmmers lern progrmming y providing feedck out semntic errors [Adm nd Lurent, 198; Murry, 1987]. Such technologies shre similr ides with our tool in the wy they mesure distnce from correct solution. 6 Conclusion We investigted the prolem of grding DFA constructions. First, we introduced MOSEL, declrtive logic le to provide succinct nd nturl descriptions of regulr lnguges ppering in utomt theory textooks. Second, we provided lgorithms for trnsforming MOSEL descriptions into DFAs nd vice-vers. Lst, we presented three grding techniques sed on three different clsses of mistkes. We evluted our tool on DFAs sumitted y rel students nd compred the grdes generted y the tool to those provided y humn grders. The results re encourging nd show tht the tool grdes s well s humn. In fct we pln on further engineering our tool, deploy it, nd soon use it in rel courses. We lso pln on storing prolem solutions in dtse in order to speed up the grding nd fix the few cses in which the tool does not ssign fir grde. We elieve our techniques cn provide foundtions for generting utomted feedck for students, nd cn e lso dpted for other types of constructions. In prticulr we re working on utomticlly generting n inductive proof tht given DFA ccepts given lnguge. Finlly, this tool will not replce the need for teching ssistnts (TAs), ut it utomtes tsk tht often consumes severl TA-hours. Although we only ddress smll prolem, once stisfctory grding tool for this prolem is constructed, it will e used for long time, since the concept of DFA will e lwys tught s it is now.

7 References [Adm nd Lurent, 198] Anne Adm nd Jen-Pierre H. Lurent. Lur, system to deug student progrms. Artificil Intelligence, 15(1-2):75 122, 198. [Alur et l., ] Rjeev Alur, Loris D Antoni, Sumit Gulwni, Dileep Kini, nd Mhesh Viswnthn. Automted grding of df constructions, full version. lorisdn/ppers/ijci13long.pdf. [Aut, 21] Automt Tutor. tutor/, [Benedikt et l., 211] Michel Benedikt, Griele Puppis, nd Cristin Riveros. Regulr repir of specifictions. In Proceedings of the 211 IEEE 26th Annul Symposium on Logic in Computer Science, pges 335 3, Wshington, DC, USA, 211. IEEE Computer Society. [Bille, 25] Philip Bille. A survey on tree edit distnce nd relted prolems. Theor. Comput. Sci., 337(1-3): , June 25. [Bodirsky et l., 2] Mnuel Bodirsky, Tois Grtner, Timo von Oertzen, nd Jn Schwinghmmer. Effciently computing the density of regulr lnguges. In Mrtin Frch-Colton, editor, In Proceedings of the 6th Ltin Americn Symposium, Buenos Aires, Argentin, 2, volume 2976 of Lecture Notes in Computer Science, pges Springer, 2. [Büchi nd Lndweer, 1969] J. Richrd Büchi nd Lwrence H. Lndweer. Definility in the mondic second-order theory of successor. Journl of Symolic Logic, 3(2):166 17, [Go et l., 21] Xino Go, Bing Xio, Dcheng To, nd Xuelong Li. A survey of grph edit distnce. Pttern Anlysis nd Applictions, 13(1): , Jnury 21. [Henriksen et l., 1995] Jesper G. Henriksen, Ole J.L. Jensen, Michel E. Jrgensen, Nils Klrlund, Roert Pige, Theis Ruhe, nd Anders B. Sndholm. Mon: Mondic second-order logic in prctice. In Tools nd Algorithms for the Construction nd Anlysis of Systems, First Interntionl Workshop, LNCS 119. Springer-Verlg, [Hopcroft et l., 26] John E. Hopcroft, Rjeev Motwni, nd Jeffrey D. Ullmn. Introduction to Automt Theory, Lnguges, nd Computtion (3rd Edition). Addison- Wesley Longmn Pulishing Co., Inc., Boston, MA, USA, 26. [Kozik, 25] Jku Kozik. Conditionl densities of regulr lnguges. Electronic Notes in Theoreticl Computer Science, 1:67 79, Novemer 25. [Murry, 1987] Willim R. Murry. Automtic progrm deugging for intelligent tutoring systems. Computtionl Intelligence, 3:1 16, [Rodger nd Finley, 26] Susn H. Rodger nd Thoms Finley. JFLAP - An Interctive Forml Lnguges nd Automt Pckge. Jones nd Brtlett, 26. [Singh et l., 213] Rishh Singh, Sumit Gulwni, nd Armndo Solr-Lezm. Automted feedck genertion for introductory progrmming ssignments [Sipser, 1996] Michel Sipser. Introduction to the Theory of Computtion. Interntionl Thomson Pulishing, 1st edition, [Thoms, 1996] Wolfgng Thoms. Lnguges, utomt, nd logic. In Hndook of Forml Lnguges, pges Springer, [Venes nd Bjørner, 212] Mrgus Venes nd Nikolj Bjørner. Symolic utomt: the toolkit. In Proceedings of the 18th interntionl conference on Tools nd Algorithms for the Construction nd Anlysis of Systems, pges 72 77, Berlin, Heidelerg, 212. Springer-Verlg.