An Application Of Multiple Linear Regression In Analysis Computer-Based Assessment System s Data.

Transcription

1 Proceedngs of the Postgraduate Annual Research Semnar An Applcaton Of Multple Lnear Regresson In Analyss Computer-Based Assessment System s Data. Mohd Azwan bn Hamza, Abd Manan Ahmad Department of Software Engneerng, Faculty of Computer Scence & Informaton System, Unverst Teknolog Malaysa, UTM Skuda, Johor Tel: , Fax: Emal : azu_79@hotmal.com, manan@fsksm.utm.my 1. Abstract A hot topc n educatonal measurement s the area of computerzed essay test evaluaton. In an envronment where human typcally evaluate the submtted student wrtng, essay evaluatons have been subject to varance n the level of agreement between readers and ndvdual bas. The computer technology for essay evaluaton has been made possble by use of Naïve Bayes assumptons n text classfcaton, that the probablty of occurrence of each word n a document s ndependent of the probablty of occurrence of other words n a document. Ths research s study about the capablty of the evaluaton system by employng multvarate Bernoull model, that s one of the method n Bayesan approaches n order to classfy dscourse elements (ntroducton statement, man ponts and concluson statement) n each essay rather than prevous technques such as Fuzzy Logc n the Module of Content Organzaton and Development. Thus, ths research s also apply Multple Lnear Regresson (MLR) Algorthm n order to optmze the weght of coeffcent n regresson equaton. Usng MLR not merely could obtan more accurate essay grade, but also provdes wdely detal responses. As a fnal result, t s show that level of agreement between evaluaton system and expert s 95% accuracy. Keyword: Multple Lnear Regresson, Multvarate Bernoull, Bayesan. 2. Introducton The ablty to communcate n natural language has long been consdered of human ntellgence [2]. The feasblty of computer to evaluate somethng as subjectve as an essay for Graduate Management Admssons Test (GMAT) [2], was poneerng a new era of the world of scence and practce of artfcal ntellgence. Its mght be gves the bg mpacts on the computer usage and feld of educaton, ndrectly. Feld of Automated Essay Gradng was ntroduced about four decades ago, n 1966 by Ells Page [2,3]. Inspred by the Natural Language Processng of that tme, some Connectcut researchers thought the computer mght play a large role n evaluatng student wrtng [4]. Day by day, the researchers keep on study on ths feld tll now t s proven that the system made the transton from research prototypes nto fully operatonal systems. Nevertheless, t s stll not completely the perfect Computer-based Assessment System (CbAS). Usng computers to ncrease our understandng of the textual features and cogntve sklls nvolved n creatng and comprehendng wrtten text wll have clear benefts [2]. There are 10 most promnent approaches to essay scorng by computer: Project Essay Grade (PEG), Intellgent Essay Assessor (IEA), Educatonal Testng servce I, Electronc Essay Rater (E-Rater), C-Rater, BETSY, Intellgent Essay Markng System, SEAR, Paperless School free text Markng Engne and Automark. Ths research attempts to develop another verson of CbAS wth the am of accuracy s as hgh as 95%. 3. Objectves The man objectves of ths research conssts of ) Usng Bayesan classfer approaches that traned by Multvarate Bernoull formula to classfy dscourse elements whch conssts of ntroducton

2 Proceedngs of the Postgraduate Annual Research Semnar statement, man ponts and concluson statement n an essay. ) Apply Multple Lnear Regresson to desgn the most optmal assessment model or weght features set to make the better predcton of essay grade based on annotated essay by expert and provde better feedback as a fnal result of ths research. 4. Methodology In ths research, we have been used the combnaton of two types of methodology that were Requrement Prototypes and Evoluton Prototypes. In the Requrement Prototypes, all datas and nformatons from the varous sources was collected. All the raw data was obtaned before t wll be manages, analyzes, and converts nto the form of meanngful and valuable nformaton. Ths prototypes conssts of four procedures, that are Intal Investgaton, System Analyss, System Desgn and System Implemenaton. Whle, the Evoluton Prototypes wll be realzes the entre processes of system development. By usng whole of the collected nformaton, system have been develop phase by phase untl t beng the fnal system wth very mnmum of bugs and fulfl all the verfed requrements. Unfed Modelng Language (UML) s a wdely used technque for the object orented development. UML wll combnes the method and technque from other methodology. It s attempts to defne uncertanty mappng from the phase of requrement analyss to the phase of desgn tll the phase of mplementaton. The man objectve of usng UML s to defne the development process of teraton and ncremental. By usng UML, t s wll mproves and optmzes the ablty of methodology. 5. Implementaton The mplementatons of the system that have been bult n ths research were focus on two doman dstrbutons. Both parts wll nfluence drectly on accuracy of the system assessment. 5.1 UPSR Scorng Gude Ths research has developed CbAS n order to evaluate UPSR student essay. Due to ths, we wll used UPSR Scorng Gude as a gudelne to markng and acqure fnal grade for each student essays. 5.2 Desgn of Knowledge Base (KB) The development of the system was begun wth the desgn of the Knowledge Base (KB). The KB needs to be sketch, plan and desgn frst to make sure t s n lne and support the further system mplementaton. 5.3 Assessment Applcaton Analyss The mplementaton of assessment applcaton was evaluates by compared system s scores to human experts scores for the same essays. Human reader scores and machne scores are consdered to "agree" f there s no more than a sngle pont dfference between the scores on the fve-pont scale (A, B, C, D or E). 5.4 Features Set The development of the feature set for ths system was based on nformaton extracted from several CbAS that have been dscovered. Features n ths system were created by standardzng some features wth regard to essay length, alterng the defntons of others to take nto account the non-monotonc relatonshp wth the human Module of Syntactc Structure In the Module of Syntactc Structure, all sentences n the essays were parsed usng the Mcrosoft Natural Language Processng (MSNLP) concept so that syntactc structure nformaton could be accessed (tokenze). After parsng has been done, system wll match each contguous token (parsed word) to recognze one-word, two-word, three-word or four-word. The program then mplemented to dentfy the number of complement clauses, subordnate clauses, nfntve clauses, relatve clauses and occurrences of the subjunctve modal auxlary verbs for each sentence n an essay. Ratos of syntactc structure types per essay and per sentence were also used as measures of syntactc varety Module of Language Errors n Vocabulary, Sentence Structure, Infx, Punctuaton Mark and Spellng Feedback about a total 33 errors n vocabulary, sentence structure, nfx, punctuaton mark and spellng are output from current CbAS system. Most of these features are dentfed usng NLP [14]. These counts

3 Proceedngs of the Postgraduate Annual Research Semnar formed the bass of fve features n ths research. In fact, t s also parallel to UPSR Scorng Gude provded by Mnstry of Malaysa Educaton (KPM). But, n ths research, ths module s not n our scope of study. Thus, we dd t manually to dentfy each mark of language mstakes (error) n an essay Module of Content Organzaton and Development In addton to the varous errors, current CbAS feedback applcaton automatcally dentfes sentences n the essay that correspond to the followng essay dscourse categores, usng NLP: Introducton, Man Ideas and Concluson methods [14]. Two features were derved from ths feedback nformaton. An overall development score s computed by summng up the counts of the ntroducton, man deas and concluson elements n the essay. Bayesan approaches are deserved to be used n specfy avalable dscourse elements n each essay. Typcally, dscourse elements are conssts of ntroductory statement, eght man ponts and concludng statement. The second feature derved from current CbASs organzaton and development module s the average length (n number of words) of the dscourse elements n the essay A Bayesan Classfer for Identfyng Dscourse Elements Two probablstc models for text classfcaton that can be used to tran Bayesan ndependence classfers have been dscussed [13]. We use multvarate Bernoull model, rather than multnomal model, where a document s represented by both the absence and presence of features. In our experments, we used three general feature types to buld the classfer: sentence poston; words commonly occurrng n thess statements; and Rhetorcal Structure Theory (RST) labels from outputs generated by an exstng rhetorcal structure parser [15]. We traned the classfer to predct thess statements n an essay. Usng the multvarate Bernoull formula, below, ths gves us the log probablty that a sentence (S) n an essay belongs to the class (T) of sentences that are thess statements. log log ( P( T S )) ( P( T )) = + ( P( A T ) P( A )) f S contans A ( P( A T ) P( A ), f S does contan A In ths formula, P(T) s the pror probablty that a sentence s n class T, P(A T) s the condtonal probablty of a sentence havng feature A, gven that the sentence s n T, and P(A) s the pror probablty that a sentence contans feature A, P( A T) s the condtonal probablty that a sentence does not have feature A, gven that t s n T, and P( A ) s the pror probablty that a sentence does not contan feature A Module of Elaboraton Style Prompt-Specfc Vocabulary Usage Ths system evaluates the topcal content of an essay by comparng the words t contans to the words found n manually graded tranng examples for each of the fve score categores. Two measures of content smlarty are computed, one based on word frequency and the other on word weght, as n nformaton retreval applcatons. For the former applcaton (EssayContent), content smlarty s computed over the essay as a whole, whle n the latter applcaton (ArgContent) content smlartes are computed for each argument n an essay Module of Supportve Lexcal Complexty Two features n ths system are related specfcally to word-based characterstc. The frst s rato of number of word types (or forms) to tokens n an essay. The second feature s the average word length n characters across all words n the essay Essay Length Essay length s the sngle most objectvely calculated varable n predctng human holstc scores and to mnmze the effect of essay length n the other features n the feature set., 5.5 Model Development and System Assessment

4 Proceedngs of the Postgraduate Annual Research Semnar Combnng Optmal and Fxed Weghts n Multple Lnear Regresson Below s the procedure for producng a regresson equaton that predcts human score wth n features of whch the frst k wll have optmzed weghts and the last n k wll have fxed predetermned weghts. Apply a sutable lnear transformaton to the features that have negatve correlatons wth the human score n order to have only postve regresson weghts. Standardze all features and the predcted human score. Apply a lnear multple regresson procedure to predct the standardzed human score from the frst k standardzed features and obtan k standardzed weghts for these features (labeled s 1 - s k ). The fxed standardzed weghts of the last n k features should be expressed as percentages of the sum of standardzed weghts for all features (labeled p k+1 p n ). Fnd the fxed standardzed weghts by applyng the followng formula to the last n - k features: s p = 1 k j= 1 n s j= k + 1 j p j, k + 1 n To fnd the un-standardzed weghts (labeled w 1 - w n ), multply s by the rato of the standard devaton of human score to standard devaton for the feature. Compute an nterm predcted score as the sum of the product of feature values and weghts w 1 w n. Regress the nterm predcted score to the human score and obtan an ntercept, a, and a weght, b. The ntercept wll be used as the fnal ntercept. The fnal un-standardzed weghts are gven by multplyng a by 1 n w ( ) 6. Result/Dscusson 1 Dscourse Precson Recall F- Elements measure Introducton Pont Pont Pont Pont Pont Pont Pont Pont Concluson Table 1.1: Precson, Recall, and F-measures Between Judge 1 (J1-human expert) and Judge 2 (J2-computer) for Dscourse Elements 1. The results n Table 1.1 show agreement between the two judges based on 200 student essays. In Table 1.1, Judge 1 (J1) was represented by human expert whle Judge 2 (J2) was represented by computer. As we can see, the average value of F-measure s 90.7% that s hgh. The Kappa between the two judges was 0.86 based on annotatons for all words. Kappa ndcates the agreement between judges wth regard to chance agreement [17]. Research n content analyss [16] suggests that Kappa values hgher than 0.8 reflect very hgh agreement, between 0.6 and 0.8 ndcate good agreement, and values between 0.4 and 0.6 show lower agreement, but stll greater than chance[13]. Essays Precson Recall F- measure A B C D E Table 1.2: Precson, Recall, and F-measures Between Judge 1 (J1-human expert) and Judge 2 (J2-computer) for fnal grade of student essays. 1 Precson = Total judge+ system agreements total system labels; Recall = Total judge + system agreements total judge labels F-measure = 2*P*R (P+R)

5 Proceedngs of the Postgraduate Annual Research Semnar In Table 1.2 shows the fnal result of 200 sample of testng essays compared by human expert and system evaluaton usng precson, recall and F-measure. From the table, we found that the average value of F- measure for all essay s grade was 95.2% that s acheved the man objectve of the research. The Kappa between the two judges was 0.93 based on annotatons for all words. Thus, our fnal result ndcates that was very hgh agreement between the two judges. 7. Concluson As a concluson, we were successfully acheves our man objectve where our CbAS accuracy s 95.2% runnng on 400 samples of student essay as the tranng and testng resources. The suggeston of further research mght be concern on Module of Language especally n part of sentence structure that s need very mpressve and complex approaches. 9. Chodorow, M. and Leacock, C. (2000) An Unsupervsed Method for Detectng Grammatcal Errors, Proc. Frst Meetng of the North Amercan Chapterof the assocaton for Computonal Lngustcs (ANLP-NAACL-2000), Morgan Kaufmann, San Francsco, pp MacDonald, N. et al. (1982) The Wrter s Workbench: Computer Ads for Text Analyss, IEEE Trans. Comm., Vol. COM- 30, No. 1, pp Mltasakak, E. and Kukch, K. (2000) Automated Evaluaton of Coherence n Student Essays, Proc. LREC-200, Lngustc Resources n Educaton Conf., Athens, Greece. 12. Red, D. B. (1979) An Algorthm for Trackng multple Targets, IEEE Trans. Automat. Contr., Vol. AC-24, pp Bursten, J. and Wolska, M. (2003) Toward Evaluaton of Wrtng Style: Fndng Overly Repettve Word Use n Student Essays, Educatonal Testng Servce, Prnceton, New Jersey. References 1. Hedberg, S.R (1999) Computers scorng GMAT essays? Impossble! Or s t?, IEEE Intellgent Systems, vol. 14, ssue. 3, pg Mart, A. H. (2000) The debate on automated essay gradng, IEEE Intellgent Systems, vol. 15, ssue. 5, pg Rudner, L. and Phlp, G. (2001) An overvew of three Aproaches to Scorng Wrtten Essays by Computer, Assessment, Research & Evaluaton, Unversty of Maryland, College Park, vol. 7(26). 4. Page, E. B. (1994) Computer Gradng of Student Prose, Usng Modern Concepts and Software, Journal of Expermental Educaton, vol. 62 Issue 2, p127, 16p, 5 charts. 5. Palmer, J. (1999) et al. Automated Essay Gradng System Appled to a Frst Year Unversty Subject How Does We do t Better?, Curtn Unversty of Technology, Perth, WA, Australa. 6. Santago Aja-Fernandez et al. (2002) A fuzzy MHT Algorthm Appled to Text-Based Informaton Trackng, IEE Transacton on Fuzzy Systems, vol. 10, No Susan, F. (1999) NLP meet the Jaberwocky, Onlne. 8. Darusz, J. K. (2002) Fuzzy Set Taggng, Insttute of Computer Scence, Warsaw Unversty of Technology, CICLng, LNCS 2276, pp Bursten, J. et al. (2003) Towards Automatc Classfcaton of Dscourse Elements n Essays, Specal Issue on Natural Language Processng of IEEE Intellgent Systems. 15. Marcu, D. (2000) The Theory and Practce of Dscourse Parsng and Summarzaton. The MIT Press. 16. Krppendorff K. (1980) Content Analyss: An Introducton to Its Methodology. Sage Publshers. 17. Uebersax, J.S. (1982) A Generalzed Kappa Coeffcent, Educatonal and Psychologcal Measurement, Vol. 42, pp