Automated Essay Scoring for Swedish
|
|
- Ilene McKenzie
- 7 years ago
- Views:
Transcription
1 Automated Essay Scoring for Swedish Robert Östling Department of Linguistics Stockholm University SE Stockholm Andre Smolentzov Department of Linguistics Stockholm University SE Stockholm Björn Tyrefors Hinnerich Department of Economics Stockholm University SE Stockholm Abstract We present the first system developed for automated grading of high school essays written in Swedish. The system uses standard text quality indicators and is able to compare vocabulary and grammar to large reference corpora of blog posts and newspaper articles. The system is evaluated on a corpus of essays, each graded independently by the student s own teacher and also in a blind re-grading process by another teacher. We show that our system s performance is fair, given the low agreement between the two human graders, and furthermore show how it could improve efficiency in a practical setting where one seeks to identify incorrectly graded essays. 1 Introduction Automated Essay Scoring (AES) is the field of automatically assigning grades to student essays (Shermis and Burstein, 2003; Dikli, 2006). Previous work on AES has primarily focused on English texts, and to the best of our knowledge no AES system for Swedish essays has been published. We exploit some peculiarities of the Swedish language, such as its compounding nature, to design new features for classification. We also use constructions in the shape of hybrid n-grams (Tsao and Wible, 2009) extracted from large corpora in the classification. Earlier results from this work have been presented in the B.A. thesis of Smolentzov (2013), where further details can be found. Source code, a trained model as well as an on-line version of our tool are Erik Höglin National Institute of Economic Research Kungsgatan Stockholm erik.hoglin@konj.se available from the website of the Department of Linguistics. 1 Due to legal restrictions, we are currently unable to publish the corpus of essays used for training the model and in our evaluation. While this is very regrettable, there are so far no suitable training corpora available for Swedish that are publicly available. We hope in the future to be able to produce an anonymized version of the corpus, to be shared with other researchers. 2 Data We use a corpus of essays from the essay writing part of the Swedish high school national exams in Swedish. 2 These were collected using random sampling by Hinnerich et al. (2011), who had them digitized, anonymized, and re-graded by high school teachers experienced with grading the national exams. The essays were originally graded by the student s own teacher. In total, essays have all the information we require: digitized text and the two grades. The size of the corpus is tokens, or an average of 656 per essay. The essays have been automatically annotated with lemma and part of speech (PoS) information using Stagger (Östling, 2012). There are four grades: IG (fail), G (pass), VG (pass with distinction) and MVG (excellent). Hinnerich et al. (2011) found that the agreement between the two human graders is rather low, and in the set of essays used in this study only 780 (45.8%) of the essays received the same grade by both Course Svenska B, fall 2005/spring Proceedings of the Eighth Workshop on Innovative Use of NLP for Building Educational Applications, pages 42 47, Atlanta, Georgia, June c 2013 Association for Computational Linguistics
2 Blind grader Teacher IG G VG MVG Sum IG G VG MVG Sum Table 1: Confusion matrix for the grades assigned by the students own teachers, and during the blind re-grading process. In total, 780 essays (45.8%) are assigned the same grade. Linear weighted κ = graders. In 148 cases (8.7%), the grade difference was more than one step. In Table 1, we can clearly see that the blind graders grades are generally lower. The disagreement is also more severe for the grades at the extremes of the scale. It is important to note that the grading guidelines for the national exams do not focus exclusively on the quality of the language used, but rather on the ability of the student to produce a coherent and convincing argument, understanding and relating to other texts, or describing personal experiences. Some work has been carried out using highlevel features in automated essay scoring. Miltsakaki and Kukich (2004) use some manual annotation to explore the role of coherence, and Attali and Burstein (2005) automatically analyze the overall structure of essays. Others take the contents of essays into account (Landauer et al., 2003), which is suitable for essay questions in non-language subjects. We will, however, focus on form rather than content. One important reason for this is that our corpus of essays is spread out over 19 different topics (in several cases with as few as essays each), where the type of text expected can vary considerably between topics. 3 Methods We use a supervised machine learning approach, based on a Linear Discriminant Analysis classifier in the implementation of Pedregosa et al. (2011). Each essay is represented by a feature vector, whose contents we will describe in some detail in the following sections. It is important to note that we are using correlations between grade and different features of the text, but the relationship between these features and the qualities of the essay on which the grade should be based may be complex. As a cautionary tale, we could mention that vocabulary related to cell phones was found to correlate strongly with essay grade. It turned out that poor students showed a strong preference for one of the given essay topics, which happened to center around cell phones. In the field of AES, it is particularly important to keep in mind that correlation does not imply causation. 3.1 Simple features We use a number of features that may be directly measured from the text. These are presented below, roughly in decreasing order of correlation with essay grade. Most of the features have been discussed in previous literature on AES (Attali and Burstein, 2005), and specifically in the context of Swedish high school essays by Hultman and Westman (1977). Some further features that did not contribute much to grading accuracy were tried, but will be omitted from this discussion. Text length Since the essays are composed in a classroom setting with a fixed amount of time allotted (five hours), a student s fluency in writing is directly mirrored in the length of an essay, which becomes the feature that most strongly correlates with grade. While one might want to exclude the length from consideration in the grading process, it is important to keep this correlation in mind since other measures may correlate with length, and therefore indirectly correlate with essay grade without contributing any new information. Average word length The average number of letters per word also correlates with grade but only weakly with the length (in words). It does however correlate strongly with the distribution of parts of speech, primarily pronouns (which tend to be short) and nouns (which tend to be long, particularly since Swedish is a compounding language). OVIX lexical diversity measure OVIX (Hultman, 1994) was in fact developed for the very purpose of analyzing lexical diversity in Swedish high school essays, and has been found to correlate 43
3 strongly with grade in this setting. At the same time, the measure is mostly independent of text length. ( OVIX = log n tokens / 2 log n ) types log n tokens Part of speech distribution The relative frequencies of different parts of speech also correlate with essay grade, although more weakly so than the related measure of average word length. 3.2 Corpus-induced features While the size our corpus of graded student essays is in the order of one million words, much larger amounts of Swedish text are available from different sources, such as opinion pieces, news articles, and blog posts. Due to the large amounts of text available, from tens of millions to several billions of words depending on the source, we can extract reliable statistics even about relatively rare language phenomena. By comparing student essays to statistics gathered from different text types, we obtain new variables that often correlate strongly with essay grades. PoS tag cross-entropy The average cross-entropy per token from a PoS trigram model (with simple additive smoothing) is used to model the similarity on a syntactic level. This includes both elements of style (e.g. frequent use of passive constructions) and mechanics (e.g. agreement errors). We use a corpus of news texts 3 to train the model. Vocabulary cross-entropy With word frequency statistics from two different text sources, we compute the average cross-entropy per token given a unigram model, and use the difference between these values for the two models to indicate which type of text the present essay is most similar to. In our experiments, the two text sources are of equal size and consist of the news texts mentioned above, and a corpus of blog posts. Hybrid n-gram cross-entropy We can generalize the vocabulary cross-entropy measure described above by using hybrid n-grams (Tsao and Wible, 2009) rather than single words. This allows for some 3 The corpus consists of ca 200 million words, crawled from the WWW editions of Dagens Nyheter and Svenska Dagbladet. patterns that are neither entirely grammatical nor entirely lexical to be used, complementing the two previous approaches. The same news and blog corpora as above are used. 3.3 Language error features Spelling errors We implemented a simple spell checker, using the SALDO lexicon (Borin and Forsberg, 2009) and statistics from a corpus of news text. On average, a misspelling was detected in 0.63% of all word tokens, or about four misspellings per essay. Manual inspection showed that the spell checker made some errors, so it is reasonable to assume that results could be improved somewhat using a more accurate tool. Split compound errors Swedish is a compounding language, with noun compounding particularly frequent. It is a fairly common error among inexperienced writers to separate the segments of a compound word. We use word uni- and bigram statistics from a corpus of news texts to find instances of these errors in the essays. Only 0.10% of word tokens are found to be incorrectly split, or less than one instance per essay on average. As expected, there is a (weak) negative correlation between split compound frequency and grade, which seems to be due to a small number of poor essays with many such errors. 3.4 Evaluation measures The simplest measure of overlap between two graders (either among humans, or between human(s) and machine) is the percentage of essays on which they agree about the grade. However, in our setting this is not so informative because there is a high chance of graders assigning the same grade by chance, and this probability varies between different pairs of graders. This makes comparisons difficult, so we instead use Cohen s kappa value (Cohen, 1968), linearly weighted according to the numeric values of grades used by the Swedish school system: IG corresponds to 0 points, G to 10, VG to 15, and MVG to 20. A kappa value of 1 would indicate perfect agreement, while 0 would mean random agreement. The 44
4 Feature Correlation n 0.25 tokens n tokens hybrid n-gram cross-entropy vocabulary cross-entropy average word length OVIX n long /n tokens spelling errors PoS cross-entropy split compound errors Table 2: Correlation between grade (average of two graders) and features. Interactions between features are not taken into account. Only features with Pearson coefficient ρ > 0.2 are included, all are highly significant. weighted kappa value is computed as: i,j κ = 1 w ijo ij i,j w ije ij where O ij is the number of times annotator 1 assigned grade i and annotator 2 assigned grade j, while E ij is the expected number of times for the same event, given that both annotators randomly assign grades according to a multinomial distribution. w ij is the difference in score between grades i and j, according to the above. 4 Results 4.1 Feature-grade correlations First, we look at the correlations between the human-assigned grades and individual features. Since a linear machine learning algorithm is used, we use the Pearson coefficient to measure linear dependence. Spearman s rank correlation coefficient gives similar results. From Table 2 we can see that only ten of the features show a correlation above 0.2. There were statistically significant (but weak) correlations below this threshold, e.g. the ratios of different parts of speech, where the strongest correlations were ρ = (pronouns) and ρ = (prepositions). 4.2 Automated grading Table 3 shows the performance of our system, using the leave-one-out evaluation method on all es- Human avg. Computer IG G VG MVG Sum IG G VG MVG Sum Table 3: Confusion matrix for the grades assigned by the system, and the average (rounded down) of the two human graders. In total, essays (62.2%) are assigned the same grade, κ = says, i.e. evaluating each essay using a model trained on all the other essays. We see that the computer s grades are biased towards the most common grade (G, pass), but that overall accuracy is quite high (62.2%, κ = 0.399) compared to 58.4% (κ = 0.249) when using only the strongest feature (4th root of essay length), 54.9% when assigning the most common grade to all essays, or the 45.8% (κ = 0.276) agreement between the two human graders. It is also encouraging to see that only 28 essays (1.6%) receive a grade by the computer that differs more than one step from the human-assigned grade. The corresponding figure is 148 essays (8.7%) between the two humans. When training and evaluating using only the grades of the blind grader, the agreement between computer and human was 57.6% (κ = 0.369), and only 53.6% (κ = 0.345) using the grades of the student s teacher. Both these figures are below the 62.2% (κ = 0.399) obtained when using the average grade, and the explanation closest at hand is that the features we model (partially) represent or correlate with the actual grading criteria of the exam. Since the teachers are affected by various sources of bias (Hinnerich et al., 2011), a weaker correlation (mirrored by a lower κ) to any kind of objective measure would be expected. Similarly, using the average of two graders should decrease the large individual variance due to the difficult and partially subjective nature of the task, leading to a stronger correlation with relevant features of the text. 45
5 4.3 Re-grading In 148 cases (8.7%) of our essays, the grade assigned in the blind re-grading process differs by more than one step from the original grade, and we performed an experiment to see how efficiently these highly deviant grades could be identified. This scenario could arise within an organization responsible for evaluating the consistency in grading a national exam, where resources are insufficient for regrading all essays manually. Given a training corpus of graded essays, our system could then be used to select candidates among the larger set of essays for further manual re-grading. In other to evaluate the usefulness of this method, we let the system re-grade all essays based on the blind grades of all other essays (leave-one-out). In the cases where the system s grade differs by more than one step from the teacher s grade, we check whether the difference between the system s grade and that of the blind grader is less than between the two human graders. It turns out that we can correctly identify 43 (29.1%) of the 148 cases in this way, with only 91 essays (5.3% of the total) considered. In a scenario where we have a large amount of essays but only the resources to manually re-grade a fraction of them, we can thus increase the ratio of highly deviant grades found from 8.7% (148/1702, by randomly choosing essays to re-grade) to 47% (43/91, by only re-grading those identified by our system). 5 Conclusions and future work We have presented a system for automatic grading of Swedish high school essays. While its accuracy is not high enough to be used in grading highstakes exams, we have demonstrated its usefulness in a practical setting of finding instances of incorrect grading (as identified by humans). Novel aspects include features based on constructions induced using unsupervised methods, and on (language-specific) compounding errors. It would be interesting to apply some of our methods to other languages and other data sets, for instance of second language learners. Since our system is quite general, all that would be needed to adapt it to another domain is a training corpus of graded essays. Adapting to another language would in addition require a PoS tagger and suitable unlabeled text corpora. Acknowledgments We would like to thank the anonymous reviewers for their useful comments. References Yigal Attali and Jill Burstein Automated essay scoring with e-rater R v.2.0. Technical report, Educational Testing Services. Lars Borin and Markus Forsberg All in the family: A comparison of SALDO and WordNet. In Proceedings of the Nodalida 2009 Workshop on WordNets and other Lexical Semantic Resources between Lexical Semantics, Lexicography, Terminology and Formal Ontologies, Odense. Jacob Cohen Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological Bulletin, 70: Semire Dikli An overview of automated scoring of essays. Journal of Technology, Learning, and Assessment, 5. Björn Tyrefors Hinnerich, Erik Höglin, and Magnus Johannesson Are boys discriminated in swedish high schools? Economics of Education Review, 30: Tor G. Hultman and Margareta Westman Gymnasistsvenska. LiberLäromedel. Tor G. Hultman Hur gick det med ovix? In Språkbruk, grammatik och språkförändring. En festskrift till Ulf Teleman, pages Lund University. Thomas K. Landauer, Darrell Laham, and Peter Foltz Automatic essay assessment. Assessment in Education, 10: E. Miltsakaki and K. Kukich Evaluation of text coherence for electronic essay scoring systems. Natural Language Engineering, 10: Robert Östling Stagger: A modern POS tagger for Swedish. In Proceedings of the Swedish Language Technology Conference (SLTC). F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12: M.D. Shermis and J. Burstein, editors Automated Essay Scoring: A Cross Disciplinary Perspective. L. Erlbaum Associates. 46
6 André Smolentzov Automated Essay Scoring: Scoring Essays in Swedish. Bachelor s thesis, Department of Linguistics, Stockholm University. Nai-Lung Tsao and David Wible A method for unsupervised broad-coverage lexical error detection and correction. In Proceedings of the Fourth Workshop on Innovative Use of NLP for Building Educational Applications, EdAppsNLP 09, pages 51 54, Stroudsburg, PA, USA. Association for Computational Linguistics. 47
Automated Essay Scoring with the E-rater System Yigal Attali Educational Testing Service yattali@ets.org
Automated Essay Scoring with the E-rater System Yigal Attali Educational Testing Service yattali@ets.org Abstract This paper provides an overview of e-rater, a state-of-the-art automated essay scoring
Testing Data-Driven Learning Algorithms for PoS Tagging of Icelandic
Testing Data-Driven Learning Algorithms for PoS Tagging of Icelandic by Sigrún Helgadóttir Abstract This paper gives the results of an experiment concerned with training three different taggers on tagged
Spotting false translation segments in translation memories
Spotting false translation segments in translation memories Abstract Eduard Barbu Translated.net eduard@translated.net The problem of spotting false translations in the bi-segments of translation memories
ETS Automated Scoring and NLP Technologies
ETS Automated Scoring and NLP Technologies Using natural language processing (NLP) and psychometric methods to develop innovative scoring technologies The growth of the use of constructed-response tasks
REVIEW OF HANDBOOK OF AUTOMATED ESSAY EVALUATION: CURRENT APPLICATIONS AND NEW DIRECTIONS
Language Learning & Technology http://llt.msu.edu/issues/june2014/review2.pdf June 2014, Volume 18, Number 2 pp. 65 69 REVIEW OF HANDBOOK OF AUTOMATED ESSAY EVALUATION: CURRENT APPLICATIONS AND NEW DIRECTIONS
Association Between Variables
Contents 11 Association Between Variables 767 11.1 Introduction............................ 767 11.1.1 Measure of Association................. 768 11.1.2 Chapter Summary.................... 769 11.2 Chi
Handbook on Test Development: Helpful Tips for Creating Reliable and Valid Classroom Tests. Allan S. Cohen. and. James A. Wollack
Handbook on Test Development: Helpful Tips for Creating Reliable and Valid Classroom Tests Allan S. Cohen and James A. Wollack Testing & Evaluation Services University of Wisconsin-Madison 1. Terminology
Modeling coherence in ESOL learner texts
University of Cambridge Computer Lab Building Educational Applications NAACL 2012 Outline 1 2 3 4 The Task: Automated Text Scoring (ATS) ATS systems Discourse coherence & cohesion The Task: Automated Text
Language Modeling. Chapter 1. 1.1 Introduction
Chapter 1 Language Modeling (Course notes for NLP by Michael Collins, Columbia University) 1.1 Introduction In this chapter we will consider the the problem of constructing a language model from a set
Criterion SM Online Essay Evaluation: An Application for Automated Evaluation of Student Essays
Criterion SM Online Essay Evaluation: An Application for Automated Evaluation of Student Essays Jill Burstein Educational Testing Service Rosedale Road, 18E Princeton, NJ 08541 jburstein@ets.org Martin
Facebook Friend Suggestion Eytan Daniyalzade and Tim Lipus
Facebook Friend Suggestion Eytan Daniyalzade and Tim Lipus 1. Introduction Facebook is a social networking website with an open platform that enables developers to extract and utilize user information
Automated Scoring for the Assessment of Common Core Standards
Automated Scoring for the Assessment of Common Core Standards David M. Williamson, Senior Research Director, Applied Research and Development, ETS Randy E. Bennett, Frederiksen Chair, Assessment Innovation,
VCU-TSA at Semeval-2016 Task 4: Sentiment Analysis in Twitter
VCU-TSA at Semeval-2016 Task 4: Sentiment Analysis in Twitter Gerard Briones and Kasun Amarasinghe and Bridget T. McInnes, PhD. Department of Computer Science Virginia Commonwealth University Richmond,
Analysing Questionnaires using Minitab (for SPSS queries contact -) Graham.Currell@uwe.ac.uk
Analysing Questionnaires using Minitab (for SPSS queries contact -) Graham.Currell@uwe.ac.uk Structure As a starting point it is useful to consider a basic questionnaire as containing three main sections:
Correlational Research. Correlational Research. Stephen E. Brock, Ph.D., NCSP EDS 250. Descriptive Research 1. Correlational Research: Scatter Plots
Correlational Research Stephen E. Brock, Ph.D., NCSP California State University, Sacramento 1 Correlational Research A quantitative methodology used to determine whether, and to what degree, a relationship
Micro blogs Oriented Word Segmentation System
Micro blogs Oriented Word Segmentation System Yijia Liu, Meishan Zhang, Wanxiang Che, Ting Liu, Yihe Deng Research Center for Social Computing and Information Retrieval Harbin Institute of Technology,
AI Magazine Volume 25 Number 3 (2004) ( AAAI) Automated Essay Evaluation: The Criterion Online Writing Service
AI Magazine Volume 25 Number 3 (2004) ( AAAI) Articles Automated Essay Evaluation: The Criterion Online Writing Service Jill Burstein, Martin Chodorow, and Claudia Leacock In this article, we describe
CHAPTER 14 ORDINAL MEASURES OF CORRELATION: SPEARMAN'S RHO AND GAMMA
CHAPTER 14 ORDINAL MEASURES OF CORRELATION: SPEARMAN'S RHO AND GAMMA Chapter 13 introduced the concept of correlation statistics and explained the use of Pearson's Correlation Coefficient when working
English II Writing. Persuasive Writing Rubric
English II Writing Persuasive Writing Rubric Score Point 1 The essay represents a very limited writing performance. The organizing structure of the essay is inappropriate to the purpose or the specific
Identifying Thesis and Conclusion Statements in Student Essays to Scaffold Peer Review
Identifying Thesis and Conclusion Statements in Student Essays to Scaffold Peer Review Mohammad H. Falakmasir, Kevin D. Ashley, Christian D. Schunn, Diane J. Litman Learning Research and Development Center,
The Oxford Learner s Dictionary of Academic English
ISEJ Advertorial The Oxford Learner s Dictionary of Academic English Oxford University Press The Oxford Learner s Dictionary of Academic English (OLDAE) is a brand new learner s dictionary aimed at students
Identifying Focus, Techniques and Domain of Scientific Papers
Identifying Focus, Techniques and Domain of Scientific Papers Sonal Gupta Department of Computer Science Stanford University Stanford, CA 94305 sonal@cs.stanford.edu Christopher D. Manning Department of
Ngram Search Engine with Patterns Combining Token, POS, Chunk and NE Information
Ngram Search Engine with Patterns Combining Token, POS, Chunk and NE Information Satoshi Sekine Computer Science Department New York University sekine@cs.nyu.edu Kapil Dalwani Computer Science Department
Sentiment Analysis of Movie Reviews and Twitter Statuses. Introduction
Sentiment Analysis of Movie Reviews and Twitter Statuses Introduction Sentiment analysis is the task of identifying whether the opinion expressed in a text is positive or negative in general, or about
Interactive Learning Environments 8(2): 111-129
Content-based feedback 1 Supporting content-based feedback in online writing evaluation with LSA Peter W. Foltz, Sara Gilliam and Scott A. Kendall New Mexico State University Interactive Learning Environments
CALCULATIONS & STATISTICS
CALCULATIONS & STATISTICS CALCULATION OF SCORES Conversion of 1-5 scale to 0-100 scores When you look at your report, you will notice that the scores are reported on a 0-100 scale, even though respondents
Classification of Electrocardiogram Anomalies
Classification of Electrocardiogram Anomalies Karthik Ravichandran, David Chiasson, Kunle Oyedele Stanford University Abstract Electrocardiography time series are a popular non-invasive tool used to classify
Improving Credit Card Fraud Detection with Calibrated Probabilities
Improving Credit Card Fraud Detection with Calibrated Probabilities Alejandro Correa Bahnsen, Aleksandar Stojanovic, Djamila Aouada and Björn Ottersten Interdisciplinary Centre for Security, Reliability
MEUSE: RECOMMENDING INTERNET RADIO STATIONS
MEUSE: RECOMMENDING INTERNET RADIO STATIONS Maurice Grant mgrant1@ithaca.edu Adeesha Ekanayake aekanay1@ithaca.edu Douglas Turnbull dturnbull@ithaca.edu ABSTRACT In this paper, we describe a novel Internet
ETS Research Spotlight. Automated Scoring: Using Entity-Based Features to Model Coherence in Student Essays
ETS Research Spotlight Automated Scoring: Using Entity-Based Features to Model Coherence in Student Essays ISSUE, AUGUST 2011 Foreword ETS Research Spotlight Issue August 2011 Editor: Jeff Johnson Contributors:
CS 533: Natural Language. Word Prediction
CS 533: Natural Language Processing Lecture 03 N-Gram Models and Algorithms CS 533: Natural Language Processing Lecture 01 1 Word Prediction Suppose you read the following sequence of words: Sue swallowed
Virginia English Standards of Learning Grade 8
A Correlation of Prentice Hall Writing Coach 2012 To the Virginia English Standards of Learning A Correlation of, 2012, Introduction This document demonstrates how, 2012, meets the objectives of the. Correlation
Language Arts Literacy Areas of Focus: Grade 6
Language Arts Literacy : Grade 6 Mission: Learning to read, write, speak, listen, and view critically, strategically and creatively enables students to discover personal and shared meaning throughout their
Rubrics for Assessing Student Writing, Listening, and Speaking High School
Rubrics for Assessing Student Writing, Listening, and Speaking High School Copyright by the McGraw-Hill Companies, Inc. All rights reserved. Permission is granted to reproduce the material contained herein
e-commerce product classification: our participation at cdiscount 2015 challenge
e-commerce product classification: our participation at cdiscount 2015 challenge Ioannis Partalas Viseo R&D, France ioannis.partalas@viseo.com Georgios Balikas University of Grenoble Alpes, France georgios.balikas@imag.fr
Writing Rubrics. Eighth Grade. Based on the California State Writing Standards. Created by Miller seventh grade team 4/05..
Writing Rubrics Eighth Grade Based on the California State Writing Standards Created by Miller seventh grade team 4/05 1 Table of Contents Section 1: Student Rubrics Six Plus One Traits: Eighth Grade Writing
Selected Topics in Applied Machine Learning: An integrating view on data analysis and learning algorithms
Selected Topics in Applied Machine Learning: An integrating view on data analysis and learning algorithms ESSLLI 2015 Barcelona, Spain http://ufal.mff.cuni.cz/esslli2015 Barbora Hladká hladka@ufal.mff.cuni.cz
TELT March 2014 Exa miners Report
TELT March 2014 Exa miners Report 1. Introduction 101 candidates sat for the TELT March 2014 examination session. 53 candidates were awarded Pass grades or higher. This is the equivalent to 52.5 % pass
Test Bias. As we have seen, psychological tests can be well-conceived and well-constructed, but
Test Bias As we have seen, psychological tests can be well-conceived and well-constructed, but none are perfect. The reliability of test scores can be compromised by random measurement error (unsystematic
Section 3 Part 1. Relationships between two numerical variables
Section 3 Part 1 Relationships between two numerical variables 1 Relationship between two variables The summary statistics covered in the previous lessons are appropriate for describing a single variable.
Key words related to the foci of the paper: master s degree, essay, admission exam, graders
Assessment on the basis of essay writing in the admission to the master s degree in the Republic of Azerbaijan Natig Aliyev Mahabbat Akbarli Javanshir Orujov The State Students Admission Commission (SSAC),
Grade 4. Expository Writing Rubric
Grade 4 Expository Writing Rubric Score Point 1 The essay represents a very limited writing performance. The organizing structure of the essay is inappropriate to the purpose or the specific demands of
DESCRIPTIVE STATISTICS. The purpose of statistics is to condense raw data to make it easier to answer specific questions; test hypotheses.
DESCRIPTIVE STATISTICS The purpose of statistics is to condense raw data to make it easier to answer specific questions; test hypotheses. DESCRIPTIVE VS. INFERENTIAL STATISTICS Descriptive To organize,
A Mixed Trigrams Approach for Context Sensitive Spell Checking
A Mixed Trigrams Approach for Context Sensitive Spell Checking Davide Fossati and Barbara Di Eugenio Department of Computer Science University of Illinois at Chicago Chicago, IL, USA dfossa1@uic.edu, bdieugen@cs.uic.edu
ONLINE RESUME PARSING SYSTEM USING TEXT ANALYTICS
ONLINE RESUME PARSING SYSTEM USING TEXT ANALYTICS Divyanshu Chandola 1, Aditya Garg 2, Ankit Maurya 3, Amit Kushwaha 4 1 Student, Department of Information Technology, ABES Engineering College, Uttar Pradesh,
..., (Data Driven Learning).
Guideline for Pre-Service Teachers.... look see, (Data Driven Learning). Both teachers and learners can use corpus linguistics in various ways within the classroom. A teacher might collect a set of student
Sentiment analysis: towards a tool for analysing real-time students feedback
Sentiment analysis: towards a tool for analysing real-time students feedback Nabeela Altrabsheh Email: nabeela.altrabsheh@port.ac.uk Mihaela Cocea Email: mihaela.cocea@port.ac.uk Sanaz Fallahkhair Email:
CELC Benchmark Essays Set 3 Prompt:
CELC Benchmark Essays Set 3 Prompt: Recently, one of your friends fell behind in several of his/her homework assignments and asked you for help. You agreed, but then you found out that your friend was
Sentiment analysis on tweets in a financial domain
Sentiment analysis on tweets in a financial domain Jasmina Smailović 1,2, Miha Grčar 1, Martin Žnidaršič 1 1 Dept of Knowledge Technologies, Jožef Stefan Institute, Ljubljana, Slovenia 2 Jožef Stefan International
Writing and Presenting a Persuasive Paper Grade Nine
Ohio Standards Connection Writing Applications Benchmark E Write a persuasive piece that states a clear position, includes relevant information and offers compelling in the form of facts and details. Indicator
Word Completion and Prediction in Hebrew
Experiments with Language Models for בס"ד Word Completion and Prediction in Hebrew 1 Yaakov HaCohen-Kerner, Asaf Applebaum, Jacob Bitterman Department of Computer Science Jerusalem College of Technology
Speaking for IELTS. About Speaking for IELTS. Vocabulary. Grammar. Pronunciation. Exam technique. English for Exams.
About Collins series has been designed to be easy to use, whether by learners studying at home on their own or in a classroom with a teacher: Instructions are easy to follow Exercises are carefully arranged
Brill s rule-based PoS tagger
Beáta Megyesi Department of Linguistics University of Stockholm Extract from D-level thesis (section 3) Brill s rule-based PoS tagger Beáta Megyesi Eric Brill introduced a PoS tagger in 1992 that was based
Scoring TELL (Test of English Language Learning)
Scoring TELL (Test of English Language Learning) By William J. Bonk Introduction to TELL Test of English Language Learning (TELL) is an interactive English language proficiency assessment designed to identify,
Segmentation and Classification of Online Chats
Segmentation and Classification of Online Chats Justin Weisz Computer Science Department Carnegie Mellon University Pittsburgh, PA 15213 jweisz@cs.cmu.edu Abstract One method for analyzing textual chat
Descriptive Statistics
Descriptive Statistics Primer Descriptive statistics Central tendency Variation Relative position Relationships Calculating descriptive statistics Descriptive Statistics Purpose to describe or summarize
Designing and Using Weighted Rubrics
Designing and Using Weighted Rubrics The following pages demonstrate one effective way for history teachers to integrate the new assessment model required by the Ontario Ministry of Education into their
Estimating Probability Distributions
Estimating Probability Distributions Readings: Manning and Schutze, Section 6.2 Jurafsky & Martin, Section 6.3 One of the central problems we face in using probability models for NLP is obtaining the actual
Statistical tests for SPSS
Statistical tests for SPSS Paolo Coletti A.Y. 2010/11 Free University of Bolzano Bozen Premise This book is a very quick, rough and fast description of statistical tests and their usage. It is explicitly
C A R I B B E A N E X A M I N A T I O N S C O U N C I L REPORT ON CANDIDATES WORK IN THE SECONDARY EDUCATION CERTIFICATE EXAMINATION MAY/JUNE 2011
C A R I B B E A N E X A M I N A T I O N S C O U N C I L REPORT ON CANDIDATES WORK IN THE SECONDARY EDUCATION CERTIFICATE EXAMINATION MAY/JUNE 2011 ECONOMICS GENERAL PROFICIENCY EXAMINATION Copyright 2011
Statistics. Measurement. Scales of Measurement 7/18/2012
Statistics Measurement Measurement is defined as a set of rules for assigning numbers to represent objects, traits, attributes, or behaviors A variableis something that varies (eye color), a constant does
Correlation Coefficient The correlation coefficient is a summary statistic that describes the linear relationship between two numerical variables 2
Lesson 4 Part 1 Relationships between two numerical variables 1 Correlation Coefficient The correlation coefficient is a summary statistic that describes the linear relationship between two numerical variables
Number Factors. Number Factors Number of factors 1 1 1 16 1, 2, 4, 8, 16 5 2 1, 2 2 17 1, 17 2 3 1, 3 2 18 1, 2, 3, 6, 9, 18 6 4 1, 2, 4 3 19 1, 19 2
Factors This problem gives you the chance to: work with factors of numbers up to 30 A factor of a number divides into the number exactly. This table shows all the factors of most of the numbers up to 30.
Diagnosis Code Assignment Support Using Random Indexing of Patient Records A Qualitative Feasibility Study
Diagnosis Code Assignment Support Using Random Indexing of Patient Records A Qualitative Feasibility Study Aron Henriksson 1, Martin Hassel 1, and Maria Kvist 1,2 1 Department of Computer and System Sciences
Smarter Balanced Assessment Consortium:
Smarter Balanced Assessment Consortium: Practice Test Scoring Guide Grade 3 Performance Task Published August 26, 2013 Prepared by the American Institutes for Research Smarter Balanced Assessment Consortium,
Pearson s Text Complexity Measure
Pearson s Text Complexity Measure White Paper Thomas K. Landauer, Ph.D. May 2011 TEXT COMPLEXITY 1 About Pearson Pearson, the global leader in education and education technology, provides innovative print
Turker-Assisted Paraphrasing for English-Arabic Machine Translation
Turker-Assisted Paraphrasing for English-Arabic Machine Translation Michael Denkowski and Hassan Al-Haj and Alon Lavie Language Technologies Institute School of Computer Science Carnegie Mellon University
THE knowledge needed by software developers
SUBMITTED TO IEEE TRANSACTIONS ON SOFTWARE ENGINEERING 1 Extracting Development Tasks to Navigate Software Documentation Christoph Treude, Martin P. Robillard and Barthélémy Dagenais Abstract Knowledge
Examiners Report June 2013. GCSE Design and Technology Graphic Products 5GR02 01
Examiners Report June 2013 GCSE Design and Technology Graphic Products 5GR02 01 Edexcel and BTEC Qualifications Edexcel and BTEC qualifications come from Pearson, the UK s largest awarding body. We provide
Language Arts Literacy Areas of Focus: Grade 5
Language Arts Literacy : Grade 5 Mission: Learning to read, write, speak, listen, and view critically, strategically and creatively enables students to discover personal and shared meaning throughout their
Author Gender Identification of English Novels
Author Gender Identification of English Novels Joseph Baena and Catherine Chen December 13, 2013 1 Introduction Machine learning algorithms have long been used in studies of authorship, particularly in
Website Marketing Audit. Example, inc. Website Marketing Audit. For. Example, INC. Provided by
Website Marketing Audit For Example, INC Provided by State of your Website Strengths We found the website to be easy to navigate and does not contain any broken links. The structure of the website is clean
AUTOLEX: An Automatic Lexicon Builder for Minority Languages Using an Open Corpus
PACLIC 24 Proceedings 63 AUTOLEX: An Automatic Lexicon Builder for Minority Languages Using an Open Corpus Evan Liz C. Buhay a, Marie Joy P. Evardone a, Hansel B. Nocon a, Davis Muhajereen D. Dimalen a,
Unit 9 Describing Relationships in Scatter Plots and Line Graphs
Unit 9 Describing Relationships in Scatter Plots and Line Graphs Objectives: To construct and interpret a scatter plot or line graph for two quantitative variables To recognize linear relationships, non-linear
DKPro TC: A Java-based Framework for Supervised Learning Experiments on Textual Data
DKPro TC: A Java-based Framework for Supervised Learning Experiments on Textual Data Johannes Daxenberger, Oliver Ferschke, Iryna Gurevych and Torsten Zesch UKP Lab, Technische Universität Darmstadt Information
Internal Usability Testing of Automated Essay Feedback in an Intelligent Writing Tutor
Proceedings of the Twenty-Fourth International Florida Artificial Intelligence Research Society Conference Internal Usability Testing of Automated Essay Feedback in an Intelligent Writing Tutor 1 Rod D.
Improve your English and increase your employability with EN Campaigns
Improve your English and increase your employability with EN Campaigns Being able to communicate in English is becoming increasingly important in today's global economy. We provie a high quality and accessible
The correlation coefficient
The correlation coefficient Clinical Biostatistics The correlation coefficient Martin Bland Correlation coefficients are used to measure the of the relationship or association between two quantitative
Correlation key concepts:
CORRELATION Correlation key concepts: Types of correlation Methods of studying correlation a) Scatter diagram b) Karl pearson s coefficient of correlation c) Spearman s Rank correlation coefficient d)
Classification of Virtual Investing-Related Community Postings
Classification of Virtual Investing-Related Community Postings Balaji Rajagopalan * Oakland University rajagopa@oakland.edu Matthew Wimble Oakland University mwwimble@oakland.edu Prabhudev Konana * University
Predict the Popularity of YouTube Videos Using Early View Data
000 001 002 003 004 005 006 007 008 009 010 011 012 013 014 015 016 017 018 019 020 021 022 023 024 025 026 027 028 029 030 031 032 033 034 035 036 037 038 039 040 041 042 043 044 045 046 047 048 049 050
interpretation and implication of Keogh, Barnes, Joiner, and Littleton s paper Gender,
This essay critiques the theoretical perspectives, research design and analysis, and interpretation and implication of Keogh, Barnes, Joiner, and Littleton s paper Gender, Pair Composition and Computer
Customizing an English-Korean Machine Translation System for Patent Translation *
Customizing an English-Korean Machine Translation System for Patent Translation * Sung-Kwon Choi, Young-Gil Kim Natural Language Processing Team, Electronics and Telecommunications Research Institute,
CORRECTING AND GIVING FEEDBACK TO WRITING
CORRECTING AND GIVING FEEDBACK TO WRITING We have all written papers for some courses to be checked and graded by our instructors. We know very well that a paper that is returned with red markings and
Rubrics for AP Histories. + Historical Thinking Skills
Rubrics for AP Histories + Historical Thinking Skills Effective Fall 2015 AP History Document-Based Question and Long Essay Rubrics AP History Document-Based Question and Long Essay Rubrics The rubrics
Mark Scheme (Results) November 2013. Pearson Edexcel GCSE in Mathematics Linear (1MA0) Higher (Non-Calculator) Paper 1H
Mark Scheme (Results) November 2013 Pearson Edexcel GCSE in Mathematics Linear (1MA0) Higher (Non-Calculator) Paper 1H Edexcel and BTEC Qualifications Edexcel and BTEC qualifications are awarded by Pearson,
Environmental Remote Sensing GEOG 2021
Environmental Remote Sensing GEOG 2021 Lecture 4 Image classification 2 Purpose categorising data data abstraction / simplification data interpretation mapping for land cover mapping use land cover class
DEFINING COMPREHENSION
Chapter Two DEFINING COMPREHENSION We define reading comprehension as the process of simultaneously extracting and constructing meaning through interaction and involvement with written language. We use
Statistical Machine Translation: IBM Models 1 and 2
Statistical Machine Translation: IBM Models 1 and 2 Michael Collins 1 Introduction The next few lectures of the course will be focused on machine translation, and in particular on statistical machine translation
Automatic identification of construction candidates for a Swedish constructicon
Automatic identification of construction candidates for a Swedish constructicon Linnea Bäckström, Lars Borin, Markus Forsberg, Benjamin Lyngfelt, Julia Prentice, and Emma Sköldberg Språkbanken University
UNIVERSITÀ DEGLI STUDI DELL AQUILA CENTRO LINGUISTICO DI ATENEO
TESTING DI LINGUA INGLESE: PROGRAMMA DI TUTTI I LIVELLI - a.a. 2010/2011 Collaboratori e Esperti Linguistici di Lingua Inglese: Dott.ssa Fatima Bassi e-mail: fatimacarla.bassi@fastwebnet.it Dott.ssa Liliana
We are often interested in the relationship between two variables. Do people with more years of full-time education earn higher salaries?
Statistics: Correlation Richard Buxton. 2008. 1 Introduction We are often interested in the relationship between two variables. Do people with more years of full-time education earn higher salaries? Do
The Michigan State University - Certificate of English Language Proficiency (MSU- CELP)
The Michigan State University - Certificate of English Language Proficiency (MSU- CELP) The Certificate of English Language Proficiency Examination from Michigan State University is a four-section test
A Decision Support Approach based on Sentiment Analysis Combined with Data Mining for Customer Satisfaction Research
145 A Decision Support Approach based on Sentiment Analysis Combined with Data Mining for Customer Satisfaction Research Nafissa Yussupova, Maxim Boyko, and Diana Bogdanova Faculty of informatics and robotics
CALIFORNIA STATE UNIVERSITY, HAYWARD DEPARTMENT OF ENGLISH Assessment of Master s Programs in English
CALIFORNIA STATE UNIVERSITY, HAYWARD DEPARTMENT OF ENGLISH Assessment of Master s Programs in English [Note: Program descriptions are given at the end of this document.] Student Learning Outcomes (SLOs):
Grammars and introduction to machine learning. Computers Playing Jeopardy! Course Stony Brook University
Grammars and introduction to machine learning Computers Playing Jeopardy! Course Stony Brook University Last class: grammars and parsing in Prolog Noun -> roller Verb thrills VP Verb NP S NP VP NP S VP
31 Case Studies: Java Natural Language Tools Available on the Web
31 Case Studies: Java Natural Language Tools Available on the Web Chapter Objectives Chapter Contents This chapter provides a number of sources for open source and free atural language understanding software
Chapter 8. Final Results on Dutch Senseval-2 Test Data
Chapter 8 Final Results on Dutch Senseval-2 Test Data The general idea of testing is to assess how well a given model works and that can only be done properly on data that has not been seen before. Supervised