Judging grammaticality, detecting preposition errors and generating language test items using a mixed-bag of NLP techniques

Size: px
Start display at page:

Download "Judging grammaticality, detecting preposition errors and generating language test items using a mixed-bag of NLP techniques"

Transcription

1 Judging grammaticality, detecting preposition errors and generating language test items using a mixed-bag of NLP techniques Jennifer Foster Natural Language Processing and Language Learning Workshop Nancy 18th June 2010 National Centre for Language Technology, Dublin City University

2 Joint work with: Joachim Wagner, Josef van Genabith, Monica Ward (DCU) Øistein Andersen (Cambridge) Joel Tetreault (ETS), Martin Chodorow (Hunter, CUNY) Montse Maritxalar (University of the Basque Country) Elaine Ui Dhonnchadha (Trinity College, Dublin)

3 Outline 1 2 GenERRate 3 Preposition Error Detection 4 Test Generation

4 Outline 1 2 GenERRate 3 Preposition Error Detection 4 Test Generation

5 Classifying a sentence as grammatical or ungrammatical Automatic essay grading First step before targeted error detection/correction (Limited) Feedback to advanced learners Mining bilingual sentences from web Evaluating MT output Robust parsing

6 Classifying a sentence as grammatical or ungrammatical Automatic essay grading First step before targeted error detection/correction (Limited) Feedback to advanced learners Mining bilingual sentences from web Evaluating MT output Robust parsing

7 Classifying a sentence as grammatical or ungrammatical Automatic essay grading First step before targeted error detection/correction (Limited) Feedback to advanced learners Mining bilingual sentences from web Evaluating MT output Robust parsing

8 Classifying a sentence as grammatical or ungrammatical Automatic essay grading First step before targeted error detection/correction (Limited) Feedback to advanced learners Mining bilingual sentences from web Evaluating MT output Robust parsing

9 Classifying a sentence as grammatical or ungrammatical Automatic essay grading First step before targeted error detection/correction (Limited) Feedback to advanced learners Mining bilingual sentences from web Evaluating MT output Robust parsing

10 Classifying a sentence as grammatical or ungrammatical Automatic essay grading First step before targeted error detection/correction (Limited) Feedback to advanced learners Mining bilingual sentences from web Evaluating MT output Robust parsing

11 Classifying a sentence as grammatical or ungrammatical Automatic essay grading First step before targeted error detection/correction (Limited) Feedback to advanced learners Mining bilingual sentences from web Evaluating MT output Robust parsing

12 Approaches to Grammaticality Judging 3 Methods 1 POS-N-gram-based classifier 2 Precision-grammar-based classifier 3 Statistical-parsing-based classifier

13 Approaches to Grammaticality Judging 3 Methods 1 POS-N-gram-based classifier 2 Precision-grammar-based classifier 3 Statistical-parsing-based classifier

14 Approaches to Grammaticality Judging 3 Methods 1 POS-N-gram-based classifier 2 Precision-grammar-based classifier 3 Statistical-parsing-based classifier

15 Approaches to Grammaticality Judging 3 Methods 1 POS-N-gram-based classifier 2 Precision-grammar-based classifier 3 Statistical-parsing-based classifier

16 N-gram-based Classifier Classifies a sentence as ungrammatical if it contains an unusual part of speech sequence The frequencies of least frequent bigram, trigram, 4-gram, 5-gram, 6-gram and 7-gram in the sentence. Frequencies obtained from a reference corpus of grammatical sentences.

17 N-gram-based Classifier Classifies a sentence as ungrammatical if it contains an unusual part of speech sequence The frequencies of least frequent bigram, trigram, 4-gram, 5-gram, 6-gram and 7-gram in the sentence. Frequencies obtained from a reference corpus of grammatical sentences.

18 N-gram-based Classifier Classifies a sentence as ungrammatical if it contains an unusual part of speech sequence The frequencies of least frequent bigram, trigram, 4-gram, 5-gram, 6-gram and 7-gram in the sentence. Frequencies obtained from a reference corpus of grammatical sentences.

19 Precision-grammar-based Classifier Classifies a sentence using a parser and a broad-coverage hand-written grammar ParGram English LFG and XLE engine Features: 1 whether or not the sentence can be parsed... without resorting to special robustness mechanisms 2 parsing time 3 number of parses found 4 number of optimal/unoptimal constraints used during parsing 5 number of words

20 Precision-grammar-based Classifier Classifies a sentence using a parser and a broad-coverage hand-written grammar ParGram English LFG and XLE engine Features: 1 whether or not the sentence can be parsed... without resorting to special robustness mechanisms 2 parsing time 3 number of parses found 4 number of optimal/unoptimal constraints used during parsing 5 number of words

21 Precision-grammar-based Classifier Classifies a sentence using a parser and a broad-coverage hand-written grammar ParGram English LFG and XLE engine Features: 1 whether or not the sentence can be parsed... without resorting to special robustness mechanisms 2 parsing time 3 number of parses found 4 number of optimal/unoptimal constraints used during parsing 5 number of words

22 Precision-grammar-based Classifier Classifies a sentence using a parser and a broad-coverage hand-written grammar ParGram English LFG and XLE engine Features: 1 whether or not the sentence can be parsed... without resorting to special robustness mechanisms 2 parsing time 3 number of parses found 4 number of optimal/unoptimal constraints used during parsing 5 number of words

23 Precision-grammar-based Classifier Classifies a sentence using a parser and a broad-coverage hand-written grammar ParGram English LFG and XLE engine Features: 1 whether or not the sentence can be parsed... without resorting to special robustness mechanisms 2 parsing time 3 number of parses found 4 number of optimal/unoptimal constraints used during parsing 5 number of words

24 Precision-grammar-based Classifier Classifies a sentence using a parser and a broad-coverage hand-written grammar ParGram English LFG and XLE engine Features: 1 whether or not the sentence can be parsed... without resorting to special robustness mechanisms 2 parsing time 3 number of parses found 4 number of optimal/unoptimal constraints used during parsing 5 number of words

25 Precision-grammar-based Classifier Classifies a sentence using a parser and a broad-coverage hand-written grammar ParGram English LFG and XLE engine Features: 1 whether or not the sentence can be parsed... without resorting to special robustness mechanisms 2 parsing time 3 number of parses found 4 number of optimal/unoptimal constraints used during parsing 5 number of words

26 Precision-grammar-based Classifier Classifies a sentence using a parser and a broad-coverage hand-written grammar ParGram English LFG and XLE engine Features: 1 whether or not the sentence can be parsed... without resorting to special robustness mechanisms 2 parsing time 3 number of parses found 4 number of optimal/unoptimal constraints used during parsing 5 number of words

27 Precision-grammar-based Classifier Classifies a sentence using a parser and a broad-coverage hand-written grammar ParGram English LFG and XLE engine Features: 1 whether or not the sentence can be parsed... without resorting to special robustness mechanisms 2 parsing time 3 number of parses found 4 number of optimal/unoptimal constraints used during parsing 5 number of words

28 Statistical-parsing-based Classifier

29 Statistical-parsing-based Classifier Charniak and Johnson reranking parser 1 Trained on grammatical Wall Street Journal sentences 2 Trained on ungrammatical versions of the Wall Street Journal sentences 3 Trained on a mixture of above Classifier Features 1 Parse probabilities assigned by the three grammars 2 Structural differences between the three trees for a sentence

30 Statistical-parsing-based Classifier Charniak and Johnson reranking parser 1 Trained on grammatical Wall Street Journal sentences 2 Trained on ungrammatical versions of the Wall Street Journal sentences 3 Trained on a mixture of above Classifier Features 1 Parse probabilities assigned by the three grammars 2 Structural differences between the three trees for a sentence

31 Statistical-parsing-based Classifier Charniak and Johnson reranking parser 1 Trained on grammatical Wall Street Journal sentences 2 Trained on ungrammatical versions of the Wall Street Journal sentences 3 Trained on a mixture of above Classifier Features 1 Parse probabilities assigned by the three grammars 2 Structural differences between the three trees for a sentence

32 Statistical-parsing-based Classifier Charniak and Johnson reranking parser 1 Trained on grammatical Wall Street Journal sentences 2 Trained on ungrammatical versions of the Wall Street Journal sentences 3 Trained on a mixture of above Classifier Features 1 Parse probabilities assigned by the three grammars 2 Structural differences between the three trees for a sentence

33 Statistical-parsing-based Classifier Charniak and Johnson reranking parser 1 Trained on grammatical Wall Street Journal sentences 2 Trained on ungrammatical versions of the Wall Street Journal sentences 3 Trained on a mixture of above Classifier Features 1 Parse probabilities assigned by the three grammars 2 Structural differences between the three trees for a sentence

34 Statistical-parsing-based Classifier Charniak and Johnson reranking parser 1 Trained on grammatical Wall Street Journal sentences 2 Trained on ungrammatical versions of the Wall Street Journal sentences 3 Trained on a mixture of above Classifier Features 1 Parse probabilities assigned by the three grammars 2 Structural differences between the three trees for a sentence

35 Statistical-parsing-based Classifier Charniak and Johnson reranking parser 1 Trained on grammatical Wall Street Journal sentences 2 Trained on ungrammatical versions of the Wall Street Journal sentences 3 Trained on a mixture of above Classifier Features 1 Parse probabilities assigned by the three grammars 2 Structural differences between the three trees for a sentence

36 Classifier Experiments Classifiers trained on sentences from the British National Corpus 50%: original grammatical sentences 50%: artificially created ungrammatical sentences 10-fold cross validation J48 decision tree machine learning algorithm

37 Classifier Experiments Classifiers trained on sentences from the British National Corpus 50%: original grammatical sentences 50%: artificially created ungrammatical sentences 10-fold cross validation J48 decision tree machine learning algorithm

38 Classifier Experiments Classifiers trained on sentences from the British National Corpus 50%: original grammatical sentences 50%: artificially created ungrammatical sentences 10-fold cross validation J48 decision tree machine learning algorithm

39 Classifier Experiments Classifiers trained on sentences from the British National Corpus 50%: original grammatical sentences 50%: artificially created ungrammatical sentences 10-fold cross validation J48 decision tree machine learning algorithm

40 Classifier Experiments Classifiers trained on sentences from the British National Corpus 50%: original grammatical sentences 50%: artificially created ungrammatical sentences 10-fold cross validation J48 decision tree machine learning algorithm

41 Classifier Experiments Classifiers trained on sentences from the British National Corpus 50%: original grammatical sentences 50%: artificially created ungrammatical sentences 10-fold cross validation J48 decision tree machine learning algorithm

42 Artificial Data Two types of evidence used in error detection 1 Positive: compare to some model of normal language 2 Negative: compare to some model of deviant language Combining both types of evidence is likely to be useful.

43 Artificial Data Two types of evidence used in error detection 1 Positive: compare to some model of normal language 2 Negative: compare to some model of deviant language Combining both types of evidence is likely to be useful.

44 Artificial Data Two types of evidence used in error detection 1 Positive: compare to some model of normal language 2 Negative: compare to some model of deviant language Combining both types of evidence is likely to be useful.

45 Artificial Data Two types of evidence used in error detection 1 Positive: compare to some model of normal language 2 Negative: compare to some model of deviant language Combining both types of evidence is likely to be useful.

46 Why Artificial Data? Positive data is easy enough to obtain Negative data is less straightforward Find the correct type of text Annotate errors A possible solution? Create negative data automatically.

47 Why Artificial Data? Positive data is easy enough to obtain Negative data is less straightforward Find the correct type of text Annotate errors A possible solution? Create negative data automatically.

48 Why Artificial Data? Positive data is easy enough to obtain Negative data is less straightforward Find the correct type of text Annotate errors A possible solution? Create negative data automatically.

49 Why Artificial Data? Positive data is easy enough to obtain Negative data is less straightforward Find the correct type of text Annotate errors A possible solution? Create negative data automatically.

50 Why Artificial Data? Positive data is easy enough to obtain Negative data is less straightforward Find the correct type of text Annotate errors A possible solution? Create negative data automatically.

51 Why Artificial Data? Cheap to create Knowledge of errors still necessary The error will appear in varied contexts - useful for training Number and type of errors can be controlled

52 Why Artificial Data? Cheap to create Knowledge of errors still necessary The error will appear in varied contexts - useful for training Number and type of errors can be controlled

53 Why Artificial Data? Cheap to create Knowledge of errors still necessary The error will appear in varied contexts - useful for training Number and type of errors can be controlled

54 Why Artificial Data? Cheap to create Knowledge of errors still necessary The error will appear in varied contexts - useful for training Number and type of errors can be controlled

55 Artificial Data: Background Targeted Error Detection Sjöbergh & Knutsson, 2005 Brockett et al., 2006 Lee & Seneff, 2008 Rozovskaya & Roth, 2010

56 Artificial Data: Background Grammaticality Rating Wagner et al., 2007 Okanahara & Tsujii, 2007

57 Artificial Data: Background Robustness Evaluation Bigert et al., 2005 Foster, 2007

58 Artificial Data: Background Unsupervised Learning Contrastive Estimation (Smith & Eisner, 2005)

59 Artificial Data

60 Artificial Data Authentic error corpus (Foster 2005) 923 ungrammatical sentences Ungrammatical sentences encountered when reading were noted and corrected Errors annotated in terms of correction operations Sources: newspapers, s, academic papers Produced by native and non-native English speakers

61 Artificial Data Authentic error corpus (Foster 2005) 923 ungrammatical sentences Ungrammatical sentences encountered when reading were noted and corrected Errors annotated in terms of correction operations Sources: newspapers, s, academic papers Produced by native and non-native English speakers

62 Artificial Data Authentic error corpus (Foster 2005) 923 ungrammatical sentences Ungrammatical sentences encountered when reading were noted and corrected Errors annotated in terms of correction operations Sources: newspapers, s, academic papers Produced by native and non-native English speakers

63 Artificial Data Authentic error corpus (Foster 2005) 923 ungrammatical sentences Ungrammatical sentences encountered when reading were noted and corrected Errors annotated in terms of correction operations Sources: newspapers, s, academic papers Produced by native and non-native English speakers

64 Artificial Data Authentic error corpus (Foster 2005) 923 ungrammatical sentences Ungrammatical sentences encountered when reading were noted and corrected Errors annotated in terms of correction operations Sources: newspapers, s, academic papers Produced by native and non-native English speakers

65 Artificial Data Authentic error corpus (Foster 2005) 923 ungrammatical sentences Ungrammatical sentences encountered when reading were noted and corrected Errors annotated in terms of correction operations Sources: newspapers, s, academic papers Produced by native and non-native English speakers

66 Artificial data Properties of error corpus (Foster 2005) Most ungrammatical sentences contain only one error Substitute (48%) > Insert (24%) > Delete (17%) > Combination (11%) Most common substitution errors: Real-word spelling errors Agreement errors Wrong verb form

67 Artificial data Properties of error corpus (Foster 2005) Most ungrammatical sentences contain only one error Substitute (48%) > Insert (24%) > Delete (17%) > Combination (11%) Most common substitution errors: Real-word spelling errors Agreement errors Wrong verb form

68 Artificial data Properties of error corpus (Foster 2005) Most ungrammatical sentences contain only one error Substitute (48%) > Insert (24%) > Delete (17%) > Combination (11%) Most common substitution errors: Real-word spelling errors Agreement errors Wrong verb form

69 Artificial data Properties of error corpus (Foster 2005) Most ungrammatical sentences contain only one error Substitute (48%) > Insert (24%) > Delete (17%) > Combination (11%) Most common substitution errors: Real-word spelling errors Agreement errors Wrong verb form

70 Artificial Data Five most common error types are replicated. Agreement: She steered Melissa around a corners. Real-word-spell: She could no comprehend. Extra word: Was that in the summer in? Missing word: What the subject? Verb Form: I didn t wanted to delete it. Error creation procedure expects a part-of-speech tagged corpus as input.

71 Artificial Data Five most common error types are replicated. Agreement: She steered Melissa around a corners. Real-word-spell: She could no comprehend. Extra word: Was that in the summer in? Missing word: What the subject? Verb Form: I didn t wanted to delete it. Error creation procedure expects a part-of-speech tagged corpus as input.

72 Artificial Data Five most common error types are replicated. Agreement: She steered Melissa around a corners. Real-word-spell: She could no comprehend. Extra word: Was that in the summer in? Missing word: What the subject? Verb Form: I didn t wanted to delete it. Error creation procedure expects a part-of-speech tagged corpus as input.

73 Artificial Data Five most common error types are replicated. Agreement: She steered Melissa around a corners. Real-word-spell: She could no comprehend. Extra word: Was that in the summer in? Missing word: What the subject? Verb Form: I didn t wanted to delete it. Error creation procedure expects a part-of-speech tagged corpus as input.

74 Artificial Data Five most common error types are replicated. Agreement: She steered Melissa around a corners. Real-word-spell: She could no comprehend. Extra word: Was that in the summer in? Missing word: What the subject? Verb Form: I didn t wanted to delete it. Error creation procedure expects a part-of-speech tagged corpus as input.

75 Artificial Data Five most common error types are replicated. Agreement: She steered Melissa around a corners. Real-word-spell: She could no comprehend. Extra word: Was that in the summer in? Missing word: What the subject? Verb Form: I didn t wanted to delete it. Error creation procedure expects a part-of-speech tagged corpus as input.

76 Evaluation Metrics Accuracy on ungrammatical data acc ungram = #correctly classified as ungrammatical #ungrammatical sentences Accuracy on grammatical data acc gram = #correctly classified as grammatical #grammatical sentences

77 Classifier Results Accuracy graph

78 Classifier Results Region of improvement

79 Classifier Results Region of degradation

80 Classifier Results Undecided

81 Comparing classifier results on artificial data

82 Comparing classifier results on artificial data

83 Comparing classifier results on artificial data

84 Accuracy Tradeoff with Voting Scheme Train multiple classifiers on overlapping subsets of data Each classifier votes whether sentence is grammatical Parameter: number of votes required for final decision Plot accuracy for all possible parameter values

85 Accuracy Tradeoff with Voting Scheme 0.9 Accuracy on grammatical data Accuracy on ungrammatical data all3 prob comb ngram xle

86 Testing on Data from Learner Corpora What happens when we apply our best classifier to sentences from learner corpora? International Corpus of Learner English (Granger, 1993) Transcribed spoken utterances (Learners of English of various levels and L1s) Microsoft mass noun corpus (Brockett et al, 2006) Performance drops!

87 Testing on Data from Learner Corpora What happens when we apply our best classifier to sentences from learner corpora? International Corpus of Learner English (Granger, 1993) Transcribed spoken utterances (Learners of English of various levels and L1s) Microsoft mass noun corpus (Brockett et al, 2006) Performance drops!

88 Testing on Data from Learner Corpora What happens when we apply our best classifier to sentences from learner corpora? International Corpus of Learner English (Granger, 1993) Transcribed spoken utterances (Learners of English of various levels and L1s) Microsoft mass noun corpus (Brockett et al, 2006) Performance drops!

89 Testing on Data from Learner Corpora What happens when we apply our best classifier to sentences from learner corpora? International Corpus of Learner English (Granger, 1993) Transcribed spoken utterances (Learners of English of various levels and L1s) Microsoft mass noun corpus (Brockett et al, 2006) Performance drops!

90 Testing on Data from Learner Corpora What happens when we apply our best classifier to sentences from learner corpora? International Corpus of Learner English (Granger, 1993) Transcribed spoken utterances (Learners of English of various levels and L1s) Microsoft mass noun corpus (Brockett et al, 2006) Performance drops!

91 Testing on Data from Learner Corpora 1 Accuracy on grammatical part Accuracy on ungrammatical part Foster 44 Spoken Essays Mass Noun

92 GenERRate Outline 1 2 GenERRate 3 Preposition Error Detection 4 Test Generation

93 GenERRate GenERRate A tool for introducing grammatical errors into text Available from

94 GenERRate GenERRate A tool for introducing grammatical errors into text Available from

95 GenERRate GenERRate A tool for introducing grammatical errors into text Available from

96 GenERRate GenERRate: Supported Error Types Error Deletion Insertion Move Subst DeletionPOS InsertionFromFileOrSentence MovePOS SubPos Form DeletionPOSWhere InsertionPOS MovePOSWhere NewPOS Specific InsertionPOSWhere

97 GenERRate GenERRate: Supported Error Types Error Deletion Insertion Move Subst DeletionPOS InsertionFromFileOrSentence MovePOS SubPos Form DeletionPOSWhere InsertionPOS MovePOSWhere NewPOS Specific InsertionPOSWhere

98 GenERRate GenERRate: Supported Error Types Error Deletion Insertion Move Subst DeletionPOS InsertionFromFileOrSentence MovePOS SubPos Form DeletionPOSWhere InsertionPOS MovePOSWhere NewPOS Specific InsertionPOSWhere

99 GenERRate GenERRate: Supported Error Types Error Deletion Insertion Move Subst DeletionPOS InsertionFromFileOrSentence MovePOS SubPos Form DeletionPOSWhere InsertionPOS MovePOSWhere NewPOS Specific InsertionPOSWhere

100 GenERRate GenERRate: Supported Error Types Error Deletion Insertion Move Subst DeletionPOS InsertionFromFileOrSentence MovePOS SubPos Form DeletionPOSWhere InsertionPOS MovePOSWhere NewPOS Specific InsertionPOSWhere

101 GenERRate GenERRate: Supported Error Types Error Deletion Insertion Move Subst DeletionPOS InsertionFromFileOrSentence MovePOS SubPos Form DeletionPOSWhere InsertionPOS MovePOSWhere NewPOS Specific InsertionPOSWhere

102 GenERRate GenERRate: Supported Error Types Error Deletion Insertion Move Subst DeletionPOS InsertionFromFileOrSentence MovePOS SubPos Form DeletionPOSWhere InsertionPOS MovePOSWhere NewPOS Specific InsertionPOSWhere

103 GenERRate GenERRate: Supported Error Types Error Deletion Insertion Move Subst DeletionPOS InsertionFromFileOrSentence MovePOS SubPos Form DeletionPOSWhere InsertionPOS MovePOSWhere NewPOS Specific InsertionPOSWhere

104 GenERRate GenERRate: Supported Error Types Error Deletion Insertion Move Subst DeletionPOS InsertionFromFileOrSentence MovePOS SubPos Form DeletionPOSWhere InsertionPOS MovePOSWhere NewPOS Specific InsertionPOSWhere

105 GenERRate GenERRate Input 1 A corpus of well-formed language 2 An error analysis file Output An error-tagged corpus

106 GenERRate GenERRate Input 1 A corpus of well-formed language 2 An error analysis file Output An error-tagged corpus

107 GenERRate GenERRate Error Analysis File subst,word,an,a subst,nns,nn subst,vbg,to delete,dt move,rb,left,1 Input Corpus The DT cats NNS are VBG also RB sitting VBG on IN the DT mat NN.. Output Corpus The cat are also sitting on the mat. The cats are also to sit on the mat. The cats are also sitting on mat. The cats also are sitting on the mat.

108 GenERRate GenERRate Error Analysis File subst,word,an,a subst,nns,nn subst,vbg,to delete,dt move,rb,left,1 Input Corpus The DT cats NNS are VBG also RB sitting VBG on IN the DT mat NN.. Output Corpus The cat are also sitting on the mat. The cats are also to sit on the mat. The cats are also sitting on mat. The cats also are sitting on the mat.

109 GenERRate GenERRate Error Analysis File subst,word,an,a subst,nns,nn subst,vbg,to delete,dt move,rb,left,1 Input Corpus The DT cats NNS are VBG also RB sitting VBG on IN the DT mat NN.. Output Corpus The cat are also sitting on the mat. The cats are also to sit on the mat. The cats are also sitting on mat. The cats also are sitting on the mat.

110 GenERRate Spoken Learner Corpus Experiment Existing classifier that uses artificial data Can we improve the classifier by using more realistic training data?

111 GenERRate Spoken Learner Corpus Experiment The existing classifier Wagner et al., 2007 n-gram frequency counts Training data BNC sentences Distorted versions of the BNC sentences Test data Sentences from a spoken language learner corpus

112 GenERRate Spoken Learner Corpus Experiment The Spoken Learner Corpus 4,295 utterances Produced by ESL learners in a classroom setting Various levels and L1s Transcribed by the teacher Approx. 500 of these have been corrected

113 GenERRate Spoken Learner Corpus Experiment The new classifier 1 Take out 200 sentences from test data 2 Perform manual error analysis 3 Produce GenERRate error analysis file 4 Use GenERRate to generate new ungrammatical training data

114 GenERRate Spoken Learner Corpus Experiment Results OLD CLASSIFIER 37.0% of the ungrammatical sentences are flagged and 95.5% of the flagged sentences are ungrammatical. NEW CLASSIFIER 51.6% of the ungrammatical sentences are flagged and 94.9% of the flagged sentences are ungrammatical.

115 GenERRate Spoken Learner Corpus Experiment Results OLD CLASSIFIER 37.0% of the ungrammatical sentences are flagged and 95.5% of the flagged sentences are ungrammatical. NEW CLASSIFIER 51.6% of the ungrammatical sentences are flagged and 94.9% of the flagged sentences are ungrammatical.

116 GenERRate Spoken Learner Corpus Experiment Results OLD CLASSIFIER 37.0% of the ungrammatical sentences are flagged and 95.5% of the flagged sentences are ungrammatical. NEW CLASSIFIER 51.6% of the ungrammatical sentences are flagged and 94.9% of the flagged sentences are ungrammatical.

117 GenERRate Next version of GenERRate Integration with WordNet Spelling errors Different ways of specifying contextual information, e.g. parsed input Introduce morphological errors

118 Preposition Error Detection Outline 1 2 GenERRate 3 Preposition Error Detection 4 Test Generation

119 Preposition Error Detection Targeted Error Detection Specific Errors Articles Prepositions Preposition Error Detection System Chodorow et al.,2007 Tetreault and Chodorow, 2008

120 Preposition Error Detection Targeted Error Detection Specific Errors Articles Prepositions Preposition Error Detection System Chodorow et al.,2007 Tetreault and Chodorow, 2008

121 Preposition Error Detection Targeted Error Detection Specific Errors Articles Prepositions Preposition Error Detection System Chodorow et al.,2007 Tetreault and Chodorow, 2008

122 Preposition Error Detection Targeted Error Detection Specific Errors Articles Prepositions Preposition Error Detection System Chodorow et al.,2007 Tetreault and Chodorow, 2008

123 Preposition Error Detection Selection and Error Detection Two Tasks Preposition selection in well-formed text There are many local groups the country. Preposition error detection in learner text I had a trip for Italy. Both tasks are trained on well-formed text.

124 Preposition Error Detection Selection and Error Detection Two Tasks Preposition selection in well-formed text There are many local groups the country. Preposition error detection in learner text I had a trip for Italy. Both tasks are trained on well-formed text.

125 Preposition Error Detection Selection and Error Detection Two Tasks Preposition selection in well-formed text There are many local groups the country. Preposition error detection in learner text I had a trip for Italy. Both tasks are trained on well-formed text.

126 Preposition Error Detection Selection and Error Detection Two Tasks Preposition selection in well-formed text There are many local groups the country. Preposition error detection in learner text I had a trip for Italy. Both tasks are trained on well-formed text.

127 Preposition Error Detection Baseline Features There are many local groups the country. Contextual features (token and POS) Preceding noun (PN) (token and POS) Preceding verb (PV) (token and POS) Following noun (FN) (token and POS) PN-PV-FN combinations PN, PV and FN are determined using POS-based heuristics

128 Preposition Error Detection Baseline Features There are many local groups the country. Contextual features (token and POS) Preceding noun (PN) (token and POS) Preceding verb (PV) (token and POS) Following noun (FN) (token and POS) PN-PV-FN combinations PN, PV and FN are determined using POS-based heuristics

129 Preposition Error Detection Baseline Features There are many local groups the country. Contextual features (token and POS) Preceding noun (PN) (token and POS) Preceding verb (PV) (token and POS) Following noun (FN) (token and POS) PN-PV-FN combinations PN, PV and FN are determined using POS-based heuristics

130 Preposition Error Detection Baseline Features There are many local groups the country. Contextual features (token and POS) Preceding noun (PN) (token and POS) Preceding verb (PV) (token and POS) Following noun (FN) (token and POS) PN-PV-FN combinations PN, PV and FN are determined using POS-based heuristics

131 Preposition Error Detection Baseline Features There are many local groups the country. Contextual features (token and POS) Preceding noun (PN) (token and POS) Preceding verb (PV) (token and POS) Following noun (FN) (token and POS) PN-PV-FN combinations PN, PV and FN are determined using POS-based heuristics

132 Preposition Error Detection Baseline Features There are many local groups the country. Contextual features (token and POS) Preceding noun (PN) (token and POS) Preceding verb (PV) (token and POS) Following noun (FN) (token and POS) PN-PV-FN combinations PN, PV and FN are determined using POS-based heuristics

133 Preposition Error Detection Baseline Features There are many local groups the country. Contextual features (token and POS) Preceding noun (PN) (token and POS) Preceding verb (PV) (token and POS) Following noun (FN) (token and POS) PN-PV-FN combinations PN, PV and FN are determined using POS-based heuristics

134 Preposition Error Detection Research Questions Can we get more informative features by carrying out full syntactic parsing? And how will the parser behave on ESL data? Stanford Parser Phrase Structure Parser (Klein and Manning, 2003) Typed Dependency Generator (de Marneffe et al, 2006)

135 Preposition Error Detection Research Questions Can we get more informative features by carrying out full syntactic parsing? And how will the parser behave on ESL data? Stanford Parser Phrase Structure Parser (Klein and Manning, 2003) Typed Dependency Generator (de Marneffe et al, 2006)

136 Preposition Error Detection Research Questions Can we get more informative features by carrying out full syntactic parsing? And how will the parser behave on ESL data? Stanford Parser Phrase Structure Parser (Klein and Manning, 2003) Typed Dependency Generator (de Marneffe et al, 2006)

137 Preposition Error Detection Research Questions Can we get more informative features by carrying out full syntactic parsing? And how will the parser behave on ESL data? Stanford Parser Phrase Structure Parser (Klein and Manning, 2003) Typed Dependency Generator (de Marneffe et al, 2006)

138 Preposition Error Detection Research Questions Can we get more informative features by carrying out full syntactic parsing? And how will the parser behave on ESL data? Stanford Parser Phrase Structure Parser (Klein and Manning, 2003) Typed Dependency Generator (de Marneffe et al, 2006)

139 Preposition Error Detection Parse Features NP DT many NP JJ local NNS groups amod(groups-3, many-1) amod(groups-3, local-2) prep(groups-3, around-4) det(country-6, the-5) pobj(around-4, country-6) IN around PP NP DT NN the country

140 Preposition Error Detection Parse Features NP DT many NP JJ local NNS groups amod(groups-3, many-1) amod(groups-3, local-2) prep(groups-3, around-4) det(country-6, the-5) pobj(around-4, country-6) IN around PP NP DT NN the country

141 Preposition Error Detection Parse Features NP DT many NP JJ local NNS groups amod(groups-3, many-1) amod(groups-3, local-2) prep(groups-3, around-4) det(country-6, the-5) pobj(around-4, country-6) IN around PP NP DT NN the country

142 Preposition Error Detection Parse Features NP DT many NP JJ local NNS groups amod(groups-3, many-1) amod(groups-3, local-2) prep(groups-3, around-4) det(country-6, the-5) pobj(around-4, country-6) IN around PP NP DT NN the country

143 Preposition Error Detection Parse Features NP DT many NP JJ local NNS groups amod(groups-3, many-1) amod(groups-3, local-2) prep(groups-3, around-4) det(country-6, the-5) pobj(around-4, country-6) IN around PP NP DT NN the country

144 Preposition Error Detection Parse Features NP DT many NP JJ local NNS groups amod(groups-3, many-1) amod(groups-3, local-2) prep(groups-3, around-4) det(country-6, the-5) pobj(around-4, country-6) IN around PP NP DT NN the country

145 Preposition Error Detection Parse Features NP DT many NP JJ local NNS groups amod(groups-3, many-1) amod(groups-3, local-2) prep(groups-3, around-4) det(country-6, the-5) pobj(around-4, country-6) IN around PP NP DT NN the country

146 Preposition Error Detection Parse Features NP DT many NP JJ local NNS groups amod(groups-3, many-1) amod(groups-3, local-2) prep(groups-3, around-4) det(country-6, the-5) pobj(around-4, country-6) IN around PP NP DT NN the country

147 Preposition Error Detection Parse Features NP DT many NP JJ local NNS groups amod(groups-3, many-1) amod(groups-3, local-2) prep(groups-3, around-4) det(country-6, the-5) pobj(around-4, country-6) IN around PP NP DT NN the country

148 Preposition Error Detection Parse Features NP DT many NP JJ local NNS groups amod(groups-3, many-1) amod(groups-3, local-2) prep(groups-3, around-4) det(country-6, the-5) pobj(around-4, country-6) IN around PP NP DT NN the country

149 Preposition Error Detection Parse Features NP DT many NP JJ local NNS groups amod(groups-3, many-1) amod(groups-3, local-2) prep(groups-3, around-4) det(country-6, the-5) pobj(around-4, country-6) IN around PP NP DT NN the country

150 Preposition Error Detection Parse Features NP DT many NP JJ local NNS groups amod(groups-3, many-1) amod(groups-3, local-2) prep(groups-3, around-4) det(country-6, the-5) pobj(around-4, country-6) IN around PP NP DT NN the country

151 Preposition Error Detection Parse Features NP DT many NP JJ local NNS groups amod(groups-3, many-1) amod(groups-3, local-2) prep(groups-3, around-4) det(country-6, the-5) pobj(around-4, country-6) IN around PP NP DT NN the country

152 Preposition Error Detection Preposition Selection Results Parsing Helps Model Accuracy T&C Phrase Structure Only Dependency Only Parse 68.5

153 Preposition Error Detection Preposition Selection Results Parsing Helps Model Accuracy T&C Phrase Structure Only Dependency Only Parse 68.5

154 Preposition Error Detection Preposition Error Detection Results Parsing Helps Somewhat! Method Precision Recall T&C Parse

155 Preposition Error Detection Parser Accuracy on ESL Data Manual Inspection of 210 Parse Trees Example Parser finds it easier to determine complement Parser is quite robust to preposition errors (S (NP A scientist) (VP devotes (NP (NP his prime part) (PP of (NP his life)) ) (PP in (NP research)) )

156 Preposition Error Detection Parser Accuracy on ESL Data Manual Inspection of 210 Parse Trees Example Parser finds it easier to determine complement Parser is quite robust to preposition errors (S (NP A scientist) (VP devotes (NP (NP his prime part) (PP of (NP his life)) ) (PP in (NP research)) )

157 Preposition Error Detection Parser Accuracy on ESL Data Manual Inspection of 210 Parse Trees Example Parser finds it easier to determine complement Parser is quite robust to preposition errors (S (NP A scientist) (VP devotes (NP (NP his prime part) (PP of (NP his life)) ) (PP in (NP research)) )

158 Preposition Error Detection Parser Accuracy on ESL Data Manual Inspection of 210 Parse Trees Example Parser finds it easier to determine complement Parser is quite robust to preposition errors (S (NP A scientist) (VP devotes (NP (NP his prime part) (PP of (NP his life)) ) (PP in (NP research)) )

159 Test Generation Outline 1 2 GenERRate 3 Preposition Error Detection 4 Test Generation

160 Test Generation Automatic Test Generation H 2 O is a chemical compound consisting of oxygen. Helium Distractor Potassium Distractor Hydrogen Key Carbon Distractor and

161 Test Generation Automatic Test Generation H 2 O is a chemical compound consisting of oxygen. Helium Distractor Potassium Distractor Hydrogen Key Carbon Distractor and

162 Test Generation Automatic Test Generation H 2 O is a chemical compound consisting of oxygen. Helium Distractor Potassium Distractor Hydrogen Key Carbon Distractor and

163 Test Generation Automatic Test Generation Developed at the University of the Basque Country Generate distractors automatically using semantic similarity measures New Project Apply this to Irish

164 Test Generation Automatic Test Generation Developed at the University of the Basque Country Generate distractors automatically using semantic similarity measures New Project Apply this to Irish

165 Test Generation Automatic Test Generation Scientific knowledge through Irish Irish language Literature Grammar Ceist Agam Ort The End

166 Test Generation Automatic Test Generation Scientific knowledge through Irish Irish language Literature Grammar Ceist Agam Ort The End

167 Test Generation Automatic Test Generation Scientific knowledge through Irish Irish language Literature Grammar Ceist Agam Ort The End

168 Test Generation Automatic Test Generation Scientific knowledge through Irish Irish language Literature Grammar Ceist Agam Ort The End

169 Test Generation Automatic Test Generation Scientific knowledge through Irish Irish language Literature Grammar Ceist Agam Ort The End

170 Test Generation The Problem of Covert Errors When to avoid them The cats are worth seeing. The cats are worth to see. The cats are also sitting on the mat. The cats are also to sit on the mat.

171 Test Generation The Problem of Covert Errors When to avoid them The cats are worth seeing. The cats are worth to see. The cats are also sitting on the mat. The cats are also to sit on the mat.

172 Test Generation The Problem of Covert Errors When to avoid them The cats are worth seeing. The cats are worth to see. The cats are also sitting on the mat. The cats are also to sit on the mat.

173 Test Generation The Problem of Covert Errors When to avoid them The cats are worth seeing. The cats are worth to see. The cats are also sitting on the mat. The cats are also to sit on the mat.

174 Test Generation The Problem of Covert Errors When to avoid them The cats are worth seeing. The cats are worth to see. The cats are also sitting on the mat. The cats are also to sit on the mat.

175 Test Generation The Problem of Covert Errors When not to avoid them What time did you go to bed in high school? I went to bed at one. What time did you go to bed in high school? I go to bed at one. When I was a high school student I went to bed at one in the morning When I was a high school student I go to bed at one in the morning

176 Test Generation The Problem of Covert Errors When not to avoid them What time did you go to bed in high school? I went to bed at one. What time did you go to bed in high school? I go to bed at one. When I was a high school student I went to bed at one in the morning When I was a high school student I go to bed at one in the morning

177 Test Generation The Problem of Covert Errors When not to avoid them What time did you go to bed in high school? I went to bed at one. What time did you go to bed in high school? I go to bed at one. When I was a high school student I went to bed at one in the morning When I was a high school student I go to bed at one in the morning

178 Test Generation The Problem of Covert Errors When not to avoid them What time did you go to bed in high school? I went to bed at one. What time did you go to bed in high school? I go to bed at one. When I was a high school student I went to bed at one in the morning When I was a high school student I go to bed at one in the morning

179 Test Generation The Problem of Covert Errors When not to avoid them What time did you go to bed in high school? I went to bed at one. What time did you go to bed in high school? I go to bed at one. When I was a high school student I went to bed at one in the morning When I was a high school student I go to bed at one in the morning

180 Test Generation Classifier Results WSJ Test Data

181 Test Generation Spoken Learner Corpus Experiment Training data examples ORIGINAL: Biogas production is growing rapidly OLD: Biogas production production is growing rapidly NEW: Biogas productions is growing rapidly

182 Test Generation Spoken Learner Corpus Experiment Training data examples ORIGINAL: Biogas production is growing rapidly OLD: Biogas production production is growing rapidly NEW: Biogas productions is growing rapidly

183 Test Generation Spoken Learner Corpus Experiment Training data examples ORIGINAL: Biogas production is growing rapidly OLD: Biogas production production is growing rapidly NEW: Biogas productions is growing rapidly

184 Test Generation Spoken Learner Corpus Experiment Training data examples ORIGINAL: Emil was courteous and helpful OLD: Emil as courteous and helpful NEW: Emil courteous and was helpful

185 Test Generation Spoken Learner Corpus Experiment Training data examples ORIGINAL: Emil was courteous and helpful OLD: Emil as courteous and helpful NEW: Emil courteous and was helpful

186 Test Generation Spoken Learner Corpus Experiment Training data examples ORIGINAL: Emil was courteous and helpful OLD: Emil as courteous and helpful NEW: Emil courteous and was helpful

Training Paradigms for Correcting Errors in Grammar and Usage

Training Paradigms for Correcting Errors in Grammar and Usage NAACL 10 Training Paradigms for Correcting Errors in Grammar and Usage Alla Rozovskaya and Dan Roth University of Illinois at Urbana-Champaign Urbana, IL 61801 {rozovska,danr}@illinois.edu Abstract This

More information

Judging Grammaticality: Experiments in Sentence Classification

Judging Grammaticality: Experiments in Sentence Classification Judging Grammaticality: Experiments in Sentence Classification Jo a c h i m Wa g n e r Je n n i f e r Fo s t e r Jo s e f v a n Ge n a b i t h National Centre for Language Technology Dublin City University

More information

Detection of Grammatical Errors Involving Prepositions

Detection of Grammatical Errors Involving Prepositions Detection of Grammatical Errors Involving Prepositions Martin Chodorow Hunter College of CUNY 695 Park Avenue New York, NY, 10021 mchodoro@hunter.cuny.edu Joel R. Tetreault and Na-Rae Han Educational Testing

More information

NTHU at the CoNLL-2014 Shared Task

NTHU at the CoNLL-2014 Shared Task NTHU at the CoNLL-2014 Shared Task Jian-Cheng Wu*, Tzu-Hsi Yen*, Jim Chang*, Guan-Cheng Huang*, Jimmy Chang*, Hsiang-Ling Hsu+, Yu-Wei Chang+, Jason S. Chang* * Department of Computer Science + Institute

More information

Phrases. Topics for Today. Phrases. POS Tagging. ! Text transformation. ! Text processing issues

Phrases. Topics for Today. Phrases. POS Tagging. ! Text transformation. ! Text processing issues Topics for Today! Text transformation Word occurrence statistics Tokenizing Stopping and stemming Phrases Document structure Link analysis Information extraction Internationalization Phrases! Many queries

More information

Self-Training for Parsing Learner Text

Self-Training for Parsing Learner Text elf-training for Parsing Learner Text Aoife Cahill, Binod Gyawali and James V. Bruno Educational Testing ervice, 660 Rosedale Road, Princeton, NJ 0854, UA {acahill, bgyawali, jbruno}@ets.org Abstract We

More information

Intelligent Systems (AI-2)

Intelligent Systems (AI-2) Intelligent Systems (AI-2) Computer Science cpsc422, Lecture 25 Nov, 9, 2016 CPSC 422, Lecture 26 Slide 1 Morphology Syntax Semantics Pragmatics Discourse and Dialogue Knowledge-Formalisms Map (including

More information

Analyzing the Dynamics of Research by Extracting Key Aspects of Scientific Papers

Analyzing the Dynamics of Research by Extracting Key Aspects of Scientific Papers Analyzing the Dynamics of Research by Extracting Key Aspects of Scientific Papers Sonal Gupta Christopher Manning Natural Language Processing Group Department of Computer Science Stanford University Columbia

More information

Effective Self-Training for Parsing

Effective Self-Training for Parsing Effective Self-Training for Parsing David McClosky dmcc@cs.brown.edu Brown Laboratory for Linguistic Information Processing (BLLIP) Joint work with Eugene Charniak and Mark Johnson David McClosky - dmcc@cs.brown.edu

More information

English Grammar Checker

English Grammar Checker International l Journal of Computer Sciences and Engineering Open Access Review Paper Volume-4, Issue-3 E-ISSN: 2347-2693 English Grammar Checker Pratik Ghosalkar 1*, Sarvesh Malagi 2, Vatsal Nagda 3,

More information

Context-Free Grammar Analysis for Arabic Sentences

Context-Free Grammar Analysis for Arabic Sentences Context-Free Grammar Analysis for Arabic Sentences Shihadeh Alqrainy Software Engineering Dept. Hasan Muaidi Computer Science Dept. Mahmud S. Alkoffash Software Engineering Dept. ABSTRACT This paper presents

More information

A Mixed Trigrams Approach for Context Sensitive Spell Checking

A Mixed Trigrams Approach for Context Sensitive Spell Checking A Mixed Trigrams Approach for Context Sensitive Spell Checking Davide Fossati and Barbara Di Eugenio Department of Computer Science University of Illinois at Chicago Chicago, IL, USA dfossa1@uic.edu, bdieugen@cs.uic.edu

More information

14 Automatic language correction

14 Automatic language correction 14 Automatic language correction IA161 Advanced Techniques of Natural Language Processing J. Švec NLP Centre, FI MU, Brno December 21, 2015 J. Švec IA161 Advanced NLP 14 Automatic language correction 1

More information

Identifying Focus, Techniques and Domain of Scientific Papers

Identifying Focus, Techniques and Domain of Scientific Papers Identifying Focus, Techniques and Domain of Scientific Papers Sonal Gupta Department of Computer Science Stanford University Stanford, CA 94305 sonal@cs.stanford.edu Christopher D. Manning Department of

More information

Measuring Language Development in Early Childhood Education: A Case Study of Grammar Checking in Child Language Transcripts

Measuring Language Development in Early Childhood Education: A Case Study of Grammar Checking in Child Language Transcripts Measuring Language Development in Early Childhood Education: A Case Study of Grammar Checking in Child Language Transcripts Khairun-nisa Hassanali Computer Science Department The University of Texas at

More information

Statistical Machine Translation

Statistical Machine Translation Statistical Machine Translation Some of the content of this lecture is taken from previous lectures and presentations given by Philipp Koehn and Andy Way. Dr. Jennifer Foster National Centre for Language

More information

Guest Editors Introduction: Machine Learning in Speech and Language Technologies

Guest Editors Introduction: Machine Learning in Speech and Language Technologies Guest Editors Introduction: Machine Learning in Speech and Language Technologies Pascale Fung (pascale@ee.ust.hk) Department of Electrical and Electronic Engineering Hong Kong University of Science and

More information

Automatic Correction of Grammatical Errors in Non-native English Text. John Sie Yuen Lee

Automatic Correction of Grammatical Errors in Non-native English Text. John Sie Yuen Lee Automatic Correction of Grammatical Errors in Non-native English Text by John Sie Yuen Lee BMath in Computer Science, University of Waterloo (2002) S.M. in Electrical Engineering and Computer Science,

More information

Joint Learning of English Spelling Error Correction and POS Tagging

Joint Learning of English Spelling Error Correction and POS Tagging 1,a) 1,b) 1,c) 1,d) Joint Learning of English Spelling Error Correction and POS Tagging Sakaguchi Keisuke 1,a) Mizumoto Tomoya 1,b) Komachi Mamoru 1,c) Matsumoto Yuji 1,d) Abstract: Automated grammatical

More information

SI485i : NLP. Set 7 Syntax and Parsing

SI485i : NLP. Set 7 Syntax and Parsing SI485i : NLP Set 7 Syntax and Parsing Syntax Grammar, or syntax: The kind of implicit knowledge of your native language that you had mastered by the time you were 3 years old Not the kind of stuff you

More information

Boosting Unsupervised Grammar Induction by Splitting Complex Sentences on Function Words

Boosting Unsupervised Grammar Induction by Splitting Complex Sentences on Function Words Boosting Unsupervised Grammar Induction by Splitting Complex Sentences on Function Words Jonathan Berant 1, Yaron Gross 1, Matan Mussel 1, Ben Sandbank 1, Eytan Ruppin, 1 and Shimon Edelman 2 1 Tel Aviv

More information

31 Case Studies: Java Natural Language Tools Available on the Web

31 Case Studies: Java Natural Language Tools Available on the Web 31 Case Studies: Java Natural Language Tools Available on the Web Chapter Objectives Chapter Contents This chapter provides a number of sources for open source and free atural language understanding software

More information

Learning Translation Rules from Bilingual English Filipino Corpus

Learning Translation Rules from Bilingual English Filipino Corpus Proceedings of PACLIC 19, the 19 th Asia-Pacific Conference on Language, Information and Computation. Learning Translation s from Bilingual English Filipino Corpus Michelle Wendy Tan, Raymond Joseph Ang,

More information

Curse or Boon? Presence of Subjunctive Mood in Opinionated Text

Curse or Boon? Presence of Subjunctive Mood in Opinionated Text Curse or Boon? Presence of Subjunctive Mood in Opinionated Text Sapna Negi and Paul Buitelaar Insight Centre for Data Analytics National University of Ireland Galway firstname.lastname@insight-centre.org

More information

Customizing an English-Korean Machine Translation System for Patent Translation *

Customizing an English-Korean Machine Translation System for Patent Translation * Customizing an English-Korean Machine Translation System for Patent Translation * Sung-Kwon Choi, Young-Gil Kim Natural Language Processing Team, Electronics and Telecommunications Research Institute,

More information

Comparison of different POS Tagging Techniques ( -Gram, HMM and

Comparison of different POS Tagging Techniques ( -Gram, HMM and Comparison of different POS Tagging Techniques ( -Gram, HMM and Brill s tagger) for Bangla Fahim Muhammad Hasan, Naushad UzZaman and Mumit Khan Center for Research on Bangla Language Processing, BRAC University,

More information

Special Topics in Computer Science

Special Topics in Computer Science Special Topics in Computer Science NLP in a Nutshell CS492B Spring Semester 2009 Jong C. Park Computer Science Department Korea Advanced Institute of Science and Technology INTRODUCTION Jong C. Park, CS

More information

Natural Language Understanding

Natural Language Understanding Natural Language Understanding We want to communicate with computers using natural language (spoken and written). unstructured natural language allow any statements, but make mistakes or failure. controlled

More information

Frequency based Spell Checking and Rule based Grammar Checking

Frequency based Spell Checking and Rule based Grammar Checking Frequency based Spell Checking and based Grammar Checking Shashi Pal Singh* 1, Ajai Kumar *2, Lenali Singh *3, Mahesh Bhargava *4, Kritika Goyal #1, Bhanu Sharma #2 * AAI, Center for development of Advanced

More information

Chunk Parsing. Steven Bird Ewan Klein Edward Loper. University of Melbourne, AUSTRALIA. University of Edinburgh, UK. University of Pennsylvania, USA

Chunk Parsing. Steven Bird Ewan Klein Edward Loper. University of Melbourne, AUSTRALIA. University of Edinburgh, UK. University of Pennsylvania, USA Chunk Parsing Steven Bird Ewan Klein Edward Loper University of Melbourne, AUSTRALIA University of Edinburgh, UK University of Pennsylvania, USA March 1, 2012 chunk parsing: efficient and robust approach

More information

Artificial Intelligence Exam DT2001 / DT2006 Ordinarie tentamen

Artificial Intelligence Exam DT2001 / DT2006 Ordinarie tentamen Artificial Intelligence Exam DT2001 / DT2006 Ordinarie tentamen Date: 2010-01-11 Time: 08:15-11:15 Teacher: Mathias Broxvall Phone: 301438 Aids: Calculator and/or a Swedish-English dictionary Points: The

More information

Building a Question Classifier for a TREC-Style Question Answering System

Building a Question Classifier for a TREC-Style Question Answering System Building a Question Classifier for a TREC-Style Question Answering System Richard May & Ari Steinberg Topic: Question Classification We define Question Classification (QC) here to be the task that, given

More information

1. Sentence Processing

1. Sentence Processing Linguistics 401, section 3 Sentence processing October 25, 2007 1. Sentence Processing Syntax tells us that sentences have a structure, but it doesn't tell us how that structure is used. s this one of

More information

Mavuno: A Scalable and Effective Hadoop-Based Paraphrase Acquisition System

Mavuno: A Scalable and Effective Hadoop-Based Paraphrase Acquisition System Mavuno: A Scalable and Effective Hadoop-Based Paraphrase Acquisition System Donald Metzler and Eduard Hovy Information Sciences Institute University of Southern California Overview Mavuno Paraphrases 101

More information

Testing Data-Driven Learning Algorithms for PoS Tagging of Icelandic

Testing Data-Driven Learning Algorithms for PoS Tagging of Icelandic Testing Data-Driven Learning Algorithms for PoS Tagging of Icelandic by Sigrún Helgadóttir Abstract This paper gives the results of an experiment concerned with training three different taggers on tagged

More information

Why language is hard. And what Linguistics has to say about it. Natalia Silveira Participation code: eagles

Why language is hard. And what Linguistics has to say about it. Natalia Silveira Participation code: eagles Why language is hard And what Linguistics has to say about it Natalia Silveira Participation code: eagles Christopher Natalia Silveira Manning Language processing is so easy for humans that it is like

More information

College-level L2 English Writing Competence: Conjunctions and Errors

College-level L2 English Writing Competence: Conjunctions and Errors College-level L2 English Writing Competence: Mo Li Abstract Learners output of the target language has been taken into consideration a great deal over the past decade. One of the issues of L2 writing research

More information

ENIN 020 Culture of American Classroom

ENIN 020 Culture of American Classroom The mission of the IELP is to offer innovative programs to international students. These programs are designed to increase English proficiency, to assist with acculturation of life in the United States,

More information

Grammars and introduction to machine learning. Computers Playing Jeopardy! Course Stony Brook University

Grammars and introduction to machine learning. Computers Playing Jeopardy! Course Stony Brook University Grammars and introduction to machine learning Computers Playing Jeopardy! Course Stony Brook University Last class: grammars and parsing in Prolog Noun -> roller Verb thrills VP Verb NP S NP VP NP S VP

More information

Assignment 2. Due date: 13h10, Tuesday 27 October 2015, on CDF. This assignment is worth 18% of your final grade.

Assignment 2. Due date: 13h10, Tuesday 27 October 2015, on CDF. This assignment is worth 18% of your final grade. University of Toronto, Department of Computer Science CSC 2501/485F Computational Linguistics, Fall 2015 Assignment 2 Due date: 13h10, Tuesday 27 October 2015, on CDF. This assignment is worth 18% of your

More information

Tense and Aspect Error Correction for ESL Learners Using Global Context

Tense and Aspect Error Correction for ESL Learners Using Global Context Tense and Aspect Error Correction for ESL Learners Using Global Context Toshikazu Tajiri Mamoru Komachi Yuji Matsumoto Graduate School of Information Science Nara Institute of Science and Technology 8916-5

More information

English Developmental Continuum P-10 Speaking and Listening Conventions of language Developmental Overview

English Developmental Continuum P-10 Speaking and Listening Conventions of language Developmental Overview English Developmental Continuum P-10 Speaking and Listening Conventions of language Developmental Overview 2007 Each table contains the progression points and standards related to the Speaking and Listening

More information

How much does word sense disambiguation help in sentiment analysis of micropost data?

How much does word sense disambiguation help in sentiment analysis of micropost data? How much does word sense disambiguation help in sentiment analysis of micropost data? Chiraag Sumanth PES Institute of Technology India Diana Inkpen University of Ottawa Canada 6th Workshop on Computational

More information

Customizing an English-Korean Machine Translation System for Patent/Technical Documents Translation * *

Customizing an English-Korean Machine Translation System for Patent/Technical Documents Translation * * Customizing an English-Korean Machine Translation System for Patent/Technical Documents Translation * * Oh-Woog Kwon, Sung-Kwon Choi, Ki-Young Lee, Yoon-Hyung Roh, and Young-Gil Kim Natural Language Processing

More information

Word Completion and Prediction in Hebrew

Word Completion and Prediction in Hebrew Experiments with Language Models for בס"ד Word Completion and Prediction in Hebrew 1 Yaakov HaCohen-Kerner, Asaf Applebaum, Jacob Bitterman Department of Computer Science Jerusalem College of Technology

More information

Automatic Detection of Sentence Fragments

Automatic Detection of Sentence Fragments Automatic Detection of Sentence Fragments Chak Yan Yeung and John Lee Halliday Centre for Intelligent Applications of Language Studies Department of Linguistics and Translation City University of Hong

More information

Training Set Properties and Decision-Tree Taggers: A Closer Look

Training Set Properties and Decision-Tree Taggers: A Closer Look Training Set Properties and Decision-Tree Taggers: A Closer Look Brian R. Hirshman Department of Computer Science Carnegie Mellon University Pittsburgh, PA 15213 hirshman@cs.cmu.edu Abstract This paper

More information

Open Domain Information Extraction. Günter Neumann, DFKI, 2012

Open Domain Information Extraction. Günter Neumann, DFKI, 2012 Open Domain Information Extraction Günter Neumann, DFKI, 2012 Improving TextRunner Wu and Weld (2010) Open Information Extraction using Wikipedia, ACL 2010 Fader et al. (2011) Identifying Relations for

More information

Feather in Her Hat: 200 English Idioms & Expressions for. Advanced English Language Learners

Feather in Her Hat: 200 English Idioms & Expressions for. Advanced English Language Learners Feather in Her Hat: 200 English Idioms & Expressions for Advanced English Language Learners By Erin N. O Reilly Copyright 2013 Strategic L2 www.strategicl2.com Amazon Edition Cover Art: istock All rights

More information

Speaking and Listening

Speaking and Listening Speaking and Listening ESLA 0310 0340/ESL 1010-1040 Speaking/Listening Competencies Levels 1 4 Level Course Descriptions: Student Learning Outcomes 1 In this course students learn basic conversational

More information

ESOL 94S- Ford Section 3.1 & 3.2: Input Analysis

ESOL 94S- Ford Section 3.1 & 3.2: Input Analysis ESOL 94S- Ford Section 3.1 & 3.2: Input Analysis 3.1: Variation in Language The English language, a phrase heard very frequently, gives the impression that English is one uniform system of communication

More information

Semi-automatically Alignment of Predicates between Speech and OntoNotes Data

Semi-automatically Alignment of Predicates between Speech and OntoNotes Data Semi-automatically Alignment of Predicates between Speech and OntoNotes Data Niraj Shrestha, Marie-Francine Moens Department of Computer Science, KU Leuven, Belgium {niraj.shrestha, Marie-Francine.Moens}@cs.kuleuven.be

More information

Clustering Semantically Similar and Related Questions

Clustering Semantically Similar and Related Questions Clustering Semantically Similar and Related Questions Deepa Paranjpe deepap@stanford.edu 1 ABSTRACT The success of online question answering communities that allow humans to answer questions posed by other

More information

Register Differences between Prefabs in Native and EFL English

Register Differences between Prefabs in Native and EFL English Register Differences between Prefabs in Native and EFL English MARIA WIKTORSSON 1 Introduction In the later stages of EFL (English as a Foreign Language) learning, and foreign language learning in general,

More information

A Method for Automatic De-identification of Medical Records

A Method for Automatic De-identification of Medical Records A Method for Automatic De-identification of Medical Records Arya Tafvizi MIT CSAIL Cambridge, MA 0239, USA tafvizi@csail.mit.edu Maciej Pacula MIT CSAIL Cambridge, MA 0239, USA mpacula@csail.mit.edu Abstract

More information

Basic Spelling Rules: Learn the four basic spelling rules and techniques for studying hard-to-spell words. Practice spelling from dictation.

Basic Spelling Rules: Learn the four basic spelling rules and techniques for studying hard-to-spell words. Practice spelling from dictation. CLAD Grammar & Writing Workshops Adjective Clauses This workshop includes: review of the rules for choice of adjective pronouns oral practice sentence combining practice practice correcting errors in adjective

More information

Sorting out the Most Confusing English Phrasal Verbs

Sorting out the Most Confusing English Phrasal Verbs STARSEM-2012 Sorting out the Most Confusing English Phrasal Verbs Yuancheng Tu Department of Linguistics University of Illinois ytu@illinois.edu Dan Roth Department of Computer Science University of Illinois

More information

Assignment 3. Due date: 13:10, Wednesday 4 November 2015, on CDF. This assignment is worth 12% of your final grade.

Assignment 3. Due date: 13:10, Wednesday 4 November 2015, on CDF. This assignment is worth 12% of your final grade. University of Toronto, Department of Computer Science CSC 2501/485F Computational Linguistics, Fall 2015 Assignment 3 Due date: 13:10, Wednesday 4 ovember 2015, on CDF. This assignment is worth 12% of

More information

Dictionary-Driven Semantic Look-up

Dictionary-Driven Semantic Look-up Computers and the Humanities 34: 193 197, 2000. 2000 Kluwer Academic Publishers. Printed in the Netherlands. 193 Dictionary-Driven Semantic Look-up FRÉDÉRIQUE SEGOND 1, ELISABETH AIMELET 1, VERONIKA LUX

More information

Discriminative Approach to Fill-in-the-Blank Quiz Generation for Language Learners

Discriminative Approach to Fill-in-the-Blank Quiz Generation for Language Learners Discriminative Approach to Fill-in-the-Blank Quiz Generation for Language Learners Keisuke Sakaguchi 1 Yuki Arase 2 Mamoru Komachi 1 1 Nara Institute of Science and Technology 8916-5 Takayama, Ikoma, Nara,

More information

Machine Learning for natural language processing

Machine Learning for natural language processing Machine Learning for natural language processing Introduction Laura Kallmeyer Heinrich-Heine-Universität Düsseldorf Summer 2016 1 / 13 Introduction Goal of machine learning: Automatically learn how to

More information

A Framework for Collocation Error Correction in Web Pages and Text Documents

A Framework for Collocation Error Correction in Web Pages and Text Documents A Framework for Collocation Error Correction in Web Pages and Text Documents Alan Varghese 1, Aparna S. Varde 1, Jing Peng 1 and Eileen Fitzpatrick 2 1. Department of Computer Science 2. Department of

More information

Sentiment analysis: towards a tool for analysing real-time students feedback

Sentiment analysis: towards a tool for analysing real-time students feedback Sentiment analysis: towards a tool for analysing real-time students feedback Nabeela Altrabsheh Email: nabeela.altrabsheh@port.ac.uk Mihaela Cocea Email: mihaela.cocea@port.ac.uk Sanaz Fallahkhair Email:

More information

Context Free Grammars

Context Free Grammars Context Free Grammars So far we have looked at models of language that capture only local phenomena, namely what we can analyze when looking at only a small window of words in a sentence. To move towards

More information

Test at a Glance. About this test

Test at a Glance. About this test English to Speakers of Other Languages (0360) Test at a Glance Test Name English to Speakers of Other Languages Test Code 0360 Time 2 hours, with a 30-minute listening section Number of Questions 120,

More information

Syntax: Phrases. 1. The phrase

Syntax: Phrases. 1. The phrase Syntax: Phrases Sentences can be divided into phrases. A phrase is a group of words forming a unit and united around a head, the most important part of the phrase. The head can be a noun NP, a verb VP,

More information

Learning and Inference for Clause Identification

Learning and Inference for Clause Identification Learning and Inference for Clause Identification Xavier Carreras Lluís Màrquez Technical University of Catalonia (UPC) Vasin Punyakanok Dan Roth University of Illinois at Urbana-Champaign (UIUC) ECML 2002

More information

Ling 130 Notes: English syntax

Ling 130 Notes: English syntax Ling 130 Notes: English syntax Sophia A. Malamud March 13, 2014 1 Introduction: syntactic composition A formal language is a set of strings - finite sequences of minimal units (words/morphemes, for natural

More information

RNN for Sentiment Analysis: Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

RNN for Sentiment Analysis: Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank RNN for Sentiment Analysis: Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank Borui(Athena) Ye University of Waterloo borui.ye@uwaterloo.ca July 15, 2015 1 / 26 Overview 1 Introduction

More information

Learning and Inference for Clause Identification

Learning and Inference for Clause Identification ECML 02 Learning and Inference for Clause Identification Xavier Carreras 1, Lluís Màrquez 1, Vasin Punyakanok 2, and Dan Roth 2 1 TALP Research Center LSI Department Universitat Politècnica de Catalunya

More information

Automatic Collocation Suggestion in Academic Writing

Automatic Collocation Suggestion in Academic Writing Automatic Collocation Suggestion in Academic Writing Jian-Cheng Wu 1 Yu-Chia Chang 1,* Teruko Mitamura 2 Jason S. Chang 1 1 National Tsing Hua University Hsinchu, Taiwan {wujc86, richtrf, jason.jschang}

More information

Dr. Markisha Smith Program Education Specialist. Ed. S. David Bautista Assistant Superintendent

Dr. Markisha Smith Program Education Specialist. Ed. S. David Bautista Assistant Superintendent Dr. Markisha Smith Program Education Specialist Ed. S. David Bautista Assistant Superintendent 1 Oregon English Learner Numbers Fastest growing population 58,000 Current and 58,000 Exited High Poverty

More information

WLN. Grammar Checkers Do Not Work Les Perelman Massachusetts Institute of Technology Cambridge, Massachusetts

WLN. Grammar Checkers Do Not Work Les Perelman Massachusetts Institute of Technology Cambridge, Massachusetts WLN Grammar Checkers Do Not Work Les Perelman Massachusetts Institute of Technology Cambridge, Massachusetts Daily I thank the powers that be for the computer spell checker. I never could spell decently.

More information

Efficient Techniques for Improved Data Classification and POS Tagging by Monitoring Extraction, Pruning and Updating of Unknown Foreign Words

Efficient Techniques for Improved Data Classification and POS Tagging by Monitoring Extraction, Pruning and Updating of Unknown Foreign Words , pp.290-295 http://dx.doi.org/10.14257/astl.2015.111.55 Efficient Techniques for Improved Data Classification and POS Tagging by Monitoring Extraction, Pruning and Updating of Unknown Foreign Words Irfan

More information

Accelerating and Evaluation of Syntactic Parsing in Natural Language Question Answering Systems

Accelerating and Evaluation of Syntactic Parsing in Natural Language Question Answering Systems Accelerating and Evaluation of Syntactic Parsing in Natural Language Question Answering Systems cation systems. For example, NLP could be used in Question Answering (QA) systems to understand users natural

More information

Multi-Engine Machine Translation by Recursive Sentence Decomposition

Multi-Engine Machine Translation by Recursive Sentence Decomposition Multi-Engine Machine Translation by Recursive Sentence Decomposition Bart Mellebeek Karolina Owczarzak Josef van Genabith Andy Way National Centre for Language Technology School of Computing Dublin City

More information

Discovery of Manner Relations and their Applicability to Question Answering

Discovery of Manner Relations and their Applicability to Question Answering Discovery of Manner Relations and their Applicability to Question Answering Roxana Girju, Manju Putcha and Dan Moldovan Human Language Technology Research Institute University of Texas at Dallas and Department

More information

Recent Innovations in NTT s Statistical Machine Translation

Recent Innovations in NTT s Statistical Machine Translation : Front-line of Speech, Language, and Hearing Research for Heartfelt Communications Recent Innovations in NTT s Statistical Machine Translation Masaaki Nagata, Katsuhito Sudoh, Jun Suzuki, Yasuhiro Akiba,

More information

Overview. Clustering. Clustering vs. Classification. Supervised vs. Unsupervised Learning. Connectionist and Statistical Language Processing

Overview. Clustering. Clustering vs. Classification. Supervised vs. Unsupervised Learning. Connectionist and Statistical Language Processing Overview Clustering Connectionist and Statistical Language Processing Frank Keller keller@coli.uni-sb.de Computerlinguistik Universität des Saarlandes clustering vs. classification supervised vs. unsupervised

More information

Listening Speaking Reading Writing

Listening Speaking Reading Writing Advanced CB 21 A One level Assess descriptions and narrations of factual and technical materials. Discriminate for accurate information while taking notes of a complex discourse. Assess the essential message

More information

Natural Language Models and Interfaces Part B, lecture 3

Natural Language Models and Interfaces Part B, lecture 3 atural Language Models and Interfaces Part B, lecture 3 Ivan Titov Institute for Logic, Language and Computation Today (Treebank) PCFG weaknesses PCFG extensions Structural annotation Lexicalization PCFG

More information

A Beam-Search Decoder for Grammatical Error Correction

A Beam-Search Decoder for Grammatical Error Correction A Beam-Search Decoder for Grammatical Error Correction Daniel Dahlmeier 1 and Hwee Tou Ng 1,2 1 NUS Graduate School for Integrative Sciences and Engineering 2 Department of Computer Science, National University

More information

D2.4: Two trained semantic decoders for the Appointment Scheduling task

D2.4: Two trained semantic decoders for the Appointment Scheduling task D2.4: Two trained semantic decoders for the Appointment Scheduling task James Henderson, François Mairesse, Lonneke van der Plas, Paola Merlo Distribution: Public CLASSiC Computational Learning in Adaptive

More information

The Specific Text Analysis Tasks at the Beginning of MDA Life Cycle

The Specific Text Analysis Tasks at the Beginning of MDA Life Cycle SCIENTIFIC PAPERS, UNIVERSITY OF LATVIA, 2010. Vol. 757 COMPUTER SCIENCE AND INFORMATION TECHNOLOGIES 11 22 P. The Specific Text Analysis Tasks at the Beginning of MDA Life Cycle Armands Šlihte Faculty

More information

Paraphrasing controlled English texts

Paraphrasing controlled English texts Paraphrasing controlled English texts Kaarel Kaljurand Institute of Computational Linguistics, University of Zurich kaljurand@gmail.com Abstract. We discuss paraphrasing controlled English texts, by defining

More information

Teaching Vocabulary to Young Learners (Linse, 2005, pp. 120-134)

Teaching Vocabulary to Young Learners (Linse, 2005, pp. 120-134) Teaching Vocabulary to Young Learners (Linse, 2005, pp. 120-134) Very young children learn vocabulary items related to the different concepts they are learning. When children learn numbers or colors in

More information

Editing Your Writing for Grammar Mistakes

Editing Your Writing for Grammar Mistakes Editing Your Writing for Grammar Mistakes Does grammar matter? In most assignment guidelines given in the Faculty of Business a requirement for clear expression is mentioned. Some assignment guidelines

More information

Dealing with Grammar Errors. Compiled by Ms. Terri Yueh

Dealing with Grammar Errors. Compiled by Ms. Terri Yueh Dealing with Grammar Errors Compiled by Ms. Terri Yueh What is your Approach to Error Correction? I correct students sometimes, but not all the time. If we re practicing one particular language point,

More information

CHARACTERISTICS FOR STUDENTS WITH: LIMITED ENGLISH PROFICIENCY (LEP)

CHARACTERISTICS FOR STUDENTS WITH: LIMITED ENGLISH PROFICIENCY (LEP) CHARACTERISTICS FOR STUDENTS WITH: LIMITED ENGLISH PROFICIENCY (LEP) Research has shown that students acquire a second language in the same way that they acquire the first language. It is an exploratory

More information

Session 2: Teaching procedures to assist students who have oral language difficulties in the classroom. John Munro

Session 2: Teaching procedures to assist students who have oral language difficulties in the classroom. John Munro Session 2: Teaching procedures to assist students who have oral language difficulties in the classroom John Munro Pathway followed in this session Using the ICPALER framework to develop teaching procedures

More information

POS Tagging 1. POS Tagging. Rule-based taggers Statistical taggers Hybrid approaches

POS Tagging 1. POS Tagging. Rule-based taggers Statistical taggers Hybrid approaches POS Tagging 1 POS Tagging Rule-based taggers Statistical taggers Hybrid approaches POS Tagging 1 POS Tagging 2 Words taken isolatedly are ambiguous regarding its POS Yo bajo con el hombre bajo a PP AQ

More information

Modern Natural Language Interfaces to Databases: Composing Statistical Parsing with Semantic Tractability

Modern Natural Language Interfaces to Databases: Composing Statistical Parsing with Semantic Tractability Modern Natural Language Interfaces to Databases: Composing Statistical Parsing with Semantic Tractability Ana-Maria Popescu Alex Armanasu Oren Etzioni University of Washington David Ko {amp, alexarm, etzioni,

More information

Turker-Assisted Paraphrasing for English-Arabic Machine Translation

Turker-Assisted Paraphrasing for English-Arabic Machine Translation Turker-Assisted Paraphrasing for English-Arabic Machine Translation Michael Denkowski and Hassan Al-Haj and Alon Lavie Language Technologies Institute School of Computer Science Carnegie Mellon University

More information

Detecting Grammatical Errors in Text using a Ngram-based Ruleset

Detecting Grammatical Errors in Text using a Ngram-based Ruleset Detecting Grammatical Errors in Text using a Ngram-based Ruleset Manu Konchady, Mustru Search Services, 118, RMV Extension, Stage 2, Block 1, Bangalore, 560094. India. mkonchady@yahoo.com Abstract Applications

More information

PROGRESS MONITORING CHECKLIST FOR ENGLISH LANGUAGE LEARNERS (ELL)

PROGRESS MONITORING CHECKLIST FOR ENGLISH LANGUAGE LEARNERS (ELL) LISTENING Standard : Students demonstrate competence in listening as a tool for learning and comprehension. Proficiency Level I: Students at this level are beginning to understand short utterances. They

More information

Dublin City University at QA@CLEF 2008

Dublin City University at QA@CLEF 2008 Dublin City University at QA@CLEF 2008 Sisay Fissaha Adafre Josef van Genabith National Center for Language Technology School of Computing, DCU IBM CAS Dublin sadafre,josef@computing.dcu.ie Abstract We

More information

Factive / non-factive predicate recognition within Question Generation systems

Factive / non-factive predicate recognition within Question Generation systems ISSN 1744-1986 T e c h n i c a l R e p o r t N O 2009/ 09 Factive / non-factive predicate recognition within Question Generation systems B Wyse 20 September, 2009 Department of Computing Faculty of Mathematics,

More information

Sentiment analysis on news articles using Natural Language Processing and Machine Learning Approach.

Sentiment analysis on news articles using Natural Language Processing and Machine Learning Approach. Sentiment analysis on news articles using Natural Language Processing and Machine Learning Approach. Pranali Chilekar 1, Swati Ubale 2, Pragati Sonkambale 3, Reema Panarkar 4, Gopal Upadhye 5 1 2 3 4 5

More information

Sentences and paragraphs

Sentences and paragraphs Sentences and paragraphs From the Skills Team, University of Hull What is a sentence? This is not an easy question to answer but we shall try to define the main characteristics of a sentence in the written

More information

Automatic Detection and Correction of Errors in Dependency Treebanks

Automatic Detection and Correction of Errors in Dependency Treebanks Automatic Detection and Correction of Errors in Dependency Treebanks Alexander Volokh DFKI Stuhlsatzenhausweg 3 66123 Saarbrücken, Germany alexander.volokh@dfki.de Günter Neumann DFKI Stuhlsatzenhausweg

More information