Frequency based Spell Checking and Rule based Grammar Checking

Transcription

1 Frequency based Spell Checking and based Grammar Checking Shashi Pal Singh* 1, Ajai Kumar *2, Lenali Singh *3, Mahesh Bhargava *4, Kritika Goyal #1, Bhanu Sharma #2 * AAI, Center for development of Advanced Computing, Pune, India * 1 shashis@cdac.in * 2 ajai@cdac.in *3 lenali@cdac.in *4 mbhargava@cdac.in # Banasthali Vidyapith, Banasthali, India #1 kratikagoyel@gmail.com #2 bh4nusharma@gmail.com Abstract- English is a language that is spoken by around million people on this planet and understanding it is not at all easy. The meaning of a sentence varies according to the context and the tone of the speaker. To convey the thoughts efficiently, the knowledge of the language and its various rules is very important as thoughts take the form of words and the words take the form of action. One should aim to minimize the errors while using the language. Lesser is the number of mistakes, better will be the communication. To aid in achieving this goal, we are creating a frequency based spell checker and a rule based grammar checker for English language. The grammar checker focuses on detecting and correcting tense related mistakes. Keywords Spell Checker, Grammar Checker, JSON (Java Script Object Notation), N-Gram, Tense, Suggestion Prediction I. INTRODUCTION To err is human, and the same is applicable for languages also. We make mistakes while writing English, which can be divided into two kinds: spelling mistakes and grammatical mistakes. Spelling mistakes are mistakes where the given word does not belong to vocabulary of the language whereas grammatical mistakes correspond to the mistakes where the sentence does not confirm to rules imposed on the language s grammar. A spell checker is a software or program or idea which is designed to spot misspellings in a text while either automatically correct them or suggest different spelling options. Spell checkers can be very helpful when you want a polished document without any spelling error. A Grammar checker is a software or program or idea which is designed to spot grammatical mistakes in the given text with the capability of either automatically correcting the mistakes or suggesting correct options. II. LITERATURE REVIEW Spell Checking and Grammar Checking are very important tasks in the process of writing. When people are in a hurry to submit an essay or abstract or any written piece in any language these aspects are recklessly abandoned without thinking of the consequences. Considering the fact that if you are given a material that has various spelling and grammatical mistakes and due to which the entire document does not deliver the idea that it must, then would you be convinced about the idea the writer wants to convey? A. Spell Checking One of the error types that can be done while writing spellings is non word error. Non word Error: These are the errors where the word doesn t exist in the dictionary. (Amit Sharma, 2013) E.g. 1. He is a gud boy. 2. She is a nce girl. Gud, nce are the words that don t exist in English Dictionary. Correct statements are: 1. He is a good boy. 2. She is a nice girl. In spell checker the basic tasks are: 1. Error Detection 2. Suggestion Prediction 1. Error Detection Technique : There are various methods of error detection. We are mainly using two methods which are : 1.1 Dictionary Lookup Method: Dictionary lookup is one of the most important methods of error detection in spell checking, it is the technique where data is arranged in rows and columns. If word is not found in dictionary then it is considered as incorrect. Advantage of Dictionary lookup technique is, it is easy to implement but the problem is cost of searching in a large dictionary is often high, and also sometimes it is difficult to extract and store the complete vocabulary of a given language in it. But this technique is good for non- word type of error. (Pratistha Mathur, 2012) 1.2 N-Gram Technique N-gram analysis is a process to detect wrongly spelled words. In this approach rather than comparing each and every word in a dictionary, n-grams are used. If we find

2 an empty or deficient n-gram, the word is considered as an incorrect word, otherwise it is assumed to be correct. An n- gram is a collection of characters of length N. If N is equal to 1 then the term used is a unigram, if N is 2 then it is called Bigram, and if N is 3 then the term is trigram and so on. Each word or string that is involved in the process of comparison is divided into pair of adjacent N-grams. The n-grams algorithm is also referred as neutral string matching or a language independent algorithm. P(X/Y) = P(X) P(Y/X)/P(Y) [5] Where P(X/Y): probability of X given Y, P(X): overall probability of X, P(Y/X): probability of Y given X, P(Y): overall probability of Y 2. Error Correction and Suggestion Prediction Techniques: 2.1 Edit Distance: This method is based on the assumption that a person usually types only a few incorrect letters while typing a word, therefore for each dictionary word, lesser the number of the basic edit operations (insertion, deletions, substitutions) necessary to convert the dictionary word into the non-word, higher the probability that the user intended to type that dictionary word in place of the misspelled word. (Pratistha Mathur, 2012) e.g. Distance between the words gud and good will be 2, since the following two edits can change one into the other, similarly: TABLE.1 EDIT DISTANCE G G Same O U Substitution of u in place of o O - Deletion of o D D Same 2.2 Grammar Checking [ based approach]: The grammar checking is done on the basis of -based checking.this is the approach where we match the text with a set of rules and that has been at least POS tagged. The rules are in accordance with the grammar of the language of interest. The major disadvantage of this approach is that there can be errors which the system will not able to detect. (Naber, 2003) Advantage of this approach is as follows: This method gives immediate feedback. It is easy to configure, as each rule can be turned on and off individually. It can give detailed error messages with comments, even can explain grammar rules. It is easily extendable by the users, as the rule system is easy to understand. Its extension is easy, starting with just one rule and then extending it rule by rule Tense Based Grammar Checker: Tenses are verb-based method used to indicate the time, sometimes the continuation or completeness, of an action. It is a method used to refer time past, present or future. The tenses we are focusing: Present Past Future Our grammar checker is based on the English grammar rules, which most of us have studied in our schools. Since these rules are key to the grammar checker, we are listing down the rules in details Present Tense It is used to tell about present time i.e. the things that are currently going on.it is further divided into 4 types: TABLE.2 GRAMMAR RULES AND CHECKER FOR PRESENT TENSE Past Tense It is used to express the conditions or actions that occurred in past. TABLE.3 GRAMMAR RULES AND CHECKER FOR PAST TENSE Simple present Simple sentence Negative Interrogative Subject + verb +s/es in Third Person Do(used with plural noun) Does(used with Singular noun) Subject + is/am/are + present are(used with plural noun) is(used with Singular noun) am(used with I) Subject + has/have + past [3rd form of verb] Simple past Simple sentence Negative Interrogative Subject + 2 nd form of verb Subject+ do/does+ not+ base form of verb Present Continuous Subject+ is/am/are+ not+ present Present Perfect have(used with plural noun) has(used with Singular noun) Subject+ has/have+ not+ past Present Perfect Continuous Subject + has/have+ Subject+ has/have+ been+ present not+ been+ present have been(used with plural noun) Has been(used with Singular noun) Subject+ did + not+ base form of verb Did(used with both singular and plural noun) Past Continuous Subject + was/were Subject+ was/were+ + present not+ present Do/does+ subject+ base form of verb +? Is/am/are + subject+ present Has/have+ subject+ past Has/have+ subject+ been+ present Did+ subject+ base form of verb +? Was/Were + subject+ present were(used with plural noun) was(used with Singular noun) I is an exception. In past tense I is considered as a singular noun and was is used with it.

3 Subject + had + past [3rd form of verb] Past Perfect Subject+ had+ not+ past had(used with both singular and plural noun) Past Perfect Continuous Subject + had+ been+ present Subject+ had+ not+ been+ present Had been(used with both Singular noun and plural noun) Had+ subject+ past Had+ subject+ been+ present Future Tense It is used to locate a situation or action in coming time. TABLE.4 GRAMMAR RULES AND CHECKER FOR FUTURE TENSE Simple Future Simple sentence Negative Interrogative Subject + will/shall +base form of verb Subject+ will/shall + not+ base form of verb Will/ Shall(used with both Singular plural noun) Future Continuous be+ present will/shall+ not+ be+ present Will/Shall+ subject+ base form of verb +? Will/Shall + subject+ be +present Will/Shall be(used with singular and plural noun) Future Perfect Will/shall+ have + past will/shall+ not+ subject+ [3rd form of have+ past have+ past verb] Will have/shall have(used with singular and plural noun) Future Perfect Continuous have+ been+ present will/shall+ not+ have+ been+ present Will/shall+ subject+ have + been+ present Will have been/shall have been(used with singular and plural noun) III. PROPOSED SYSTEM A. Spell Checker Frequency Based Suggestion Prediction For each word, in our dictionary we maintain the usage frequency. For the words, which are within a threshold edit distance of the given incorrect word, we look at the frequency of each suggested word. The word with higher frequency is presented higher in the suggestion list. TABLE.5 FREQUENCY TABLE Words Frequency A 196 Alike 150 abuse 500 The proper training should be given to the corpus so that it gives efficient and accurate results. Fig 1: Flow of Spell Checker A.1 Text Retrieval Extract the text when the user enters it and the cleaning of the data should be done so that it can be checked against the dictionary efficiently. Further, tags are removed in the cleaning part A.2 Dictionary Lookup A dictionary is maintained, where each word should be matched with it. If the word is there in the dictionary then it is considered as correct and we simply increase its frequency by 1, but if not there in the dictionary then we will further proceed. A.3 Edit Distance We apply edit distance only on those words which are not present in the dictionary. By using proper queries and query optimization techniques we extract the words which are close to incorrect word and then edit distance algorithm applied on the incorrect word and the words which are obtains as result of query optimization. A.4 Frequency Based Suggestion Prediction The words, whose distance is minimum with the incorrect word and whose frequency is maximum are shown to the user as suggestions. B. Grammar Checker On the Basis of tense rules, certain rules are formed which can be understood by the system. B.1 Representation of rules: The rules are stored in a JSON file. Below is a sample of how the rules are structured: "key": "singular", "nextallowedvalues":[ "type": "word", "value": "will", "nextallowedvalues": [ "value": "VB", "exception": "be", }, "type": "word", "value": "be", "nextallowedvalues": [ "value": "VBG", } ] }, } The above presented rule can be broken down as:

4 For a singular noun, the values which are allowed is an array labeled as nextallowedvalues. Each array entry is a valid value for the singular noun. E.g. Consider He will., He is a singular noun and will is a legal value that can come after a singular noun. So we have an entry for will in our next allowed values. Consider He will go to market The above sentence is correct. But we don t have any rule in our JSON (JavaScript Object Notation) [6] which says that A singular noun followed by will followed by goes is allowed. If we have to store such rules for all possible verbs in the world, our rules would be really unmanageable. So we have a different approach to store these rules. The sentence above is a simple future tense. We know that a simple future tense has first form of the verb (represented by tag VB). So instead of having a rule for every verb, we have a generic rule that a VB tag is allowed after will. Consider the sentence They is They is a plural noun. A plural noun cannot be followed by is according to our tense rules. So there won t be any rule in our JSON which says that a plural noun followed by is is valid. Process (NP (NNS sweets)))) (..))) With the help of parsing we can identify whether the subject is singular or plural and also the verb phrase. With this we can easy apply English tense rules on a sentence and find which part is correct and what will be the suggestions for that corresponding sentence. Using the parse tree, classify the noun as singular/plural (I/we etc.) and find out the word from which we need to start matching our rules. Noun: people, which is a plural noun Verb: is It means that we should start matching the rules for plural noun against the word is. B.4 Match with the Tense s Using the noun type select the root in the rules (JSON file), start matching the words in sentence against the nextallowedvalues. If matched, proceed to next word and continue the process until we reach the leaf node in our rules tree. If we reach to a leaf node, the sentence is correct. If the current word does not match with any of the rules in nextallowedvalues, the sentence is incorrect C. Find Suggestions If the sentence is not correct then we give suggestions based on the next allowed values. A. Spell Checker IV. RESULT TABLE 6. SPELL CHECKER AND SUGGESTION Fig. 2: Flow of Grammar Checker B.2 POS Tagging Parts-of-speech tagging, also known as grammatical tagging disambiguation is a technique in corpus linguistics to assign the part-of-speech to the words. Tagging may be based on the definition of the word or the context of the word it is being used in. The various parts-of-speech include noun, main verb, auxiliary verb, proper-noun, adjective, adverb etc. (Naber, 2003) Example- Those people is eating sweets. POS Tag-Those/DT People/NNS is/vbz eating/vbg Sweets/NNS Tag the incoming sentence using POS tagger B.3 Parsing Parsing is the process of analyzing a string or symbols, conforming to the rules of grammar. After parsing we get a tree which is known as parsed tree which will divide noun phrase and verb phrase. (Naber, 2003) Example- The tree for the sentence Those people is eating sweets. is below. (S (NP (DT Those) (NNS people)) (VP (VBZ is) (VP (VBG eating) Incorrect Word Gud Hile B. Grammar Checker good god dug hole hike kite Suggestion (in order) TABLE 7. GRAMMAR CHECKER AND SUGGESTION S.No. Input Output Suggestions 1. Those People is do their work Those People is do their work. Is do-> will Do shall Do had Done were Doing have Done are Doing ignore 2. Those people are travel to Pune. They are doing their work. They have do it. Dogs are run. Those people are travel to Pune. They are doing their work. They have do it. Dogs are run. Travel-> Travelling Do-> Done Run-> Running 3. one of them is one of them is Correct going going People standing People standing on Correct

5 4. on the hill are singing. the hill are singing. V. CONCLUSION We have developed an English language processing system which comprises of a Spell Checker and a Grammar Checker. The grammar checker works on the rule-based approach whereas the spell checker is based on frequency based approach. VI. FUTURE SCOPE The accuracy of the grammar can be further increased by refining the grammar based rules. The application of grammar checking can be applied to the various aspects of the grammar apart from the tenses in English language, such as the Active and Passive voice, Direct and Indirect speech, etc. A smartest spell checker can be made which along with Error Detection, Suggestion Prediction and Automatic error correction, also helps user to learn spellings. For instances, there are certain patterns like ii that can t come together, so on observing the patterns of mistakes committed buy the users the application can give them appropriate suggestions so that user can learn spelling rules and can reduce spelling mistakes. The current system can be extended for supporting languages other than English. REFERENCES [1] Amit Sharma, P. J. (2013). Hindi Spell Checker. [2] Naber, D. (2003). A -Based Style and Grammar Checker. [3] Pratistha Mathur, N. G. (2012). Spell Checking Techniques in NLP: A Survey. International Journal of Advanced Research in Computer Science and Software Engineering. [4] Amanjot Kaur, Dr. Paramjeet Singh, Dr. Shaveta Rani(2012). Spell Checking and Error Correcting System for text paragraphs written in Punjabi Language using Hybrid approach. [5] [6] [7]