VOL. 3, NO.10 Oct, 2012 ISSN Journal of Emerging Trends in Computing and Information Sciences CIS Journal. All rights reserved.

Transcription

1 Spelling Err Trends and Patterns in Sindhi 1 Zeeshan Bhatti, 2 Imdad Ali Ismaili, 3 Asad Ali Shaikh, 4 Waseem Javaid 1,2,3,4 Institute of Infmation and Communication Technology, University of Sindh, Jamsho { 1 zeeshanbhatti22@hotmail.com, 2 iai_a@yahoo.com, 3 dr.asadshaikh@yahoo.com, 4 waseem.javaid@usindh.edu.pk} ABSTRACT Statistical err ion technique is the most accurate and widely used approach today, but f a language like Sindhi which is a low resourced language the trained cpa s are not available, so the statistical techniques are not possible at all. Instead a useful alternative would be to exploit various spelling err trends in Sindhi by using a Rule based approach. F designing such technique an essential prerequisite would be to study the various err patterns in a language. This paper presents various studies of spelling err trends and their types in Sindhi Language. The research shows that the err trends common to all languages are also encountered in Sindhi but their do exist some err patters that are catered specifically to a Sindhi language. Keywds: Sindhi, Err Trends, Phonetic Errs, Cognitive Errs, Typographic Errs, Visual Errs. 1. INTRODUCTION Spell Checker have been developed f many wld languages. The research in the field of detecting errs in typed wds started in 1960 [2]. The investigations have been made that the first spell checkers were only Verifiers. They were not perfming the task of s, in that they offered no suggestions f increct spelling wd. The investigation showed that f logical phonetic errs those verifiers were not helpful as they only helped in typo mistakes. Thus the early developers faced immense challenges and difficulties in offering useful suggestions f misspelled wds. Thus f the understanding and implementation of an advance spell checking algithm various techniques and algithms have been developed f various languages of the wld. Sindhi language spoken in Pakistan and some parts of India is one of the oldest language of the wld, using Persio-Arabic and Devnagri script. Due to its script and ligature complexity, it s very essential to understand the ligature structure and analyze the various err patterns trends in Sindhi language befe any attempt on developing a spell checker could be made. 2. THE WORD ERROR There are two distinct categies in which the wd Err can be placed, first being the Non-Wd Err and the second categy is the Real-Wd Err. Consider a candidate string- which is collection of characters and usually separated by hard space punctuation marks, if this string wd has a meaning then it would be called as a candidate string otherwise this string is referred to as a Non-Wd Err [2]. Real wd errs are considered to be such type of errs that occur when the typist writer types a valid wd which is spelled crectly but it is not the actually wd intended to be written with respect to context. Therefe the sentence would be considered increct syntactically semantically [3]. 3. CLASSIFICATIONS OF ERRORS In the literature, there are several theies on various classifications of errs and each they does not exclude another. Several of these err classification theies have been discussed by Kukich [4]. In the next subsection (3.1), I review the various different classification theies from several different studies. In section 3.2, the classifications of errs from these studies are discussed. 3.1 Typing Errs And Spelling Errs The basic type of classification of errs is between typing errs (commonly known as typos ) and spelling errs. Typing errs occurs when the typist the auth knows the actual and crect spelling of the wd but mistakenly by slip of finger presses an invalid key. It has been observed that since the writer is aware of the actually spelling of the wds so the possibilities of multiple mistakes within the wd are very low and the resulting misspelled wd will usually resemble thographically to the intended wd. (e.g. {cupreme court} f {Supreme Court (noun): supreme kourt}). Whereas the term spelling errs refers to the errs that occur due to the ignance of the auth typist regardless of the fact that the actual spelling of the intended wd is known not. Usually the terms spelling errs and typing errs are considered to be same and referred to simply as a spelling errs, this sometimes also creates confusion among the users. It has been observed that there are three general possibilities of spelling errs: i. Phonetic err are errs due to the similar pronounced characters in a language where the increct misspelled wd is phonetically ماڻهوماڻ حو similar to the intended wd (e.g. rof پاڪستان f ناتصڪاپ {human: mardhon} {Pakistan (noun): Pakistan}) and so the phonetically increct wd is sometimes also referred to as a homophone of the actual intended wd. ii. The other reason f the spelling mistake is that the increct misspelled wd is semantically حيحص (e.g. similar to the actual intended wd وحالص f وتحالص {crect: sahi} يحص f ). يحالص {guider : salahoo} iii. The last reason behind the spelling errs is that the writer the typist is ignant unaware of the basic grammatical rules of a language while 1435

2 typing the text. Accding to researchers and linguistics the grammatical errs do not fall under the categy of spelling errs but accding to them such type of errs should be considered under a separate categy and so usually most of the general spelling err detection tools and software do not consider grammatical errs and igne them. 3.2 Single Err Wds And The Multiple Err Wds From the literature review of kukich [4] the second categy of errs discussed is the distinction between the wds that contain multiple errs against just a single character err, hence the terms single err wds and multiple err wds. Kukick in his studies describes that single err mistakes usually occur due to one of the following four reasons [4]: i. A single character inserted at invalid position in a string. ii. A single character deleted fm an intended wd. iii. A character substituted by another increct undesired character. iv. Character transposition among two letters in a wd. These are also sometimes collectively referred to as single transfmations and are also known as Damerau transfmations [5]. Wds that include me than one of the above mentioned err types are simply known as multiple err wds. character letter can be inserted befe the first initial character حج) rof اظت ح فاظت { noitcetp :ف hifazat}). An nth err of a character in a wd occurs when there is an err in certain characters of wds other than the first initial character. In nth err type first character must be crect. 3.5 Errs Within One Wd And The Wd Boundary Errs In this type of err classifications, the distinction between errs within a Sindhi wd and errs at the boundary of multiple wds, are discussed. The errs due to the Wd boundary are such type of errs that occur due to the result of increct misplaced spacing of wds. There are generally two types reasons behind the wd boundary err: i. Increct splits, when the space character is inserted at an invalid position of a wd hence splitting a single wd into two, e.g. f {Jamsho (noun): jamsho} ii. Run-ons, when the space character is omitted not inserted between two separate wds hence resulting in the joining of two wds as a single wd e.g. f {University(noun) : uoniversity}. 15% of all the errs discovered by Kukich [9] in his study were actually wd boundary errs and similarly Mitton [10] in his research also found that 13% of the errs generated were wd boundary errs. 3.3 Errs In Long Wds And The Errs In Sht Wds The next categy of err classification is the distinction between the errs in sht wds and the errs that are found in long wds a valid string that consists of large number of characters. It has been analyzed from the studies of Pollock and Zama [6] that in sht small wds - with three four characters, there is an extensive variance in the amount of errs that occur that constitutes 9.2% of all errs. Accding to Yannakoudakis and Fawthrop [7] the frequency of errs in sht wds is 1.5%, however Kukich [8] in his studies disagrees with previous two analysis and emphasizes that the frequency of errs is 63% in small wds of two, three and four characters. 3.4 First Character Errs And The Nth Character Errs The next type of err classification discussed in the literature is the distinction between the err that occurs in the first initial character of the wd and the err that occurs in rest of the wd in all other characters of the wd except the first. The first the initial character of the ف اظت rof ح فاظت ( deleted intended wd can be تظافح f تظافھ) {protection: hifazat}, substituted {protection: hifazat}), transposed with the second character : protectionح hifazat}) some other فاظت } ف حاظت!) 3.6 The Real-Wd Errs And Non-Wd Errs In the sixth and the last classification of errs a difference is established between the real-wd errs and non-wd errs. The Non-wd errs are such type of errs that are completely invalid and increct wds not to be found in any dictionary in any other cpus, (e.g. ھ فاظت rof ح فاظت noitcetp} eht saerehw.({tazafih : Real-wd errs actually yields a string of characters which is a valid and crect wd itself found within a dictionary with some meaning (e.g. حيحص f w rekcehc lleps eht ecneh ({ihas : صحي{ tcerroc ill not be able to identify such type of errs. Peterson [11] is amongst the few who had investigated and produced several theies on the frequency of real-wd errs. Peterson [11] in his research wk studied and analyzed the relationship between the lexicon size and real-wd err rate, considering only single err misspellings [5]. 4. SPELLING ERROR TRENDS IN SINDHI The common technique of developing the spell crecting algithm is by initially studying and analyzing he spelling err trends (also known as err patterns) in a 1436

3 language. Accding to Damerau [5] and Peterson [11] the spelling errs in language are divided into two main categies, typographic errs and cognitive errs. Through research and analysis I have ascertained that in Sindhi Language Four types of Errs usually occur. Typographic Errs Cognitive Phonetic Errs Visual Errs Space Related Errs Single-Err is therefe refereed to any type of errs produced from above editing operations [4]. The proclamation made by the Damerau s was later confirmed by a number of researchers and scholar including Peterson [11]. Table 1 shows the results of a study by Peterson [11]. The source of data and infmation were taken actually from the Webster s Dictionary and Government Printing Office (GPO) documents, which were in-fact retyped by college students [1]. Table 1: Statistics of the Four Basic Types of Errs (f language) Err Type GPO Web7 Transposition 4 (2.6%) 47 (13.1%) Insertion 29 (18.7%) 73 (20.3%) Deletion 49 (31.6%) 124 (34.4%) Substitution 62 (40.0%) 97 (26.9%) Total 144 (92.9%) 4.7%) 5. TYPOGRAPHIC ERRORS Typographic Errs occur when a wd is mistyped by mistake usually because a finger was placed in the wrong position, whereas the actual and crect spelling of the intended wd is known. These type of errs are usually and most often related and dependent on the keyboard, along with the typing knowledge and accuracy of the typist and so such type of errs do not fall under the criteria of any linguistics rule [12][13]. The err patterns here are somewhat dependent on the language and keyboard layout. Damerau [5] in his study emphasized that approximately 80% of the typographic errs fall into one of the following four categies Insertion Err: While writing a wd if a character is pressed twice some other character is pressed along with the iginal character, the undesired character is considered to be inserted e.g. سنهڌيجح فاظت rof ح فاظت noitcetp} ro { tazafih : f سنڌي {Sindhi(noun) : sindhi}. Deletion Err: While writing a wd if any me character is missing then it is called to be deleted and the err is considered to be a deletion err Omission err e.g. تظاف f پاڪستان f پاڪتانح فاظت{ noitcetp ro { tazafih : {Pakistan (noun): Pakistan}. Substitution Err: When the desired character in a wd got replaced by some other character that is the actual character is substituted. e.g. پارلعامينٽ f parliament }.پارليامينٽ (noun) : parliament} Transposition Err: As the name suggest, transposition errs occur when characters have "transposed" - that is, they have switched places. Transposition errs are almost always human in igin. The most common way f characters to be transposed is when a user is touch typing at a speed that makes them input one character, befe the other. This may be caused by their brain being one تظافح f تظاحف e.g. step ahead of their body. {protection : hifazat } قئاداعظم f - Quaid }.قائداعظe-Azam (noun) : quaideazam} Err Type Insertion Omission Substitution Table 2: Various Typographic Err Types Transposition of two adjacent letters Wrong Wd Wd سنڌي Sindhi سنهڌي پاڪستان Pakistan پاڪتان پارليامينٽ Parliament پارلعامينٽ قائداعظم Quaideazam قئاداعظم 6. COGNITIVE OR PHONETIC ERRORS Cognitive Errs Phonetic errs usually occur due to similarly pronounced characters homophone characters in Sindhi sometimes also referred to as Hearing Errs. Generally the writer is unaware does not know the actual spelling of the wd and the writer tries to write it with probable spelling guessing it with similar pronunciation of the letters that sounds somewhat similar to the actual wd. The similar sounding homophone letters are often substituted in this type of err (e.g. ھ {hah} f ڪيرات e.g. {toye}. ط {tey} f ت {hey} ح {darkness: tareeik} replaced by. ڪيراط Table 3 shows the list of all the Sindhi letters groped accding to their similar sound of pronunciation Table 3: Similarly pronounced letters in Sindhi Sound trgui tmira C uiuos i limis code ا آ ء ي ئ 1 پ 2 ڀ ب ٻ 3 ت ط 4 ٿ 5 ٽ ٺ

4 ث س ص 7 ج ڇ چ 8 ڃ ڄ 9 ح ھ ہ 10 خ ک 11 د ڌ ڏ ڍ 12 ذ ز ض ظ 13 ڙ ڻ ر 14 ش 15 ف ڦ 16 ق ڪ 71 گ غ 71 ل 71 م ن 00 ڱ ڳ Some of the err that usually occurs due to the similar sound of Sindhi letters is shown in the table 4 below. Here the writer tries to guess the crect wd by inserting a letter which sounds appropriate f the wd. The crect wd is shown in the left column and their misspelled wd is shown in the opposite column. reason could be that the writer is unaware of the crect spelling of the wd and tries to guess the wd based on the similar shape of characters. Table 5 shows few examples of such type of Visual errs in Sindhi language. Table 5: Types of Err that occur Wrong Wd ڌندو Dhandho دندو دنڌو ڄڃ Jyunj جڃ ڃڄ Wd ٻڪري Bakri بڪري ٻڪزي اختيارات Akhtyrait اخٺيارات احتيازات The various shape groups f Sindhi are given in Table 6. Some of the characters in Sindhi language do not share their shapes with any other character and so are so placed alone in the table, there are ten such characters (i.e.:. Other then these all other )ي,ء,ھ,و,م,ل,گه,ڪ,ق, جه, ا characters in Sindhi alphabet have similar shape characters as shown in Table 6 below. Table 6: Basic Shape groups in Sindhi Table 4: Various Examples of cognitive errs Wrong Wd مفروزو مفروذو Mafroozo wd 1 مفروضو حڪومط ھڪومت قرارداڌ ڪرارداد موجودگهي موجهودگي 2 حڪومت Hukomat 3 قرارداد Qrrardaad 4 موجودگي Moujeedgy 7. VISUAL ERRORS Visual errs are the errs that occur due to the similar shapes of various Sindhi characters. They could also be termed as Reading Errs similar shape errs. Accding to Tahira Naseem Similar shape based errs are not cognitive in nature [1]. There could be several reasons f this type of errs. First, the professional typists, when typing fm a hand written draft of a document will usually type in the text exactly as it looks in his first glaze without giving much consideration to its meaning wd structure and thus characters with similar visual representation are confused with each other rendering in spelling mistakes. Second, when a novice typist writer of Sindhi language writes a document he may know the crect spelling of the wd but could confuse the letters due to shape similarity f example ڃ {jey} could be mistyped f ڄ {chey} ٽ ymi r f ث.{Tey}. Third S.No: Base No: of Shape Group Shape Elements 1 ا ا 1 2 ٮ, ٿ, ٽ, ٺ, ث 9 ب, ٻ, پ, ڀ, ت 3 ح, ڄ, ڃ, چ, ڇ, ح, خ 7 ج 4 حه جه 1 5 د د, ڌ, ڏ, ڊ, ڍ, ذ 6 6 ر 3 ر, ڙ,ز 7 س س, ش 2 ض ص 8 ص, 9 ط 2 ط, ظ 10 ع 2 ع, غ 11 ڡ 2 ف, ڦ 12 ٯ ق 1 13 ڪ ڪ 1 14 ک ک, گ, ڳ, ڱ 4 15 گه گه 1 16 ل ل 1 17 م م 1 18 ں ن, ڻ 2 19 و و 1 20 ه ھ 1 21 ء ء

5 22 ى ي 1 Total SPACE RELATED ERRORS Space is one of the most commonly and habitually used character in document drafting and composition. Due to its immense use the involuntary insertion occurs during typing. In Sindhi the space insertion is not as common as space omission which is much me frequent. One reason behind this could be that we usually want to type quickly and want to minimize the effts hence omitting the space character ( any other character ) intentionally out of ignance but at the same time we insert a space character either mistakenly of necessity. This second reason of omitting a space is found to be most common specially when we intent to write complex and long wds as Sindhi language has several wds that use spaces between two wds to make the sentence clear while reading. F example consider the wd شهباز Lal } لعل Shabaz (noun): lal shahbaaz}, if f instance the space character is omitted deleted from the middle of the wds then the wd will become لعلشهباز {LalShabaz(noun): lalshaabaz} which is increct and misspelled. Table 7 shows some of the space related errs in Sindhi Language. Table 7: Shows the various space related errs Err Wrong wd پاڻي Pardein پا ڻي Insertion wd لعل شهباز Lal Shabaz لعلشهباز Omission زندگي Zindegi زن دگي Transposition 9. CONCLUSION From the discussion and studies given in this paper it can be summarized that in Sindhi, few unique script specific errs trends exists which cannot be found in a language like. Amongst the various err trends discussed f Sindhi language the one found to be most frequent is substitution errs caused due to the shape similarity of the letters in Sindhi alphabet and also due to the similar pronunciation of various letters. The other type of err found in Sindhi language are the omission deletion of space character at the wd boundaries. From the studied presented f Sindhi it can be assumed that these results and err trends and patters will also apply to other languages that are similar to Persio-Arabic script. REFERENCES [1] NASEEM, T., HUSSAIN, S. (2007). Spelling Err Trends in Urdu. In the Proceedings of Conference on Language Technology (CLT07), University of Peshawar, Pakistan. [2] B.B. CHAUDHURI, (2001) Reversed wd dictionary and phonetically similar wd grouping based spellchecker to Bangla text, Proc. LESAL Wkshop, Mumbai,. [3] A. BHATT, M. CHOUDHURY, S. SARKAR AND A. BASU, Expling the Limits of Spellcheckers: A comparative Study in Bengali and, Proc. The Second Symposium on Indian Mphology, Phonology and Language Engineering (SIMPLE'05). Published by CIIL Myse, Kharagpur, INDIA, February 2005, pp [4] KUKICH, K. (1992B), Techniques f automatically crecting wds in text, in.acm Computing Surveys. Vol: 24 Issue 4, pp : [5] DAMERAU, F.J. A Technique f Computer Detection and ion of Spelling Errs. In Communications of ACM, Vol. 7, No. 3, pp , March, [6] POLLOCK, J. J. AND ZAMORA, A. (1983), Collection and characterization of spelling errs in scientific and scholarly text, in.journal of American Society of Infmatics and Science., 34(1): [7] YANNAKOUDAKIS, E. J. AND FAWTHROP, D. (1983), the rules of spelling errs, in.infmation processing and management. 19(12): [8] KUKICH, K. (1990), A comparison of some novel and traditional lexical distance metrics f spelling crection, in.proceedings of INNC-90-Paris., [9] KUKICH, K. (1992a), Spelling crection f the telecommunications netwk f the deaf, in.communications of the ACM., 35(5): [10] MITTON, R. (1987), Spelling Checkers, Spelling s, and the Misspellings of Po Spellers, in Infmation Processing & Management., 23(5): [11] PETERSON, J. L. (1986), A note on undetected typing errs, in.communications of the ACM., 29(7): [12] ETHNOLOGUE.COM. Sindhi: A language of Pakistan. Retrieved from e=snd (Accessed on: 26th April, 2011). [13] Sindhi Language Authity Sindhi Script. Website: (Accessed: November 2011). 1439