TrinityHall, Cambridge, England.

Transcription

1 usingrecurrentneuralnetworks HandwritingRecognition O-lineCursive AndrewWilliamSenior A TrinityHall, Cambridge, England. Thisthesisissubmittedforconsideration forthedegreeofdoctorofphilosophy attheuniversityofcambridge. September1994

2 Computerhandwritingrecognitionoersanewwayofimprovingthehumancomputerinterfaceandofenablingcomputerstoreadandprocessthemany handwrittendocumentsthatmustcurrentlybeprocessedmanually.this thesisdescribesthedesignofasystemthatcantranscribehandwrittendocuments. Summary forthenormalizationandrepresentationofhandwrittenwordsaredescribed, scannedfromahandwrittenpageandproducesword-leveloutput.methods nitionispresented,followedbyadescriptionofrelevantpsychologicalre- writingrecognitionarethendescribed.acompletesystemforautomatic, o-linerecognitionofhandwritingisthendetailed,whichtakeswordimages search.previousresearchers'approachestotheproblemsofo-linehand- First,areviewoftheaimsandapplicationsofcomputerhandwritingrecog- includinganoveltechniquefordetectingstroke-likefeatures.threeprobwritingrecognitioninvestigated.themethodofcombiningtheprobability estimatestochoosethemostlikelywordisdescribed,andperformanceimprovementsaremadebymodellingthelengthsoflettersandthefrequency ofwordsinthecorpus.thesystemistestedonadatabaseoftranscriptsfrom abilityestimationtechniquesaredescribed,andtheirapplicationtohand- acorpusofmodernenglishandrecognitionresultsareshown.recognition isdescribedbothwiththesearchconstrainedtoaxedvocabularyandwith anunlimitedvocabulary. beforeassessingwherefutureworkismostlikelytobringaboutimprovements. Keywords O-linecursivescript,handwritingrecognition,OCR,recurrentneuralnet- Thenalchaptersummarizesthesystemandhighlightstheadvancesmade works,forward-backwardalgorithm,hiddenmarkovmodels,durationmod- elling. O-linehandwritingrecognition 1

3 ThisthesisdescribesresearchcarriedoutatCambridgeUniversityEngineeringDepartmentbetweenOctober1991andSeptember1994.Itistheresult ofmyownworkandcontainsnoworkdoneincollaboration.thelengthof thisthesis,includingreferencesandgurecaptions,isthirty-seventhousand words. Acknowledgements Firstofall,IwouldliketoexpressmygratitudetothelateProfessorFrank Robinsonwhohassupervisedmeadmirablyforthelatterhalfofthisthesis vidingtheoriginalinspirationforthiswork.iamalsoindebtedtodrtony withenthusiasticguidanceandsupport,particularlyinthelastfewweeks. Fallside,forsupervisingmeduringthersthalfofthisthesisandforpro- groupwhichfrankfallsidecreated.thegrouphasbeenanidealenvironment,bothsociallyandtechnically,inwhichtoconductresearch.thosein thegroupwhohavehelpedinthecreationofthisthesisaretoonumerousto mentionindividually. IwouldliketothankeveryoneelseintheSpeech,VisionandRobotics Declaration whosefriendshiphasbeeninvaluableinthelastthreeyears. Fuzzywhoproof-readatsuchshortnotice,andparticularlytoTimJervis forprovidingthenancialsupportnecessaryformetocarryoutthiswork. AT&THolmdelforrecentfruitfuldiscussionsthathavehelpedshapethewritingofthisthesisoryofmyfather.MyparentshavealwayssupportedmeandtothemIowe everything. Finally,Iwouldliketodedicatethisthesistomymotherandtothemem- IwouldalsoliketothankeveryoneImetatLexicus,IBMHawthorneand TheformerScienceandEngineeringResearchCouncilistobethanked SpecialthanksmustgotoChenThamandAndyPiperforfriendship;to O-linehandwritingrecognition 2

4 Contents 1Introduction 2Handwritingrecognition 1.1Thisthesis::::::::::::::::::::::::::::::::8 1.2Originalcontribution::::::::::::::::::::::::::9 1.3Notation:::::::::::::::::::::::::::::::::9 2.1Ataxonomyofhandwritingrecognitionproblems::::::::: Applications::::::::::::::::::::::::::::::: On-lineversuso-line::::::::::::::::::::: Authoridenticationversuscontentdetermination:::: Writerindependence:::::::::::::::::::::: Vocabularysize::::::::::::::::::::::::: Isolatedcharacters::::::::::::::::::::::: Opticalcharacterrecognition::::::::::::::::: Cheques::::::::::::::::::::::::::::: Frompostcodestoaddresses:::::::::::::::::16 3Psychologyofreading 2.3Existingo-linehandwritingrecognitionsystems::::::::: Formprocessing::::::::::::::::::::::::: Otherapplications::::::::::::::::::::::: Isolatedcharactersordigits:::::::::::::::::: O-linecursivescript:::::::::::::::::::::19 4Overviewofthesystem 3.1Readingbyfeatures:::::::::::::::::::::::::::23 3.2Readingbylettersandreadingbywords::::::::::::::25 3.3Lexiconandcontext:::::::::::::::::::::::::::26 3.4Summary::::::::::::::::::::::::::::::::: Summaryofparts::::::::::::::::::::::::::::29 4.2Imageacquisitionandcorpuschoice:::::::::::::::::30 4.3Anoteonresults::::::::::::::::::::::::::::32 4.4Theremainingchapters::::::::::::::::::::::::33 O-linehandwritingrecognition 3

5 5Normalizationandrepresentation 5.1Normalization::::::::::::::::::::::::::::::34 CONTENTS 5.2Parametrization::::::::::::::::::::::::::::: Baselineestimationandslopecorrection::::::::: Slantcorrection::::::::::::::::::::::::: Smoothingandthinning:::::::::::::::::::: Skeletoncoding:::::::::::::::::::::::::40 6Findinglarge-scalefeatureswithsnakes 5.3Findinghandwritingfeatures:::::::::::::::::::::45 5.4Summary:::::::::::::::::::::::::::::::::46 6.1Findingstrokes::::::::::::::::::::::::::::: Non-uniformquantization::::::::::::::::::43 6.2Snakes:::::::::::::::::::::::::::::::::: Analternativeapproach::::::::::::::::::::44 6.3Pointdistributionmodelsandconstraints:::::::::::::50 6.4Trainingfeaturemodels::::::::::::::::::::::::52 6.5Findingfeaturematches::::::::::::::::::::::::53 6.6Discussion:::::::::::::::::::::::::::::::: Recognitionmethods 7.1Recurrentnetworks:::::::::::::::::::::::::::58 7.2Time-delayneuralnetworks::::::::::::::::::::: Training::::::::::::::::::::::::::::: Networktargets::::::::::::::::::::::::: Generalization:::::::::::::::::::::::::: Understandingthenetwork:::::::::::::::::: Discreteprobabilityestimation::::::::::::::::::::74 8HiddenMarkovmodelling 7.4Summary::::::::::::::::::::::::::::::::: Asimplesystem:::::::::::::::::::::::::75 8.1AbasichiddenMarkovmodel:::::::::::::::::::: Vectorquantization::::::::::::::::::::::: Training::::::::::::::::::::::::::::: Discussion::::::::::::::::::::::::::::78 8.2Durationmodelling::::::::::::::::::::::::::: Labelling::::::::::::::::::::::::::::: Decoding::::::::::::::::::::::::::::: Enforcingaminimumduration:::::::::::::::: Parametricdistributions::::::::::::::::::::88 80 O-linehandwritingrecognition 8.3Targetre-estimation::::::::::::::::::::::::::90 8.4Languagemodelling:::::::::::::::::::::::::: Results:::::::::::::::::::::::::::::: Forward-backwardretraining:::::::::::::::::934

6 8.4.1Vocabularychoice:::::::::::::::::::::::: Grammars:::::::::::::::::::::::::::: Experimentalconditions:::::::::::::::::::: Coverage:::::::::::::::::::::::::::::102 CONTENTS 9Conclusions 8.5Rejection::::::::::::::::::::::::::::::::: Out-of-vocabularywordrecognition::::::::::::::::: Summary::::::::::::::::::::::::::::::::: Searchissues::::::::::::::::::::::::::103 Bibliography 9.1Furtherwork::::::::::::::::::::::::::::::: O-linehandwritingrecognition 5

7 Chapter1 Introduction Theworldisllingwithcomputers.Whetherwelikeitornot,theyare becomingubiquitous.asevermorepeopleareforcedintocontactwithcomputersandourdependenceuponthemcontinuestoincrease,itisessential thattheybecomeeasiertouse.asmoreoftheworld'sinformationprocessingisdoneelectronically,itbecomesmoreimportanttomakethetransfer Bythisartyoumaycontemplatethevariationofthe23letters. ofinformationbetweenpeopleandmachinessimpleandreliable. RobertBurton.TheAnatomyofMelancholy. andtoactinhumansocietyinalessconstrainedmannerthanhaspreviously computerindustrytomakecomputersincreasingly`userfriendly'.inthis uraltopeople.thuscomputersshouldbebetterabletointeractwithpeople beenpossible.theseaimsarereectedinthemoremodestattemptbythe intelligence,issimplytoenablecomputerstoaccomplishtaskswhicharenat- forthetimebeingthelonger-termgoalsofanalysingandemulatinghuman Oneoftheaspirationsoftheeldofarticialintelligence,ifoneignores vein,computershavecomeoutoflaboratoriesandintohomesandoces; wecommunicatewiththemusingmiceandkeyboardsratherthanpunched cardsandtoggleswitches.handwritingisanaturalmeansofcommunication whichnearlyeveryonelearnsatanearlyage.1thusitwouldprovideaneasy wayofinteractingwithacomputer,requiringnospecialtrainingtouseeectively.acomputerabletoreadhandwritingwouldbeabletoprocessahost ofdatawhichatthemomentisnotaccessibletocomputermanipulation. intothecomputerrecognitionofhandwriting.onereasonadvancedisthat theoptimismaboutthecapabilitiesofimminentspeechrecognitionmachines madepeoplefeelthatotherapproacheswereunnecessary.whilesomeof thepromisesofspeechrecognitionbymachinehavealreadybeenfullled, andresearchersarestilloptimistic,someofthebenetshavebeenslowto materializeandpeoplehavethoughtagainaboutwhatisrequiredofhuman- Afterthisargument,itseemssurprisinghowlittleresearchtherehasbeen O-linehandwritingrecognition peoplecomingintocontactwithcomputers,theguremustbehigher. computerinterfaces.thoughspeechisaveryconvenientformofcommu- 1DowningandLeong(1982:p.299)quoteanestimatedworldliteracyrateof71%.Inthose 6

8 wheresilenceisimportant,orwherealargenumberofpeoplemustwork withcomputers,itisclearthatvoiceinputisnotthebestsolution.though nication,itisnotalwaysthemostpractical.innoisyenvironments,those computerprofessionalsandsecretarieswouldbelothtogiveuptheconvenienceandspeedofakeyboard,forthosenotfamiliarwithkeyboards,and forportableoroccasionaluse,handwritingentryisclearlyofpracticalvalue. ofhandwrittendocumentsalreadyincirculation.fromchequesandletters totaxreturnsandmarketresearchsurveys,handwritingrecognitionhasa Thishasleadtothegrowthinthelastyearortwoof`pencomputing' the useofcomputerswhichallowinputfromanelectronicstylus(geake1992). hugepotentialtoimproveeciencyandtoobviatetedioustranscription.as theeconomistrecentlysuggested,\today'sbiggestprizeincomputervision, howeveristextandhandwriting..."(browning1992). Inadditiontoapotentialmodeofdirectcommunicationwithcomputers, CHAPTER1.INTRODUCTION handwritingrecognitionisessentialtoautomatetheprocessingofamyriad 1.1Thisthesis Thisthesisinvestigatestheuseofhandwritingrecognitionasamediumof contentsofthethesis.thenextchaptersummarizestheaimsandachievementsofotherworkintheeldofhandwritingrecognitionandestablishes handwrittendocuments.laterchapterspresentresearchcarriedouttode- communicationbetweenpeopleandcomputers.afterpresentingageneral overviewofhandwritingrecognition,itfocusesontheproblemofreading ataxonomyoftheeldintowhichtheoriginalworkofthisthesiscanbetted.applicationsforhandwritingrecognitionarealsoexamined.chaptervelopacomputersystemwhichtacklesthisproblem.thesystemhasbeen describedinearlierpapers(seniorandfallside1993a;senior1993). studiesworkinthepsychologyofreading,todiscoverknowledgewhichcan beputtouseinthedesignofamachinehandwritingrecognitionsystem. Thethesisisdividedinto9chapters.Thischapterdescribestheaimand ofindividualpartsofthatsystem,includingnormalizationandrepresentation(senior1994);feature-nding(seniorandfallside1993b);probabilitsentedandadiscussionoftheirvalidity. thathasbeendesigned,andthefollowingchaptersdescribetheworkings Chapter4presentsanoverviewofthehandwritingrecognitionsystem estimationandlanguagemodelling.eachofthesechaptersincludesdetails ofexperimentscarriedouttoassesstheperformanceofthetechniquespre- thehandwritingsystemandsummarizeswhathasbeenachievedinthisprogrammeofresearch.furtherworkwhichcouldbecarriedonfromthisthesis isalsosuggested. Thenalchapterdrawstogethertheconclusionsofthechaptersabout O-linehandwritingrecognition 7

9 Thisthesisdescribesanew,completeo-linehandwritingrecognitionsystem.Themajororiginalcontributionsdescribedinthisthesisareasfollows: CHAPTER1.INTRODUCTION 1.2Originalcontribution Thesystemappliesanovelapproach,usingrecurrentneuralnetworks Thepsychologyofreadingliteratureisreviewed,showinghowthestudy Thetrainingofarecurrentneuralnetworkwiththeforward-backward forprobabilityestimation.whiletherecurrentneuralnetworkhaspreviouslybeenusedforspeechrecognition,ithasnotbeforebeenapplied ofhumanreadingandwritinggivesanindicationofthecharacteristics totherecognitionofhandwriting. algorithmisdescribedhereforthersttime. Themethodsusedheretonormalizehandwrittenwordsareanoriginalsynthesisofnewandestablishedtechniques.Previouslypublished methodsarecomparedandimprovedupon. whichmightproveusefulinareadingmachine. Wordsareencodedinanoriginalmannerwhichisshowntobebetterthanthecommonbit-maprepresentation,andanovelmethodof featuredetection,basedupontheuseofsnakesisdescribed. Chapter8investigatestheuseofdurationmodellingforo-linehandwritingrecognitionandinvestigatestheproblemsofout-of-vocabulary Throughoutthisthesis,thedistinctionismadebetweenahandwrittenword, 1.3Notation andtheideaofthatword.tomakethisdistinction,thefollowingtypo- wordswithlexicaoflimitedsize. graphicalconventionisemployed.torepresentahandwrittenwordorlet- ter,thefollowingfontisused:`abc fghijklmnopqr uvwxyz';andtode- notethelettersorwordsasconcepts(mcgrawetal.1994),thisfontisused: `abcdefghijklmnopqrstuvwxyz'.thepurposeofthesystemdescribedhereis totranscribe`word'into`words'.whentheinternalrepresentationofthe systemisreferredto(section5.2),asingleframeofdataisshownthus:xt; andthedatarepresentingawholewordareshownasx0.thesetoflettersasconceptsisdenotedandanarbitraryindividualletterisshowni. i P(xtji)orP(xt0ji)respectively;theprobabilitythatframetrepresents ThediscreteprobabilitiesusedthroughoutaredenotedP.Theseincludethe probabilityofoneorseveralframesofdatagiventhatframetispartofletter letterigiventhedataoftheframe P(ijxt);andtheprobabilityofthejth elementofaframext,giventhatthatframerepresentsletteri P((xt)jji). O-linehandwritingrecognition 8

10 Chapter2 Handwritingrecognition Ascomputerpowerhasincreasedovertheyears,andtheirrangeofapplicabilityhassimilarlyincreased,oneofthemajorgoalsofresearchintocomputershasbeentomakecomputerseasiertocommunicatewithandthusto :::avastpopulationabletoreadbutunabletodistinguishwhatis maketheirbenetsavailabletoamuchgreaternumberofpeople.oneof worthreading. themajorobstaclestotheintegrationofcomputersasuniversalinformation G.M.Trevelyan.EnglishSocialHistory. annotationstoprinteddocuments,maybehandwritten;inmanysituations itwouldbehighlydesirabletoprocessthecontentsofthesedocumentsby machine,forwhichhandwritingrecognitionisessential. onpaper.particularlywhendealingwiththegeneralpublic,ahugeamount ofocepaperworkishandwritten.lettersandfaxes,aswellasformsor processingsystemsisthefactthatmostusefulbusinessdataisstillstored havebeendeveloped,andmuchworkhasbeencarriedoutintocomputer municationbetweencomputersandawiderclassofusersinagreatervariety ofcircumstances.whileideassuchasthemouseandtouch-sensitivescreens speechrecognition,thereisstillmuchscopeformakingtheinterfacemore naturalforuserswhoarenotfamiliarwithcomputers.handwritingranks veryhighlyasawayofcommunicatinglinguisticinformationinawaywhich Similarly,computeruserinterfacesneedtobeimprovedtoenablecom- isnaturaltoverymanypeople.thoughspeechrecognitionhasbeenclaimed asthepanaceaforuser-interfaceproblems,ithasbeenslowtoachieveits promise,particularlyinnoisyenvironments,andthelimitationsofspeech morepopular.notonlyaremoreresearcherstryingtotackletheproblems recognitionhavebecomeclearerasresearchhasadvanced. havebecomeavailablewithhandwritingrecognitionsoftwareforisolated charactersandmorerecentlyforcursivescript.handwritingrecognitionsystemshavealreadystartedtobeusedforreadingzipcodesonenvelopesanableandareactuallybeingsoldasusefulproducts.oflate,pencomputers Inthelastfewyearstheeldofhandwritingrecognitionhasbecomemuch thatitpresents,butsolutionstotheseproblemsareslowlybecomingavail- O-linehandwritingrecognition 9

11 itisworthpresentingheretheeldofautomatichandwritingrecognitionin itsentirety.afterdescribingataxonomyoftheeld,applicationsenvisaged amountsoncheques. Beforedescribinganewhandwritingrecognitionsysteminlaterchapters, CHAPTER2.HANDWRITINGRECOGNITION Havingestablishedtheneedforautomatichandwritingrecognitioningeneral,itisusefultoexaminetheeldmorecloselyandtoidentifyseveral 2.1Ataxonomyofhandwritingrecognitionproblems ispresentedtodemonstratetheapproachestaken. forhandwritingrecognitionsystemsarediscussedandworkbyotherauthors areaswithdierentapplicationsandrequiringdierentapproaches.though methodscouldbedistinguished,handwritingrecognitionsystemsaregenerallypolarizedbetweenthosereceivingtheirdatadirectlyfromsomesortof researchers,eachconcentratingonaspecialareaofhandwritingrecognition On-lineversuso-line Themajordivisionisbetweenon-lineando-linesystems.Whileother manytechniquescanbeshared,theliteraturetendstodivideintogroupsof line,problemwherethetimeorderingofstrokesisavailableaswellaspen up/downinformation;overlappingstrokescaneasilybedistinguishedand matter.intheliteraturedynamicissometimesusedtomeanon-lineand statico-line.sofar,themajorityofsystemshavetackledtheeasier,on- pendeviceattachedtothecomputer,andthosewhichrecognizehandwritingalreadypresentonapieceofpaper ahandwritingequivalentofoptical CharacterRecognition(OCR)whichisalreadywidelyusedforreadingprinted strokepositionsareaccuratelyknown.ontheotherhand,o-linesystems streamofinformation,techniquesfromspeechrecognitionhavebeensuccessfullyappliedtothisproblem,includinghiddenmarkovmodels(bellegardaetal.1994)andtime-delayneuralnetworks(schenkeletal.1994).the datafromthetabletareusually(x;y)coordinatessampledataconstantfre- hasseenmuchinvestmentinon-linesystems,andthedicultyofo-line recognitionhasdeterredresearchuntilrecently. quencyintime,thoughtheyareoftenre-parametrizedtobeequally-spaced, andrepresentedintermsofarc-length,curvature,andangle,withinforma- Sincetheon-linedatafromanelectronicstylusareaone-dimensional overlapandalackoforderinginformation.thegrowthofpencomputing havetocopewiththevagariesofdierentpentypes,widestrokeswhich tionaboutwhetherthepenistouchingthetablet.aparticularproblemof on-linerecognitionishowtohandledelayedstrokes strokeswhichare writtenaftertherestoftheword,asindotting`i'sandcrossing`t's.some authorschoosetomanagewithoutthisextradata;schenkeletal.record itsexistenceasa`hat'featureassociatedwiththestrokesoverwhichthe O-linehandwritingrecognition 10

12 CHAPTER2.HANDWRITINGRECOGNITION RECOGNITIONTEXTON-LINEOFF-LINE HANDWRITING IDENTIFICATIONSIGNATURE delayedstrokesoccur,andbengioetal.(1994a)representthesurrounding Figure2.1:Subdivisionsofmachinehandwritingrecognition(afterPlamondonandLorette(1989)). VERIFICATION asinthemodellingofhandwritingproductionorintheapplicationofprobabilisticrecognizersandgrammaticalconstraintscernedwitho-linehandwritingrecognition,parallelworkfromon-lineresearchisbroughtinthroughoutwhenthereisacommunityofinterests,such ingure2.1anddescribedinthefollowingsection.whilethisthesisiscononomyofbotho-andon-linehandwritinganalysisissimilar;asisshown Althoughapplicationsandtechniquesvaryconsiderably,thegeneraltax- visualcontextofallstrokessothatthedotisseenabovethecuspofthe`i' Authoridenticationversuscontentdetermination Aseconddichotomyintheeld,orthogonaltotheon-line/o-linedivision isaccordingtotheinformationtobeextractedfromthehandwriting.from bothon-lineando-linedata,itmaybenecessarytodeterminetheauthorshipofthewriting,thecontentofwhathasbeenwritten,orboth.inboth approaches.techniquesalsodierdependingonwhethertheauthoristobe recognizedfromasignatureorfromapieceoftext. cases,theeectsofsomevariationsshouldbeignored.todeterminetheauthorship,dierencesinpersonalstyleshouldbehighlighted,tocapturewhat ischaracteristicaboutoneperson'swriting(theiridioscript).conversely,to determinethecontentofthewriting,thevariationsduetoidioscriptshould beeliminatedandignored.thesetworequirementsresultinverydierent poolofknownauthors,forinstanceinawriter-adaptivehandwritingrecognitionsystemwhichusesdierentparametersforwordrecognitionaccording Iftheauthorofapieceoftextorsignaturemustbedetermined,thedis- totheauthor.theformeristhemoreuseful,butofcoursetheharder,prob- tinctionismadebetweenverifyingthattheauthoristheclaimedauthor(for instanceinsecurityorbankingapplications)ormerelydecidingbetweena O-linehandwritingrecognition 11

13 andathoroughreviewofsignaturevericationsystems Writerindependence lem.plamondonandlorette(1989)giveanoverviewofhandwritingsystems, CHAPTER2.HANDWRITINGRECOGNITION continuousspeech.analoguestoeachoftheseexistinhandwritingrecognition,andarediscussedinthisandthefollowingsections. Thewholeeldofhandwritingrecognitionissimilartothealreadywelldevelopedsubjectofautomaticspeechrecognition,whichisoftenclassied alongthelinesofspeakerdependence,vocabularysizeandisolatedwordvs. whichneedonlyrecognizethewritingofasingleauthor.insteadofcreating diculttodeviseasystemtorecognizemanypeoples'handwritingthanone asystemwhichcanrecognizeanybody'shandwriting,theproblemofmultiplewriterscouldbetackledbyasystemwhichisabletoadapttothecurrent (correspondingtospokenaccentsandidiolects).becauseofthis,itismore usedtoteachhandwritingtoanindividualandontheindividual'sidioscript Handwritingstylesareextremelydiverse,dependingbothonthepattern lotofmaterialbythesameauthor,butwouldbeofnousewhenidentifyingthecitynamesonenvelopes.alternatively,manysimilarsubsystems writer.adaptationtothewriter'sstylecouldbeusedwhenrecognizinga couldbecreated,eachrecognizingonestyleofhandwriting(oroneindividual'shandwriting).thenaglobalsystemwouldselectthesubsystemwhich alargelexicon(wherewordsaremorelikelytobesimilartoeachother). Thetaskofrecognizingwordsfromasmalllexiconismucheasierthanfrom 2.1.4Vocabularysize correspondedtoaparticularhandwritingsample. Thus,animportantcriterioninassessingsystemperformanceisthesizeofthe lexiconused.thelexiconwilldependontheapplicationoftherecognition system.forageneraltexttranscriptionsystem,alexiconof60,000words words,orpostaltownsfromenvelopes,thevocabularycanbemuchsmaller. Alternatively,itmaybenecessaryforthesystemtorecognizenon-wordsif (thenumberofreferencesinamedium-sizeddictionary),wouldcoverabout 98%ofoccurrences,andforspecicdomains,suchasreadingchequevaluesin tobeverydicultsinceinnaturalspeechwordsruntogetherwithnosilence foreignwordsornames.thisissueisdiscussedagaininsection Isolatedcharacters Segmentationofcontinuousspeechintoitscomponentwordshasbeenfound theuserislikelytowritewordsnotinthelexicon,suchasabbreviations, O-linehandwritingrecognition tinguishtheboundariesbetweenletters thedierencebetween`ui'and between.forsimplertaskstherecognitionismadeeasierbyforcingthe speakertopausebetweenwords.similarly,incursivescriptitishardtodis- 12

14 `iu'orbetween`v]'and`^'isveryslight.thetaskcanbesimpliedbyforcing thewritertoseparateletters(discretehandwriting),towriteincapitalsorfor thegreatestclarity,towriteclearlyseparatedcapitalsinpre-printedboxes. Whenhighreliabilityisrequired,thelatterconstraintsmaybeunavoidable CHAPTER2.HANDWRITINGRECOGNITION sincetheyarealreadynecessarytoenablehumanreaderstodecipherresponsesonforms.anumberofauthorshaveinvestigatedtheproblemof writeeachwordinaseparatebox,oronaguideline.theseconstraintsare mustbeseparate)orpurecursivescript. mainlytoencourageclaritysincethewordsegmentationproblemprovesless recognizingisolatedcharacters(section2.3.1),particularlyfortheproblem ofreadingpostalcodes.otherauthorshaveresearchedtherecognitionof discretehandwriting(`handprint'wherelower-caselettersarewrittenbut dicultthansegmentationintocharacters,andlessstrictconstraintscould stillensurehighaccuracysegmentationofapageintoitscomponentwords. Otherauthorshavedescribedmethodsofsegmentingpagesintowordsand distinguishingbetweengapsinwordsandgapsbetweenwords(sriharietal. Similarconstraintscanbeplacedoncursivescript,forcingtheauthorto 1993) Opticalcharacterrecognition O-linehandwritingrecognitionhasmuchincommonwithopticalcharacter recognition(ocr) thereadingofprintbycomputer.thisapplicationreceivedmuchattentionduringthe1980sandsuccessfulsolutionshavebeen found,withcommercialpackagesavailableformicrocomputerswhichcan readtypeinavarietyoffontsandinacertainamountofnoise.thehistory (1993).Inmoredicultsituations,thesecommercialpackagesarestillnot satisfactory.authorsdescribeproblemsworkingwithunusualcharactersets andfonts,poorqualitydocumentsordocumentsinspecialformats(bosand vandermoer1993;mcveigh1993).indeed,itisnotclearthatocriseconomicallyviableinagreatmanycaseswhenhighaccuracyisessential(olsen andcurrentstatusofocrarereviewedbymorietal.(1992)andpavlidis formssuchasblurring,mergingandslightpositionalvariations.theprocess ofhandwritingismuchmorevariableinalloftheseprocessesandsuers fromvariationsduetoothereectssuchasco-articulation theinuence onthepage,onlybeingcorruptedbyarelativelysmallamountofnoisein allletters`a'areproducedfromasinglearchetype,andthusareverysimilar recognitionisthegreatvariabilityinhandwriting.fortypeinaxedfont, ThereasonwhythesuccessofOCRhasnotcarriedoverintohandwriting ofoneletteronanother.also,withtype,thesymbolsareusuallydistinct (exceptcertainligatures,as`',whichcanbelearntasaseparatesymbol)so theproblemofsegmentationisnotpresent. suchastemplatematching,areinadequatewhenpresentedwiththegreater O-linehandwritingrecognition AsaconsequenceofthistherelativelysimpletechniquesusedinOCR, 13

15 carriesovertohandwritingrecognition. variabilityinhandwritingsorelativelylittleresearchintheocrliterature CHAPTER2.HANDWRITINGRECOGNITION 2.2Applications befordata-entrytoobviateakeyboardasinpencomputers,butcanalsobe usedforspecialpurposessuchasusingdynamicsignaturestoverifyidentity. Thissectionreviewssomeofthemoreimportantapplicationsthatmaybe envisagedforo-linehandwritingrecognition.on-linerecognitiontendsto developmentcosts.thisconvergencecanbeseeninthemodel-basedapproachesnowbeingused(pettierandcamillerapp1993;doermann1993), foron-linehandwritingrecognition.currently,o-lineperformancelagsbehindthatofon-linerecognitionsystems,butoverthenextfewyears,asthingrecognitionwillconverge,leadingtomoregeneralsystemsandreduced technologyimprovesitislikelythatmethodsforbothtypesofhandwrit- Onepotentialapplicationinthelongtermisinusingo-linetechniques whichinterpreto-linehandwritingasapathofinklaiddownovertime, bylookingatthepsychologyofreading(chapter3) thewaypeopleread treatbotho-lineandon-linewordsasatwo-dimensionalimage,andnotas aone-dimensionalstreamoftrajectorydata.thereasonforthiscanbeseen duction.thedatathatcanbederivedbysuchalgorithmsisverysimilarto thedataavailabletoanon-linerecognizer. ratherthanasanimagetobeanalysedindependentlyofitsmethodofpro- thewriting.sincethisinvolvesignoringthetimeinformation,atrstthis seemstobeapoormethodofanalysingon-linedata.however,theinformationinhandwritingisnottransmittedinthetimingofthepentrajectory.it isbylookingatanimage,notbyanalysingthepenpathusedtoproduce Inthelongertermthough,itwouldseemthattheconvergenceislikelyto linesystems,an`o'writtenclockwisemustberecognizeddierentlyfroman `o'writtenanticlockwise,forinthetimesequenceinformation,theyappear dierent.someonewhowrites` a'maysubsequentlyreturntoextendthe doesnotmatterwhetherthestrokesofawordarewrittenquicklyorslowly, nal`a'stroketomakethewordread` d',butthischangewouldbeloston withchangingspeed,oreveninrandomorder,sinceitistheappearanceof amachinerelyingonthetime-orderingofstrokes.ano-lineapproachignoresthesefactorsandsimplylooksatthenalpositionofthestrokes,just sourceofmis-informationisactuallyavoided.forinstance,incurrenton- thenishedwordthatmatters.thus,bydiscardingthetimesequence,a inginformationisveryusefulwhencreatinganauthorvericationsystem totheproblemofdelayedstrokes(section2.1.1).afterthesearguments,it maybeseenthat,whileon-linerecognitionisbetterthano-linenow,becausethetiminginformationgenerallyisconsistent,agoodo-lineapproach mightultimatelycopewithawidervarietyofvariation.conversely,thetim- asahumanreaderwould.thisapproachalsogivesasatisfactorysolution O-linehandwritingrecognition 14

16 on-linesignaturesaremuchhardertoforgethano-linesignatures,sincethe dynamicsofstrokes(withpenbothupanddown)arehardertoforgethan thenishedappearance. CHAPTER2.HANDWRITINGRECOGNITION 2.2.1Cheques Oneimportantcommercialapplicationforo-linecursivescriptisinthemachinereadingofbankcheques.Whiletheamountinguresiseasiertoread, alsoincludesignatureverication,bringingaboutanincreaseinsecuritywith thepayeecorrespondedtotheaccounttobecredited.suchasystemmight thereductionindrudgeryandtime.giventhenumberofchequespassing itshouldbecheckedthattheamountinwordsisthesame,andthiscanbe throughthebankingsystemeachday,achequereadingsystem,evenifonly asystemthatachievedhighaccuracywithoutalexicon,onecouldcheckthat usedforconrmationwherethenumericalamountisunclear.suchasystem abletocondentlyverifyhalfofthecheques,wouldsavemuchlabouron wouldonlyneedtohaveasmallvocabulary(aboutthirty-vewords).given ofachievinga1in100,000errorratefromthecombinedrecognitionofliteralandnumericalamounts,butpermitting50%ofchequestoberejectedfor manualsorting(lerouxetal.1991). maintained.theprojectsupportedbythefrenchpostocehasthegoal atediousandunpleasantjob.chequeswhichcouldnotbecondentlyveriedbymachinewouldstillbeprocessedmanually,soaccuracywouldbe 2.2.2Frompostcodestoaddresses O-linesystemscapableofrecognizingisolatedhandwrittendigitshavealreadybeencreatedandinstalledinmanypostocesaroundtheworld,as partofautomaticmail-sortingmachines.givenasystemtolocatethepostcodeonanenvelope(wangandsrihari1988;martinsandallinson1991; Palumboetal.1992)thiscanbereadandusedtodirectmailautomatically. ClearlycertaincountriessuchastheUSAareatanadvantageinhavingdigitonlyzip-codesandmanyresearchershavealreadytackledthisproblemwith thepostalcodeclassicationtoberemovedbycomparingcandidatezipcodes withcandidateaddressesinadatabaseofalladdress/zipcodecombinations, givingmorehighcondenceclassications.furthermore,forcountrieswith limitedresolutioninthepostcode,theaddresscanbeusedtoincreasethe reasonablesuccess(section2.3.1). mationcontainedintherestoftheaddress.thisallowstheuncertaintyin resolutionofsorting.u.s.postalserviceprojectsaimtousetheaddressto Toprocessmor automatically,systemsmustbegintousetheinfor- whenonlythevedigitzipcodewasprovided. handwritingrecognition,sinceithasawidevarietyoflevelsofdiculty,from determinean11digitdeliverypointcodewhichspeciesasinglehouseeven O-linehandwritingrecognition Mailsortingcanbeseenasanidealapplicationforwriter-independent 15

17 pletedeterminationofanaddresswithoutapostcode.addressrecognition isolateddigitswrittenatpredeterminedlocationsonanenvelope,uptocom- alsoadmitsofacertainamountoferrorwhileallowingalargerejectionrate. Sincetherewillalwaysbesomeaddressesthatareillegibleorincomprehensibletoamachine,a`don'tknow'answercanbegivenandtheitemsentto CHAPTER2.HANDWRITINGRECOGNITION Anothermajorapplicationwhichisnowreceivingattentionistheautomatic postalserviceisconsideredfallibleandtheconsequentdelaysarealready tolerated Formprocessing abinforhumansorting.further,som isalreadymisrouted,sothe public.foranythingmorethanthemostsimpleinformation,forwhichcheck boxescanbeused,repliesarehandwritteninspacesprovided.muchofthis processingofforms.formsarewidelyusedtocollectdatafromthegeneral informationmustbestoredindatabasesandcanbeprocessedautomatically onceenteredintothecomputer.dataentryiscurrentlythebottle-neckinthe process.severalauthorshavewrittensystemstosegmentthehandwritten datafromthepre-printedformandthentotranscribethehandwrittendata. Insomeapplications,thismaybeisolatedcapitalletterswritteninboxes, butworkisnowmovingontohandprint(breuel1994;garrisetal.1994). Althoughformsmustusuallybehandprintedtokeepthewritingaslegibleas possible,forhumanaswellasmachineprocessing,cursiverecognitionwould stillbeusefulforprocessingthoseformsthathavemistakenlybeenlledout incursivescript Otherapplications recordingthestyleofwriting).documentswouldthenbeeasilysearchable writingrecognitioncaneasilybeenvisaged.alreadymanycompaniesuse agesofdocumentsratherthanthedocumentsthemselves.thisisclearlya verydata-intensivetask,butonewayofreducingthedatastorageistoextracttheinformationandstoretextinascii(orperhapsinaricherformat Avarietyofotherocedocumentprocessingsystemsusingo-linehand- andindexconstructionwouldbemadepossible.furtherpossibilitiesexistin readinghandwrittendocumentsfortheblindorinautomaticreadingoffaxes. electronicdocumentprocessingsystemswhichmanipulatethescannedim- Faxedorderscouldbeprocessedanddispatchedautomaticallyandstandard mostresearch.intheliteraturethereisawiderangeofpapersdescribing enquiriesrepliedtowithouthumanintervention.otherfaxescouldbefed directlyintoanelectronicmailsystem,providingattheveryleastautomatic O-linehandwritingrecognition noticationoffaxarrivalbyreadingthecoversheet,ifnotthefulltextofthe toenglishortotheromanalphabet,thoughthesehaveprobablyattracted document. Ofcourse,theadvantagesofhandwritingrecognitionarenotrestricted 16

18 handwritingrecognitioninamultitudeoflanguages.thebasicproblems ofhandwritingrecognitionarecommontoalllanguages,butthediversity ofscriptsmeansthatverydierentapproachesmaybeused.forexample, JapaneseKanji(MoriandYokosawa1988)andChinese(Luetal.1991)charactersarestronglystroke-based,andcharactersareeasytosegmentfrom CHAPTER2.HANDWRITINGRECOGNITION somehebrewrequireaccuraterecognitionofdiacriticmarks.govindanand Shivaprasad(1990)citemanymorelanguages. todistinguish.arabicandromanalphabetscanbecursive,andarabicand oneanother,butcharactersareverycomplexandtherearemanyclasses Thissectionreviewssomeoftheo-linehandwritingsystemswhichhave beendetailedinprint.todothisitisconvenienttoclassifythem,asdescribedabove,intoisolatedcharacterandcursivescriptsystems.hereonly 2.3Existingo-linehandwritingrecognitionsystems 2.3.1Isolatedcharactersordigits Suenetal.(1980)provideagoodreviewofhandwritingrecognitionupto laterchapterswhenparticularissuesarediscussed. 1980,concentratingonisolatedcharacterrecognition whichhadbeenthe abriefoverviewofthesesystemsisgiven.specicdetailsareprovidedin focusofresearchuntilthen.theydescribeavarietyoffeaturebasedapproachesanddividetheseintoglobalfeatures(templatesortransformations suchasfourier,walshorhadamard);pointdistributions(zoning,moments, n-tuples,characteristiclociandcrossingsanddistances)andgeometricalor lartechniques,andinvolveseparatedetectorsforeachofseveraltypesof topologicalfeatures.thelatterwere,andhaveremained,themostpopu- featuressuchasloops,curves,straightsections,endpoints,anglesandintersections.forinstance,impedovoetal.(1990)usecross-points,end-points andbend-pointsastheirfeatures,codingtheseastotheirlocationinthree horizontalandthreeverticalzoneswithineachcharacter.theencodedcharactersarethenidentiedusingadecisiontreeclassier.ellimanandbanks beingdecodedinaneuralnetwork(afeed-forwardneuralnetworkoran (1991)alsousefeatures(end-point,junction,curveandloop)eachofwhich isassociatedwithanumericalquantity,suchascurvatureorlength,before adaptivefeedbackclassier). phologicalfeaturescreatedbyseparatelyexaminingtheleft,right,topand anormalizedbitmapimageofthecharactertoberecognizedintotheirnetworks(multi-layeredperceptronandneocognitronrespectively).boththese bottomedgesofeachcharacter.theproleofthecharacterfromeachedge iscodedasaseparatefeatureforclassicationbyaneuralnetwork. LeCunetal.(1989)andFukushima(1980)taketheapproachoffeeding NellisandStonham(1991)andHepp(1991)bothusesetsofglobalmor- O-linehandwritingrecognition 17

19 becomemorespecializedandlesslocationspecicdeeperinthenetwork, untiltheoutputsofthenallayercorrespondtocharacters,independentof locationintheimage. networksareconstructedfromlayersofidenticalfeaturedetectors,which CHAPTER2.HANDWRITINGRECOGNITION patternrecognitionmethods(simardetal.1993;hintonetal.1992;boser Impedovoetal.1990;Lanitisetal.1993),particularlysincetheincreasingavailabilityofdatahasmadethisastandardtestproblemfortesting 1994).Isolateddigitclassiershavenowbecomesogoodthatresearchis digitsorcharactersinthelastfewyears(hepp1991;idanandchevalier1991; concentratingonreadingwholezipcodeswherethedigitsareoftentouching Ahostofotherauthorshavetackledtheproblemofrecognizingisolated (FontaineandShastri1992;KimuraandShridhar1991;Matanetal.1992), andndingoptimalcombinationsofmultipleclassiersnowseemsamore promisingwayofreducingerrorratesthanndingbetterclassiers.huang andsuen(1993)citeseveralpaperstakingthisapproach.performanceisnow tionuntilrecently,partlybecauseofthedicultyoftheproblem,butalso beinglimitedbythenumberofdigitswhichareentirelyambiguousandcould notbecondentlyclassiedbyhumanreaders O-linecursivescript Theproblemofo-linecursivescriptrecognitionhasreceivedlittleatten- becauseofthelackofdata.simon(1992)andsuenetal.(1993)givebrief reviewsofscriptrecognizers,butthebestreviewisprobablybylecolinet andbaret(1994).simonmakesthedistinctionbetweenthesegmentation approachandtheglobalapproach,accordingtowhetherwordsareidentied veryfewauthorstakethelatterstrategy.plessisetal.(1993)useaholistic match,butonlytoreducethesizeoftheirlexiconbeforeusingamoredetailedrecognitionmethod.lecolinetandcrettez(1991)usethetermsexplicit byrecognizingindividuallettersorbyrecognizingwordsasawhole.infact, onadierentunitofwriting.bothapproachesusestrongevidencefrom well-writtenpartsofwords,togetherwitharestrictedlexicon,torecognize ually,orifthesegmentationisaby-productofarecognitionprocessworking wordswhicharepartiallybadlywritten. segmentationandimplicitsegmentationaccordingtowhetheranattemptis madetodividethewordintoseparatecharactersandrecognizetheseindivid- tonormalizeandcleanthedata.somepreprocessingmethodsaredescribed approachesusealexicontoconstraintheresponsestoaknownvocabulary. inchapter5.ineachcase,arecognitionstrategythenhypothesizescharacterorwordidentities,andbecauseexactrecognitionisverydicult,allthe Alltheauthorsdescribedbelowincorporatesomeformofpreprocessing O-linehandwritingrecognition cityorstatenamesinaddresses.theseauthorstakeadualapproach,with arst,quickclassicationtoreducethelexiconsize,followedbyamoreac- thatofkimuraetal.(1993b,1993a)whohavecreatedasystemforreading Perhapsthemostsuccessfulo-linehandwritingrecognitionsystemis 18

20 aroughexplicitsegmentationandeachsegmentisclassiedasaletter.the secondstagendsadierentexplicitsegmentationbysplittingthewordinto disjointboxesandjoiningtheboxestogetherusingdynamicprogrammingto curatesecondclassicationusingdierenttechniques.therststagends CHAPTER2.HANDWRITINGRECOGNITION Theseauthorsreportresultsof91.5%recognitionwithalexiconof1000words onthecedardatabaseofwordssegmentedfromaddressesintheu.s.mail (Hull1993). formcompletecharacters.thesearethenpassedtoacharacterclassier. intheirpaper,identifyingthesekeylettersmightbesucienttoidentify mostwords,buttheauthorsproposetheirtechniquesasawayofltering, toreducethenumberofwordsinthelexiconofpossiblematches. approachistoextractanumberofkeylettersfromeachcursiveword particularlytheinitialletterandthoseclearlyidentiablebyascenders,descendersorloops.forasmallvocabularytask(readingcheques)asdescribed CherietandSuen's(1993)approachisalsoletter-based.However,their theboundariesbetweencharacters,butalsosplitsomecharactersintotwo anexplicitsegmentationapproach,buthereeachsegmentneednotcorrespondtoacharacter.theyndpresegmentationpointswhichincludeall strokes,loopsandcusps)withinthesegmentsbyaseriesofeventdetectors PapersbySrihariandBozinovic(1987;BozinovicandSrihari1989)take ormorepieces.theythenndfeatures(16inall,includingdots,curves, andusethefeaturestoconstructletterhypothesesaccordingtostatisticsof featureoccurrencesgatheredduringtraining.wordsarehypothesizedviaa follow.theresultanttwo-lettersequencesareputontothestack,tobeexpandedwhentheyarethemostlikelysequences.attheendoftheword, stackmethod,wherethemostlikelyprexesarestoredandexpandeduntil thewordendisreached.aftertherstiterationofthisprocedure,thestack thelexicallycorrectwordthatishighestonthestackischosenasthebest match. containsallthehypothesesfortherstletterinorderoflikelihood.thetop entwritersanddierentlexica(780and7800words).testingonasingle- authordatabaseofhorizontal,non-slantingwriting,a77%recognitionrate wasobtainedonthesmalllexicon,48%onthelarge.asecondsingle-author databaseyieldeda71%recognitionrateonthesmallerlexicon. SrihariandBozinovicconductedanumberofexperiments,usingdier- (mostlikely)hypothesisisthenexpandedbylookingatwhatletterscould ofuptothreesegmentswithaneuralnetworkclassiertrainedonisolated charactersegmentationpointsandattempttoclassifysegmentsorgroups letters.incorrectsegmentationstendtogetlowerclassicationscoresthan whenaletteriscorrectlysegmented,andwhenthescoresarecombinedin ahiddenmarkovmodel,thebesthypothesisforthegroupingsofsegments andtheiridentitiesisfound.resultsof70%forsingle-authorcursiveword YanikogluandSandon(1993)takeasimilarapproach.Theyndpossible recognitionarequotedforalexiconof30,000words. O-linehandwritingrecognition Edelmanetal.(1990)havedevelopedahandwritingreaderwhichrelies

21 turningpointsatthetop,bottom,leftorrightofacharacter)arefoundin onthealignmentofletterprototypes.here,anchorpoints(e.g.endpoints; prototypecurves,codedassplines,whichcanbecomposedintolower-case thetestwordandthesepointsareusedtomatchthewordagainstasetof CHAPTER2.HANDWRITINGRECOGNITION characters.thesystemishand-designedandisnottrainedautomatically. Usinga30,000wordlexicon,theseauthorsobtainedan81%recognitionrate onthetrainingsetandaround50%ontestsetsbythreeauthors.thestress ofthissystemisonrecognitionwithoutalexicon,however,andrecognition ratesof8{22%aregivenforthreeauthorsincludingtheauthorwhosewriting thesetoasetofreferencewordswithdynamicprogramming.theidentied wasusedtodevelopthesystem. wordsareusedtogetherwithagrammartoverifytheamountingures.with oce.thetaskhereistorecognizeamountswritten(inwords)onpostal chequesandtousethesetoverifytheamountswritteningures.moreau tackledbyanumberofauthorsintheproblemposedbythefrenchpost etal.(1991)identifyafewcharacteristicsofthecursivewordsandmatch Theproblemofreadingtheamountoncheques(section2.3.2)hasbeen a60%rejectionrate,theerrorrateachievedis0.2%.paquetandlecourtier (1991)reduceeachwordtoaseriesofcurveswhichtheymatchtoexamples inalexicon.theyachieve60%correctonthe50%ofwordswhicharewellsegmentedandlater(paquetandlecourtier1993)achieveanerrorrateof 59%whenrejecting9.5%ofwords.Lerouxetal.(1991)taketwoparallelapproaches oneistorecognizethewordasawhole,byndingafewfeatures andcomparingwithreferencewords.thesecondisaletter-by-letterapproachwherethedesireistorecognizeonlysomeoftheletters,andtouse thisinformationtorestrictthelexicon.theirsystemcorrectlyidenties62% ofwords.thesystemdescribedbysimon(1992)achievesa0.15%errorrate witharejectrateof24%usinga25wordvocabulary. O-linehandwritingrecognition 20

22 Chapter3 Psychologyofreading Thereisanartofreadingaswellasanartofthinkingandanartof writing. derstoodwhatinformationpeopleusetorecognizehandwrittenwords,then Beforeattemptingthemachinerecognitionofhandwriting,itisworthwhile consideringthewaythatpeoplereadandwrite.consideringhumanreadingmayleadtoanincreasedunderstandingofthetransferofinformation throughthemediumofhandwriting,sothatitcanbeseenwhichprocesses playausefulrole,andwhicharemerelyepiphenomena.ifitcanbeun- D'Israeli. aclueisfoundastowhatfeaturesmightbeusefulforamachinerecognitionsystem.otherfeaturesarelikelytobepoorlypreservedsincetheyplay insightsastowhichfeaturesofhandwritingarerepresentationsoftheinformationandwhichmereartefactsofthegenerationprocess. nousefulrole.understandinghandwritingproductionmaysimilarlygive involvedinreadingtype,someofwhichisapplicabletocursivescript.taylorandtaylor(1983),downingandleong(1982)andraynerandpollatsek (1989)givethoroughreviewsofthepsychologyofreading.Mostresearchso farhasconcentratedonreadingindividuallettersorwordsoutofcontext.it Alargebodyofpsychologicaldatahasbeengatheredontheprocesses couldbearguedthatthisgiveslittleindicationoftheprocessesoccurringin normalreadingwheremanywordsarevisibleanditisthetextasawhole, notindividualwords,thatisimportant.howeverresultsarehardtoprove insuchanaturalenvironmentwithmanyvariables,anditisonlyunderrestrictedexperimentalconditionsthathypothesescanberigorouslytestedingwhaterrorsaremadeunderdicultconditions.onetechniqueistheuse oftachistoscopestoashawordinfrontofasubjectforaveryshorttimefollowedbyapatternedmasktoinhibiticonicmemory,whichotherwiseallows thesubjecttopreserveanimageofthewordmentallyforanuncontrolled periodoftime. Researchintoreading,asinmuchofpsychology,reliesheavilyonobserv- O-linehandwritingrecognition 21

23 Aswillbeseenlater,manyapproachestohandwritingrecognitionrelyon detectingfeaturesinthewriting,suchasthestrokeswhichgotomakeup 3.1Readingbyfeatures CHAPTER3.PSYCHOLOGYOFREADING ofbarsandedgesandprovideacompactrepresentationoflineswhichis particularlyappropriatetotherepresentationofwritingandprint.anumber individualletters.hubelandwiesel(1962)describetheprocessesearlyin thevisualcortex.thecomplexcellsthattheydiscoveredcodethepresence ofauthorshavesoughttodeterminewhathigher-levelrepresentationmight latedcharactersbyexaminingtheconfusionsbetweenletterspresentedei- theratadistanceorforashorttime,eccentricallyinthesubject'seldof beusedspecicallyforletters. view.boumausestheerrorsmadebysubjectstoidentifygroupsofconfusable,or`psychologicallyclose',letters.bouma'sclassicationisshownin table3.1. Bouma(1971)investigatedthefeatureswhichpeopleusetorecognizeiso- OutercontourBoumashape Short Tall innerpartsandrectangularenvelope1aszx roundenvelope obliqueouterparts CodeLetters Projecting verticalouterparts ascendingextensions slenderness descender 2eoc 3rvw 4nmu 5dhkb 6tilf Shapetype Table3.1:Boumashapes. Numberofwordssharingthesameshape 7gjpqy Outercontour+initial Boumashape+initial Table3.2:Worddiscriminationusingwordshapemeasureson ` 'wouldbecome527,butsoalsowould`bo'whichisseentobesimilar inshape.taylorandtaylorusedtheseboumashapesforastudyonthetext oftheirownbook.table3.2showsasimilarexperimentonthetextofthis Usingtheseclasses,wordscanbeencodedaccordingtotheirshape,so thetextofthisthesis. O-linehandwritingrecognition 22

24 asshort,tallorprojecting.theoutercontourisenoughtospecify1389ofthe techniques,andthenumberofwordsofeachshapeiscounted.theouter contourisacoarsercodingthantheboumashape,simplyclassifyingletters thesis.thewordsareclassiedaccordingtoeachoffourshapedescription CHAPTER3.PSYCHOLOGYOFREADING 3444wordsuniquely,butthereare36shapessharedbytenormorewords Boumashape,havingmoreclassesthanoutercontour,givesmoreunique shapes 3201wordsareuniquelylabelled. each.iftherstletterisknown,theambiguityisfurtherreduced.the gatethevariabilityofsomehandwritingfeaturescomparingvariationinan afewsimplefeaturescanidentifymostwords,withouttheneedtorecognizetheindividualletters.haberandhaber(1981)havecarriedoutsimilasiontreewhichmightbeusedtodistinguishthelettersofthehelveticafont byobservingonlyalimitedsetoffeatures.eldridgeetal.(1984)investi- workintotheeectivenessoflettershapeforreading,andalsogiveadeci- Thisstudyshowsthat,inconjunctionwithalexiconofpermittedwords, tioniscarriedoutbyndingwordfeaturesthatllrolesininternalmodels individual'shandwritingwiththatbetweenindividuals. inrepresentingcharacters.althoughtheirexperimentsareconductedwith machine-generatedlettersmadeupofstraightlinesegments,theyinvestigatetherecognitionoflettersatthelimitsofclass-boundaries,sotheirwork ofletters.thusaletter`b'couldbedescribedasaloopwithashortstroke isofrelevancetohandwritingrecognition.theysuggestthatletterrecogni- McGrawetal.(1994)furtherinvestigatethefeaturesthatmightbeused lowerright.theseauthorsdonotconsiderthepossibilityofoverlapping featureswhichmightcharacterizetheletteraswellifnotbetter.forinstance,a`b'couldalsobedescribedasatallstrokeoverlappingalooptothe aboveandtotheleft,orasatallstrokewithacurvedsectionjoinedatthe right.theymaketheimportantpointthatthehigher-levelfeaturesusedfor readingarenotlikelytosimplyarisebottom-upfromthevisualprocessing Ifanaccuratemodeloftheseprocessescanbefound,thenitcouldbeusedfor system,ashubelandwieselcellsdo,buttobedenedtop-downdependingontheclassestobedistinguished.thisdependsinturnonthewriting andgroupingsmadeinthatlanguage. representationofhandwritinginacompactform,andforrecognition.alimi tweenphonemeshavetobere-learntaccordingtothedierentdistinctions andplamondon(1993)discussavarietyofmodelsforhandwritinggeneration,andabbinketal.(1993)andsingerandtishby(1993)haveusedthe Hollerbach(1981)modelformodellinghandwritingforrecognition.Singer andtishbyderiveaverycompactcodewhichrepresentsthehandwriting Manystudieshavealsobeenmadeintotheprocessesinvolvedinwriting. systemtoberead,justaswhenlearninganewlanguagetheboundariesbe- butalsoallowstheeasyremovalofslant,slopeandothervariation,making thewritingmorelegible.teulings(1994)discussesfeatureextractionfrom O-linehandwritingrecognition on-linecursivescript.asyettheseapproacheshaveusuallybeenappliedto on-linescriptwherethepentrajectoryisaccuratelyknown.thestaticnature 23

25 (1993)showsthato-linescriptcanbeconsideredinthisway.Howeverit seemsthat,whilecompactrepresentationscanbefoundusingthemodel- ofo-linewritingdoesnotlenditselftotheseapproaches,thoughdoermann basedapproach,readingisavisualprocessanddynamicapproacheswillal- waysfailtorepresentdatasuchasthedotsonaletter`i'appropriately,for CHAPTER3.PSYCHOLOGYOFREADING Oneofthefundamentalndingsofreadingresearchistheimportanceof 3.2Readingbylettersandreadingbywords recognitionofwordsassingleentitiesandnotastheconjunctionoftheir hereitisimportantwherethedotoccurs,notwhenorhow. onetonameletters theneachgroupwasswitchedtotheothertask.noevidencewasfoundthatlearningonetaskimprovedperformanceintheother, thusonemayconcludethat\relativelyuentreadingrequiresfamiliarity withtheshapesofwords,butnotwiththelettersinthosewords."(p.195) tersareallupside-down).theytrainedtwogroups onetoreadwordsand componentcharacters.taylorandtaylorciteworkbykolers&magee, whoseexperimentsinvolvedtrainingsubjectsoninvertedtext(wheretheletrectlywhenpresentationtimeisshortenoughtoinduceerrors)whenpresentedaspartofawordthanwhenpresentedeitheronitsownorsurrounded describedbyraynerandpollatsekp.77). givenbythewordsuperiorityeect.thisisthetermusedforthephe- byarbitrarycharactersinanon-word(forinstanceinreicher'sexperiments Furtherevidenceforreadingbywordsratherthanindividuallettersis nomenonthataletterisbetterrecognized(morefrequentlyrecognizedcor- mustbeusedforthetwoscripttypesandindicatesthatthemechanismof readingismorecomplexthanitmightatrstappear.downingandleong phemic)scriptismuchlessaected.thisshowsthatdierentbrainpathways discussthepossibilityofphonological,visualorbothpathwaysforindexinganinternallexicon,andtheevidenceseemstosuggestthatpeopleuse bothacodingofthesoundsofwordsandacodingofthevisualimagewhen erscanseverelyimpairreadingofkana(syllabic)scriptwhereaskanji(mor- whichshowsthatdamagetocertainareasofthebrainsofjapaneseread- ItisinterestingtonotetheworkbyYamadori(1975)andSasanuma(1984) recognizingwordswhilereading. Letter-basedprocessFrom50msafterawordispresented,theindividual Whole-wordprocessThisisarapidprocesstakingperhaps50-100mswhich TaylorandTaylorproposeareadingmechanismwiththreepaths: dozenlettersoflongerwords. letteridentitiesarebecomingavailable.(thiscouldbeunderstoodasa progressiveincreaseinthefrequencyofthelterusedassuggestedin workbymarr(1982)).outerlettersareidentiedrst,andmaybeused isbasedonlyonthepatternofthewordasawhole,orthersthalf- O-linehandwritingrecognition 24