Transcription

1 VericationoftheSyllableBasedText AGeneticAlgorithmApproachfor GokturkUcoluk CompressionTechnique DepartmentofComputerEngineering _I.HakkToroslu MiddleEastTechnicalUniversity composeatextintostringsthathavelengthsgreaterthanoneandoccur Providedthataneasymechanismexistsforit,itispossibletode- Abstract frequently.havinginonehandthesetoffrequentlyoccurringsuchstrings thetextusinghumancodingoveranalphabetwhichisasubsetofthe unionofthesetwosets.observationsrevealthatinmostcasesthemaximalinclusionofthestringsleadstoanoptimallengthofcompressedtext. andintheotherthesetoflettersandsymbolsitispossibletocompress Howeverthevericationofthispredictionrequirestheconsiderationofall subsetsinordertondtheonethatleadstothebestcompression.ageneticalgorithm(ga)isdevisedandusedforthissearchprocess.turkish regularsyllableformation,areusedasatestbad. texts,whereduetoitsaglunitativenature,thelanguageprovidesahighly 1

2 11.1Introduction Thoughthecostofcomputerstoragedeviceshasdroppeddramaticallyandfar TextCompressionandHumanCoding morestorageisavailabletotheuseritcanstillbearguedthattheneedfordata compressionispreservingitssignicancesincetheinformationnowadaysbeing storedonelectronicmediaisalsoexponentiallygrowing.datacompressionis usedinpracticefortwopurposes,namelydatastorageanddatatransmission.by makinguseofvariouscompressiontechniquestypicallystoragesavingsof20%to 50%fortextlesmightbeachieved.Duetolargehomogeneouspatternsthese Run-lengthencodingissimpleandisbasedonstoringthenumbersofsuccessively useindatacompression:run-lengthencodingandvariable-lengthencoding[4]. guresescalateto50%to%90forbinaryles[9].twomaintechniquesaremade repeatingpatterns.thereforeitisnotverysuitablefortextcompressionsince theonlyrepeatingcharacterthatislikelytobefoundinatextistheblank character.variable-lengthencodingismakinguseofthefrequencyinformationof thepatterns.afterastatisticalanalysisofthedata,frequentlyoccuringpatterns areassignedshortercodesthantheonesthatoccurlessfrequently.d.human discoveredageneralmethodforndingtheoptimalcodingofthedatausingthe patternsisconstructed(suchatreeiscalleda`trie').thewaytoreachaleafis frequenciesofthepatterns[9].inthiswayabinarytreewhichhasatitsleafsthe tostartfromtherootandbranchleftorrightoneachnodeonthepath.this sequenceofbranchinginformation(letssaya1foraleftanda0foraright)is placedclosertotherootaretheonesthatfrequentlyoccurinthedata.hence reachingthemrequireslesserbranchingswhichresultsinshortercodesforthem. theencodingofthatspecicleaf.inahumantriepatternsattheleafsthatare Hereisanexample: Assumeweareencodingthetext ababababababababccbbbbbbddaaaaaacdcdcdacac 2

3 Theletterfrequenciesareasfollows: LetterFrequency bca 15 d 13 7 letterwouldrequire2bits(lg4=2)1(uncompresseddenotationisdierentthan Sinceouralphabeta,b,c,dhasfourelementsforuncompresseddenotationeach 5 theasciirepresentationwhichrequires8bits/char.commitingtotheascii 240=80bitswouldberequired.TheHumantreeisconstructedas representationimplicitelymeansthatthereare256distinctcharacters.insection 5thisisexplainedinmoredetails).Thetextcontained40characters.So,intotal, 1 0 Sothecoding(atbits)levelisasfollows: a LetterCodeCodelength b a 1 c d (1bit) 7 5 bc (2bits) Fortheabovegiventextthesumofrequiredbitscanbecalculatedasfollows: d 000 (3bits) Soacompressionratioof77=80=0:9625isobtained. 151 {z} a +132 {z} b+73 {z} c+53 {z} d =77(bits) 1lgrepresentslog2 3

4 1.2 Makinguseoftherepeatingpatternsinthetext:Extendingthealphabet Againconsidertheabovegiventextexample,butthistimeweindicatetherepeatingpatternsforsakeofeasyrecognition(superscriptstandsforrepetition, parenthesisforgrouping): Withoutmakinguseofanyadditionalorderinginformationinthetextotherthan (ab)7c2b6d2a6(cd)3(ac)2 canexpresstheabovetext.theseare thegroupedsymbolstherearevariouspossiblealphabetsintermsofwhichwe fa;b;c;dg fa;b;c;d;acg fa;b;c;d;ab;acgfa;b;c;d;ab;cdg fa;b;c;d;abg fa;b;c;d;ac;cdgfa;b;c;d;ab;ac;cdg fa;b;c;d;cdg ConstructingthecorrespondingHumantreesoneobtains Alphabet fa;b;c;dg fa;b;c;d;abg Uncompr.Len.Compr.Len.Compr.Ratio fa;b;c;d;acg fa;b;c;d;cdg fa;b;c;d;ab;acg fa;b;c;d;ab;cdg fa;b;c;d;ac;cdg fa;b;c;d;ab;ac;cdg basedhumancoding.furthermorethereisnotheoreticaleasymechanismwhich Thissmallexampleprovesthatitispossibletoperformbetterthanonlyletter letterremainsalmostconstantandatmostlooses1-2%.butnogeneralrulehas pinpointsthealphabetthatwillleadtothebestcompression.experiencewith lengthytextsshowsthatthesubsetofthealphabetsetwhichconstitutesofsingle 4

5 beenidentiedforthestringswithlengthsgreaterthanone.thisresultsina searchprocessoverallmembersofthepowersetofthesetofsuchstrings(with lengthsgreaterthanone).evenwithveryregularlanguagesthathaverelatively smallnumberofsyllablesthatrepeat,theorderofthesetsizeisofmagnitude 103.Aswellknown,theenumerationofallthemembersofapowersetofaset withnmembersisano(2n)process.hence,aproductivesearchmethodis neededtoapproachtheoptimal. requireamechanismforevaluationofthesuccessofcandidatesolutionswhich isinourcasethehumancompressionlengthsobtainedbyusingthecandidate AGeneticAlgorithms(GA)techniqueisemployedforthispurpose.GA duetoitscomplicateddatastructure.so,atheoreticalapproximationforthe isknowntobeofordero(nlgn)withaconsiderablylargeconstanttimefactor alphabets.butthisrequiresextensivehumantreeconstructionseachofwhich usingonlymathematicaloperationsontherepetitioncountsofthealphabet membersinthetext.thecomputationalcomplexityofthisapproximationis compressedtextlengthisusedwhichenablestoestimatethecompressedlength calculation. ofordero(n)wheretheconstantfactorcanconsiderablybereducedbypre- syllablesinahugesizedcorporaremainsoftheorderofafewthousands.itis ularsyllablestructure.duetothisregularityofthelanguagethecountofused TheproposedmethodistestedwithTurkishcorporawhichpossessesareg- hyphenationwhichproducesthesyllableboundariesofanyword.thefollowing moreturkishlanguageprovidesaverygrammaticalandsimplealgorithmfor observedthatabouttwohundredofthemoccurwithahighfrequency.further- owchartsummarizesthewholecompressionprocess. 5

6 Apply the hypenation algorithm to the text and obtain the constituent syllables and their frequencies. Construct a chromosome pool in which each gene position corresponds to one of the syllables (with length>1) and the allele values (1/0) have the meaning of keeping that syllable as a member of the Huffman alphabet or disolving that syllable into its letter constituents. Run the Genetic Engine to get the syllables which shall be included into the coding alphabet in order to, when used in the Huffman coding of the subject text, provides the best Huffman compression ratio. GAispresented.Section4brieysummarizesthesyllableformationofTurkish Perform a Huffman coding of the text over an alphabet that includes the syllables determined by the and all the individual characters required to code the text parts comming from the unincluded syllables andthealgorithmusedtoobtainthesyllables.in5,thelastsection,wepresent Thenextsectionexplainsthisapproximationtechniqueandinthesection3the punctuation characters. theresults/observationshenceobtainedandconclude. 2 TheoreticalApproximationfortheCompressed BasedonShannon'scontribution,itcanbeproven[4,8]inCodingTheorythat TextLength IftheentropyofagiventextisH,thenthegreatestlowerboundof thecompressioncoecientforallpossiblecodesish=lgmwhere Ontheotherhandweknowthat,modulocodingalphabet,Humancodingis misthenumberofdierentsymbolsofthetext. optimal.so,wecanconcludethat =?1 6 lgmmxi=1pilgpi (1)

7 isagoodapproximationforhumancompression.herepiistheprobabilityofa ofsymbolsinthetext(n=pmi=1ni).therealhumancompressioncoecient niisthecountoftheithalphabetmemberinthetextandnisthetotalcount symbol,memberofthealphabet,tooccurinthetextandisdenedasni=nwhere willbeequaltoorslightlygreaterthanthistheoreticalupperbound. theusefulnessofanydatacompression,thiscompressioncoecientmustbemultipliedbythebit-lengthoftheuncompressedtextinordertoobtainthebitlengthofthecompressedtext.thebit-lengthoftheuncompressedtextisnlgm. Hence,givenanalphabet,thetheoreticallowerboundforthebit-lengthofthe compressedtextis:lcompressed=nlgn?mxi=1nilgni[bits]. Notethatthisquantityisnotinvariantunderalphabetchangessincen,mand (2) Ifweareinterestedinthenalcodelengthwhichistheactualmeasurefor nivalueswillvaryfromalphabettoalphabet. 3GeneticAlgorithmsareemployedforalgorithmicsearchesbymimickingtheway GARepresentation value,aretransformedintoanewgenerationofthepopulationusingdarwinian naturegeneticallysearchesforsolutionsofthesurvivalproblemofitsspecies. principleofreproduction.hereusingthegeneticoperationssuchasrecombination(crossover)andmutationthepopulationisbreadedandevaluatedwitha possiblesolutiontothegivenproblem.thegaattemptstondtheoptimalor `survivalofthettests'criteria.eachindividualofthepopulationrepresentsa InGAapopulationofindividualobjects,eachpossessinganassociatedtness ingcomplex,highlynonlinear,multidimensionalsearchspaces.excellentreviews thepopulationofindividuals.inpracticegaisastonishinglyecientinsearch- nearlyoptimalsolutionstotheproblembyapplyingthosegeneticoperationsto canbefoundin[2,1,10]. 7

8 encodingoftheparametersofapossiblesolutionofthesubjectproblem.this encodingcanbedoneinvariouswaysbutmostlyaxedlengthstringischosen ThemainingredientofaGAimplementationisthechromosomewhichisthe agene.thesetofpossiblevalidvaluesagenecanholdiscalledthealleleofthat parametercorrespondstoaxedpositioninthechromosomewhichisnamedas duetotherelativeeasinessoftheimplementationofgeneticoperations.each chromosomestor.therunofagaisatwostepprocess.firstaninitialization isdevised.thisfunctionisnamedasthetnessfunctionandmapsthespaceof gene.afunctionwhichevaluatesthesuccessofasolution,namelyachromosome, ofapoolofsolutionsisperformed.thiscreationoftheinitialpoolismostlydone calledageneration.ineachgenerationthepopulationofchromosomesaremutated(randomlygenesarereplacedbyotherallelemembers)andthenmatedfor ofaprocesswhichconstitutesofthreebasicsteps.eachiterationofthiscycleis randomly.thenthemainstepofthegaisentered.thisisacontrolledcycle mannerwithpredeterminedprobabilities.followingthis,anevaluationprocess crossoverinwhichchromosomesexchangegeneswiththeirpartnersinarandom thetnessfunctionadecisionismadeaboutthechromosomeswhethertoexist iscarriedoutoverthenewgeneratedpopulation.usingthevaluesobtainedfrom (tolive)inthenextgenerationornot(todie). (namelysyllable)withlengthgreaterthanoneageneisassumedtocorrespond. Achromosomeisaxedlengthvectorofallelevalues1or0.Achromosomecorrespondstoacandidatesolution(inourcaseacandidatealphabet).Agenevalue of1meansthecorrespondingstringisincludedintothealphabet.a0meansthe correspondingstringwillbedissolvedintoitslettersresultinginanincremental contributiontothecountofthoseletters.atnessfunction,devised,evaluates achromosomebyrstdissolvingthenotincludedstrings,andthencalculating Inourproblemtoeverypredeterminedpossibleconstituentstringofthetext intheprevioussection.thecompressionlengthservesasthetnessvaluewhich Humancoding.Thiscalculationusesthetheoreticalapproximationexplained thecompressedlengthwhichwouldbeobtainedbyusingthatalphabetinthe willbeusedtodeterminewhichchromosomeisgoingtoliveandwhichisgoingto 8

9 die.sincetheoptimalcompressionlengthisnotknownthetnessvaluecannot relation.henceitisusedtoobtainasortingamongthesolutions.afteranew beconvertedtoanabsolutetnesscriteria,butratherbeusedasanordering generationisproduceditissortedaccordingtoeachchromosome'stnessvalue. Thebestsofthepreviousgenerationreplacetheworstsofthenewgeneration providedthat Thisreplacementtakesplaceforapredeterminedpercentageofthepoolat Thetnessofthereplacingchromosomeisbetterthanthereplacedone. most(whichisabout5%-10%). inwhichabout2700genes/chromosomeexistedarealsomentionedinitalics. aboutthetuningoftheenginethedynamicsettingsusedintherealapplication TheGAenginehasthefollowingalgorithmicoutline.Inordertogiveanidea Generatearandompopulation(seeexplanationbelow),evaluate Repeat: Poolsize=100chromosomes itandstoreitalsoastheformergeneration. Mutate. ChangesperChromosome=Fliponerandomlyselectedgene MutationRate=Onceeach10Generationonerandomchro- Mateallthepoolformingrandompairs,thenperformrandom crossoversamongpairs.henceformanewgeneration. Evaluatethenewgeneration,usingthetheoreticalapproximation. CrossOver=At10randomchosenrandomlengthgeneintervals thepreviousgeneration.keepratio=atmost10%(seeabove Replacethepoolbythenewgenerationkeepingthe`realbests'of Display/Recordperformanceresult. textforreplacementcondition) Ifitwasnotthelastgenerationtheuserdemanded,gotoRepeat. 9

10 proachedeventually,thespeedofconvergenceheavilydependsonsomeaspects ofthegaprocess.theinitialpoolcreationisoneoftheseaspects.iftheglobal AlthoughGA,bynature,isamechanisminwhichtheoptimalsolutionisap- searchspacehastobesearchedfortheoptimal,thenitiswisetoincludemembersfromalmostallregionsintheinitialpopulation.inthesubjectproblem,the propertiesofthespecicproblemdoesnotprovidescluesabouttheregionthe numberofmembersofalmostallpossiblecardinalities.so,forexample,thepool initialpopulationhadtobecreatedfromtheelementsofthepowersetofallconstituentsofthetext.theevennessoftheselectionisobtainedbyincludingequal thelettersandotherswhichhaveinclusionratioslinearlydistributedbetween stituents,anotherwhichcorrespondedtotheexclusionofallconstituentsexcept certainlycontainedamemberwhichcorrespondedtotheinclusionofallthecon- thesetwoextremes.thepoolsizeof100isagoodvalueacceptedempirically bymanygaresearchers[3].theuseofthecrossoveroperatorisencouraged mationofuseful`buildingblocks'underthisoperation.theencodingusedin forgaproblemswithencodingschemesthatpromotesthepropagationandfor- uniformcrossover.asknown,inuniformcrossovereachgeneiscrossedwitha thisapplicationisofsuchanature.thecrossoverusedisarestrictedversionof probabilityto0:5andthetotalcountofswappedblocksofgenesto10.various experimentationswiththesevaluesprovedthat10isanappropriate(thoughnot certainprobability(foragoodreviewofthesubjectsee[6,5]).werestrictthe approachwhereatmost10elementsofthepreviousgenerationarekept`alive' varycritical)choiceforproperconvergence.theselectionphaseusesanelitist Thenumberofiterationtoconvergetoasolutionisabout generations. IthasbeenobservedthatsuchanapproachstabilizestheconvergenceoftheGA. providedthattheyarebetterthantheworstelementsofthecurrentgeneration. Turkishisanaglunitativelanguage.Inections,tensesareallgeneratedbyappendingseveralsuxestorootwords.Althoughnotverycommon,itispossible 4 TurkishSyllableFormation 10

11 simplealgorithmforhyphenation.turkishlanguagehas8vowelsand21consonants.anysyllableofthelanguagehastohaveexactlyonevowel.inasyllable atmosttwoadjacentconsonantsisallowed.someadditionalruleslimitthe possibilitiestothefollowings(v:vowel,c:consonant): SyllablePattern VCV,VC NumberofPossibilities syllableboundariesinawordarethehyphenationpointsandthelanguagehasa toobservemeaningfulwordformationswhichmakeuseof10-15suxes.the VCC,CVC Thoughthiscombinatorialcalculationreveals8104possibilities,thelanguage usesabout3%ofthemonly.themostfrequenttypesofoccurancesarethe CVCC extremelyrare(oforder102)andoccurinatextwithprobabilitiesoforder10?4. ofcv,vc,cvctypepatterns.thefourlettercombinationsare scansawordandproducesthesyllables. ThefollowingsimplealgorithmwhichisO(n),(denotedinCsyntax),linearly 11

12 Assumethecharacterarrayturkishword[]holdsthewordtobe cessivesyllables.furthermoreaglobalintegervariablesylcount ofpointerssyl[]thathaselementspointingtotheendsofsuc- hyphenatedandthefunctionhyphen()uponacallllsoutanarray getssetbyhyphen()tothecountofsyllablesformed. char*twp; voidhyphen() {syl_count=0; twp=turkish_word-1; do{if(next_is_vowel()) if(next_is_vowel())mark(1); while(*twp); elseloop:if(next_is_vowel())mark(2); while(!*--twp); elseif(*twp)gotoloop;} }voidmark(chark) mark(0); intnext_is_vowel() {syl[syl_count++]=(twp-=k)+1;} {if(is_vowel(*(++twp))return1elsereturn0;} intis_vowel(charc) smallnumberofdistinctsyllablesthatareusedfrequentlythroughoutthelanguageenablesanecientuseoftheproposedcompressiontechniqueforturkish texts. Theexistenceofsuchasimplesyllableformationalgorithmandtherelatively returns1ifcisavowelelsereturns0 12

13 5magazinearticles,arecompressedthroughthisalgorithmandcomparedsizewise Atotalof5MBytesofTurkishcorpora,mainlygatheredfromnewsitemsand ResultsforTurkishTextsandConclusion withthestandardadaptivehumancodingoveraletteralphabet.thefound resultsaretabulatedbelow. (8bits/char) textlesize OriginalASCII#ofbytes Corpus1Corpus22Corpus3Corpus4 Lengthofuncompressedrepresenta tionwithmin.#ofbits/char(see text) Length(bits) Humancodingoversingle character alphabet Compressionratio Compressionratio ()w.r.t.minimalbitrepresentation Humancod-Length(bitsing overgadeterminelableextended alphabetsyl-compressionratio ()w.r.t.minimalbitrepresentation w.r.t.asciisize (#ofkeptsyllables)/(total#of w.r.t.asciisize Comparedtothestandardcodingthemethodhasprovidedanupto21% syllables) 1976/ / / / 2594 bettercompressionratio(excludingtheoverheadwhichisnegligible).theworst referenceismadebyputtingdownthevalueofreductioninthelesize.text quantitywiththeusually(butmistakenly)usedquantity.usuallyacompression improvementobservedwas13%.itisworthtopointoutthedierenceofthis lesaremadeofbyteswhereeachcharacter(letter,punctuationsymbol,etc.)is representedbyauniquebyte(usuallytheasciicode).butthewholebunchof 2containsonlylowercaseletters,hencecanbecodedwith5bits. 13

14 bytepatternshavenocharacterassociationatallorarenotfoundinthetext the`used'charactersaremuchlessthen28soactuallyaconsiderablenumberof hasagreaterinformationcontentthanitactuallypossesses).thereforethe consistsof28distinctcharacters(thisisassumingthattheuncompresseddata le.henceitiswrongtoassumethattheoriginalalphabetofthecompression compressionratio(whichwerefertoas)isproperlycalculatedwithrespectto therealinformationsizeoftheuncompressedtext.thismeansiftheoriginal LlgNbits.Aftercompression,assumethatthecompressionyieldsCnumber textconsistsoflcharactersfromasetofndistinctcharacters,sincedlgne ofbits,thenthecompressionratioiscalculatedastheratioofthecompressed bitswouldbesucienttorepresenteachcharacter,thewholetextwouldrequire bitlengthtotheuncompressedbitlength: Thetabulationabovedisplaysthiscorrectcompressionratioaswellastheobserveddecreaseinthetext(ASCII)lesize. = LlgN C valuescouldbeidentiedforexclusionfromthealphabet.themechanismthat codingincludesalmostallthepossiblesyllables.norulebasedonfrequency IthasbeenobservedthatforTurkishtextsthealphabetfortheHuman theoverallentropy.so,usingagaapproachseemstobeappropriatethatsuits leadstoinclusion/exclusionisheavilybasedonthewaythedissolvingaectsthe frequenciesoftheremainingalphabetmembersinfavorofincreasing/reducing syllabicationprocessissubstitutedbyaniterativetri-gramprocessinwhichat thepurposeofdecision. eachsteptri-gramsareattemptedtobereplacedbytokensaccordingtoaga's Itwouldbeinteresting,asafuturework,toconsidertheproblemwherethe successfulcompressionofanykindofdata. decision.webelievethatthiswillleadtoamoregenerallyapplicableandmore 14

15 References [1]L.D.Davis,HandbookofGeneticAlgorithms.(VanNonstrandReinhold, [2]D.E.Goldberg,GeneticAlgorithms(Addison{WesleyCo.,Reading,MA, 1991) [3]D.E.Goldberg,SizingPopulationsforSerialandParallelGeneticAlgorithms,Proc.ICGA'89, ). [4]R.W.Hamming,CodingandInformationTheory(Prentice{Hall,EnglewoodClis,NJ,1986). [5]T.Jones,EvolutionaryAlgorithms,tnessLandscapeandSearch.PhDthesis,(TheUniversityofNewMexico,NewMexico,1995). [6]K.A.DeJongandW.M.Spears,AnAnalysisofMulti-PointCrossover. [7]G.Lewis,TurkishGrammar(OxfordUniversityPress,Oxford,1991). FGA, ,1991. [8]S.Roman,CodingandInformationTheory(Springer{Verlag,NY,1992). [10]A.Wright,ed.FoundationsofGeneticAlgorithms.(Morgan-Kaufmann, [9]R.Sedgewick,Algorithms(Addison{WesleyCo.,Reading,MA,1988). 1991). 15