Testing The Torah Code Hypothesis: The Experimental Protocol

Transcription

1 Testng The Torah Code Hypothess: The Expermental Protocol Robert M. Haralck Computer Scence, Graduate Center Cty Unversty of New York 365 Ffth Avenue New York, NY Abstract Ths s the second part of a tutoral dscussng the expermental protocol ssues n Testng the Torah Code Hypothess. The prncpal concept s the test statstc whch s used to do the actual hypothess testng of the Null hypothess aganst a smple alternatve or aganst a complex of alternatves. We llustrate the methodology usng the data sets from the WRR[3] experment. We use the WRR key word sets of lst 1 and 2 combned. The experment produces a p-value of less than 1/100,000 n the Geness text. We performed another experment parng rule based translteratons for the spellngs of the names of the Amercan presdents nto Hebrew wth the Hebrew word for presdent. Takng nto account Bonferron, the resultng p-value of the 100,000 tral experment was less than 1/66, Introducton In order to have an experment that s reproducble, there has to be an expermental protocol whch descrbes n suffcently precse detal all the steps and calculatons so that another researcher can ndependently perform the experment and expect to get results that are nsgnfcantly dfferent from that of the orgnal experment. It s ths knd of replcaton that the scentfc methodology demands. In ths paper we provde exact descrptons of expermental protocols that can test dfferent varatons of the Torah Code hypothess. The notaton and concepts n ths paper follow that of Haralck[1] and we do not repeat here any of the defntons or concepts dscussed there. Just as n pattern recognton, where t s well known that some features wll work better for a partcular task so n testng the Torah code hypothess, some protocols work better than others, better meanng lower false alarm and msdetecton rates. At ths tme t s not known what the best protocol s, but n ths paper we are able to demonstrate a protocol wth an mproved false alarm rate compared to the orgnal WRR protocol. Prncpal concepts nvolve the control populaton, here called the monkey text populaton and the test statstc for actually dong the hypothess testng of the Null hypothess aganst a smple alternatve and aganst a complex of alternatves each assocated wth the Torah Code hypothess. Our test statstc uses multple compactness features and ts formula s motvated by a probablty dervaton. Our expermental protocol uses the test statstc as a score n Monte Carlo experment We frst llustrate the applcaton of the expermental protocol n an experment of the McKay key word set n the War and Peace text and the WRR key word set of lst 1 and lst 2 combned n the Geness text. The McKay experments were an attempt to llustrate that the codes found n the Torah text could be replcated n an ordnary text by suffcent wgglng and fddlng wth the key word choces and spellngs. Wth our protocol, we are not able to reject the Null hypothess for the McKay key word set n the War and Peace text and we must reject the Null hypothess for the WRR key word set n the Geness text. Then we llustrate the applcaton of the expermental protocol n an experment orgnally suggested by 13 year old Davd Roffman n December 2005: the relatonshp between the names of the Amercan presdents and the Hebrew word `$p, meanng presdent. 2. Hypothess Testng The formal way n whch the sgnfcance of an encodng s evaluated s by a test of Hypotheses. The Null hypothess of No Torah Code Effect s tested aganst an alternatve hypothess that there s an encodng. The statstcal computaton nvolved n the test of hypotheses amounts to determnng the fracton of monkey texts that have at least as good an encodng as the Torah text. Or sayng t another way, f the compactness value of the gven key word set n the Torah text s v 1 and the com-

2 pactness value of the gven key word set n Monkey texts 2,..., N s v 2,..., v N then the estmated probablty that a monkey text would have as good an encodng as the Torah text s the normalzed rank of v 1 among v 1,..., v N. Ths normalzed rank s called the p-value of the experment. To do the test of hypothess, the p-value of the experment s compared to a sgnfcance level α 0. If the p-value s smaller than α 0, then the Null hypothess of No Torah Code effect s rejected n favor of the alternatve that the key word set has ELSs n an unusually compact arrangement. If the p-value s larger than the sgnfcane level, the Null hypothess s not rejected Test of Null Hypothess Aganst A Complex Alternatve Hypothess An experment about a partcular hstorcal event s descrbed by a set of what are consdered to be the key words relevant to that event. However, not all of the key words thought about mght have correspondng ELSs n a relatvely compact arrangement. Hence an hypothess test of the Null hypothess aganst the Alternatve hypothess that all of the key words have ELSs that are n a relatvely compact arrangement wll most lkely not be rejected. Therefore, the experment s set up as a test of the Null hypothess aganst multple alternatves hypotheses. Each alternatve hypothess s specfed by some subset of the gven total set of key words. The formal test of the Null hypothess s aganst the alternatve hypothess that at least one of the alternatve hypotheses s true Bonferron When K separate experments are done, each testng the Null hypothess aganst a dfferent Alternatve hypothess, yeldng p-values p 1,..., p K, the smallest p-value s not the p-value of the complex of the K separate experments. Indeed, f the experments are separate, then the exact p-value of the complex of K separate experments cannot be determned f they are not ndependent. Ths s the usual case. However, t can be bounded. The Bonferron upper bound s K mn{p 1,..., p K }. The p-value of the K separate experments must be smaller than the Bonferron bound. Therefore, f the Bonferron bound whch s necessarly hgher than the p-value of the complex of experments, s smaller than the sgnfcance level, then t necessarly follows that the p-value of the complex of experments s also smaller than the sgnfcance level. In ths case the Null hypothess can be rejected at the gven sgnfcance level. The problem wth the Bonferonn bound s that t s an upper bound and n many nstances s much hgher than the true p-value of the complex of K separate experments. Ths s partcularly true when the K Alternatve hypothess are statstcally dependent K Scores And Combne There s a statstcally economcal alternatve to usng the Bonferron bound when testng the Null hypothess aganst a complex of K Alternatve hypotheses. The alternatve s on a tral by tral bass, to use K scorng schemes, one approprate for each of the K Alternatve hypotheses, and then combne the scores together n a sutable way. Suppose there are K key word sets, each descrbng the same hstorcal event. In ths case t s expected that every par of key word sets wll have a substantal fracton of ts key words n common to both sets. In ths case the Alternatve hypotheses wll necessarly have statstcal dependence. Each tral of an N tral experment randomly samples a monkey text from the monkey text populaton. In accordance wth a specfed protocol, on tral n, the compactness of the ELSs from each of the K key word sets s computed, resultng n c 1n,..., c Kn. For the k th key word set, the compactness values c k1,..., c kn of the N trals are rank normalzed to r k1,..., r kn. The p-value assocated wth test of the Null hypothess aganst the Alternatve that the k th key word set has ts ELSs n a more compact relatonshp than expected by chance s gven by r k1. The Bonferron bound B on the test of the Null hypothess aganst the K alternatve hypotheses s then B = K mn{r 11,..., r K1 }. The K scores and combne feature, would defne a combnng functon F actng on the rank normalzed values r 1n,..., r Kn, for the n th tral of the experment. In ths stuaton, combnng functons ought to be symmetrc n ts arguments. For example, one combnng functon could be the mnmum: f n = F (r 1n,..., r Kn ) = mn{r 1n,..., r Kn }. The scores f 1,..., f N are rank normalzed and the rank normalzed value, p 1, assocated wth f 1 s the p-value of the experment. For the mn combnng functon, p 1 s nessarly smaller than the Bonferonn bound B. However, the mn combnng functon s not necessarly the statstcally most optmal. For example, a combnng functon may be motvated by a probablty dervaton that has even some unwarranted condtonal ndependence assumptons. 1 One such combnng functon s F 1 (r 1n,..., r Kn ; θ) = 1 K K p(r kn ; θ) k=1 = f 1n (1) 1 The unwarranted assumptons are not used n makng any probablty calculatons for p-value. The probablty derved by usng the unwarranted assumptons gves a motvaton and a formula for performng a calculaton of a score functon. It s the score functon that s used n a proper Monte Carlo experment for determnng the p-values.

3 where p(r; θ) s the probablty under the Alternatve hypothess of observng a normalzed relatve rank of r n an N tral experment and 1/K s the pror probablty of any one of the K alternatve hypotheses of beng true. We base p(r; θ) on log because for small relatve ranks log wll be large (log(2n)) for an N tral experment where the smallest and unque relatve rank s 1/2N. For larger relatve ranks, log wll be small and ndeed be 0 for a relatve rank of 1. { β(θ) log(r) when r < θ p(r; θ) = (2) 0 otherwse Ths combnng functon arses (up to a fxed constant of proportonalty) when exactly one of the K alternatve hypothess s assumed to be true, and for each tral n, and for each k, the random varables r 1n,..., r Kn are condtonally ndependent gven that Alternatve Hypothess k s true. When the Null hypothess holds the remanng K 1 possbltes are assumed to follow the dscrete unform on the normalzed relatve ranks of an N tral experment. The threshold θ specfes that the probablty of observng a normalzed rank greater than θ under the alternatve hypothess s 0. We call ths method the frst order combnng method. If two of the K alternatve hypotheses are assumed to be true, wth the pror probablty for any par to be true to be 2/(K(K 1)), then under the same condtons as the prevous dervaton, the combnng functon should be 2 F 2 (r 1n,..., r Kn ; θ) = K(K 1) p(r jn ; θ)p(r kn ; θ) {(j,k) k>j} = f 2n (3) We call ths method the second order combnng method. If t s assumed that when there s an encodng ether one or two of the K alternatve hypotheses s true, and the pror probablty for exactly one alternatve beng encoded s the same as the pror probablty for exactly two alternatves beng encoded, then under the same condtons as the frst probablty dervaton, the score s n of the n th tral should be s n = f 1n /N + f 2n (4) We call ths method the non-rank normalzed method of combnng. Another possble way of combnng F 1 wth F 2 would be to take the N values f 11,..., f 1N and rank normalze them formng the N normalzed ranks t 11,..., t 1N. Also rank normalze the N values f 21,..., f 2N formng the N normalzed ranks t 21,..., t 2N. Defne the rank normalzed score s n for the n th tral by the convex combnaton s n = wt 1n + (1 w)t 2n (5) for a specfed weght w. In ether combnng method, the p-value of the experment s the normalzed relatve rank of the score for the frst tral Composte Experments Composte experments are assocated wth multple events. Suppose that there are M events. Each event has assocated wth t a collecton of key word sets. The m th such collecton s assocated wth a test of the Null hypothess aganst the K m alternatve hypotheses formed by each one of the K m key word sets n the collecton. In the composte experment, we are nterested n a test of hypotheses at two levels. Frst we wsh to test the Null hypothess aganst the Alternatve that more of the M events have ther ELSs n a more compact arrangement than expected by chance. Second we wsh to test the Null hypothess aganst the Alternatve that more of the K K M key word sets have ther ELSs n a more compact arrangement than expect by chance. In the frst case, we treat each event as an experment that produces n each tral a score whch s the normalzed relatve rank of the compactness assocated wth the tral. Thus each tral produces M scores. These scores then combned together n a test statstc approprate for a test of the Null hypothess aganst M Alternatve hypotheses, where one of the M Alternatve hypotheses s assumed to be true. We know that n ths stuaton, the probablty of small scores under the Alternatve hypothess for each of the M alternatves s not as hgh as n the case when consderng the probablty of small scores of an alternatve n the complex of alternatve stuaton. In ths case, sutable combnng functon choces nclude g n = G(s 1n,..., s Mn ) = β M or g n = G(s 1n,..., s Mn ) = β M M 1 exp( s mn /q m ) (6) m=1 M log(1. s mn /q m ) (7) m=1 where s mn s the score of the m th event of the n th tral and q m s a scale factor computed as the largest of the scores for the m th rabbnc personalty. q m = max n=1,...,n s mn The p-value of the composte experment s the normalzed relatve rank of g 1.

4 3. Compactness Measures It has been anecdotally notced that sometmes ELSs have compact meetngs n accordance wth one compactness measure and other tmes n accordance wth a dfferent compactness measure. Therefore an experment can select multple knds of measures that tend to score well for many of the knds of compact meetngs notced. These dfferent compactness measures are n effect assocated wth dfferent alternatve hypotheses of Torah code effect. The raw rank normalzed data set of the M collectons of key word sets, one collecton per event, for the n th tral then can be represented by r c 11n,..., r c 1K 1n; r c 21n,..., r c 2K 2n;... ; r c M1n,..., r c MK M n c = 1,..., C; n = 1,..., N where C s the number of compactness measures, N s the number of trals, and K m s the number of key word sets n the collecton of key words sets assocated wth the m th event. In accordance wth ether the non-rank normalzed method or the rank normalzed method, the raw data set s processed to produce scores for each event, tral, and compactness measure. We denote by s c mn the score assocated wth event m, tral n, and compactness measure c. If the alternatve hypothess s that the encodng occurs wth at least one of the compactness measures, then t s reasonable to form a score s mn by s mn = max c=1,...,c sc mn (8) If the alternatve hypothess s that each encodng occurs wth all of of the compactness measures smultaneously, then t s reasonable to form a score s mn by s mn = 4. Our Expermental Protocol mn c=1,...,c sc mn (9) We use the combned data from lst 1 and lst 2 of the WRR paper[3]. Ths data set has become a standard n Torah code work, smlar to the status of how the Fsher Irs data set s used n the classc dscrmnaton experments. For the WRR data, each key word set has one appellaton and one date of ether death or brth. There are 53 rabbnc personaltes for whch at least one of ts key word sets has at least one ELS for each of the key words n the set. These rabbnc personaltes represent our events. The collecton of key word sets assocated wth each rabbnc personalty consttute the possble event descrptons. There are a total of 321 key word sets for these 53 rabbnc personaltes. For our skp specfcaton σ, we set the largest skp permtted for ELSs of a gven key word to be such that the expected number of ELSs searchng from a mnmum skp of 2 would be 10. Ths s smlar to the protocol of WRR. And we set the mnmum skp for ELSs to be 1 (WRR sets the mnmum skp to be 2). For our resonance specfcaton φ, we requre that at least one ELS from each key word n a key word set be resonant on a cylnder sze and on the resonant cylnder sze the skp of the ELS must be no more than 10 rows and no more than 10 columns. Ths dffers from WRR who nssted that for one ELS the row skp on the cylnder be no more than 10 rows. For our monkey text populaton we use the ELS random placement populaton wth 100,000 trals. The Monte Carlo experment s carred out wth an ndependent executon for each rabb. The random number seed was obtaned from the dgts of π. Startng from the frst dgt after the decmal pont, the dgts were broken up nto strngs of seven long. Each successve strng of seven π dgts was used as the random number seed for each successve rabb Monte Carlo experment. We use both frst order (1) and second order (3) methods of combnng over the key word sets of each rabb. We set our threshold θ =.2 n (2), a value used by WRR[3] n a slghtly dfferent context, but n the same sprt as we used t. For our compactness features we choose two knds of compactnesses that measure essentally dfferent knds of geometrc arrangement. The 1D compactness measure searches over all ELS sets ζ satsfyng the skp specfcaton, and fnds the ELS set havng the smallest span length and the correspondng text segment. The span length of an ELS set s the dfference between the largest endng poston taken over all ELSs n the set and the smallest begnnng poston taken over all ELSs n the set. Ths compactness feature s essentally the area of the table formed on a cylnder of 1 column. Our second compactness feature s dstance based. Followng the notaton n Haralck[1], t s formed by R 12 followed by Ψ harm. For each of the 100,000 trals, each of the 321 key word sets 2 has ts ELSs evaluated by the two compactness measures. Each of the resultng 642 compactness values are then rank normalzed producng the raw rank normalzed data for performng the hypothess testng. On each tral, for each rabbnc personalty, for each of the two compactness measures, we combne usng (1) and (3) over the key word sets assocated wth the rabbnc personalty. We use the normalzed rank method (5) of combnng the frst order and second order ranks together. As n (9) the mnmum of the resultng two compactness values s then the score for the tral and rabbnc personalty. On each of the 100,000 trals, these 53 scores are com- 2 We are only countng those whch have at least one ELS for each key word n the set.

5 bned usng the g combnng method usng the exp functon. (6) The relatve rank of the g-score for the frst tral s the p-value assocated wth the test of the Null hypothess aganst the alternatve that more of the 53 rabbnc personaltes have ELSs from one or two of ts key word sets n a more compact arrangement by both the 1D and the dstance compactness measure than expected by chance. The p-value of ths experment was For reference purposes the p-value of an dentcal experment usng the WRR lst 1 was and for WRR lst 2 was None of the WRR lsts produced a sgnfcant p-value on the Hebrew translaton of the War and Peace text. 5. The McKay Demonstraton McKay[2] tred to llustrate that the successful experment of WRR[3] could be re-enacted n the Hebrew translaton of War and Peace f one fddled and wggled enough n makng changes n spellngs (ncludng ncorrect spellngs) and choces of appellatons (ncludng ncorrect appelatons). Indeed, ther demonstraton n the War and Peace text produced a p-value of about 1/1,000,000 usng the same protocol as WRR. Ther concluson was that the success of the WRR experment was due to choce n the nput data of appellatons and dates and not due to a genune ELS phenomena n Geness. There s no space here to explan the varous techncal problems wth the McKay et. al. paper. We just want to note that the McKay data set of appellatons for War and Peace produces a p-value of n an experment of 10,000 trals wth exactly the same expermental protocol as employed n our experment n the Geness text. Clearly, the McKay data set does not produce a sgnfcant p-value n the Geness text. There s an mportant nterpretaton that one can make from these results: there s a structural/geometrc dfference between the ELS arrangements of the McKay appellatons n War and Peace, whch do not consttute any encodng, from that of the WRR appellatons n Geness, whch are hypotheszed to be an encodng. It follows from ths result that the protocol used by WRR and McKay was not senstve enough to detect ths geometrc dfference. Or sayng ths another way, the nherent false alarm rate wth the WRR protocol s hgher than wth our protocol. That was the reason McKay was able to make hs demonstraton succeed. 6. The Amercan Presdents In ths secton we report on an experment parng the names of the Amercan presdents, translterated nto Hebrew wth the key word `sp, meanng presdent. There are 42 people who have served as presdents, some multple tmes. Due to the varous ways non-hebrac names can be spelled n Hebrew, we devsed a rule base system to provde a reasonable set of Hebrew spellngs for each presdent s name. The rule base s descrbed n the appendx. As an example, the name Lncoln s translterated as olewpl, olewpl, olwpl, or olwpl. In addton we use two varatons: the last name alone and the frst character of the presdent s frst name as a prefx to the spellng of the last name. The total number of spellngs havng ELSs was 248, on the average nearly sx spellngs per name. The p-value usng the dentcal protocol as n secton 4 wth 1000 trals was not sgnfcant. We performed a second experment usng just the compactness measure defned by R 12 followed by ψ mn followed by Ψ harm. The p-value was Ths ndcated that somethng nterestng was happenng. So we explored further. We examned the dstance measure formed by Ω 2 followed by ψ mn followed by Ψ harm n a 100,000 tral experment. Ths s a compactness measure reported on at the Internatonal Torah code conference a few years ago. Wth ths compactness measure and our protocol, the Amercan presdent experment tests the Null hypothess of No Torah Code Effect aganst the complex alternatve hypothess that 1. n accordance wth the Hebrew to Englsh translteraton rules of the appendx (secton 8) 2. and n accordance wth the skp specfcaton, and the resonance specfcaton stated n secton 4 3. for nearly all the presdents 4. each presdent has one or two Hebrew spellngs of hs name 5. that have ELSs whch are n a more compact arrangement wth ELSs of the Hebrew word `sp, meanng presdent 6. n the 5 books of the Chumash 7. by compactness measure Ω 2 followed by ψ mn followed by Ψ harm In a 100,000 tral experment, the resultng p-value was , the smallest p-value possble. Clearly, the experment has to be repeated wth more trals to get a better estmate of the p-value. At ths pont we have done three experments. By Bonferron, we can only bound the true p-value to be less than 3/200,000=1/66, Concludng Dscusson We have dscussed expermental protocol possbltes by whch an experment can be done that tests the Null hypothess of No Torah Code Effect aganst a composte alternatve. The composte alternatve s that more of the 53

6 rabbnc personaltes have ELSs of ther key word sets n a more compact arrangement by both a 1D compactness measure and a dstance compactness measure than expected by chance. For ths purpose we developed a score functon based on a probablty dervaton of what the probablty would be f one or f one or two of a fxed number of choces follows a gven probablty functon whle the remanng follow a dscrete unform probablty functon. For varous reasons, that we dd not dscuss due to space lmtatons, our methodology s more conservatve than that employed by WRR[3]. We performed a 100,000 tral experment that took more than 36 hours on an AMD 64 X processor. The experment produced a p-value of.5/100,000. It s clear that the Null hypothess of No Torah Code Effect has to be rejected. The resultng p-values were so small that a 15 day experment of 1,000,000 trals needs to be done to get better estmates of how small they really are. The protocol used n ths experment was developed (traned on) the WRR data set. The protocol s drect, statstcally motvated, self normalzng, consstent wth the nature of the alternatve hypothess, and (n our opnon) aesthetcally smple. No part of the protocol has large numbers of varables or parameters whose values can be set to memorze the pattern of the ELS data from the Torah text versus that from the monkey texts. The parameters of the protocol tself were three: maxmum skp set so that the expected number of ELSs was about 10; the maxmum row and column skp of an ELS on a cylnder was 10. The probablty threshold was.2. The rest of the freedom n the protocol came from methodologcal choces: the monkey text populaton, the compactness measures, the varous rank normalzatons, the combnng method over key word sets of an event, the combnng method over scores of events. For our future work, we wll be applyng ths protocol to new data sets. 8. Appendx: Translteraton of Englsh Names Into Hebrew B a P t C (see) v Q w C, ck (kay) w R x D c S (ess) q F t S (zee) q, f G b Sh s H d T h J b Th, Ta z K w V (next to long vowel) a Kn p V (next to short vowel) e L l W e M n X qw N p Z f Table of translteraton of Englsh consonants nto Hebrew consonants Englsh Long Hebrew Short Hebrew Vowel Vowel Vowel Cake ` cat - A Hayes, Buchanan ` Taylor Adams, ` Reagen Arthur ` Taft, ` Grant ` Carter ` E seek, bead set - feld, Perce Jefferson - bke, ` Fllmore I Tyler, ` Madson Clnton Harrson Nxon Wlson boat, rose e Wlson, e Roosevelt e Clnton e O Polk e U Truman, e pup, e Hoover e Roosevelt e Table of translteraton of Englsh vowels nto Hebrew consonants Here we gve the prncples by whch the Englsh names of the presdents were translterated nto Hebrew n all the possble forms. The translteraton of the consonants are shown n the frst table and the translteraton of the long and short vowels are shown n the second table. The rule we used s to translterate each name wth every combnaton of the vowels formed by keepng or omttng the Hebrew wowel. Hence a name wth two vowels wll have four possble translteratons. References [1] R. Haralck. Testng the torah code hypothess: Basc concepts. In ICPR, [2] B. McKay, D. B. Natan, M. B. Hllel, and G. Kala. Solvng the bble codes puzzle. Statstcal Scence, pages , May [3] D. Wtztum, E. Rps, and Y. Rosenberg. Equdstant letter sequences n the book of Geness. Statstcal Scence, 9(3): , August 1994.