Engenharia Informática e de Computadores

Transcription

1 Classical Checkers Jã Carls Crreia Guerra Dissertaçã para btençã d Grau de Mestre em Engenharia Infrmática e de Cmputadres Júri Presidente: Orientadr: Vgais: Dutr Pedr Manuel Mreira Vaz Antunes de Susa Dutra Maria Inês Camarate de Camps Lynce de Faria Dutr Faust Jrge Mrgad Pereira de Almeida Dutr Rui Filipe Fernandes Prada Nvembr 2011

2

3 Once the game is ver, the king and the pawn g back in the same b. Italian Prverb

4

5 Abstract Fr the last 60 years, games have been an imprtant research subject in the field f Artificial Intelligence. Fr sme time the fcus was mainly n creating strng cmputer Chess prgrams. With the rise f master-level game-playing prgrams the trend changed. Nw, the fcus is n determining which games can effectively be slved and n slving them. The recent breakthrugh with Mnte-Carl Tree Search (MCTS) techniques in cmple games, such as G and Amazns, als led the research away frm alphabeta. Hwever, MCTS has nt yet beaten alpha-beta in mre simple games, such as Checkers and Chess. In this dissertatin, we describe a cmputer prgram created t play the game f Classical Checkers, Turska, and, fr the first time, we estimate and analyse the cmpleity f Classical Checkers. Additinally, sme eperiments were made with the prgram. The results are analysed in rder t better understand hw each ne f the studied search algrithms and techniques behave in the dmain under study, and hw they can be imprved. Given that the enhancements that take advantage f transpsitins were the nes that accunted fr mre savings, we tried t imprve them. We did s by trying t imprve the infrmatin stred in the transpsitin table and hw it was managed. The prgram was evaluated by playing sme matches against anther Classical Checkers cmputer prgram. In shrt, the results indicate that Turska s evaluatin functin needs further wrk. This is re-stated in the future wrk, alng with ther research suggestins. Keywrds Classical Checkers Game Playing Adversarial Search Alpha-Beta Search

6

7 Resum Ns últims 60 ans, s jgs, na sua generalidade, têm sid um imprtante tema de pesquisa na área da Inteligência Artificial. Durante muit temp bjectiv era criar prgramas de cmputadr que jgassem bem Xadrez. Cm surgiment de prgramas que jgam a um nível bastante elevad a tendência mudu. O fc está agra em determinar que jgs pdem ser reslvids e em reslvê-ls. Os recentes avançs cm métd de Mnte-Carl em prcura (MCTS) em jgs mais cmples, cm G e Jg das Amaznas, clcaram a prcura alfa-beta em segund plan. N entant, MCTS ainda nã bateu a prcura alfa-beta em jgs mais simples, cm Jg das Damas e Xadrez. Nesta dissertaçã descrevems um prgrama de cmputadr, denminad Turska, criad para jgar a variante clássica d Jg das Damas. O prgrama fi também utilizad para, através de algumas eperiências, se cmpreender melhr cm cada um ds algritms e técnicas de prcura estudads se cmprtam n dmíni em estud, e cm pdem ser melhrads. Tend em cnta que as técnicas que tiram partid de transpsições fram as respnsáveis pr mais e maires ganhs, tentáms melhrá-las. Pela primeira vez, a cmpleidade d jg é estimada e analisada. Para avaliar nss prgrama jgáms alguns jgs cntra utr prgrama. Os resultads indicam que a funçã de avaliaçã d Turska precisa de ser mais trabalhada. Esta bservaçã é reafirmada nas sugestões de trabalh futur, juntamente cm utras sugestões de investigaçã. Palavras-Chave Jg das Damas Jgs Prcura Adversarial Prcura Alfa-Beta

8

9 Preface Fr the past year r s this thesis has been a cnstant part f my life. Nw that it is finished, I wish t thank several peple fr helping and supprting me thrughut the prcess. First f all, I wuld like t thank my supervisr, prfessr Inês Lynce, fr guiding me thrugh the whle prcess and fr cmmenting n the prgress f the thesis every week. I wuld als like t thank prfessrs Rui Prada and Faust Almeida fr their many cmments and suggestins, which have greatly enhanced the quality f this dissertatin. Furthermre, I wuld like t epress my gratitude twards Veríssim Dias fr being available t discuss and share sme f his Checkers knwledge, and fr prviding feedback n Turska. Additinally, I want t thank my fellw friends and clleagues, wh were als ding their master s theses, especially Miguel Miranda, fr being a reliable beta tester and fr the several discussins n research in general. Mrever, I thank my friend Céline Csta fr helping me find a particularly nasty bug. I als want t thank Liliana Fernandes and Tiag Ferreira fr reading drafts f this dissertatin. Their cmments and suggestins helped imprve the readability f sme parts f this dcument. In additin, I wuld als like t thank Catarina Crreia Rcha fr always being there when I needed a gd friend t talk t, and fr helping me take my mind ff wrk when I needed t. Last, but nt least, I wuld like t thank my family fr their cntinuus supprt during the whle studies. Acknwledgements The research reprted in this dissertatin is supprted, in part, by Institut de Engenharia de Sistemas e Cmputadres, Investigaçã e Desenvlviment em Lisba (INESC-ID), with financial supprt frm the Fundaçã para a Ciência e a Tecnlgia (FCT). The research has been carried ut under the auspices f the SAT Grup at INESC-ID (Sftware Algrithms and Tls fr Cnstraint Slving). Jã Guerra Lisbn, Nvember 2011

10

11 Cntents 1 Intrductin Prblem Statement and Cntributins Dcument Outline Games and Adversarial Search Classical Checkers Game Prperties Adversarial Search Minima Alpha-Beta The Minimal Tree Imperfect Decisins Search Enhancements Transpsitin Tables Killer and Histry Heuristics Iterative Deepening Search Windw Reductins Selective Search Search Instability Slving Games Game Cmpleity Case Study: Chink English Checkers The Wrld Man-Machine Checkers Champin Search Algrithm i

12 3.2.2 Search Etensins Evaluatin Functin Bk Knwledge Endgame Databases Draw Differentiatin Slving Checkers Prf Tree Manager Prf Slver Cmpleity f Classical Checkers State-Space Cmpleity Game-Tree Cmpleity Cmparisn with Other Games Search in Turska The Test Set Transpsitin Tables NegaScut Iterative Deepening Aspiratin Search Enhanced Transpsitin Cutffs Replacement Schemes fr Transpsitin Tables Infrmatin in Transpsitin Tables Other Implementatin Details Summary Knwledge in Turska Evaluatin Functin Static Evaluatin Eample Eperimental Evaluatin Quiescence Search Results Against Other Prgrams ii

13 7 Cncluding Remarks Cnclusins Future Wrk A The Rules f Checkers in Brief 57 B Prf-Number Search 59 B.1 AND/OR Trees B.2 Prf-Number Search Bibligraphy 63 iii

14 iv

15 List f Figures 2.1 The pssible lines f play frm a certain Tic-Tac-Te game psitin (a) and its minima tree (b) Value prpagatin in game trees The alpha-beta algrithm A game tree anntated with the final values f α and β. In each cutff, the type is indicated. The principal variatin is shwn with a thicker line The structure f the minimal tree. The principal variatin is shwn with a thicker line The negama versin f the (heuristic) alpha-beta algrithm The alpha-beta algrithm using a transpsitin table An enhanced transpsitin cutff The NegaScut algrithm The prf prcedure Cmparisn f the ttal nde cunts fr each search eperiment in the pening Cmparisn f the ttal nde cunts fr each search eperiment in the middle game Cmparisn f the ttal nde cunts fr each search eperiment in the endgame Identifying bridge eplits (a). The river (b) and the triangle frmatins (c) A game psitin A.1 Several situatins in Classical and English Checkers B.1 An AND/OR tree anntated with its prf and disprf numbers B.2 The Prf-Number Search algrithm v

16 vi

17 List f Tables 3.1 The fudge heuristic The state-space cmpleity f Classical Checkers The average game length and average branching factr estimatins fr Classical Checkers The cmpleity f Classical Checkers cmpared t that f ther games Alpha-beta vs alpha-beta using a transpsitin table Alpha-beta vs NegaScut NegaScut vs NegaScut with iterative deepening NegaScut vs NegaScut with aspiratin search NegaScut vs NegaScut with enhanced transpsitin cutffs Cmparisn f several replacement schemes fr transpsitin tables String additinal infrmatin in the transpsitin table Game phases in Turska Turska s heuristics and their values Cmparisn f evaluatin functins Cmparisn f quiescence searches Turska vs Windamas vii

18 viii

19 List f Abbreviatins α-β AS ETC ID MT MTD NS PNS PV PVS TNC TT Alpha-Beta Aspiratin Search Enhanced Transpsitin Cutffs Iterative Deepening Memry-Enhanced Test Memry-Enhanced Test Driver Framewrk NegaScut Prf-Number Search Principal Variatin Principal Variatin Search Ttal Nde Cunt Transpsitin Table i

20

21 1 Intrductin Games have been a leisure fr human beings almst as lng as civilisatin eists. Nt nly are they fun t play, but they can als be challenging and cmpetitive. Games have als been knwn t stimulate the human brain and help in the develpment f mathematical reasning. Because f this, certain games, such as Checkers r Chess, have been taught in schls in sme cuntries. In the past centuries, sme games, mst ntably Chess, have gained a certain level f status and even a place in culture. Thus, there is a stimulus fr peple t ecel in the games they like. Chess is the mst etreme eample, with lts f cmpetitins and champinships happening regularly all arund the glbe. The reputatin fr wning a Chess title is very high. Althugh in general mst games cntinue t be played as a leisure, fr sme peple it has becme their prfessin, the best eample being Pker. Given the increased interest and cmpetitiveness, sme games are being studied thrughly. This led t an increased literature abut sme games. Additinally, ver the past years the use f cmputers as a tl t study games has been increasing. In the early 1950s, advances in the fields f cmputer science and artificial intelligence raised a special interest in games. The hypthesis f creating artificial players fr games had just becme real. A large number f game-playing prgrams has emerged ever since. A small number f games has been slved, that is, cnsidering that the players play the best they can, the utcme f the game has been determined. Althugh in sme games the artificial players have challenged and even cnquered the human supremacy, in thers they are n match fr humans. The first cmputer prgram t fficially cnquer the human supremacy was Chink. Chink became the first cmputer prgram t win a champinship title frm a human back in The game in questin was English Checkers. Games are interesting research subjects fr several reasns: Games have a clsed dmain with well-defined rules, in cntrast t real-wrld prblems, which makes them easier t represent and analyse. Althugh mst games are easy t learn, they are difficult t master. Creating a strng game-playing prgram can smetimes prvide insights int hw peple reasn, as the way game eperts think is usually studied t devise knwledge that cmputers can understand. Games can be used t test new ideas in prblem slving. 1 Chink did nt actually wn the title frm the human. A new champinship was created t address cmpetitins between humans and cmputers, as the title f Wrld Champin is usually disputed slely between human beings. Instead, Chink was declared the Wrld Man-Machine Champin. 1

22 Usually, advances in game research are nly applicable t the game in study and t similar games. Only smetimes can the ideas, algrithms, and techniques be translated t ther dmains. 1.1 Prblem Statement and Cntributins In game-playing research, and search research in general, the bjective is t create a cmputer prgram that can autnmusly slve a given prblem. In ur case the prblem is t play the game f Classical Checkers. The main prblem addressed in this thesis can be stated as: Prblem Statement 1 Hw can we develp a cmputer prgram t play Classical Checkers? T answer this questin we will study alpha-beta, a game-playing methd that has been shwn t perfrm well in Checkers. Then, we will use Chink as a case study t better understand hw alpha-beta and enhancements are used in practice, as well as ther cmpnents, such as pening bks and endgame databases. Fllwing that, we will estimate the cmpleity f Classical Checkers and cmpare it t that f ther games, particularly f ther Checkers variants. T the best f ur knwledge, this is the first attempt at estimating the cmpleity f Classical Checkers. After building ur Classical Checkers cmputer prgram, Turska, tw ther questins will arise: Prblem Statement 2 Hw d the studied search algrithms and techniques behave in the dmain under study, and hw can they be imprved? Prblem Statement 3 What game-specific knwledge is required t imprve ur prgram? T answer the first questin we will cmpare several f the studied search algrithms and techniques. Additinally, we will eperiment several replacement schemes fr transpsitin tables, including a new ne, and investigate sme f the suggestins mentined by Breuker fr additinal infrmatin in the transpsitin table [8]. The research will always be perfrmed in the dmain f Classical Checkers. T answer the secnd questin we will gather sme specific knwledge abut the game, either by studying knwledge available in ther Checkers variants r by btaining feedback frm a Checkers epert, Veríssim Dias. We will evaluate the knwledge f ur prgram by running sme eperiments and by playing sme matches against anther prgram. 1.2 Dcument Outline In chapter 2 we present the game in which we fcus, Classical Checkers, alng with sme frmal backgrund behind games: sme f their features and categrisatins. Then, we intrduce the field f adversarial search, specialising in alpha-beta search. The chapter includes an eplanatin f bunds, the minimal tree, nde types, search enhancements, and prblems related with imperfect decisins in games. The chapter ends with the descriptin f sme relevant issues related with slving games and game cmpleity. The ntins intrduced in this chapter are used thrughut the dcument. 2

23 In chapter 3 we take a lk at Chink, a state-f-the-art English Checkers cmputer prgram, and at the prf prcedure used t weakly slve the game f English Checkers. Other cmpnents that are used in game-playing prgrams, such as pening bks and endgame databases, are intrduced as well. The game f English Checkers is als presented, alng with an verview f the histry f cmputer Checkers. In chapter 4 we estimate and analyse the state-space and game-tree cmpleities f Classical Checkers, which are then cmpared t the nes f ther games, particularly f ther Checkers variants. In chapter 5 and 6 we describe Turska s search and knwledge cmpnents, similarly t hw it was dne with Chink, in chapter 3. In chapter 5 we analyse the results f the eperiments made with several search algrithms and techniques. The purpse is t determine hw the enhancements behave fr different search depths and game phases, as well as the influence f the enhancements n the search perfrmance. In chapter 6 we describe the prgram s evaluatin functin and quiescence search, and cmpare sme versins f them. We cnclude the chapter by analysing the matches played between Turska and Windamas. We cnclude this dissertatin with chapter 7. We discuss the results btained, draw sme cnclusins, and discuss future wrk directins. In appendi A we eplain the rules f Checkers, in brief, and intrduce the standard bard ntatins fr bth Classical and English Checkers. In appendi B we describe AND/OR trees and the basic Prf- Number Search algrithm. The reading f appendi A is required t better understand chapter 6, and the reading f appendi B is advised befre prceeding with the secnd part f chapter 3. 3

24 4

25 2 Games and Adversarial Search Since the birth f Artificial Intelligence games have been ne f the main tpics f research in the field. We start by presenting the game in which we will fcus, Classical Checkers, alng with its prperties. Then, we intrduce a methd that allws cmputers t play games, minima, and an imprvement t it, alpha-beta. Net, we discuss sme techniques t enhance alpha-beta and finish by describing sme relevant issues related with slving games and game cmpleity. 2.1 Classical Checkers Checkers 1 is a kind f bard game played between tw players that invlves diagnal mves f unifrm pieces and mandatry leap captures. Checkers is believed t riginate frm Alquerque 2. The rules fr the game f Checkers can be cnsulted in appendi A. Classical Checkers, als knwn as Prtuguese r Spanish Checkers, is the mst played Checkers variant in Prtugal, Spain, and in sme cuntries in Nrthern Africa and Suthern America. This variant is cnsidered t be the ldest and thus is cnsidered as the riginal Checkers game 3. Sme cmputer prgrams eist fr this variant 4. Prfund, by Álvar Cards, and Windamas, by Jean- Bernard Alemanni, are recgnized as being the strngest [18] Game Prperties In game thery, games can have several prperties and categrisatins: Cperative (r calitinal) vs nn-cperative. The mst basic entity in any game mdel is the player. A player, r agent, may be interpreted as an individual r as a grup f individuals making a decisin. Once the set f players is defined, we may distinguish between tw types f mdels: cperative and nn-cperative. In cperative game mdels, players can frm calitins and are suppsed t be able t discuss the situatins and agree n a ratinal jint plan f actin, an agreement that shuld be assumed t be enfrceable. On the ther hand, in nn-cperative game mdels, it is assumed that each player acts independently, withut cllabratin r cmmunicatin with any f the thers. 1 Depending n the frm f the English language used, the term used t refer t the game may differ. Checkers is the term used in American English, whilst Draughts is the ne used in British English. In this dcument we will use the American term. In the literature, the terms Checkers and Draughts are ften incrrectly used t refer t the games f English Checkers and Internatinal Checkers, respectively. 2 As investigated by Arie van der Step, 3 As investigated by Arie van der Step, 4 A small survey can be fund at 5

26 Sequential (r dynamic) vs simultaneus (r static). In sequential games, each player chses his actins befre the thers can chse theirs, i.e. players take turns playing. An imprtant prperty f sequential games is that the latter players have sme infrmatin abut the actins taken by the frmer players. Simultaneus games are the cunterpart f sequential games in the sense that there is n sequentiality, and each player chses his actins withut any knwledge f the ther players actins. Deterministic vs stchastic. A game is deterministic if there is n randmness invlved in the game, e.g. the rll f a die. If a game includes a randm element, then it is stchastic. Perfect infrmatin vs imperfect infrmatin. A game is said t have perfect infrmatin if each player, when making any decisin, is perfectly infrmed f all the events that have previusly ccurred. Checkers is an eample f such game as each player can see all the pieces n the bard at any time. A game has imperfect infrmatin if it allws players t have nly partial r n infrmatin abut the actins taken previusly when making decisins. Zer-sum vs nn-zer-sum. Infrmally, in a zer-sum game, the gain r lss f a player, at any mment in the game, is balanced by the lsses and gains f the ther players, at that same mment. This means that if the gains and lsses f all players are summed, the result will be equal t zer 5. In nn-zer-sum games, the result f the sum will be different frm zer. Game thery and many f the previus cncepts were first intrduced and develped by vn Neumann and Mrgenstern [61]. Since then, game thery has been a field under cnstant develpment. Checkers is a tw-player deterministic zer-sum nn-cperative sequential game with perfect infrmatin. Games with these prperties are called abstract strategy games and are studied within cmbinatrial game thery [16, 4], which is a branch f game thery. 2.2 Adversarial Search Humans play games by cnsidering different cmbinatins f pssible mves fr themselves and pssible replies, r cunter-mves, by their ppnents. Each legal cmbinatin f mves and replies is a pssible line f play. The mst prbable line f play is called the principal variatin (PV). Cmputers try t mimic humans by systematically analysing the different lines f play and building a game tree with that infrmatin. As an eample, let us lk at Tic-Tac-Te. Figure 2.1(a) shws the pssible lines f play frm a certain Tic-Tac-Te game psitin. The rt f the tree is the current psitin, the ndes are legal psitins, and the branches are legal mves. Leaf ndes are called terminal ndes and represent psitins where the game ends. Cnsidering the pssible lines f play t find the best ne is usually referred t as searching the tree and a mve by ne player is called a ply. 5 Assuming that the gain f a player is represented by a psitive number and the lss by a negative ne. 6

27 draw lss draw lss lss draw win draw win lss draw win draw win (a) Ma 0 a Min b -1 c 0 d -1 Ma Min e -1 f g h i j k l m n (b) Figure 2.1: The pssible lines f play frm a certain Tic-Tac-Te game psitin (a) and its minima tree (b) Minima In tw-player zer-sum nn-cperative sequential games with perfect infrmatin, the first player is called Ma and the secnd, Ma s ppnent, is called Min. A minima tree is a game tree where each nde is anntated with its minima value. The minima value f a psitin is the utility f that psitin fr Ma. The utility, r payff, f a psitin is a numeric value that represents the best pssible game utcme frm that psitin fr a certain player. Fr eample, in Tic-Tac-Te, as in Checkers, there are three pssible utcmes fr a game: win, lss, r draw, with utilities +1, 1, and 0, respectively. There are games with a wider variety f pssible utcmes. Figure 2.1(b) shws the minima tree fr ur 7

28 Tic-Tac-Te eample. Ndes where it is Ma s turn t play are depicted with squares, ndes where it is Min s turn t play are depicted with circles. The minima value f a psitin can be cmputed with the fllwing frmula: Minima(psitin) = Utility(psitin) Maimum p Successrs(psitin) Minima(p) Minimum p Successrs(psitin) Minima(p) if psitin is terminal, if psitin is a Ma psitin, if psitin is a Min psitin. Minima uses backward inductin t cmpute the value f a psitin. It starts by cmputing the values f the terminal psitins and then backs up these values using maimum and minimum peratins. When all psitins have been analysed, we have the minima value f the tree (n.b. the minima values f the tree and f the rt are the eact same thing). Figure 2.1(b) shws an eample f a minima tree with value 0. The principal variatin is shwn with a thicker line. Nte that the minima value f a tree is a functin f its leaves and nt f the rt. The principal variatin is the path frm the rt t the leaf that gives the rt its value. There may be mre than ne principal variatin as it is pssible that mre than ne leaf prpagates its value back t the rt. Minima assumes that bth players play ptimally, i.e. every player always tries t maimize its utility. If ne f the players des nt play ptimally, then the ther shuld perfrm as gd r even better. Althugh there may be ther strategies that perfrm better than minima against subptimal, r fallible, ppnents, we will cnsider that the ppnent always plays ptimally. Ntice that Ma s gal is t maimize its wn utility, whilst Min s gal is t minimize Ma s utility. T btain the utility f a psitin fr Min we nly have t negate its value fr Ma, due t the zer-sum prperty. In fact, the names f the players cme frm this relatin between their utilities. It is als this ppsitin between the players utilities that makes the situatin adversarial. The name minima cmes frm the fact that each player maimizes its wn utility while minimizing the ppnent s utility at the same time. Althugh minima was first intrduced by Zermel [63], nly years later was the minima therem crrectly described [61]. If we take advantage f the zer-sum prperty f games, we can refrmulate minima slightly differently. This new frmulatin is called negama [28]: Negama(psitin) = Utility(psitin) Maimum p Successrs(psitin) Negama(p) if psitin is terminal, therwise. We shuld make clear that when using the minima frmulatin Utility returns the utility f a psitin fr Ma, whilst when using the negama frmulatin it returns the utility f the psitin relative t the side playing; therwise the frmulatins are nt equivalent. 8

29 a m1 m2 b c... d... y Figure 2.2: Value prpagatin in game trees. Befre cntinuing we shuld pint ut that we have been using the term game as it meant abstract strategy game. As in this dcument we are nly cncerned with abstract strategy games, mre particularly the game f Classical Checkers, the algrithms and techniques we are ging t present are intrduced in this cntet. Nevertheless, sme f the algrithms can be etended t ther kinds f games. Fr the subset f games that can be represented with game trees, adversarial search is ften referred t as game-tree search Alpha-Beta T find the value f a game tree, minima eamines all f its ndes. In mst cases, this is nt nly infeasible due t the massive number f ndes that may eist in a game tree, but als nn-ptimal because there are many ndes that d nt affect the value f the rt. Alpha-beta (α-β) [28] 6 is an imprvement t the minima algrithm that can pssibly avid the eaminatin f the entire game tree. The idea is nt t analyse parts f the tree that d nt affect the value f the rt. This technique is called pruning. Cnsider the eample in figure 2.2. After finding the utility f the first mve, m1, Ma shuld nly be cncerned with mves that have a utility greater than, as he is trying t maimize his utility. Nw cnsider the situatin where Ma makes a secnd mve, m2, and the first reply f Min returns an utility y, such that y. Since Min is trying t minimize Ma s utility, the eventual utility f m2 will be at mst y. Hence, Ma will never play m2 since it already has a better ptin, m1. In ther wrds, nde a, a Ma nde, has impsed a lwer bund n its child c, a Min nde. Cnversely, a Min nde can impse an upper bund n its Ma children. Because f this, the alpha-beta algrithm intrduces tw bunds: The lwer bund, α, equals the highest minima value fund s far, i.e. the minimum value that Ma is guaranteed t have. The upper bund, β, equals the lwest minima value fund s far, i.e. the maimum value that Min is guaranteed t have. The alpha-beta algrithm is shwn in figure 2.3. dentes a value that is lwer than r equal t any pssible utility fr a terminal nde, and + a value that is greater than r equal t any pssible 6 The early histry f α-β is smewhat bscure. Knuth and Mre prvide an verview f this histry [28]. 9

30 Alpha-Beta(nde, α, β) if nde is terminal then return Utility(nde) if nde is MAX then utility = fr each nde n Children(nde) d utility = Ma(utility, Alpha-Beta(n, α, β)) if utility β then // β cutff break α = Ma(α, utility) else // nde is MIN utility = + fr each nde n Children(nde) d utility = Min(utility, Alpha-Beta(n, α, β)) if utility α then // α cutff break β = Min(β, utility) return utility Figure 2.3: The alpha-beta algrithm. utility fr a terminal nde. The interval between α and β is called the search windw. If the value f a nde falls utside the search windw, the crrespnding subtree is pruned. After searching a nde using the alpha-beta algrithm, ne f three things can happen [20]: 1. utility α. The result is belw the search windw (failing lw). 2. utility β. The result is abve the search windw (failing high). 3. α < utility < β. The result is within the search windw (success). If the value f a nde falls utside the search windw, it represents a bund n its minima value, therwise the value f the nde has been fund. Success can be assured by calling alpha-beta with the full windw, i.e. [, + ]. The versin f alpha-beta shwn in figure 2.3 is called fail-sft alpha-beta [20] because it allws the search t return values that lie utside the search windw (bunds) when the search fails, either lw r high. Other versins f alpha-beta return α r β in the same situatins. In this dcument, we will nly cnsider the fail-sft versins f the algrithms presented. The reasn fr chsing these versins will be given later. Figure 2.4 shws a game tree anntated with its final α and β values. The alpha-beta algrithm updates the values f α and β as it traverses the tree (frm left t right) and prunes away branches with values knwn t be wrse than the current values f α r β. The initial values f α and β are and +, respectively. Frm this eample, it is pssible t see that the effectiveness f the algrithm is highly dependent n the rder in which ndes are eamined, e.g. if nde h was eamined befre nde g, there wuld have been n cutff in the subtree rted at h. Later we will see sme techniques t imprve the rder in which ndes are eamined, and thus increase the chance f cutffs. Such techniques are called mve rdering techniques. 10

31 α: - / β: + 0 / + a b - / -7 c -7 / 0 d 0 / + α e f g h -7 / 0 i j β -1 1 k l 4 7 Figure 2.4: A game tree anntated with the final values f α and β. In each cutff, the type is indicated. The principal variatin is shwn with a thicker line The Minimal Tree The minimal tree, r critical tree, is the minimal part f a game tree necessary t determine the minima value f the rt. Thus, a game tree may have several minimal trees. The minima value f the rt depends nly n the ndes f the minimal tree, the remaining ndes d nt affect its value. Fr a game tree with fied depth d and unifrm branching factr w, the number f leaf ndes is w d, and the number f leaf ndes f its minimal tree is w d/2 + w d/2 1 [28]. Althugh the minimal tree is a significant imprvement ver the minima tree, its size is still epnential with respect t its depth. In an alpha-beta minima tree, we can identify three types f ndes [28, 32]: PV, r type-1. The rt f the tree is a PV nde. At a PV nde all the children have t be eplred. Furthermre, at least ne f the children gives the parent its value. Pick ne f such children. That child is a PV nde, all the thers are CUT ndes. CUT, r type-2. At a CUT nde there is a cutff. The child that causes the cutff is an ALL nde. If there are several such children, pick ne arbitrarily. In a perfectly rdered game tree nly ne child f a CUT nde needs t be eplred. A perfectly rdered game tree is a game tree in which the best mve is searched first and a cutff ccurs immediately after, at CUT ndes. ALL, r type-3. At an ALL nde all the children have t be eplred. Every child f an ALL nde is a CUT nde. Befre searching a nde we d nt knw its type. We can nly knw the type f a nde after it has been eplred. Thus, befre searching a nde we refer t it as epected-pv, epected-cut, r epected-all nde. If there is n cutff at an epected-cut nde, then the nde becmes an ALL nde. If at least ne child f an epected-all nde turns ut nt t be a CUT nde, then the epected-all nde becmes a CUT nde. If all the epected-cut ndes n a path frm the rt t a leaf becme ALL ndes, a new principal variatin emerges (all the ndes n the path have becme PV 11

32 pv pv cut cut pv cut cut all all pv cut cut all all cut cut cut cut cut cut Figure 2.5: The structure f the minimal tree. The principal variatin is shwn with a thicker line. ndes). Figure 2.5 shws the structure f the minimal tree. Every child f a PV r ALL nde is a part f the minimal tree, but nly ne f the children f a CUT nde is Imperfect Decisins Fr the game f Tic-Tac-Te, the game tree is small enugh s that it can be entirely eplred within a reasnable amunt f time. In mst games, this is nt pssible due t the massive size f the game tree. Thus, the search must stp at sme pint. When the game tree is t large t be generated cmpletely, a search tree is generated instead. Instead f stpping nly at terminal psitins, the search can nw stp at any psitin. Using this new apprach, we may nt be able t identify the utility f a psitin, and s we resrt t an evaluatin functin. An evaluatin functin returns a heuristic assessment f a psitin, i.e. an estimate f the real utility f a psitin. Fr terminal psitins the evaluatin functin shuld return the true utility, fr nn-terminal psitins the value returned shuld be strngly crrelated with the actual chances f winning. The range f values returned by an evaluatin functin is usually significantly wider than just win, lss, r draw. This allws fr a better discriminatin between mves. The value returned by an evaluatin functin is called scre. Evaluatin functins are necessarily applicatin dependent. In Checkers, they typically include features such as material balance, piece mbility, and psitinal cntrl, amngst thers. The new versin f the (heuristic) alpha-beta algrithm is shwn in figure 2.6. This versin uses the negama frmulatin. As with utility functins, if an algrithm uses the negama frmulatin, the evaluatin functin must return the scre relative t the side playing; therwise the algrithm will be incrrect. Ntice als that fr the algrithm t be crrect the values f α and β have t be swapped and negated at each recursin. This is a cnsequence f using the negama frmulatin. Heuristic Search Prblems One f the prblems with heuristic search is hw t decide when t stp the search and apply the evaluatin functin. Usually the search depth is used as the primary criterin. Anther cmmnly used 12

33 Alpha-Beta(nde, α, β, depth) if nde is terminal r depth == 0 then return Evaluate(nde) // r perfrm a quiescence search scre = fr each nde n Children(nde) and scre < β d scre = Ma(scre, Alpha-Beta(n, β, α, depth 1)) α = Ma(α, scre) return scre Figure 2.6: The negama versin f the (heuristic) alpha-beta algrithm. criterin is time cntrl stpping the search when the time run uts. A majr depth-related prblem is the hrizn effect [5]. The hrizn effect arises when sme negative event is inevitable but pstpnable. Because nly a part f the game tree has been analysed, it will appear that the event can be avided when in fact it cannt. As the event is nw ver the hrizn (beynd the depth f search), it is n lnger a prblem. Thus, the scre reprted fr the mve will nt reflect its eventual utcme. One way f mitigating the hrizn effect is t increase the search depth. Anther ne is t perfrm a quiescence search [54] when the maimum search depth is achieved, instead f immediately evaluating the psitin. A quiescence search etends the search at a leaf psitin until a stable psitin is reached. A stable, r quiet, psitin is ne that has a scre that is unlikely t ehibit wild swings in the near future. Fr eample, in Checkers, a psitin in which the ppnent is threatening t capture several pieces is unstable. It wuld nt make sense t stp the search in such a psitin, as the scre returned by the evaluatin functin wuld nt take the lss f the pieces int accunt, as the capture is ver the hrizn. In general, the quality f play imprves with the depth f search, at least up t sme pint (see, e.g., [26]). Hwever, there are sme games where the ppsite is true. This phenmenn is called search pathlgy [33] and is nt bserved in mst cmmn games, such as Checkers, Chess, and Othell. 2.3 Search Enhancements The perfrmance f alpha-beta depends largely n the rder in which psitins are eamined. In the wrst case alpha-beta eamines the entire game tree, while in the best case nly the minimal tree is eplred. We will discuss sme techniques t enhance alpha-beta in rder t bring it clser t the theretical limit Transpsitin Tables In games, different sequences f mves may lead t the same psitin. Such sequences are called transpsitins. Thus, game trees are nt really trees but instead graphs, as sme psitins appear in varius places thrughut the tree (e.g. in figure 2.1 ndes k and m represent the same psitin, as well as ndes l and n). If we detect these repeated psitins, we can eliminate the redundant effrt that is wasted by re-searching them. A transpsitin table (TT) [22] serves as a cache f recently eplred psitins. 13

34 Alpha-Beta(nde, α, β, depth) * ld-α = α * tt = Retrieve(nde) * if tt.depth depth then * if tt.flag == lwer-bund then α = Ma(α, tt.scre) * else if tt.flag == upper-bund then β = Min(β, tt.scre) * else /* tt.flag == eact-value */ return tt.scre * if α β then return tt.scre if nde is terminal r depth == 0 then return Evaluate(nde) * if tt.mve then // try the mve frm the transpsitin table first * scre = Alpha-Beta(tt.mve, β, α, depth 1) * mve = tt.mve * α = Ma(α, scre) else scre = mve = null fr each nde n Children(nde) \ tt.mve and scre < β d temp = Alpha-Beta(n, β, α, depth 1) if temp > scre scre = temp mve = n α = Ma(α, scre) * if scre ld-α then flag = upper-bund * else if scre β then flag = lwer-bund * else /* ld-α < scre < β */ flag = eact-value * Stre(nde, flag, scre, depth, mve) return scre Figure 2.7: The alpha-beta algrithm using a transpsitin table. Fr each psitin, the table stres infrmatin such as the psitin scre, the search depth at which it was encuntered, and the best mve. Befre searching a psitin, the table is cnsulted t see if it has been already eplred. If s, the infrmatin stred may be sufficient t stp further searching at this psitin (cause a cutff). If the infrmatin is insufficient t cause a cutff, then the search windw may be narrwed and the best mve frm a previus search can be cnsidered first. Since the mve was cnsidered the best befre, there is a gd prbability that it is still the best. Figure 2.7 shws the alpha-beta algrithm fr use with a transpsitin table. The transpsitin table cde is marked with an asterisk (*). Details cncerning the retrieval, strage, and replacement f infrmatin are mitted fr clarity. Breuker has published an etensive study n transpsitin tables and their use [8]. Zbrist Hashing Transpsitin tables are usually implemented as hash tables. A cmmnly used hash functin in gameplaying prgrams is the ne prpsed by Zbrist [64]. In Checkers there are 4 different pieces (2 kinds 2 clurs) and 32 squares. Fr every cmbinatin f piece and square a pseud-randm number is 14

35 generated. This gives us a ttal f 128 pseud-randm numbers. Additinally, there is an etra pseudrandm number that indicates if it is the secnd player s turn t play. T cmpute the hash value f a psitin, we d an eclusive-r (XOR) peratin with the pseud-randm numbers assciated with that psitin. This methd allws us t cmpute the hash value f a psitin incrementally. Fr eample, in Checkers, given the hash value f a psitin, we can btain the hash value f ne f its successrs easily. Fr that, we nly need t d an XOR peratin with the pseud-randm numbers assciated with the crrespnding mve, i.e. the square frm which the piece is mving and the square t which the piece is mving, plus the squares f the captured pieces, if there are any. T und a mve we prceed similarly. If we implement a transpsitin table as a hash table, we have t tackle with tw types f errrs [64]. A type-1 errr ccurs when tw different psitins have the same hash value. Type-1 errrs happen because the number f hash values is usually much lwer than the number f psitins in the game. This can intrduce search errrs, as the infrmatin stred in the transpsitin table can be used fr the wrng psitins. We can cmpletely avid this type f errrs by string the whle psitin in the transpsitin table. Hwever, in many games this takes up t much memry, and thus is nt used in practice. In practice, type-1 errrs are mitigated by using a hash key t distinguish between psitins with the same hash value. Because a transpsitin table s size is limited, we cannt stre all the psitins encuntered during the search. A type-2 errr, r cllisin, ccurs when a psitin is abut t be stred in the transpsitin table and the crrespnding entry is already ccupied. Cnsequently, a chice has t be made abut which psitin t keep in the transpsitin table. Such chices are based n a replacement scheme. Replacement schemes are discussed later in chapter 5. Enhanced Transpsitin Cutffs Often, when the infrmatin stred in the transpsitin table des nt cause an immediate cutff, the eplratin f the best mve des. Hwever, the subtree under the mve is still searched. If there was anther mve whse stred infrmatin culd cause an immediate cutff, then we wuld save search effrt, as n subtree wuld be eplred. Thus, befre searching any mve at a psitin, we check if the infrmatin stred fr any f the psitin s successrs can cause a cutff. If s, unnecessary search effrt has been avided. This technique is called enhanced transpsitin cutffs (ETC) [36]. Due t the increased cmputatinal effrt, ETC is usually nt perfrmed at all ndes, but just at the ndes that are fr eample mre than 2 ply away frm the hrizn. Figure 2.8 shws an eample f the kind f cutffs pssible with ETC. Cnsider nde c, whse infrmatin stred in the transpsitin table des nt cause an immediate cutff. The transpsitin table suggests that mve m shuld be tried first at c. If the eplratin f m causes a cutff, the ther mves will nt be eplred. Nw cnsider that ne f the ther children f c, nde f, transpses int anther part f the game tree, nde d, which has already been searched. If d s stred infrmatin is enugh t cause a cutff at c, nde e des nt need t be eplred. Befre searching any mve at a 15

36 a b c m d e... f Figure 2.8: An enhanced transpsitin cutff. nde, ETC accesses the infrmatin stred in the transpsitin table fr each ne f the nde s children lking fr transpsitins Killer and Histry Heuristics Often, during the search mst mves tried in a psitin are quickly refuted, usually by the same mve. The killer heuristic [1] invlves remembering, at each search depth, the mves that are causing mre cutffs (the killers). When the same depth is reached again in the game tree, the killer mves are retrieved and used, if valid. By trying the killer mves befre the ther mves, we increase the chance f cutffs. The mve searched just befre a cutff is usually referred t as the refutatin, because it refutes its predecessr. The histry heuristic [41] is a generalisatin f the killer heuristic. Instead f nly remembering the best killer mves at each depth, the histry heuristic maintains a recrd fr every mve seen s far. Each mve has an assciated histry scre that indicates hw ften it has been gd (caused a cutff r had the best scre). Mves are then rdered by their histry scre and eamined by this rder. Bth the killer and histry heuristics are applicatin-independent techniques t perfrm mve rdering. Transpsitin tables stre the eact cntet under which a mve was cnsidered the best. The histry heuristic recrds which mves are mst ften best, but has n knwledge abut the cntet in which they were cnsidered the best Iterative Deepening Iterative deepening (ID) [17, 55] is a technique used in many game-playing prgrams. The idea behind iterative deepening is that the best mve fr a (d 1)-depth search is likely t als be the best mve fr a d-depth search. Hence, a series f searches is perfrmed, each ne with increasing depth. At the end f each search, the mves are rdered based n their scres and this rdering is used in the net search. The ratinale behind iterative deepening is that a shallw search is a gd apprimatin f a deeper search. At each iteratin, the previus iteratin s slutin is refined. In cntrast, withut iterative deepening the search can g ff in the wrng directin and waste lts f effrt befre stumbling n the best line f play. Mve rdering can be further imprved if iterative deepening is used alng with transpsitin tables 16

37 and heuristics such as the killer and histry heuristics, as the previus searches fill the tables with useful infrmatin. Mve rdering is mre imprtant at the rt f the game tree. Cnsidering the best mves first narrws the search windw, increasing the chance f cutffs. Anther advantage f using iterative deepening is fr time cntrl. If the search is being cnstrained by time, when time runs ut, the best mve (up t the current search depth) is knwn. If there is time left, a deeper search can be started Search Windw Reductins The mst bvius way t imprve the effectiveness f alpha-beta is t imprve the rder in which psitins are eamined. If the mst relevant psitins are eamined first, the number f cutffs dne by alpha-beta may increase. Anther way f imprving alpha-beta s effectiveness is t reduce its search windw. If the search windw is smaller, the search will be mre efficient as the likelihd f pruning parts f the game tree increases. If the search windw is reduced artificially, there is a risk that alpha-beta may nt be able t find the scre. Hence, a re-search with the crrect windw may be necessary. In practice, the savings f smaller search windws utweigh the verhead f additinal re-searches (see, e.g., [13, 27]). Aspiratin Search Aspiratin search (AS) [10, 21] uses a search windw centred arund an epected value, [EV δ, EV + δ], where EV represents the epected value (the epected result f the search), and δ a reasnable range f uncertainty, r epected errr. Such windws are referred t as aspiratin windws. An aspiratin search can result in ne f three cases: 1. scre EV δ. The result is belw the aspiratin windw. Anther search, with windw [, scre], is required t find the actual scre. 2. scre EV + δ. The result is abve the aspiratin windw. Anther search, with windw [scre, + ], is required t find the actual scre. 3. EV δ < scre < EV + δ. The result is within the aspiratin windw and it was determined with less effrt than with a search using a full windw. Aspiratin search is usually cmbined with iterative deepening. The result f the (d 1)-depth search can be used as the epected value fr the d-depth search. Nte that δ is applicatin dependent. Aspiratin search is a gamble. If the result is within the aspiratin windw, then the enhancement wins, therwise an additinal search is needed. Aspiratin search is usually nly used at the rt f the game tree. Null-Windw Search The null-windw search [34, 20], als knwn as minimal-windw search r zer-windw search, takes the aspiratin search idea t an etreme: it uses a search windw f width ne 7, i.e. β α = 1, 7 Assuming that the smallest pssible difference between any tw values returned by the evaluatin functin is 1. 17

38 NegaScut(nde, α, β, depth) if nde is terminal r depth == 0 then return Evaluate(nde) first-mve = FirstChild(nde) scre = NegaScut(first-mve, β, α, depth 1) α = Ma(α, scre) fr each nde n Children(nde) \ first-mve and scre < β d temp = NegaScut(n, α 1, α, depth 1) if temp > α and temp < β and depth > 2 then // re-search temp = NegaScut(n, β, temp, depth 1) scre = Ma(scre, temp) α = Ma(α, scre) return scre Figure 2.9: The NegaScut algrithm. called null windw, minimal windw, r zer windw. A null-windw search always returns a bund n the scre. The algrithm assumes that the first mve cnsidered at each psitin is the best, and cnsequently, the remaining are inferir, until prven therwise. The algrithm uses a full r aspiratin windw t scre the first mve, then a null windw is used fr the remainder f the search. As with aspiratin search, a re-search may be necessary t find the actual scre. In the best-case scenari, the first mve cnsidered is really the best and all the thers will fail, either lw r high. The biggest advantage f using a null-windw search ver a full-windw search is the pssibility f generating cutffs at ALL ndes. NegaScut and Principal Variatin Search The NegaScut (NS) algrithm [37, 38] is an eample f a null-windw search algrithm. It is an enhanced versin f the fail-sft alpha-beta algrithm that uses null windws. NegaScut uses a wide search windw fr the first mve, hping it will lead t a PV nde (it assumes that the first mve cnsidered at each nde is the best), and a null windw fr the thers, as it epects them nt t be part f the principal variatin (it may be useful t remember the structure f the minimal tree, figure 2.5). If ne f the null-windw searches fails high, then the crrespnding nde may becme a new PV candidate and may have t be re-searched with a wider search windw. The NegaScut algrithm is shwn in figure 2.9. The searching f the first mve has been unrlled frm the lp fr readability purpses. Ntice that a fail-sft search f the last tw levels f a game tree always returns an eact scre, and thus n re-search is needed (underlined cde in the figure). This is nly true fr fied-depth game trees. If we ignre this etra enhancement, then Principal Variatin Search (PVS) [31] and NegaScut are identical. As with alpha-beta, there is als a versin f NegaScut that is nt fail-sft. The fail-sft enhancement lets the search return values that lie utside the search windw. These can be used t tighten the search windw fr re-searches and t make the transpsitin table s pruning mre effective. Thus, a fail-sft versin is usually preferred ver a nn fail-sft. 18