Computig First-Order Logic Programs by Fibrig Artificial Neural Netorks Sebastia Bader Departmet of Computer Sciece Techische Uiversität Dresde Germay Artur S. d Avila Garcez Departmet of Computig City Uiversity Lodo UK Pascal Hitzler stitute AFB Uiversity of Karlsruhe Germay Abstract The itegratio of symbolic ad eural-etork-based artificial itelligece paradigms costitutes a very challegig area of research. The overall aim is to merge these to very differet major approaches to itelliget systems egieerig hile retaiig their respective stregths. For symbolic paradigms that use the sytax of some first-order laguage this appears to be particularly difficult. this paper, e ill exted o a idea proposed by Garcez ad Gabbay (2004) ad sho ho first-order logic programs ca be represeted by fibred eural etorks. The idea is to use a eural etork to iterate a global couter. For each clause C i i the logic program, this couter is combied (fibred) ith aother eural etork, hich determies hether C i outputs a atom of level for a give iterpretatio. As a result, the fibred etork approximates the sigle-step operator T P of the logic program, thus capturig the sematics of the program. troductio telliget systems based o artificial eural etorks differ substatially from those based o symbolic koledge processig like logic programmig. Neural etorks are traiable from ra data ad are robust, but practically impossible to read declaratively. Logic programs ca be implemeted from problem specificatios ad ca be highly recursive, hile lackig good traiig methods ad robustess, particularly he data are oisy (Thru & others 99). t is obvious that a itegratio of both paradigms ito sigle systems ould be very beeficiary if the respective stregths could be retaied. There exists a otable body of ork ivestigatig the itegratio of eural etorks ith propositioal or similarly fiitistic logic. We refer to (Broe & Su 200; d Avila Garcez, Broda, & Gabbay 2002) for overvies. For Sebastia Bader is supported by the GK334 of the Germa Research Foudatio. Artur Garcez is partly supported by The Nuffield Foudatio. Pascal Hitzler is supported by the Germa Federal Miistry of Educatio ad Research uder the SmartWeb project ad by the Europea Uio uder the KoledgeWeb Netork of Excellece. Copyright c 2005, America Associatio for Artificial telligece (.aaai.org). All rights reserved. first-order logic, hoever, it is much less clear ho a reasoable itegratio ca be achieved, ad there are systematic difficulties hich slo do recet research efforts, as spelled out i (Bader, Hitzler, & Hölldobler 2004). Differet techiques for overcomig these obstacles are curretly uder ivestigatio, icludig the use of metric spaces ad topology, ad of iterated fuctio systems (Hitzler, Hölldobler, & Seda 2004; Bader & Hitzler 2004). At the heart of these itegratio efforts is the questio of ho first-order koledge ca be represeted by eural etork architectures. this paper, e preset a ovel approach usig fibrig eural etorks as proposed by (d Avila Garcez & Gabbay 2004). For each clause C i of a logic program, a eural etork that iterates a couter is combied (fibred) ith aother eural etork, hich determies hether C i outputs a atom of level for a give iterpretatio. Fibrig offers a modular ay of performig complex fuctios by usig relatively simple etorks (modules) i a esemble. The paper is orgaized as follos. the ext sectio e briefly revie fibrig eural etorks ad logic programs. We the preset the fudametal ideas uderlyig our represetatio results, before givig the details of our implemetatio ad a orked example. We coclude ith some discussios. Prelimiaries We itroduce stadard termiology for artificial eural etorks, fibrig eural etorks, ad logic programs. We refer the reader to (Bishop 995; d Avila Garcez & Gabbay 2004; Lloyd 988), respectively, for further backgroud. Artificial Neural Netorks Artificial eural etorks cosist of simple computatioal uits (euros), hich receive real umbers as iputs via eighted coectios ad perform simple operatios: the eighted iputs are added ad simple fuctios like threshold, sigmoidal, idetity or trucate are applied to the sum. The euros are usually orgaized i layers. Neuros hich do ot receive iput from other euros are called iput euros, ad those ithout outgoig coectios to other euros are output euros. So a etork computes a fuctio from R to R m, here ad m are the umber of iput, respectively, output uits. A key to the success of
x[t] x2[t] x3[t] x[t] 2 3 θ Figure : A artificial euro (left) ad a simple fibrig etork (right) eural etork architectures rests o the fact that they ca be traied effectively usig traiig samples i the form of iput-output pairs. For coveiece, e make the folloig assumptios for the etorks depicted i this paper: The layers are updated sequetially from left to right ad ithi a layer the euros are updated from top to bottom. Recetly, (d Avila Garcez & Gabbay 2004) itroduced a e model of eural etorks, amely fibrig eural etorks. Briefly, the activatio of a certai uit may ifluece the behaviour of other uits by chagig their eights. Our particular architecture is a slight variat of the origial proposal, hich appears to be more atural for our purposes. Defiitio A fibrig fuctio i associated ith euro i maps some eights of the etork to e values, depedig o ad the iput x of euro i. Fibrig fuctios ca be uderstood as modelig presyaptic eights, hich play a importat role i biological eural etorks. Certaily, a ecessary requiremet for biological plausibility is that fibrig fuctios compute either simple fuctios or tasks hich ca i tur be performed by eural etorks. We ill retur to this poit later. Throughout this paper e ill use dashed lies, as i Figure, to idicate the eights hich may be chaged by some fibrig fuctio. As described above, e ill use a update dyamics from left to right, ad top to bottom. Ad, as soo as the activatio of a fibrig euro is (re)calculated, the correspodig fibrig fuctio is applied ad the respective eights are modified. Example 2 A simple fibrig etork for squarig umbers. Each ode computes the eighted sum of its iputs ad performs the operatio idetity o it. The fibrig fuctio takes iput x ad multiplies it by W. f W, the output ill be y x 2 : x y x[t+] 4 2 3 : (, x) x Example 3 A simple fibrig etork implemetig a gatelike behaviour. Nodes behave as i Example 2: x z y : (, x) { if x > 0 0 otherise The questio of plausible types of fibrig fuctios, as ell as the computatioal poer of those etorks, ill be studied separately ad are touched here oly slightly. We ill start ith very geeral fibrig fuctios, but later e restrict ourselves to simple oes oly, e.g. the fibred eight is simply multiplied by the activatio. Sometimes e ill use the output of a euro istead of the activatio, or apply liear trasformatios to it, ad it is clear that such modificatios could also be achieved by addig aother euro to the etork ad use this for the fibrig. Therefore these modificatios ca be uderstood as abbreviatios to keep the etorks simple. Logic Programs A logic program is a fiite set of clauses H L L, here N may differ for each clause, H is a atom i a first order laguage L ad L,..., L are literals, that is, atoms or egated atoms, i L. The clauses of a program are uderstood as beig uiversally quatified. H is called the head of the clause, each L i is called a body literal ad their cojuctio L L is called the body of the clause. We allo 0, by a abuse of otatio, hich idicates that the body is empty; i this case the clause is called a uit clause or a fact. A atom, literal or clause is said to be groud if it does ot cotai variables, ad groud(p) deotes the groud istatiatio of the program P. The Herbrad base uderlyig a give program P is defied as the set of all groud istaces of atoms, deoted B P. Example 4 shos a logic program ad its correspodig Herbrad base. Subsets of the Herbrad base are called (Herbrad) iterpretatios of P, ad e ca thik of such a set as cotaiig those atoms hich are true uder the iterpretatio. The set P of all iterpretatios of a program P ca thus be idetified ith the poer set of B P. Example 4 The atural umbers program P, the uderlyig laguage L ad the correspodig Herbrad base B P. The iteded meaig of s is the successor fuctio: P : at(0). at(s(x)) at(x). L : costats: C {0} fuctios: F {s/} relatios: R {at/} B P : at(0), at(s(0)), at(s(s(0))),... Logic programs are accepted as a coveiet tool for koledge represetatio i logical form. Furthermore, the koledge represeted by a logic program P ca essetially be captured by the immediate cosequece or siglestep operator T P, hich is defied as a mappig o P here for ay P e have that T P () is the set of all H B P for hich there exists a groud istace H A A m B B of a clause i P such that for all i e have A i ad for all j e have
B j. Fixed poits of T P are called supported models of P, hich ca be uderstood to represet the declarative sematics of P. the sequel of this paper e ill ofte eed to eumerate the Herbrad base, hich is doe via level mappigs: Defiitio 5 Give a logic program P, a level mappig is a fuctio : B P N +, here N + deotes the set of positive itegers excludig zero. Level mappigs i slightly more geeral form are commoly used for cotrollig recursive depedecies betee atoms, ad the most promiet otio is probably the folloig. Defiitio 6 Let P be a logic program ad be a level mappig. f for all clauses A L L 2... L groud(p) ad all i e have that A > L i, the P is called acyclic ith respect to. A program is called acyclic, if there exists such a level mappig. Acyclic programs are ko to have uique supported models (Cavedo 99). The programs from Examples 4 ad 7 belo are acyclic. Example 7 The eve ad odd umbers program ad a level mappig: P : : eve(0). eve(s(x)) eve(x). odd(s(x)) eve(x). A { 2 + if A eve(s (0)) 2 + 2 if A odd(s (0)) Throughout this paper e ill assume that level mappigs are bijective, i.e. for each N + there is exactly oe A B P, such that A. Thus, for the purposes of our paper, a level mappig is simply a eumeratio of the Herbrad base. Sice level mappigs iduce a order o the atoms, e ca use them to defie a prefix-fuctio o iterpretatios, returig oly the first atoms: Defiitio 8 The prefix of legth of a give iterpretatio is defied as pref : P N + P (, ) {A A ad A }. We ill rite pref () for pref(, ). For acyclic programs, it follos that i order to decide hether the atom ith level + must be icluded i T P (), it is sufficiet to cosider pref () oly. From Logic Programs to Fibrig Netorks We ill sho ho to represet acyclic logic programs by meas of fibrig eural etorks. We follo up o the basic idea from (Hölldobler & Kalike 994; Hölldobler, Kalike, & Störr 999), ad further developed i (Hitzler, Hölldobler, & Seda 2004; Bader & Hitzler 2004), to represet the siglestep operator T P by a etork, istead of the program P itself. This is a reasoable thig to do sice the sigle-step operator essetially captures the sematics of the program it is associated ith, as metioed before. order to represet T P by the iput-output mappig of a etork, e also eed a ecodig of P as a suitable subset of the real umbers. We also use a idea from (Hölldobler, Kalike, & Störr 999) for this purpose. Let B > 2 be some iteger, ad let be a bijective level mappig. Defie R : P R : A B A. We exclude B 2, because i this case R ould ot be ijective. t ill be coveiet to assume B 3 throughout the paper, but our results do ot deped o this. We deote the rage of R by rage(r). There are systematic reasos hy this ay of embeddig P ito the reals is reasoable, ad they ca be foud i (Hitzler, Hölldobler, & Seda 2004; Bader & Hitzler 2004), but ill ot cocer us here. Usig R, the prefix operatio ca be expressed aturally o the reals. Propositio 9 For P ad x rage(r) e have ( truc(r() B pref(, ) R ) ) B ad R(pref(R (x), )) truc(x B ) B. For coveiece, e overload pref ad set pref(x, ) R(pref(R (x), )) ad pref (x) pref(x, ). We ill o tur to the costructio of fibrig etorks hich approximate give programs. We ill first describe our approach i geeral terms, ad spell it out i a more formal ad detailed ay later o. The goal is to costruct a eural etork, hich ill compute R(T P )(x) R(T P (R (x))) for a give x rage(r). The etork is desiged i such a ay that it successively approximates R (T P ) (x) hile ruig. There ill be a mai loop iteratig a global couter. This couter fibres the kerel, hich ill evaluate hether the atom of level is cotaied i T P () or ot, i.e. the kerel ill output B if the atom is cotaied, ad 0 otherise. Furthermore, there ill be a iput subetork providig R() all the time, ad the output subetork hich ill accumulate the outputs of the kerel, ad hece coverge to R(T P ()). Figure 2 shos the geeral architecture. For each clause C i there is a subetork, hich determies hether C i outputs the atom of level for the give iterpretatio, or ot. This is doe by fibrig the subetork such that it computes the correspodig groud istace C () i, ith head of level, if existet. f there is o
such groud istace, this subetork ill output 0, otherise it ill determie hether the body is true uder the iterpretatio. A detailed descriptio of these clause etorks ill be give i the ext sectio. Note that this costructio is oly possible for programs hich are covered. This meas that they do ot have ay local variables, i.e. every variable occurig i some body also occurs i the correspodig head. Obviously, programs hich are acyclic ith respect to a bijective level mappig are alays covered. + Clause Clause2 TP() + Clausex Clause Figure 3: Recurret architecture for acyclic programs Clause2 TP() Gate Clausex Filter for L Filter for L2 Figure 2: Geeral architecture Filter for Lk f P is acyclic e ca compute the uique supported model of the program directly, by coectig the output ad the iput regio of the etork as sho i Figure 3. This is simply due to the above metioed fact: f e at to decide hether the atom of level should be icluded i T P (), it is sufficiet to look at the atoms A ith level <. We also have the folloig result. Propositio 0 Let P be a program hich is acyclic ith respect to a bijective level mappig, let A B P ith A. The for each P e have that A TP () iff A is true ith respect to the uique supported model of P. Proof This is a immediate result from the applicatio of the Baach cotractio mappig priciple to the sematic aalysis of acyclic programs, see (Hitzler & Seda 2003). So, for acyclic programs, e ca start ith the empty (or ay other) iterpretatio ad let the (recurret) etork ru. mplemetig Clauses order to complete the costructio from the previous sectio, e give a implemetatio of the clauses. For a clause C of the form H L L 2... L k, let C () deote the groud istace of C for hich the head has level, assumig it exists. The idea of the folloig costructio is to create a etork hich implemets C, ad ill be fibred by the couter such that it implemets C (). case that there is o groud istace of C ith head of level, the etork ill output 0, otherise it ill output if the body is true ith respect to the iterpretatio, ad 0 if it is ot. Figure 4: mplemetig clauses The idea, as sho i Figure 4, is that each subetork implemetig a clause C : H L... L k ith k body literals, cosists of k + parts oe gate ad k filters. The gate ill output, if the clause C has a groud istace C () here the level of the head is. Furthermore there is a filter for each body literal L i, hich outputs, if the correspodig groud literal L i is true uder. f all coditios are satisfied the fial cojuctio-euro ill become active, i.e. the subetork outputs. Note that this costructio agai is sufficiet oly for programs hich are covered. f e alloed local variables, the more tha oe (i fact ifiitely may) groud istaces of C ith a head of level could exist. Let us have a closer look at the type of fibrig fuctio eeded for our costructio. For the gate, it implicitly performs a very simple patter matchig operatio, checkig hether the atom ith level uifies ith the head of the clause. For the filters, it checks hether correspodig istaces of body literals are true i the give iterpretatio, i.e. it implicitly performs a variable bidig ad a elemetary check of set-iclusio. We argue that the operatios performed by the fibrig fuctio are ideed biologically feasible. The perspective hich e take i this paper is that they should be uderstood as fuctios performed by a separate etork, hich e do ot give explicitly, although e ill substatiate this poit to a certai extet i the ext sectio. Ad patter matchig is ideed a task that coectioist etorks perform ell.
The variable bidig task ill also be addressed i the ext sectio he e give examples for implemetig the filters. Neural Gates As specified above, the gate for a clause C : H L... L k fires if there is a groud istace C () of C ith head is of level, as depicted i Figure 5. The decisio based o - 0 l iit (, x) m Figure 7: A simple gate for poers o if groud istace ith (, ) head of level exists 0 otherise Figure 5: A eural gate simple patter matchig is embedded ito the fibrig fuctio. hat follos, e ill discuss a umber of differet cases of ho to ufold this fibrig fuctio ito a etork, i order to give plausible etork topologies ad yet simpler fibrig fuctios. Other implemetatios are possible, ad the cases preseted here shall serve as examples oly. Groud-headed clauses. Let us first cosider a clause for hich the head does ot cotai variables, i.e. a groud clause, like for example the first clause give i Example 7 above. Sice the level of the head i this case is fixed to some value, say m, the correspodig gate subetork should fire if ad oly if the geeral couter is equal to m. This ca be doe usig the etork sho i Figure 6 (left): The euro! ill alays output ad the euro 0 ill output if ad oly if the eighted iputs sum up to 0. This ca easily be implemeted usig e.g. threshold uits.! -m 0 Figure 6: Simple gates for groud-headed clauses (left) ad remaider classes (right) Remaider classes. f the levels l i of groud istatiated heads for a certai clause ca be expressed as multiples of a certai fixed umber m, i.e. l i i m for all i (like clauses umber 2 ad 3 of Example 7), e ca costruct a simple subetork, as depicted i Figure 6 (right). The euros symbolize the equivalece classes for the remaiders of the devisio by 3. The etork ill be iitialized by activatig. Every time it is reevaluated the activatio simply proceeds to the ext ro. Poers. f the level l i of groud istatiated heads for a certai clause ca be expressed as poers of a certai fixed umber m, i.e. l i m i for all i, e ca costruct a simple subetork as sho i Figure 7. 0 2 0 2 Filterig terpretatios For a etork to implemet the groud istace C () : H L ()... L () k of a clause C ith head of level, e eed to ko the distace betee the head ad the body literals i terms of levels as a fuctio i, i.e. e eed a set of fuctios {b i : N N i,..., k} oe for each body literal here b i computes the level of the literal L i, takig as iput the level of the head, as illustrated i Example. Example For the eve ad odd umbers program from Example 7, e ca use the folloig b i -fuctios: eve(0). {} eve(s(x)) eve(x). {b : 2} odd(s(x)) eve(x). {b : } For each body literal e ill o costruct a filter subetork, that fires if the correspodig groud body literal L () i of C () is icluded i. Give a iterpretatio, e eed to decide hether a certai atom A is icluded or ot. The uderlyig idea is the folloig. order to decide hether the atom A of level is icluded i the iterpretatio, e costruct a iterpretatio J cotaiig all atoms of ith level smaller tha, ad the atom A, i.e. J pref () {A}, or, expressed o the reals, R(J) pref (R())+B. f e evaluate R() R(J) the result ill be o-egative if ad oly if A is icluded i. This ca be doe usig the etork sho i Figure 8. truc 2 3 O (, ) B ( 2, ) B ( 3, ) B Figure 8: Schematic plot ad fibrig fuctio of a filter for the atom of level t is clear that e ca costruct etorks to filter a atom of level b i (), if the fuctio b i ca itself be implemeted i a eural etork. Sice fibrig etorks ca implemet ay polyomial fuctio, as sho i (d Avila Garcez & Gabbay 2004) ad idicated i Example 2, our approach is flexible ad very geeral.
A Worked Example Let us o give a complete example by extedig o the logic program ad the level mappig from Example 7 above. For the first clause e eed a groud-headed gate oly. To implemet the secod clause a remaider-class gate for the devisio by 2 is eeded, hich returs for all odd umbers. Furthermore, e eed a filter hich returs if the atom of level 2 is ot icluded i. For the last clause of the example, e eed a gate returig for all eve umbers ad a similar filter as for clause umber 2. Combiig all three parts ad takig ito accout that P is acyclic, e get the etork sho i Figure 9. f ru o ay iitial value, its +! 0 0 truc 0 truc Figure 9: Neural implemetatio of the hole example outputs coverge to the uique supported model of P, i.e. the sequece of outputs of the right-most euro is a sequece of real umbers hich coverges to R(M), here M is the uique supported model of P. Coclusios This paper cotributes to advace the state of the art o eural-symbolic itegratio by shoig ho first-order logic programs ca be implemeted i fibrig eural etorks. Geeric ays for represetig the eeded fibrig fuctios i a biologically plausible fashio remai to be ivestigated i detail, as ell as the task of extedig our proposal toards a fully fuctioal eural-symbolic learig ad reasoig system, icludig learig capabilities. Fibrig offers a modular ay of performig complex fuctios, such as logical reasoig, by combiig relatively simple modules (etorks) i a esemble. f each module is kept simple eough, e should be able to apply stadard eural learig algorithms to them. Ultimately, this may provide a itegrated system ith robust learig ad expressive reasoig capability. Refereces Bader, S., ad Hitzler, P. 2004. Logic programs, iterated fuctio systems, ad recurret radial basis fuctio etorks. Joural of Applied Logic 2(3):273 300. Bader, S.; Hitzler, P.; ad Hölldobler, S. 2004. The itegratio of coectioism ad first-order koledge represetatio ad reasoig as a challege for artificial itelligece. Proceedigs of the Third teratioal Coferece o formatio, Tokyo, Japa. To appear. Bishop, C. M. 995. Neural Netorks for Patter Recogitio. Oxford Uiversity Press. Broe, A., ad Su, R. 200. Coectioist iferece models. Neural Netorks 4(0):33 355. Cavedo, L. 99. Acyclic programs ad the completeess of SLDNF-resolutio. Theoretical Computer Sciece 86:8 92. d Avila Garcez, A. S., ad Gabbay, D. M. 2004. Fibrig eural etorks. McGuiess, D. L., ad Ferguso, G., eds., Proceedigs of the Nieteeth Natioal Coferece o Artificial telligece, Sixteeth Coferece o ovative Applicatios of Artificial telligece, July 25-29, 2004, Sa Jose, Califoria, USA, 342 347. AAA Press / The MT Press. d Avila Garcez, A. S.; Broda, K. B.; ad Gabbay, D. M. 2002. Neural-Symbolic Learig Systems Foudatios ad Applicatios. Perspectives i Neural Computig. Spriger, Berli. Hitzler, P., ad Seda, A. K. 2003. Geeralized metrics ad uiquely determied logic programs. Theoretical Computer Sciece 305( 3):87 29. Hitzler, P.; Hölldobler, S.; ad Seda, A. K. 2004. Logic programs ad coectioist etorks. Joural of Applied Logic 2(3):245 272. Hölldobler, S., ad Kalike, Y. 994. Toards a massively parallel computatioal model for logic programmig. Proceedigs ECA94 Workshop o Combiig Symbolic ad Coectioist Processig, 68 77. ECCA. Hölldobler, S.; Kalike, Y.; ad Störr, H.-P. 999. Approximatig the sematics of logic programs by recurret eural etorks. Applied telligece :45 58. Lloyd, J. W. 988. Foudatios of Logic Programmig. Spriger, Berli. Thru, S. B., et al. 99. The MONK s problems: A performace compariso of differet learig algorithms. Techical Report CMU-CS-9-97, Caregie Mello Uiversity.