CS838-1 Advanced NLP: Conditional Random Fields

Transcription

1 CS838-1 Advanced NLP: Condtonal Random Felds Xaojn Zhu 2007 Send comments to 1 Informaton Extracton Current NLP technques cannot fully understand general natural language artcles. However, they can stll be useful on restrcted tasks. One example s Informaton Extracton. For example, one mght want to extract the ttle, authors, year, and conference names from a researcher s Web page. Or one mght want to dentfy person, locaton, organzaton names from news artcles (NER, named entty recognton). These are useful to automatcally turn free text on the Web nto knowledge databases, and form the bass of many Web servces. The basc Informaton Extracton technque s to treat the problem as a text sequence taggng problem. The tag sets can be {ttle, author, year, conference, other}, or {person, locaton, organzaton, other}, for nstance. Therefore HMMs has been naturally and successfully appled to Informaton Extracton. However, HMMs have dffculty modelng overlappng, non-ndependent features of the output. For example, an HMM mght specfy whch words are lkely for a gven state (tag) va p(x z). But often the part-of-speech of the word, as well as that of the surroundng words, character n-grams, captalzaton patterns all carry mportant nformaton. HMMs cannot easly model these, because the generatve story lmts what can be generated by a state varable. Condtonal Random Feld (CRF) can model these overlappng, non-ndependent features. A specal case, lnear chan CRF, can be thought of as the undrected graphcal model verson of HMM. It s as effcent as HMMs, where the sumproduct algorthm and max-product algorthm stll apply. 2 The CRF Model Let x 1:N be the observatons (e.g., words n a document), and z 1:N the hdden labels (e.g., tags). A lnear chan Condtonal Random Feld defnes a condtonal 1

2 probablty (whereas HMM defnes the jont) p(z 1:N x 1:N ) = 1 Z exp ( N n=1 =1 ) λ f (z n 1, z n, x 1:N, n). (1) Let us walk through the model n detal. The scalar Z s the normalzaton factor, or partton functon, to make t a vald probablty. Z s defned as the sum of exponental number of sequences, Z = ( N ) exp λ f (z n 1, z n, x 1:N, n), (2) z 1:N n=1 =1 therefore s dffcult to compute n general. Note Z mplctly depends on x 1:N and the parameters λ. The bg exp() functon s there for hstorcal reasons, wth connecton to the exponental famly dstrbuton. For now, t s suffcent to note that λ and f() can take arbtrary real values, and the whole exp functon wll be non-negatve. Wthn the exp() functon, we sum over n = 1,..., N word postons n the sequence. For each poston, we sum over = 1,..., F weghted features. The scalar λ s the weght for feature f (). The λ s are the parameters of the CRF model, and must be learned, smlar to θ = {π, φ, A} n HMMs. 3 Feature Functons The feature functons are the key components of CRF. In our specal case of lnear-chan CRF, the general form of a feature functon s f (z n 1, z n, x 1:N, n), whch looks at a par of adjacent states z n 1, z n, the whole nput sequence x 1:N, and where we are n the sequence (n). These are arbtrary functons that produce a real value. For example, we can defne a smple feature functon whch produces bnary values: t s 1 f the current word s John, and f the current state z n s PERSON: { 1 f zn = PERSON and x f 1 (z n 1, z n, x 1:N, n) = n = John 0 otherwse (3) How s ths feature used? It depends on ts correspondng weght λ 1. If λ 1 > 0, whenever f 1 s actve (.e. we see the word John n the sentence and we assgn t tag PERSON), t ncreases the probablty of the tag sequence z 1:N. Ths s another way of sayng the CRF model should prefer the tag PERSON for the word John. If on the other hand λ 1 < 0, the CRF model wll try to avod the tag PERSON for John. Whch way s correct? One may set λ 1 by doman knowledge (we know t should probably be postve), or learn λ 1 from corpus (let the data tell us), or both (treatng doman knowledge as pror on λ 1 ). Note λ 1, f 1 () together s equvalent to (the log of) HMM s φ parameter p(x = John z = PERSON). 2

3 As another example, consder { 1 f zn = PERSON and x f 2 (z n 1, z n, x 1:N, n) = n+1 = sad 0 otherwse (4) Ths feature s actve f the current tag s PERSON and the next word s sad. One would therefore expect a postve λ 2 to go wth the feature. Furthermore, note f 1 and f 2 can be both actve for a sentence lke John sad so. and z 1 = PERSON. Ths s an example of overlappng features. It boosts up the belef of z 1 = PERSON to λ 1 +λ 2. Ths s somethng HMMs cannot do: HMMs cannot look at the next word, nor can they use overlappng features. The next feature example s rather lke the transton matrx A n HMMs. We can defne { 1 f zn 1 = OTHER and z f 3 (z n 1, z n, x 1:N, n) = n = PERSON 0 otherwse (5) Ths feature s actve f we see the partcular tag transton (OTHER, PER- SON). Note t s the value of λ 3 that actually specfes the equvalent of (log) transton probablty from OTHER to PERSON, or A OTHER, PERSON n HMM notaton. In a smlar fashon, we can defne all K 2 transton features, where K s the sze of tag set. Of course the features are not lmted to bnary functons. Any real-valued functon s allowed. 4 Undrected Graphcal Models (Markov Random Felds) CRF s a specal case of undrected graphcal models, also known as Markov Random Felds. A clque s a subset of nodes n the graph that are fully connected (havng an edge between any two nodes). A maxmum clque s a clque that s not a subset of any other clque. Let X c be the set of nodes nvolved n a maxmum clque c. Let ψ(x c ) be an arbtrary non-negatve real-valued functon, called the potental functon. In partcular ψ(x c ) does not need to be normalzed. The Markov Random Feld defnes a probablty dstrbuton over the node states as the normalzed product of potental functons of all maxmum clques n the graph: p(x) = 1 ψ(x c ), (6) Z c where Z s the normalzaton factor. In the specal case of lnear-chan CRFs, the clques correspond to a par of states z n 1, z n as well as the correspondng x nodes, wth ψ = exp (λf). (7) 3

4 Ths s ndeed the drect connecton to factor graph representaton as well. Each clque can be represented by a factor node wth the factor ψ(x c ), and the factor node connects to every node n X c. There s one addton specal factor node whch represents Z. A welcome consequence s that the sum-product algorthm and max-sum algorthm mmedately apply to Markov Random Felds (and CRFs n partcular). The factor correspondng to Z can be gnored durng message passng. 5 CRF tranng Tranng nvolves fndng the λ parameters. For ths we need fully labeled data sequences {(x (1), z (1) ),..., (x (m), z (m) )}, where x (1) = x (1) 1:N 1 the frst observaton sequence, and so on 1. Snce CRFs defne the condtonal probablty p(z x), the approprate objectve for parameter learnng s to maxmze the condtonal lkelhood of the tranng data m log p(z (j) x (j) ). (8) Often one can also put a Gaussan pror on the λ s to regularze the tranng (.e., smoothng). If λ N(0, σ 2 ), the objectve becomes m log p(z (j) x (j) λ 2 ) 2σ 2. (9) The good news s that the objectve s concave, so the λ s have a unque set of optmal values. The bad news s that there s no closed form soluton 2. The standard parameter learnng approach s to compute the gradent of the objectve functon, and use the gradent n an optmzaton algorthm lke L-BFGS. The gradent of the objectve functon s computed as follows: m log p(z (j) x (j) λ 2 ) λ k 2σ 2 (10) = = λ k ( m n m f k (z (j) n 1, z(j) n, x (j), n) n n λ f (z (j) n 1, z(j) n, x (j), n) log Z (j) ) λ 2 2σ 2 (11) m E z n 1,z n [f k(z n 1, z n, x (j), n)] λ k σ 2, (12) 1 Unlke HMMs whch can use the Baum-Welch (EM) algorthm to tran on unlabeled data x only, CRFs tranng on unlabeled data s dffcult 2 If ths remnds you of logstc regresson, you are rght: logstc regresson s a specal case of CRF where there s no edges among hdden states. In contrast, HMMs when traned on fully labeled data have smple and ntutve closed form solutons. 4

5 where we used the fact λ k log Z = E z [ n f k (z n 1, z n, x, n)] (13) = n = n E z n 1,z n [f k(z n 1, z n, x, n)] (14) p(z n 1, z n x)f k (z n 1, z n, x, n). (15) z n 1,z n Note the edge margnal probablty p(z n 1, z n x) s under the current parameters, and ths s exactly what the sum-product algorthm can compute. The partal dervatve n (12) has an ntutve explanaton. Let us gnore the term λ k /σ 2 from the pror. The dervatve has the form of (observed counts of feature f k ) mnus (expected counts of feature f k ). When the two are the same, the dervatve s zero, and there s no longer an ncentve to change λ k. Therefore we see that tranng can be thought of as fndng λ s that match the two counts. 6 Feature Selecton A common practce n NLP s to defne a very large number of canddate features, and let the data select a small subset to use n the fnal CRF model n a process known as feature selecton. Often the canddate features are proposed n two stages: 1. Atomc canddate features. These are usually a smple test on a specfc combnaton of words and tags, e.g.(x =John, z =PERSON), (x =John, z =LOCATION), (x =John, z =ORGANIZATION), etc. There are V K such word dentty canddate features, whch s obvously a large number. Although t s called the word dentty test, t should be understood as n combnaton wth each tag value. Smlarly one can test whether the word s captalzed, the dentty of the neghborng words, the partof-speech of the word, and so on. The state transton features are also atomc. From the large number of atomc canddate features, a small number of features are selected by how much they mprove the CRF model (e.g., ncrease n the tranng set lkelhood). 2. Grow canddate features. It s natural to combne features to form more complex features. For example, one can test for current word beng captalzed, the next word beng Inc., and both tags beng ORGANI- ZATION. However, the number of complex features grows exponentally. A compromse s to only grow canddate features on selected features so far, by extendng them wth one atomc addtons, or other smple boolean operatons. 5

6 Often any remanng atomc canddate features are added to the grown set. A small number of features are selected, and added to the exstng feature set. Ths stage s repeated untl enough features have been added. 6