The IBM Translation Models. Michael Collins, Columbia University

Transcription

1 The IBM Translaton Models Mchael Collns, Columba Unversty

2 Recap: The Nosy Channel Model Goal: translaton system from French to Englsh Have a model p(e f) whch estmates condtonal probablty of any Englsh sentence e gven the French sentence f. Use the tranng corpus to set the parameters. A Nosy Channel Model has two components: p(e) p(f e) the language model the translaton model Gvng: and p(e f) = p(e, f) p(f) = p(e)p(f e) p(e)p(f e) e argmax e p(e f) = argmax e p(e)p(f e)

3 Roadmap for the Next Few Lectures IBM Models 1 and 2 Phrase-based models

4 Overvew IBM Model 1 IBM Model 2 EM Tranng of Models 1 and 2

5 IBM Model 1: Algnments How do we model p(f e)? Englsh sentence e has l words e 1... e l, French sentence f has m words f 1... f m. An algnment a dentfes whch Englsh word each French word orgnated from Formally, an algnment a s {a 1,... a m }, where each a {0... l}. There are (l + 1) m possble algnments.

6 IBM Model 1: Algnments e.g., l = 6, m = 7 e = And the program has been mplemented f = Le programme a ete ms en applcaton One algnment s {2, 3, 4, 5, 6, 6, 6} Another (bad!) algnment s {1, 1, 1, 1, 1, 1, 1}

7 Algnments n the IBM Models We ll defne models for p(a e, m) and p(f a, e, m), gvng p(f, a e, m) = p(a e, m)p(f a, e, m) Also, p(f e, m) = a A p(a e, m)p(f a, e, m) where A s the set of all possble algnments

8 A By-Product: Most Lkely Algnments Once we have a model p(f, a e, m) = p(a e)p(f a, e, m) we can also calculate for any algnment a p(a f, e, m) = p(f, a e, m) p(f, a e, m) a A For a gven f, e par, we can also compute the most lkely algnment, a = arg max p(a f, e, m) a Nowadays, the orgnal IBM models are rarely (f ever) used for translaton, but they are used for recoverng algnments

9 An Example Algnment French: le consel a rendu son avs, et nous devons à présent adopter un nouvel avs sur la base de la premère poston. Englsh: the councl has stated ts poston, and now, on the bass of the frst poston, we agan have to gve our opnon. Algnment: the/le councl/consel has/à stated/rendu ts/son poston/avs,/, and/et now/présent,/null on/sur the/le bass/base of/de the/la frst/premère poston/poston,/null we/nous agan/null have/devons to/a gve/adopter our/nouvel opnon/avs./.

10 IBM Model 1: Algnments In IBM model 1 all allgnments a are equally lkely: p(a e, m) = 1 (l + 1) m Ths s a maor smplfyng assumpton, but t gets thngs started...

11 IBM Model 1: Translaton Probabltes Next step: come up wth an estmate for p(f a, e, m) In model 1, ths s: m p(f a, e, m) = t(f e a ) =1

12 e.g., l = 6, m = 7 e = And the program has been mplemented f = Le programme a ete ms en applcaton a = {2, 3, 4, 5, 6, 6, 6} p(f a, e) = t(le the) t(programme program) t(a has) t(ete been) t(ms mplemented) t(en mplemented) t(applcaton mplemented)

13 IBM Model 1: The Generatve Process To generate a French strng f from an Englsh strng e: Step 1: Pck an algnment a wth probablty 1 (l+1) m Step 2: Pck the French words wth probablty p(f a, e, m) = m t(f e a ) =1 The fnal result: p(f, a e, m) = p(a e, m) p(f a, e, m) = 1 (l + 1) m m t(f e a ) =1

14 An Example Lexcal Entry Englsh French Probablty poston poston poston stuaton poston mesure poston vue poston pont poston atttude de la stuaton au nveau des négocatons de l omp of the current poston n the wpo negotatons... nous ne sommes pas en mesure de décder,... we are not n a poston to decde, le pont de vue de la commsson face à ce problème complexe.... the commsson s poston on ths complex problem.

15 Overvew IBM Model 1 IBM Model 2 EM Tranng of Models 1 and 2

16 IBM Model 2 Only dfference: we now ntroduce algnment or dstorton parameters q(, l, m) = Probablty that th French word s connected Defne to th Englsh word, gven sentence lengths of e and f are l and m respectvely p(a e, m) = where a = {a 1,... a m } Gves p(f, a e, m) = m q(a, l, m) =1 m q(a, l, m)t(f e a ) =1

17 An Example l = 6 m = 7 e = And the program has been mplemented f = Le programme a ete ms en applcaton a = {2, 3, 4, 5, 6, 6, 6} p(a e, 7) = q(2 1, 6, 7) q(3 2, 6, 7) q(4 3, 6, 7) q(5 4, 6, 7) q(6 5, 6, 7) q(6 6, 6, 7) q(6 7, 6, 7)

18 An Example l = 6 m = 7 e = And the program has been mplemented f = Le programme a ete ms en applcaton a = {2, 3, 4, 5, 6, 6, 6} p(f a, e, 7) = t(le the) t(programme program) t(a has) t(ete been) t(ms mplemented) t(en mplemented) t(applcaton mplemented)

19 IBM Model 2: The Generatve Process To generate a French strng f from an Englsh strng e: Step 1: Pck an algnment a = {a 1, a 2... a m } wth probablty m q(a, l, m) =1 Step 3: Pck the French words wth probablty m p(f a, e, m) = t(f e a ) The fnal result: =1 p(f, a e, m) = p(a e, m)p(f a, e, m) = m q(a, l, m)t(f e a ) =1

20 Recoverng Algnments If we have parameters q and t, we can easly recover the most lkely algnment for any sentence par Gven a sentence par e 1, e 2,..., e l, f 1, f 2,..., f m, defne for = 1... m a = arg max a {0...l} q(a, l, m) t(f e a ) e = And the program has been mplemented f = Le programme a ete ms en applcaton

21 Overvew IBM Model 1 IBM Model 2 EM Tranng of Models 1 and 2

22 The Parameter Estmaton Problem Input to the parameter estmaton algorthm: (e (k), f (k) ) for k = 1... n. Each e (k) s an Englsh sentence, each f (k) s a French sentence Output: parameters t(f e) and q(, l, m) A key challenge: we do not have algnments on our tranng examples, e.g., e (100) = And the program has been mplemented f (100) = Le programme a ete ms en applcaton

23 Parameter Estmaton f the Algnments are Observed Frst: case where algnments are observed n tranng data. E.g., e (100) = And the program has been mplemented f (100) = Le programme a ete ms en applcaton a (100) = 2, 3, 4, 5, 6, 6, 6 Tranng data s (e (k), f (k), a (k) ) for k = 1... n. Each e (k) s an Englsh sentence, each f (k) s a French sentence, each a (k) s an algnment Maxmum-lkelhood parameter estmates n ths case are trval: t ML (f e) = Count(e, f) Count(e) q ML (, l, m) = Count(, l, m) Count(, l, m)

24 Input: A tranng corpus (f (k), e (k), a (k) ) for k = 1... n, where f (k) = f (k) 1... f m (k) k, e (k) = e (k) 1... e (k), a (k) = a (k) 1... a (k) m k. Algorthm: Set all counts c(...) = 0 For k = 1... n For = 1... mk, For = 0... l k, l k c(e (k), f (k) ) c(e (k), f (k) ) + δ(k,, ) c(e (k) ) c(e (k) ) + δ(k,, ) c(, l, m) c(, l, m) + δ(k,, ) c(, l, m) c(, l, m) + δ(k,, ) where δ(k,, ) = 1 f a (k) =, 0 otherwse. Output: t ML (f e) = c(e,f) c(e), q ML(, l, m) = c(,l,m) c(,l,m)

25 Parameter Estmaton wth the EM Algorthm Tranng examples are (e (k), f (k) ) for k = 1... n. Each e (k) s an Englsh sentence, each f (k) s a French sentence The algorthm s related to algorthm when algnments are observed, but two key dfferences: 1. The algorthm s teratve. We start wth some ntal (e.g., random) choce for the q and t parameters. At each teraton we compute some counts based on the data together wth our current parameter estmates. We then re-estmate our parameters wth these counts, and terate. 2. We use the followng defnton for δ(k,, ) at each teraton: δ(k,, ) = lk q(, l k, m k )t(f (k) e (k) ) =0 q(, l k, m k )t(f (k) e (k) )

26 Input: A tranng corpus (f (k), e (k) ) for k = 1... n, where f (k) = f (k) 1... f m (k) k, e (k) = e (k) 1... e (k) l k. Intalzaton: Intalze t(f e) and q(, l, m) parameters (e.g., to random values).

27 For s = 1... S Set all counts c(...) = 0 For k = 1... n For = 1... mk, For = 0... l k where c(e (k), f (k) ) c(e (k), f (k) ) + δ(k,, ) c(e (k) ) c(e (k) ) + δ(k,, ) c(, l, m) c(, l, m) + δ(k,, ) c(, l, m) c(, l, m) + δ(k,, ) q(, l k, m k )t(f (k) e (k) ) δ(k,, ) = lk =0 q(, l k, m k )t(f (k) e (k) ) Recalculate the parameters: t(f e) = c(e, f) c(e) q(, l, m) = c(, l, m) c(, l, m)

28 The EM Algorthm for IBM Model 1 For s = 1... S Set all counts c(...) = 0 For k = 1... n For = 1... mk, For = 0... l k where δ(k,, ) = c(e (k), f (k) ) c(e (k), f (k) ) + δ(k,, ) c(e (k) ) c(e (k) ) + δ(k,, ) c(, l, m) c(, l, m) + δ(k,, ) c(, l, m) c(, l, m) + δ(k,, ) 1 (1+l k ) lk =0 1 (1+l k ) (k) t(f e (k) ) (k) t(f e (k) ) = t(f (k) e (k) ) e (k) ) lk =0 t(f (k) Recalculate the parameters: t(f e) = c(e, f)/c(e)

29 δ(k,, ) = lk q(, l k, m k )t(f (k) e (k) ) =0 q(, l k, m k )t(f (k) e (k) ) e (100) = And the program has been mplemented f (100) = Le programme a ete ms en applcaton

30 Justfcaton for the Algorthm Tranng examples are (e (k), f (k) ) for k = 1... n. Each e (k) s an Englsh sentence, each f (k) s a French sentence The log-lkelhood functon: L(t, q) = n log p(f (k) e (k) ) = n log a p(f (k), a e (k) ) k=1 k=1 The maxmum-lkelhood estmates are arg max L(t, q) t,q The EM algorthm wll converge to a local maxmum of the log-lkelhood functon

31 Summary Key deas n the IBM translaton models: Algnment varables Translaton parameters, e.g., t(chen dog) Dstorton parameters, e.g., q(2 1, 6, 7) The EM algorthm: an teratve algorthm for tranng the q and t parameters Once the parameters are traned, we can recover the most lkely algnments on our tranng examples e = And the program has been mplemented f = Le programme a ete ms en applcaton