Latest Developments in (S)MT

1. Latest Developments in (S)MT MT Wars II: The Empire (Linguistics) strikes back Harold Somers University of Manchester

2. Overview • The story so far • EBMT • SMT • Latest developments in RBMT • Is there convergence? • Some attempts to classify MT • (Carl and Wu’s MT model spaces) • Has the empire struck back?

3. The story so far: EBMT • Early history well known • Nagao (1981/3) • Early development as part of RBMT • Relationship with Translation Memories • Focus (cf. Somers 1998) on • Matching algorithms • Selection and storage of examples • Mainly sentence-based • TL generation (Recombination) not much addressed Somers, H. (1998) New paradigms in MT, 10th European Summer School in Logic, Language and Information, Workshop on MT, Saarbrücken; revised version in Machine Translation 14 (1999) and 2nd revised version in M. Carl & A. Way (2003) Recent Advances in EBMT (Kluwer).

4. EBMT in a nutshell (In case you’ve been on Tatooine for the last 15 years) • Database of paired examples • Translation involves • Finding the best example(s) (matching) • Identifying which bits do(n’t) match (alignment) • Replacing the non-matching bits (if multiple examples, gluing them together) (recombination) • All of the above at run-time

5. EBMT in a nutshell (cont.) • Main difficulty is “boundary friction” in two senses: • The old man is dead : Le vieil homme est mort • The old woman is dead : • * Le vieil femme est mort • The operation was interrupted because the file was hidden. • a. The operation was interrupted because the Ctrl-c key was pressed. • L’opération a été interrompue car la touché Ctrl-c a été enfoncée. • b. The specified method failed because the file is hidden. • La méthode spécifiée a échoué car le fichier est masqué

6. EBMT later developments • Example generalisation (templates) • Incorporation of linguistic resources and/or statistical measures • Structured representation of examples • Use of statistical techniques

7. Example generalisation (Furuse & Iida, Kaji et al., Matsumoto et al., Carl, Cicekli & Güvenir, Brown, McTait, Way et al.) • Similar examples can be combined to give a more general example • Can be seen as a way of generating transfer rules (and lexicons) • Process may be entirely automatic, based on string matching … • … or “seeded” using linguistic information (POS tags) or resources (bilingual dictionary)

8. Example generalisation (cont.) The monkey ate a peach saru wa momo o tabeta The man ate a peach hito wa momo o tabeta monkey saru man hito The … ate a peach  … wa momo o tabeta The dog ate a rabbit inu wa usagi o tabeta dog inu rabbit usagi The …x ate a ...y…xwa …y tabeta

9. Example generalisation (cont.) • That’s too simple (e.g. because of boundary friction) • Need to introduce constraints on the slots, e.g. using POS tags and morphological information (which implies some other processing) • Can use clustering algorithms to infer substitution sets

10. Incorporation of linguistic resources • Actually, early EBMT used all sorts of linguistic resources • Briefly there was a move towards more “pure” approaches • Now we see much use of POS tags (sometimes only partial, e.g. marker words – Way et al.), morphological analysis (as just mentioned), bilingual lexicons • Target-language grammars for recombination/generation phase

11. Incorporation of statistical measures • Example database preprocessed to assign weights (probabilities) to fragments and their translations (Aramaki et al.) • Good way of handling “ambiguities” due to alternative translations • Clustering words into equivalence classes for example generalization (Brown) • Using statistical tools to extract translation knowledge from parallel corpora (Yamamoto & Matsumoto) • Statistically induced grammars for translation or generation, as in ...

12. Use of structured representations • Again, a feature of early EBMT, now reappearing • Translation grammars induced from the example set • Examples stored as tree structures (overwhelmingly: dependency structures)

13. Translation grammars • Carl: generates translation grammars from aligned linguistically annotated texts • Way:Data-Oriented Translation based on Poutsma’s DOP, using both PS and LFG models)

14. Structured examples • Use of tree comparison algorithms to extract translation patterns from parsed corpora/tree banks (Watanabe et al.) • Translation pairings extracted from aligned parsed examples (Menezes & Richardson) • Tree-to-string approach used by Langlais & Gotti and Liu et al. (+ statistical generation model)

15. Typical use of structured examples • Rule-based analysis and generation + example-based transfer • Input is parsed into representation using a traditional or statistics-based analyser • TL representation constructed by combining translation mappings learned from the parallel corpus • TL sentence generated using a hand-written or machine-learned generation grammar • Is this still EBMT? • Note that the only example-based part is use of mappings which are learned, not computed at run-time

16. Pure EBMT (Lepage & Denoual) • In contrast (but now something of an oddity): pure analogy-based EBMT • Use of proportional analogies A:B::C:D • Terms in the analogies are translation pairs A→A’: B→B’:: C→C’: D→D’

18. Pure EBMT • No explicit transfer • No extraction of symbolic knowledge • No use of templates • Analogies do not always represent any sort of linguistic reality • No training or preprocessing • Solving the proportional analogies is done at run-time

19. The story so far (SMT) • Early history well known • IBM group inspired by improved results in speech recognition when non-linguistic approach taken • Availability of Canadian Hansards inspired purely statistical approach to MT (1988) • Immediate partial success (60%) to the dismay of MT people • Early observers (Wilks) predicted hybrid methods (“stone soup”) would evolve • Later developments • Phrase-based SMT • Syntax-based SMT

20. SMT in a nutshell (In case you’ve been on Kamino for the last 15 years) • From parallel corpus two sets of statistical data are extracted • Translation model: probabilities that a given word e in the SL gives rise to a word f in the TL • (Target) language model: most probable word-order for the words predicted by the translation model • These two models are computed off-line • Given an input sentence, a “decoder” applies the two models, and juggles the probabilities to get the best score; various methods have been proposed

21. SMT in a nutshell (cont.) • The translation model has to take into account the fact that • for a given e in there may be various different fsdepending on context (grammatical variants as well as alternatives due to polysemy or homonymy) • a given e may not necessarily correspond to a single f, or any f at all:“fertility” (e.g. may have → aurait; implemented → mis en application)

22. SMT in a nutshell (cont.) • The language model has to take into account the fact that • The TL words predicted by the translation model will not occur in the same order as the SL words: “distortion” • TL word choices can depend on neighbouring words (which may be easy to model) or, especially because of distortion, more distant words: “long-distance dependencies”, much harder to model

23. SMT in a nutshell (cont.) • Main difficulty: combination of fertility and distortion: Zeitmangel erschwert das Problem. Lack of time makes the problem more difficult. Eine Diskussion erübrigt sich demnach. Therefore there is no point in discussion. Das ist der Sache nicht angemessen. That is not appropriate for this matter. Den Vorschlag lehnt die Kommission ab. The Commission rejects the proposal.

24. SMT later developments • Phrase-based SMT • Extend models beyond individual words to word sequences (phrases) • Direct phrase alignment • Word alignment induced phrase model • Alignment templates • Results better than word-based models, and show improvement proportional (log-linear) to corpus size • Phrases do not correspond to constituents, and limiting them to do so hurts results

25. Direct phrase alignment (Wang & Waible 1998, Och et al., 1999, Marcu & Wong 2002) • Enhance word translation model by adding joint probabilities, i.e. probabilities for phrases • Phrase probabilities compensate for missing lexical probabilities • Easy to integrate probabilities from different sources/methods, allows for mutual compensation

26. Word alignment induced model Koehn et al. 2003; example stolen from Knight & Koehn http://www.iccs.inf.ed.ac.uk/~pkoehn/publications/tutorial2003.pdf Maria did not slap the green witch Maria no daba una botefada a la bruja verda Start with all phrase pairs justified by the word alignment

27. Word alignment induced model Koehn et al. 2003; example stolen from Knight & Koehn http://www.iccs.inf.ed.ac.uk/~pkoehn/publications/tutorial2003.pdf (Maria, Maria), (no, did not) (daba una botefada, slap), (a la, the), (verde, green), (bruja, witch)

28. Word alignment induced model Koehn et al. 2003; example stolen from Knight & Koehn http://www.iccs.inf.ed.ac.uk/~pkoehn/publications/tutorial2003.pdf (Maria, Maria), (no, did not) (daba una botefada, slap), (a la, the), (verde, green) (bruja, witch), (Maria no, Maria did not), (no daba una botefada, did not slap), (daba una botefada a la, slap the), (bruja verde, green witch) etc.

29. Word alignment induced model Koehn et al. 2003; example stolen from Knight & Koehn http://www.iccs.inf.ed.ac.uk/~pkoehn/publications/tutorial2003.pdf (Maria, Maria), (no, did not), (slap, daba una bofetada), (a la, the), (bruja, witch), (verde, green), (Maria no, Maria did not), (no daba una bofetada, did not slap), (daba una bofetada a la, slap the), (bruja verde, green witch), (Maria no daba una bofetada, Maria did not slap), (no daba una bofetada a la, did not slap the), (a la bruja verde, the green witch), (Maria no daba una bofetada a la, Maria did not slap the), (daba una bofetada a la bruja verde, slap the green witch), (no daba una bofetada a la bruja verde, did not slap the green witch), (Maria no daba una bofetada a la bruja verde, Maria did not slap the green witch)

30. Word alignment induced model • Given the phrase pairs collected, estimate the phrase translation probability distribution by relative frequency (without smoothing)

31. Alignment templates Och et al. 1999; further developed by Marcu and Wong 2002, Koehn and Knight 2003, Koehn et al. 2003) • Problem of sparse data worse for phrases • So use word classes instead of words • alignment templates instead of phrases • more reliable statistics for translation table • smaller translation table • more complex decoding • Word classes are induced (by distributional statistics), so may not correspond to intuitive (linguistic) classes • Takes context into account

32. Problems with phrase-based models • Still do not handle very well ... • dependencies (especially long-distance) • distortion • discontinuities (e.g. bought = habe ... gekauft) • More promising seems to be ...

33. Syntax-based SMT • Better able to handle • Constituents • Function words • Grammatical context (e.g. case marking) • Inversion Transduction Grammars • Hierarchical transduction model • Tree-to-string translation • Tree-to-tree translation

34. Inversion transduction grammars • Wu and colleagues (1997 onwards) • Grammar generates two trees in parallel and mappings between them • Rules can specify order changes • Restriction to binary rules limits complexity

35. Inversion transduction grammars

36. Inversion transduction grammars • Grammar is trained on word-aligned bilingual corpus: Note that all the rules are learned automatically • Translation uses a decoder which effectively works like traditional RBMT: • Parser uses source side of transduction rules to build a parse tree • Transduction rules are applied to transform the tree • The target text is generated by linearizing the tree

39. Almost all possible mappings can be handled • Missing ones (crossing constraints) are not found in Wu’s corpus • But examples can be found, apparently

40. Hierarchical transduction model (Alshawi et al. 1998) • Based on finite-state transducers, also uses binary notation • Uses automatically induced dependency structure • Initial head-word pair is chosen • Sentence is then expanded by translating the dependent structures

41. Tree-to-string translation (Yamada & Knight 2001, Charniak 2003) • Uses (statistical) parser on input side only • Tree is then subject to reordering and insertion according to models learned from data • Lexical translation is then done, again according to probability models

42. reorder insert translate wa wa linearize kare ha ongaku wo kiku no ga daisuki desu wa

43. Tree-to-tree translation (Gildea 2003) • Use parser on both sides to capture structurual differences • Subtree cloning (Habash 2002, Čmejrek et al. 2003) • Full morphology/syntactic/semantic parsing • All based on stachastic grammars

44. Latest developments in RBMT • RBMT making a come-back (e.g. METIS) • Perhaps it was always there, just wasn’t represented in CL journals/conferences • There is some activity, but around the periphery • Open-source systems • development for low-density languages • Much use made of corpus-derived modules, eg tagging, chunking • SMT is now RBMT, only the rules are learned rather than written by linguists

45. Overview • The story so far • EBMT • SMT • Latest developments in RBMT • Is there convergence? • Some attempts to classify MT • (Carl and Wu’s MT model spaces) • Has the empire struck back?

46. Classifications of MT • Empirical vs. Rationalist • data- vs theory-driven • use (or not) of symbolic representation • From MLIM chapter 4: • high vs. low coverage • low vs. high quality/fluency • shallow vs. deep representation • Distinguish in the above • design vs. consequence • How true are they anyway?

47. ! EBMT~SMT: Is there convergence? • Lively debate on mtlist • Articles by • Somers, Turcato & Popowich in Carl & Way (2003) • Hutchins, Carl, Wu (2006) in special issue of Machine Translation • Slides marked need your input!

48. Essential features of EBMT • Use of bilingual corpus data as the main (only?) source of knowledge (Somers) • Most early EBMT systems were hybrids • We do not know a priori which parts of example are relevant (Turcato & Popowich) • Raw data is consulted at run-time: (little or) no preprocessing • Therefore template-based EBMT is already a hybrid (with RBMT) • Act of matching the input against the examples, regardless of how they are stored (Hutchins)

49. Pros (and cons) of analogy model • Like CBR: • Library of cases used during task performance • Analogous examples broken down, adapted, recombined • In contrast with other machine learning methods • Offline learning to compile abstract performance model • No loss of coverage due to incorrect generalization during training • Guaranteed correct when input is exactly like an example in the training set (not true of SMT) • But: Lack of generalization leads to potential runtime inefficiency (Wu, 2006)

50. EBMT~SMT: Common features • Easily agreed • Use of bilingual corpus data as the main (only?) source of knowledge • Translation relations are derived automatically from the data • Underlying methods are independent of language-pair, and hence of language similarity • More contentious • Bilingual corpus data should be real (a practical issue for SMT, but some EBMT systems use “hand-crafted” examples) • System can be easily extended just by adding more data