Probabilistic models for corpora and graphs

1. Probabilistic models for corpora and graphs

2. Review: some generative models • Multinomial Naïve Bayes  C • For each document d = 1,, M • Generate Cd ~ Mult( C | ) • For each position n = 1,, Nd • Generate wn ~ Mult(W | γ) ….. WN W1 W2 W3 M γ

3. Review: some generative models • Multinomial Naïve Bayes  C • For each document d = 1,, M • Generate Cd ~ Mult( C | ) • For each position n = 1,, Nd • Generate wn ~ Mult(W | g[Cd]) ….. WN W1 W2 W3 M γ γ K

4. Review: some generative models • Multinomial Naïve Bayes  • For each class k=1,…K • Generate g[k] ~ … • For each document d = 1,, M • Generate Cd ~ Mult( C | ) • For each position n = 1,, Nd • Generate wn ~ Mult(W | g[Cd]) C ….. WN W1 W2 W3 M γ γ K

5. Review: some generative models • Multinomial Naïve Bayes  • For each class k=1,…K • Generate g[k] ~ Dir(b) • For each document d = 1,, M • Generate Cd ~ Mult( C | ) • For each position n = 1,, Nd • Generate wn ~ Mult(W | g[Cd]) C Dirichlet is a prior for multinomials, defined by k params (b1,…, bK) and b0=b1+…,+bK MAP for P(g=i|b) = (nk + bi)/(n +b0) ….. WN W1 W2 W3 M γ γ b K Symmetric Dirichlet: all bi ‘s are equal

6. Review – unsupervised Naïve Bayes • Mixture model: unsupervised naïve Bayes model  • For each class k=1,…K • Generate g[k] ~ Dir(b) • For each document d = 1,, M • Generate Cd ~ Mult( C | ) • For each position n = 1,, Nd • Generate wn ~ Mult(W | g[Cd]) C Z W N Same generative story – different learning problem… M b γ K

7. “Mixed membership” • Latent Dirichlet Allocation  • For each document d = 1,,M • Generate d ~ Dir( | ) • For each position n = 1,, Nd • generate zn ~ Mult( | d) • generate wn ~ Mult( | gzn) a z w N M γ b K

8. LDA’s view of a document

9. Review - LDA • Gibbs sampling • Applicable when joint distribution is hard to evaluate but conditional distribution is known • Sequence of samples comprises a Markov Chain • Stationary distribution of the chain is the joint distribution Key capability: estimate distribution of one latent variables given the other latent variables and observed variables.

10. D1: “Happy Thanksgiving!” D2: “Thanksgiving turkey” D3: “Istambul, Turkey K=2

11. D1: “Happy Thanksgiving!” D2: “Thanksgiving turkey” D3: “Istambul, Turkey K=2 a 3 2 1 z z z w w w N1 N2 N3 γ b K=2

12. D1: “Happy Thanksgiving!” D2: “Thanksgiving turkey” D3: “Istambul, Turkey K=2 a 1 2 3 z11 z12 z21 z22 z31 z32 happy thank thank turkey istam turkey γ b K=2

13. D1: “Happy Thanksgiving!” D2: “Thanksgiving turkey” D3: “Istambul, Turkey K=2 a 3 2 1 Step 1: initialize z’s randomly 1 2 2 1 1 1 happy thank thank turkey istam turkey γ b K=2

14. D1: “Happy Thanksgiving!” D2: “Thanksgiving turkey” D3: “Istambul, Turkey K=2 a 2 1 3 Step 1: initialize z’s randomly Step 2: sample z11 from: 1 2 2 1 1 1 happy thank thank turkey istam turkey γ b K=2 slow derivation: https://lingpipe.files.wordpress.com/2010/07/lda3.pdf

15. D1: “Happy Thanksgiving!” D2: “Thanksgiving turkey” D3: “Istambul, Turkey K=2 a 3 2 1 #times ‘happy’ in topic t -------------------------------- #words in topic t (smoothed byb) 2 2 1 1 1 happy thank thank turkey istam turkey γ b K=2

16. D1: “Happy Thanksgiving!” D2: “Thanksgiving turkey” D3: “Istambul, Turkey K=2 a 3 2 1 #times topic t in D1 ---------------------- #words in D1 (smoothed bya) 2 2 1 1 1 happy thank thank turkey istam turkey γ b K=2

17. D1: “Happy Thanksgiving!” D2: “Thanksgiving turkey” D3: “Istambul, Turkey K=2 a 3 2 1 Step 1: initialize z’s randomly Step 2: sample z11 from: Pr(z11=k|z12,…,z32,a,b) Step 3: sample z12 2 2 1 1 1 happy thank thank turkey istam turkey γ b K=2

19. Models for corpora and graphs

21. Stochastic Block models: assume 1) nodes w/in a block z and 2) edges between blocks zp,zq are exchangeable a b zi zi zj u auv N N2 for each node i, pick a latent class zi for each node pair i,j, pick an edge weight a by a ~ Pr(a | b[zi,zj])

22. Another mixed membership block model • pick a multinomial θ over pairs of node topics k1,k2 • for each node topic k, pick a multinomial ϕk over node id’s • for each edge e: • pick z=(k1,k2) • pick i~ Pr(. | ϕk1 ) • pick i~ Pr(. | ϕk2 )

23. Speeding up modeling for corpora and graphs

24. Solutions • Parallelize • IPM (Newman et al, AD-LDA) • Parameter server (Qiao et al, HW7) • Speedups: • Exploit sparsity • Fast sampling techniques • State of the art methods use both….

25. z=1 This is almost all the run-time, and it’s linear in the number of topics….and bigger corpora need more topics…. z=2 random z=3 unit height … …

26. KDD 09

27. z=t1 z=t1 z=t1 z=t2 z=t2 z=t2 z=t2 random z=t3 unit height z=t3 z=t3 … z=t3 … … … … … … …

28. z=s+r+q

29. If U<s: • lookup U on line segment with tic-marks at α1β/(βV + n.|1), α2β/(βV + n.|2), … • If s<U<r: • lookup U on line segment for r Only need to check t such that nt|d>0 z=s+r+q

30. If U<s: • lookup U on line segment with tic-marks at α1β/(βV + n.|1), α2β/(βV + n.|2), … • If s<U<s+r: • lookup U on line segment for r • If s+r<U: • lookup U on line segment for q Only need to check t such that nw|t>0 z=s+r+q

31. Only need to check occasionally (< 10% of the time) Only need to check t such that nt|d>0 Only need to check t such that nw|t>0 z=s+r+q

32. Trick; count up nt|dfor d when you start working on d and update incrementally Only need to store (and maintain) total words per topic and α’s,β,V Only need to storent|dfor current d Need to store nw|t for each word, topic pair …??? z=s+r+q

33. 1. Precompute, for each t, 2. Quickly find t’s such that nw|t is large for w Most (>90%) of the time and space is here… Need to store nw|t for each word, topic pair …??? z=s+r+q

34. 1. Precompute, for each t, 2. Quickly find t’s such that nw|t is large for w • map w to an int array • no larger than frequency of w • no larger than #topics • encode (t,n) as a bit vector • n in the high-order bits • t in the low-order bits • keep ints sorted in descending order Most (>90%) of the time and space is here… Need to store nw|t for each word, topic pair …???

36. Outline • LDA/Gibbs algorithm details • How to speed it up by parallelizing • How to speed it up by faster sampling • Why sampling is key • Some sampling ideas for LDA • The Mimno/McCallum decomposition (SparseLDA) • Alias tables (Walker 1977; Li, Ahmed, Ravi, Smola KDD 2014)

37. Alias tables O(K) Basic problem: how can we sample from a biased coin quickly? If the distribution changes slowly maybe we can do some preprocessing and then sample multiple times. Proof of concept: generate r~uniformand use a binary tree r in (23/40,7/10] O(log2K) http://www.keithschwarz.com/darts-dice-coins/

38. Alias tables Another idea… Simulate the dart with two drawn values: rx int(u1*K) ry  u1*pmax keep throwing till you hit a stripe http://www.keithschwarz.com/darts-dice-coins/

39. Alias tables An even more clever idea: minimize the brown space (where the dart “misses”) by sizing the rectangle’s height to the average probability, not the maximum probability, and cutting and pasting a bit. You can always do this using only two colors in each column of the final alias table and the dart never misses! mathematically speaking… http://www.keithschwarz.com/darts-dice-coins/

40. KDD 2014 • Key ideas • use variant of Mimno/McCallum decomposition • Use alias tables to sample from the dense parts • Since the alias table gradually goes stale, use Metropolis-Hastings sampling instead of Gibbs

41. KDD 2014 else the dart missed • q is stale, easy-to-draw from distribution • p is updated distribution • computing ratios p(i)/q(i) is cheap • usually the ratio is close to one

42. KDD 2014