(Some issues in) Text Ranking

1. (Some issues in) Text Ranking

2. Recall General Framework • Crawl • Use XML structure • Follow links to get new pages • Retrieve relevant documents • Today • Rank • PageRank, HITS • Rank Aggregation

3. Relevant documents • Usually: relevant with respect to a keyword, set of keywords, logical expression.. • Closely related to ranking • “How” relevant is it can be considered another measure • Usually done as a separate step • Recall the Online vs. offline issue.. • But some techniques are reusable

4. Defining Relevant Documents • Common strategy: treat text documents as “bag of words” (BOW) • Denote BOW(D) for a document D • Bag rather than set (i.e. multiplicity is kept) • Words are typically stemmed • Reduced to root form • Loses structure, but simplifies life • Simple definition: • A document D is relevant to a keyword W if W is in BOW(D)

5. Cont. • Simple variant • The level of relevance of D to W is the multiplicity of W in BOW(D) • Problem: Bias towards long documents • So divide by the document length |BOW(D)| • This is called term frequency (TF)

6. A different angle • Given a document D, what are the “most important” words in D? • Clearly high term frequency should be considered • Rank terms according to TF?

7. Ranking according to TF A 2022 Is 1023 He 350 . . . Liverpool 25 Beatles 12

8. IDF • Observation: if w is rare in the documents set, but appears many times in a document D, then w is “important” for D • IDF(w) = log(|Docs| / |Docs’|) • Docs is the set of all documents in the corpus, Docs’ is the subset of documents that contain w • TFIDF(D,W)=TF(W,D)*IDF(W) • “Correlation” of D and W

9. Inverted Index • For every term we keep a list of all documents in which it appears • The list is sorted by TFIDF scores • Scores are also kept • Given a keyword it is then easy to give the top-k

10. Ranking • Now assume that these documents are web pages • How do we return the most relevant? • How do we combine with other rankings? (e.g. PR?) • How do we answer boolean queries? • X1 AND (X2 OR X3)

11. Rank Aggregation • To combine TFIDF, PageRank.. • To combine TFIDF with respect to different keywords

12. Part-of-Speech Tagging • So far we have considered documents only as bags-of-words • Computationally efficient, easy to program, BUT • We lost the structure that may be very important: • E.g. perhaps we are interested (more) in documents for which W is often the sentence subject? • Part-of-speech tagging • Useful for ranking • For machine translation • Word-Sense Disambiguation • …

13. Part-of-Speech Tagging • Tag this word. This word is a tag. • He dogs like a flea • The can is in the fridge • The sailor dogs me every day

14. A Learning Problem • Training set: tagged corpus • Most famous is the Brown Corpus with about 1M words • The goal is to learn a model from the training set, and then perform tagging of untagged text • Performance tested on a test-set

15. Simple Algorithm • Assign to each word its most popular tag in the training set • Problem: Ignores context • Dogs, tag will always be tagged as a noun… • Can will be tagged as a verb • Still, achieves around 80% correctness for real-life test-sets • Goes up to as high as 90% when combined with some simple rules

16. (HMM)Hidden Markov Model • Model: sentences are generated by a probabilistic process • In particular, a Markov Chain whose states correspond to Parts-of-Speech • Transitions are probabilistic • In each state a word is outputted • The output word is again chosen probabilistically based on the state

17. HMM • HMM is: • A set of N states • A set of M symbols (words) • A matrix NXN of transition probabilities Ptrans • A vector of size N of initial state probabilities Pstart • A matrix NXM of emissions probabilities Pout • “Hidden” because we see only the outputs, not the sequence of states traversed

18. Example

19. 3 Fundamental Problems 1) Compute the probability of a given observation Sequence (=sentence) 2) Given an observation sequence, find the most likely hidden state sequence This is tagging 3) Given a training set find the model that would make the observations most likely

20. Tagging • Find the most likely sequence of states that led to an observed output sequence • Problem: exponentially many possible sequences!

21. Viterbi Algorithm • Dynamic Programming • Vt,k is the probability of the most probable state sequence • Generating the first t + 1 observations (X0,..Xt) • And terminating at state k

22. Viterbi Algorithm • Dynamic Programming • Vt,k is the probability of the most probable state sequence • Generating the first t + 1 observations (X0,..Xt) • And terminating at state k • V0,k = Pstart(k)*Pout(k,X0) • Vt,k= Pout(k,Xt)*max{Vt-1k’ *Ptrans(k’,k)}

23. Finding the path • Note that we are interested in the most likely path, not only in its probability • So we need to keep track at each point of the argmax • Combine them to form a sequence • What about top-k?

24. Complexity • O(T*|S|^2) • Where T is the sequence (=sentence) length, |S| is the number of states (= number of possible tags)

25. Computing the probability of a sequence • Forward probabilities: αt(k) is the probability of seeing the sequence X1…Xt and terminating at state k • Backward probabilities: βt(k) is the probability of seeing the sequence Xt+1…Xn given that the Markov process is at state k at time t.

26. Computing the probabilities Forward algorithm α0(k)= Pstart(k)*Pout(k,X0) αt(k)= Pout(k,Xt)*Σk’{αt-1k’ *Ptrans(k’,k)} P(O1,…On)= Σk αn(k) Backward algorithm βt(k) = P(Ot+1…On| state at time t is k) βt(k) = Σk’{Ptrans(k,k’)* Pout(k’,Xt+1)* βt+1(k’)} βn(k) = 1 for all k P(O)= Σk β0(k)* Pstart(k)

27. Learning the HMM probabilities • Expectation-Maximization Algorithm • Start with initial probabilities • Compute Eij the expected number of transitions from i to j while generating a sequence, for each i,j (see next) • Set the probability of transition from i to j to be Eij/ (ΣkEik) 4. Similarly for omission probability 5. Repeat 2-4 using the new model, until convergence

28. Estimating the expectancies • By sampling • Re-run a random a execution of the model 100 times • Count transitions • By analysis • Use Bayes rule on the formula for sequence probability • Called the Forward-backward algorithm

29. Accuracy • Tested experimentally • Exceeds 96% for the Brown corpus • Trained on half and tested on the other half • Compare with the 80-90% by the trivial algorithm • The hard cases are few but are very hard..

30. NLTK • http://www.nltk.org/ • Natrual Language ToolKit • Open source python modules for NLP tasks • Including stemming, POS tagging and much more