Comparing and Combining Sentiment Analysis Methods

1. Comparing and Combining Sentiment Analysis Methods Pollyanna Gonçalves (UFMG, Brazil) MatheusAraújo (UFMG, Brazil) FabrícioBenevenuto(UFMG, Brazil) Meeyoung Cha (KAIST, Korea)

2. Sentiment Analysis on Social Networks • Key component of a new wave of applications that explore social network data • Summary of public opinion about: • politics, products, services (e.g. a new car, a movie), etc. • Monitor social network data (in real-time) • Common as polarity analysis (positive or negative)

3. Sentiment Analysis Methods • Which method to use? • There are several methods proposed for different contexts • There are several popular methods • Validations based on examples, comparisons with baseline, with use of limited datasets • There is not a proper comparison among methods • Advantages? Disadvantages? Limitations?

4. This talk • Compare 8 popular sentiment analysis methods • Focus on the task of detecting polarity: positivevs. negative • Combine methods • Deploy the methods in a system --- www.ifeel.dcc.ufmg.br

5. Methods & Methodology Comparing & Combining Ifeel System & Conclusions

6. Emoticons • Extracted from instant messages services • Skype, MSN, Yahoo Messages, etc. • Grouped as positive and negative

7. Linguistic Inquiry and Word Count (LIWC) • Lexical method (paid software) • Allows to optimize the lexical dictionary -> we used the default • Measures various emotional, cognitive, and structural components • We only consider sentiment-relevant categories such as positivity, negativity

8. SentiWordNet • Lexical approach based on the WordNet dictionary • Groups words in synonyms • Detects positivity, negativity, and neutralityof texts

9. PANAS-t • Lexical method adapted from a psychometric scale • Consists of a dictionary of adjectives associated to sentiments • Positive: Joviality, assurance, serenity, and surprise • Negative: Fear, sadness, guilt, hostility, shyness and fatigue

10. Happiness Index • Uses a well-known lexical dictionary namely Affective Norms for English Words (ANEW) • Produces a scale of happiness • 1 (extremely happy) to 9 (extremely unhappy) • We consider [1..5) for negativeand [5..9] for positive

11. SentiStrengh • Combines 9 supervised machine learning methods • Estimates the strength of positive and negative sentiment in a text • We used the trained model provided by the authors

12. SAIL/AIL SentimentAnalyzer (SASA) • Machine learning method, trained with Naïve Bayes’ model • Trained model implemented as a python library • Classify tweets in JSON format for positive, negative, neutral and unsure

13. SenticNet • Extract cognitive and affective information using natural language processing techniques • Uses the affective categorization model Hourglass of Emotions • Provides an approach that classify messages as positive and negative

14. Methodology • Comparison of coverage and prediction performance across different datasets • Dataset 1: human labeled • About 12,000 messages labeled with Amazon Mechanical Turk: • Twitter, MySpace, YouTube and Digg comments, BBC and Runners World forums • Dataset 2: unlabeled • Complete snapshot from Twitter (collected in 2009) ~2 billion tweets • Extracted tragedies, disasters, movie releases, and political events • Focus on the English messages

15. Methods & Methodology Comparing & Combining Ifeel System & Conclusions

16. What is the coverage of each method?

17. Coverage vs. Prediction Performance • Emoticons: best prediction and worst coverage • SentiStrenght: second in prediction and third in coverage

18. Prediction Performance across datasets • Strong variations across datasets

19. Prediction Performance across datasets • Worst performance for datasets containing formal text

20. Polarity Analysis • Detected only • positive • Sentiments! • Methods tend to detect more positive sentiments • Positive as positive is usually greater than negative as negative • Even disasters were • classified predominantly as positive

21. Combined Method • Combines 7, of the 8 methods analyzed • Emoticons, SentiStrength, Happiness Index, SenticNet, SentiWordNet, PANAS-t, SASA • Removed LIWC (paid method) • Weights are distributed according to the rank of prediction performance: • Higher weight for the method with highest F-measure • Emoticon received weight 7 and PANAS-t 1

22. Combined Method • Best coverage and second in prediction performance • 4 methods combined are sufficient

23. Methods & Methodology Comparing & Combining Ifeel System & Conclusions

24. iFeel(Beta version)www.ifeel.dcc.ufmg.br • Example for: • “Feeling too happy today :)“ • Deploys all methods, except LIWC • Allows to evaluate an entire file • Allows to change parameters on the methods

25. Conclusions • We compare 8 popular sentiment analysis methods for detecting polarity • No method had the best results in all analysis • Prediction performance largely varies according to the dataset • Most methods are biased towards positivity • We propose a combined method • Achieves high coverage and high prediction performance • Ifeel: methods deployed and easily available • Future work: Compare others methods like POMS and EMOLEX

26. Thank you! Questions? www.dcc.ufmg.br/~fabricio www.ifeel.dcc.ufmg.br fabricio@dcc.ufmg.br