QA and Opinions

1. QA and Opinions Eduard Hovy USC/Information Sciences Institute Kathy McKeown Columbia University

2. What’s your opinion? • Given any opinion-oriented topic, your opinion is • pro • anti • undecided • some combination • and occasionally, something altogether different • Should there be term limits? • Article NYT021109.15-311 says: • 27% statements pro • 12% statements anti • 3% statements undecided • 10% some combination • 0% statements for something different • So what else? • Who holds the opinion • What is their reasoning (sources, evidence, and steps)

3. Opinions come in arguments Abstract argument structure • Lots of literature: Toulmin, Aristotle, etc.

4. Simple WHO • US public pro Should there be term limits? • <S SNTNO="6">Last month's Wall Street Journal/NBC News poll showed Americans back term limits by 75% to 21% nationwide.</S> • <S SNTNO="7">Those earning less than $20,000 a year supported term limits by 77% to 16%. Democrats and blacks both gave term limits 71% support.</S> • <S SNTNO="8">Women favored term limits more than men.</S> • Should NAFTA be accepted? • <S HSNTNO="1">FT 12 NOV 93 / US public opinion swings behind Nafta</S> • 75% Americans pro • Those earning under $22K pro • Democrats and blacks pro • More women than men pro

5. Some groups anti • Reason: • Should the census count illegal aliens? • <S SNTNO="4">Groups which have filed suit to ignore the aliens contend large concentrations of them could result insome states gaining seats in the House of Representatives at the expense of other states.</S> • <S SNTNO="5">Meanwhile, other groups want the final census totals to be increased to account for people who may be overlooked in the census—most often blacks and Hispanics living in urban areas.</S> • Some groups pro • Reason: Simple WHY • Should the census count illegal aliens? • <S SNTNO="4">Groups which have filed suit to ignore the aliens contend large concentrations of them could result in some states gaining seats in the House of Representatives at the expense of other states.</S> • <S SNTNO="5">Meanwhile, other groups want the final census totals to be increased to account for people who may be overlooked in the census—most often blacks and Hispanics living in urban areas.</S> …but usually it’s not this simple…

6. A little inference Should the census count include illegal aliens? • <S SNTNO="15">But Reps. Tom Petri, R-Wis. and William F. Goodling, R-Pa., asserted that counting illegal aliens violates citizens' basic right to equal representation by giving •  Petri, Wis, Golding = anti — by “violating basic rights” • Should the census count include illegal aliens? • <S SNTNO="3">``This is a fairness issue,'' said Rep. Thomas J. Ridge, R-Pa., who contended that states with large numbers of illegal aliens benefit unfairly when their large population totals give then extra seats in the House.</S> •  Ridge = anti — by “benefiting unfairly” • Should there be term limits? • <S SNTNO="1">As an economist, Robert Barro ("A Free Marketeer's Case Against Term Limits" editorial page, Dec. 24) sees no difference between minimum-wage legislation, rent control and other government interventions in the free market, and the intervention of congressional term limitation.</S> •  Barro = anti — by “no difference…and other interventions”

7. More complex inference • Should NAFTA be accepted? • <S SNTNO="7">'We are absolutely for free trade,' says Mr Bill Diggitt, the state director of Mr Perot's United We Stand.</S> • <S SNTNO="8">'But Nafta undermines our constitution.</S> • <S SNTNO="9">It puts decision-making in the hands of international panels, which undermines our judicial system.'</S> • <S SNTNO="10">He says he is concerned about the loss of Virginia's tax base as companies move production to Mexico and place continuous downward pressure on wages.</S> •  Bill Diggett = anti — by“We are absolutely for free trade, BUT Nafta undermines our constitution”

8. # annotators / doc 2 3 A small pilot study • Source material: 20 editorials + 9 rants from web • Annotation: 4 students classified sentences • Results: Sometimes high agreement, but sometimes very little • Then tried to learn classifiers (Naïve Bayes and C4.5) to automatically annotate sentences 0: argument 1: claim 2: source 3: reason (4: other)

9. Results of classifiers • Precision (varying POS, sent positions, ngrams) • BUT: dumb baseline (always choose 4) gives average score of 0.643! • SO: reweight counts to reward non-4 overlaps… • …Naïve Bayes better than C4.5

10. The plan • Download trial 25 texts (5 topics) from http://www.isi.edu/natural-language/projects/Opinions/opinion-sources.tar • Challenge problems: 1. Given a topic (e.g., “Should abortion be banned?”) and a set of texts about the topic, return sets of sentences (or clauses), where each set represent a distinct opinion 2. Also return the holder(s) 3. Also return their reason(s) • By hand produce an answer set for each opinion • Answer following questions

11. Questions • What is the granularity of an opinion? (If one person is for abortion and another is for it but only in cases when the mother’s life is at risk, are these two opinions or one?) • How easy or necessary is it to collect additional information as well? (Holders; Justification/Reasoning; Source/Authority; Conditions; etc.) • How can we evaluate the results? Proposal: • number of opinions/sets, as created by people • correctness of each found set (set precision) • number of missing sets (set recall) • correctness of each sentence within each set (sentence precision) • number of sentence missing in each set (sentence recall)

12. Next steps • Help define the task • Do a hand simulation • Download the training material • From the website • New material from NIST • Participate in the pilot evaluation next semester • By hand • By machine

13. So what’s your opinion?What to do with opinions?