A Taxonomy of Questions for Question Generation

1. A Taxonomy of Questions for Question Generation Rodney D. Nielsen1,2, Jason Buckingham, Gary Knoll, Ben Marsh and Leysia Palen 1 Boulder Language Technologies, Boulder, CO 2 Center for Computational Language and Education Research, CU, Boulder

2. The Goal of the Taxonomy • HCI design for a Socratic tutoring system • Key output was mapping from a classification of learner interactions to an appropriate tutoring response • Included elaborate taxonomies for learner & tutor • Process • Start with current research on tutoring • Analyze human tutoring transcripts • Six subject areas spanning elementary school reading comprehension through college-level conceptual physics

3. Relation to Question Generation • Roadmap for question generation • View QG as a 3-step process: • Concept Selection • Question Type Determination • Question Construction • Pedagogical theory • Automatic question classification

4. Question Branch of Taxonomy • Started with question type list in Graesser & Person (1994), adapted from Lehnert (1978) • Added Bloom’s (1956) Taxonomy of Educational Objectives and other schemes • Iterative process (4 annotators): • Analyzed human tutoring transcripts • Annotated dialog acts according to taxonomy • Separately, revised taxonomy per transcripts • Met to review and integrate separate taxonomies • Redundancy, usefulness, completeness • Would change facilitate more effective tutoring / learning

5. Comparison with Prior Work • Addition of secondary dimensions • Hierarchical structure of primary taxonomy • Added a few question classes • Moved some classes to secondary dimensions • Verification and Disjunctive questions

6. Question Taxonomy ORTHOGONAL CLASSIFICATIONS Collins’ Question Type (Collins, 1985): Form hypothesis, Test hypothesis, Make prediction, Trace consequences, Entrapment, or None. Bloom’s Taxonomy of Educational Objectives, top level (Bloom, 1956): Knowledge, Comprehension, Application, Analysis, Synthesis, or Evaluation. Content Level: Amount of answer content in the question Example Usage (adapted from Collins, 1985): Positive Paradigm Case, Negative Paradigm Case, Negative Exemplar for a Necessary Factor (near miss), Positive Exemplar for an Unnecessary Factor (near hit), Generalization Exemplar for a Factor (maximal pair), Differentiation Exemplar for a Factor (minimal pair), Exemplar to Show the Variability of a Factor, Exemplar to Show the Variability of the Dependent Variable, Counterexample for Insufficient Factors, Counterexample for Unnecessary Factors, Analogy, Continuation of Example, Reuse of Example, None. Response Form: This indicates the type and length of the expected response: Boolean, Multiple Choice, Word, Phrase, Sentence or Paragraph. Question Relation: Relationship of the current question to preceding questions Connection Question PRIMARY CLASSIFICATION • Description Questions • Concept Completion: Who, what, when, where? • Definition: What does X mean? • Feature Specification: What features does X have? • Composition: What is the composition of X? • Example: What is an example of X? • Method Questions • Calculation: Compute or calculate X. • Procedural: How do you perform X? • Explanation Questions • Causal Antecedent: What caused X? • Causal Consequence: What will X cause? • Enablement: What enables the achievement of X? • Rationale Questions • Goal Orientation: What is the goal of X? • Justification: Why is X the case? • Comparison Questions • Concept Comparison: Compare X to Y? • Judgment: What do you think of X? • Improvement: How could you improve upon X? • Preference Questions • Free Creation: requires a subjective creation. • Free Option: select from a set of valid options.

7. Question Taxonomy PRIMARY CLASSIFICATION • Description Questions • … • Method Questions • … • Explanation Questions • … • Comparison Questions • … • Preference Questions • … • ORTHOGONAL CLASSIFICATIONS • Collins’ Question Type • Bloom’s Taxonomy of Educational Objectives • Content Level • Example Usage • Response Form • Question Relation • Connection Question

8. Examples • What happens to the pitch as a guitar string gets longer? It becomes: a) higher, b) lower, c) louder, or d) softer • Causal Consequence, Comprehension, Multiple Choice, … • Describe one way to lower the pitch produced by plucking a wire? • Causal Antecedent, Application, Sentence, …

9. Roadmap • Most current question generation • Bloom’s Knowledge level • Subclasses of Description questions • Progress across Bloom’s Taxonomy • Some Method & Comparison questions • Explanation questions & question series • Finally Collins’ categories & examples • All types are within reach w restrictions

10. Summary of Taxonomy Uses • Roadmap • QG as a 3-step process • Concept Selection • Question Type Determination - output • Question Construction – input • Question Classification • Learning appropriate question types • Evaluating system output

11. Thanks! • Thanks to Steve Bethard, the CU Computational Semantics Group and the anonymous reviewers for helpful feedback. • This work was partially funded by Award Numbers: • NSF 0551723, • IES R305B070434, and • NSF DRL-0733323.