1 / 21

Error Detection with Error Rules

Error Detection with Error Rules. Chong Min Lee. Error Diagnosis Methods (1). Spell Checker Pattern matching It has been widely used in CALL The author/teacher provides CALL system with a pattern. Student answer should be identical with the pattern Finite State Automata

pink
Download Presentation

Error Detection with Error Rules

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Error Detection with Error Rules Chong Min Lee

  2. Error Diagnosis Methods (1) • Spell Checker • Pattern matching • It has been widely used in CALL • The author/teacher provides CALL system with a pattern. • Student answer should be identical with the pattern • Finite State Automata • Constraint relaxation • Probably the most widely used technique for parsing ungrammatical sentences • Constraints exist in grammar. • S  NP VP with number agreement constraint • Some predicates can violate the constraints. • Detailed explanation will be given in the next class

  3. Error Diagnosis Methods (2) • Error Grammar • Use of an error grammar implies a rule-based system which has a set of grammar rules to treat grammatical input and another set to handle ill-formed input. • Only errors planned in advance can be detected with error grammars • Error messages must be attached to the error rules to be displayed whenever an error rule succeeds.

  4. Systems using error grammars (1) • German Tutor • Weischedel, Voge, and James (1978) • German Tutor • It has rules for common student mistakes. • When it can produce a correct parse only by invoking one of its mistake-rules, it makes a note that a particular mistake has been detected. • This system had only a small grammar and a vocabulary of less than 200 words. • It demonstrated the possibility of using an error grammar to detect some syntactic error types.

  5. Systems using error grammars (2) • EAES (Error Analysis and Explanation System) • It uses error rules to detect three types of errors: low-level and high-level syntactic errors, and semantic errors. • Low level syntactic errors: the omission or the addition of functional words such as articles or prepositions • High level syntactic errors: the permutation of groups of words such as the misplacement of the verbs or of adjectives. • Semantic errors: the violation of semantic restrictions on verbs and their complements.

  6. Systems using error grammars (3) • The FROG – FGA – LINGER – ISCA family • Each new system is built, at least in part, on its predecessor. • Syntactic structure is built first. • Then, errors that are stored on the tree are reported. • Error types: word is in wrong place; word is missing; word should not be present; one word should not be present and another word is missing. • ICICLE • A CALL system focusing on the needs of American Sign Language (ASL) native signers learning English • Foster and Vogel (2004) (F&V) • A robust parsing approach • An error grammar is derived from a conventional grammar on the basis of an analysis of a corpus of observed ill-formed sentences.

  7. The components of error grammar system Input Lexicon Rules for correct sentences Parser Error Rules

  8. The components of error grammar system Input Lexicon Grammatical Rules Parser Error Rules ICICLE: Grammatical and error rules are simultaneously applied to parsing process

  9. The components of error grammar system Input Lexicon Grammatical Rules Parser Error Rules F&V: Error rules are applied to sentence that parser fails to analyze

  10. Error corpus (from F&V) • The taxonomy is the result of the analysis on error corpus composed of 16,000 word tokens. • Error corpus construction process • Error sentence is added to the corpus • A note is made of where the sentence occurred. • The error in the sentence is diagnosed and based on this diagnosis, the sentence is corrected. • The corrected sentence is added to a parallel corpus of well-formed sentences. • A note is made of what was done to correct the sentence. For example, to correct the sentence • Are people really capable to understanding them? • The infinitival marker to is replaced by the preposition of.

  11. Error correction taxonomy (from F&V) • An interesting point to note is that 89% of the corrections made involved the application of just one correction operator. (F&V) • “Closer examination of our writing samples indicates that, except for determiners, our users generally leave out at most one word (constituent) per sentence.” (Schneider & McCoy) • Replace a word • Spelling: nor  not (changing at most two letters) • Agreement: He am  He is • Verb Form: tell  telling • Same root/different syntactic category: syntactic  syntactically • Prepositions: for  of • Case: rivers  river’s • Auxiliary verbs, Synonymous verbs, etc.

  12. Error correction taxonomy (from F&V) • Adding a word • Determiners, verbs, prepositions, pronouns, infinitival, etc. • Deleting a word • Repeated words • Repeated similar words: Why is do they appear? • Unnecessary words: most of them are prepositions, determiners and verbs. • Schneider & McCoy (1998): omission and number disagreement error types were explained in the paper.

  13. Error rules of F&V • Error rules are generated for all the words. • Replace a word • Verb(sing, third)  [is] • Verb(sing, third) spellop [in] • Verb(sing, third) spellop [it] • … • Verb(sing, third) agreeop [are] • … • Verb(sing,third) vformop [be] • …

  14. Error rules of F&V • Add a word • An error rule is generated which has the same right-hand side as the original rule except that a pre-terminal symbol on the right-hand side is removed. • Np(Num,Per)  det(Num,Per),nbar(Num,Per) • Np(Num,Per) missingop nbar(Num,Per) • For unary rules where the only thing on the right-hand side is a pre-terminal symbol no error rules with an empty right-hand side are generated. • Nbar(Num,Per)  noun(Num,Per) • For each rule which has the pre-terminal symbol somewhere on its right-hand side, a corresponding error rule is generated which omits this category. • Np(Num,Per)  det(Num,Per),nbar(Num,Per) • Np(Num,Per) missingop det(Num,Per)

  15. Error rules of F&V • Add a word (continued) • An error rule isn’t generated if its right-hand side is the same as the right-hand side of a conventional rule. • Nbar(num,Per)  adj, n(Num,Per) • Nbar(Num,Per) missingop n(num,Per) => not generated • Delete a word • For each rule in the grammar, an error rule is generated which has the same right-hand side as the original rule except that a symbol is added after a preterminal symbol. • This symbol must be capable of matching with any pre-terminal symbol in the grammar. • Vp(Num,Per)  v_trans(Num,Per), np(_,_) • Vp(Num,Per) extraop v_trans(Num,Per),preterm,np(_,_)

  16. Error rules of Schneider & McCoy (1998) • “Closer examination of our writing samples indicates that, except for determiners, our users generally leave out at most one word (constituent) per sentence.” (Schneider & McCoy) • Agreement • Grammatical rule: • DP(agr ?a)  Det(agr ?a) NP(agr ?a) • Mal-rules: • DP(agr s)(error + )  Det(agr (?! s)) NP(agr s) • DP(agr p)(error + )  Det(agr (?! p)) NP(agr p) • Omission (Add a word) • One suggestion • VP(missing ?a)  V NP NP(missing ?a) • VP(missing ?a)  V NP(missing ?a) NP • VP(missing ?a)  V(missing ?a) NP NP

  17. Error rules of Schneider & McCoy (1998) • Omission (continued) • Another suggestion • (missing +) feature can be used as foot feature • A foot feature moves features from any child to the parent. • Problem: If there are two omissions in a sentence. • ‘Student always bothering me while I am at dorm.’ • => Studentsare always bothering me while I am at the dorm.

  18. S&M processing example • Grammar

  19. S&M processing example Input: 0a1 boys2

  20. Evaluation • S&M • Test set: 79 sentences with errors • 44 (56%) parse with the expected type of error. • 23 (29%) fails to parse • 12 (15%) parse as having no errors at all • 9 sentences have semantic/pragmatic errors • 44 (66%) parse with the expected type of error • 23 (33%) fails to parse • 3 (4%) parse as having no errors at all • F&V • Test set: 50 ill-formed sentences • Accuracy: 84% • 8 sentences are failed • 5: more than one error • 3: chart parse passes the sentences

  21. Challenges • Schneider & McCoy (1998) • The focus of the paper was on recognition of agreement-type problems. So, the system only covered the following errors • NUM: Number problems, which are typically errors in subject-verb agreement • ED: extra determiner • MD: missing determiner for an NP that requires a determiner. • ID: incorrect determinder • F&V • The system can process only one error in a sentence. If there are more than one error, the system fails to process the sentence.

More Related