1 / 44

Automated Assessment for Adaptive Learning of Complex Tasks

Automated Assessment for Adaptive Learning of Complex Tasks. Alan Koenig Markus Iseli Allen Munro. CCT. Characteristics of Cognitively Complex Tasks. Often require multiple, non-trivial steps to complete Performance can be highly variable Often include interdependent task features

xanto
Download Presentation

Automated Assessment for Adaptive Learning of Complex Tasks

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Automated Assessment for Adaptive Learning of Complex Tasks Alan Koenig Markus Iseli Allen Munro CCT

  2. Characteristics of Cognitively Complex Tasks • Often require multiple, non-trivial steps to complete • Performance can be highly variable • Often include interdependent task features • Often well suited to representation in games & simulations • Multiple scenarios • Low cost • Low risk • But… assessment can be difficult Examples: Tactical Planning Simulation Damage Control Simulation

  3. Anti-Submarine Warfare Anti-submarine warfare (ASW) is a branch of naval warfare that uses surface warships, aircraft, or other submarines to find, track and deter, damage, or destroy enemy submarines. • Carried out by teams of people within and across ships • Requires synthesis of large quantities of information in short periods of time • Heavy emphasis on situation awareness and decision making skills (especially among TAO’s)

  4. Assessment Methodology Overview

  5. ASW OntologyConstruction Experts Rules / Doctrine DOMAIN Facts / Observations Procedures / Processes

  6. Ontology Excerpt

  7. Bayesian Network Development Ontology Nodes Latent Variables SonarPlanning Directionality of Links PassiveSonar Planning Active Sonar Planning Strength of Relationships Observable Actions UseShipPassiveSonar DropPassiveSonobuoys Observables / SME Input UseDippingSonar DropActiveSonobuoys UseShipActiveSonar Characterizing the nature of relationships is essential

  8. Bayesian Network for TDA Avoidance ASW Skilled MEUProtection TDA Avoidance Torpedo CounterMeasures Sub Hunting TorpedoEscape TorpedoDeception TDA AvoidManeuvers SonarPlanning Plan for Opposition TorpedoEvasion TorpedoCounter NixieDropped MICDropped Opposition Estimation Intell on Opposition AlterCourseToAvoidFast AlterCourseToAvoidSlow PassiveSonar Use Active Sonar Skill Datum Estimation DontEverEnterTDA ShowMissionInfo ShowBriefing LimLines Estimation UseShipPassiveSonar DropPassiveSonobuoys 0.5 EstDatumSubSpeed EstDatumTorpedoRange DropActiveSonobuoys UseShipActiveSonar EstLLSubSpeed EstLLTorpedoRange UseDippingSonar

  9. Bayesian Network for TDA Avoidance ASW Skilled MEUProtection TDA Avoidance Torpedo CounterMeasures Sub Hunting TorpedoEscape TorpedoDeception TDA AvoidManeuvers SonarPlanning Plan for Opposition TorpedoEvasion TorpedoCounter NixieDropped MICDropped Opposition Estimation Intell on Opposition AlterCourseToAvoidFast AlterCourseToAvoidSlow PassiveSonar Use Active Sonar Skill Datum Estimation DontEverEnterTDA ShowMissionInfo ShowBriefing LimLines Estimation UseShipPassiveSonar DropPassiveSonobuoys 0.5 EstDatumSubSpeed EstDatumTorpedoRange DropActiveSonobuoys UseShipActiveSonar EstLLSubSpeed EstLLTorpedoRange UseDippingSonar

  10. Conditions for TDA Avoidance to be considered by Bayesian network

  11. The Advantage of Bayesian Networks • Probability of mastery of the latent variables is inferred from the scored observable actions Latent Variables Example: Determining probabilities SonarPlanning P(ASP): Probability of mastery of concept Active Sonar Planning P(UDS | ASP): Conditional probability. Probability of mastery of concept UseDippingSonar, given information about mastery of concept Active Sonar Planning PassiveSonar Planning Active Sonar Planning P(UDS):Probability of mastery of concept UseDippingSonar P(ASP) x P(UDS | ASP) Observable Actions P(ASP | UDS) = P(UDS) UseShipPassiveSonar DropPassiveSonobuoys UseDippingSonar DropActiveSonobuoys UseShipActiveSonar

  12. Real-Time Formative Assessment Observable Actions & Events CAA(with Bayesian Network) • FORMATIVE ASSESSMENT • Provide performance feedback • Provide practice / resources • Add / change tasks • Add / change affordances Simulation(Sandbox) • ADAPTIVE OPTIONS

  13. 1st PROBLEM: Chinese Kilos Threat Problem Check the Briefing—Good! 00:00:00 .k.Announce. ShowBriefing 0.95 The fact that we’re up against Kilos implies a silent sub speed of 6 Kt.and a torpedo range of 12,000 yds.

  14. 1st PROBLEM: Chinese Kilos Threat Problem Failed to Correct Estimates—Bad! 07:26:32 .k.Announce. EstSubSpeed 0.207:26:32 .k.Announce. EstTorpedoRange 0.2 Scored at end, because these values were never changed. The estimated sub speed was left at 3 Kt. So the limit lines were not accurate, relative to the briefing.

  15. 1st PROBLEM: Chinese Kilos Threat Problem Failed to Correct Estimates—Bad! 07:26:32 .k.Announce. DontEverEnterTDA 0.1 Uh-oh! The MEU entered the TDZ.

  16. Problem 1

  17. Problem 1

  18. Problem 1

  19. Problem 1

  20. Problem 1

  21. 2nd PROBLEM: Threat West of Gibraltar Check the Briefing—Good! 00:00:00 .k.Announce. ShowBriefing 0.95 This time the mission brief gives specific estimates of hostile sub speed and torpedo range.

  22. 2nd PROBLEM: Threat West of Gibraltar Datum Estimates The datum estimates forsub speed and torpedorange don’t match the briefing.

  23. 2nd PROBLEM: Threat West of Gibraltar Datum Estimates Corrected—Good! 01:14:03 .k.Announce. EstSubSpeed 0.9501:14:03 .k.Announce. EstTorpedoRange 0.95 Scored at end. Set to the correct values.

  24. 2nd PROBLEM: Threat West of Gibraltar Check Limit Lines Perhaps not a wide enough berth…

  25. 2nd PROBLEM: Threat West of Gibraltar Plot a New Course A safer course?

  26. 2nd PROBLEM: Threat West of Gibraltar Check Limit Lines Again Looks safer

  27. 2nd PROBLEM: Threat West of Gibraltar Successful Avoidance 01:14:03 .k.Announce. DontEverEnterTDA 0.95 Success!

  28. Problem 2

  29. Problem 2

  30. Problem 2

  31. Problem 2

  32. TDA Avoidance Probability of Mastery Entered TDA time Plan for Opposition Sub speed: Wrong estimate Probability of Mastery time

  33. TDA avoidance Bayesian network sub-net ASW Skilled TDA Avoidance Torpedo CounterM. Sub Hunting MEUProtection Plan for Opposition TDA AvoidManeuvers SonarPlanning Opposition Estimation Intelligence on Opposition ManeuverSlow ManeuverFast B16: score 0.95 D16: score 0.7 F16: score 0.05 B20: score 0.95 D20: score 0.6 F20: score 0.1 GetMissionInfo GetBriefing EstSubSpeed EstTorpedoRange B5: score 0.9 D5: score 0.7 B12: score 0.95 D12: score 0.8 F12: score 0.2 B6: score 0.95 D6: score 0.7 B11: score 0.95 D11: score 0.8 F11: score 0.2 DontEnterTDAIfPos B24: score 0.95 C24: score 0.9 F24: score 0.1 Sense Loudly Sense Quietly ShipsActivSonar ActiveSonobuoys DippingSonar ShipsPassivSonar PassiveSonobuoys B40: score 0.95 C40: score 0.4 E40: score 0.3 C44: score 0.9 D44: score 0.6 F44: score 0.05 B36: score 0.9 E36: score 0.3 F36: score 0.2 B28: score 0.95 D28: score 0.6 B32: score 0.95 E32: score 0.3

  34. Initial Evaluation Feedback Instructional Feedback High Level Assessments CAA Debugging Output

  35. First Successful Problem

  36. Into the Next Problem

  37. Adaptive Problem Selection A new problem is advised.

  38. Loading the Problem The advised problem is selected

  39. Initial Actions in the New Problem

  40. Preparing to Find the Sub Launching a helicopter

  41. Detecting the Sub Sonobuoys detect sub

  42. Recording of Performances & Assessments CAA Report Generation (CAA Monitor) Assessment History (Per Student) Log Files (Actions and Events) These performance recordscan be replayed. Simulation(Sandbox) Report

  43. Questions / Comments? CCT

More Related