How Does CRTD Support Software Learning?

1. How Does CRTDSupport Software Learning? Criterion Referenced Objectives as Foundation for Evaluation Julianne O’Neil, M.Ed. Siemens Healthcare

2. Goals • Recognize criterion-referenced test development (CRTD) integration within overall evaluation framework • Describe components of criterion-referenced objectives • Write criterion-referenced objectives with those components and consider use of mind mapping • Identify possible solutions to challenges implementing performance-based tests

3. 4 Levels of Evaluation Level 1(Reaction) Level 2 (Learning) Level 4 (Results) Level 3 (Behavior) • Learner satisfaction/reaction following training • Customer Satisfaction • Knowledge acquisition, skill development, attitude change • Measurement of learning acquisition • Organizational results attributed to training • Reduced costs, reduced support time, increased profits,improved productivity • Behavior change demonstrated on the job after training • “Training Transfer”

4. CRTD in Evaluation Framework Level 1(Reaction) Level 4 (Results) Level 3 (Behavior) • Learner satisfaction/reaction following training • Customer Satisfaction • Organizational results attributed to training • Reduced costs, reduced support time, increased profits,improved productivity • Behavior change demonstrated on the job after training • “Training Transfer” Level 2 (Learning) • HOW DO YOU KNOW WHAT THEY HAVE LEARNED? • Criterion-referenced objectives/tests

5. Objectives’ Focus on Performance • Traditional objectives and tests focus on declarative knowledge • Knowledge not guarantee of performance improvement • What are learners going to DO on job after training? • Want to improve performance to drive organizational success

6. CRTD in Evaluation Framework • Goal of Siemens Healthcare Evaluation Framework: • To ultimately evaluate Levels 1 thru 4 • Level 2 Evaluation Focus: • Identify desired performance • Establish/maintain standards for objectives/tests • Restructure objectives format criterion-referenced • Declarative knowledge performance improvement • Level 3 Evaluation Focus: • Data collection by written/phone survey • Survey based on CRTD objectives

7. Components of Criterion-Referenced Objectives • A – Audience • Learner/role • B – Behavior • Desired performance/skill • C – Condition • Circumstances under which behavior is performed • D – Degree • Extent to which the new behavior is to be demonstrated

8. Template for a CRTD Objective • Given <Under what CONDITION the learner will perform>, • The <WHO? AUDIENCEperforming the behavior> • Will <Do WHAT? BEHAVIOR> • Correctly <DEGREE - HOW OFTEN the learner must perform correctly>.

9. Examples of CRTD Objectives • Given an out-of-balance Patient Census in which the new Month-to-Date amount is not equal to the Actual Month-to-Date Total, the financial analyst will write the procedures required to balance the patient census with 100% accuracy. • Audience: Financial Analyst • Behavior: Write procedures to balance patient census • Condition: Out-of-balance patient census • Degree: 100% Accuracy

10. Examples of CRTD Objectives • The reporting analyst will track the progress of report output when presented with the Output Job Status utility to effectivelyreprint reports 100% of the time. • Audience: Reporting Analyst • Behavior: Track progress of report output and print reports • Condition:Output Job Status Utility • Degree: Effectively reprint 100% of time

11. Examples of CRTD Objectives • Using 3 sample pairs of sterile examination gloves, the nurse will demonstrate the proper technique for donning sterile gloves without tearing or touching the outside of the gloves at least 1 out of 3 tries. • Audience: Nurse • Behavior: Demonstrate proper technique for donning sterile gloves • Condition: Using sample sterile examination gloves • Degree: Without tearing or touching the outside of the gloves 1 out of 3 tries

12. Defining 2 Types of Objectives • Topic complexity determines: • Depth/breadth of objectives • What to test • Standard for what is to be tested (Siemens standard: Testterminal objectives) • Terminal Objectives: • Determine performance desired upon training completion • Include at beginning of course material • Describe what learners are to demonstrate • Include all content to be tested • Describe learning “above memory level” • Enabling Objectives: • Enable achievement of terminal objectives. • Define “teachable” content at chapter or modular level • Describe prerequisite knowledge/skills needed to satisfy terminal objectives

13. Siemens-Defined Standards for CRTD • CRTD OBJECTIVES: • Terminal objectives • Must be tested • Should be performance-based whenever possible • Must be defined using CRTD format • Enabling objectives • May be written if needed • May optionally be tested • May optionally be defined using CRTD format • Audience, Condition and Degree may be defined once for all objectives in group (vs. stated for each objective) • YOUR STANDARDS MAY BE DIFFERENT!

14. CRTD Objectives Activity • Instructional Goal: Select one from list in handout • Write 1 terminal objectives in CRTD format to address chosen goal • CRTD Template: Given <CONDITIONfor learner performance>, The <learner AUDIENCEperforming the behavior> Will <desired BEHAVIOR> Correctly <DEGREEof success>.

15. CRTD Objectives Activity • Write 2-3 enabling objectives to support the terminal objective just written • NOTE: These enabling objectives do not have to be written in CRTD format.

16. Siemens-Defined Standards for CRTD • CRTD END-OF-COURSE TESTS: • Test items • Must test terminal objectives • Optionally test enabling objectives (depending on complexity) • For performance-based objectives, practice opportunities must be provided during class • Test item types allowed • Performance-based • When “traditional” performance-based can’t be used: • Multiple-choice • Matching • Sequencing • NO: • Fill-in • Essay • True/False • YOUR STANDARDS MAY BE DIFFERENT!

17. Corresponding Test Items • Performance-based Test Item:Can test one or more objectives just by producing an end product/service. • SAMPLE TEST ITEM: • Download a model report, modify the report, and add it as a custom report: • Download report from the Report Catalog. • Save report as “LA<n>_ORE_Report_Catalog.rpt,” where <n>is your Student Number. • Open report in Crystal and change report header name to “Operational Reports Report Catalog List for <your name>.” • Add new report to the Report Catalog using the following values: • Report Name: “LA-<n>,” where <n>is your Student Number • Template: Your LA<n>_ORE_Report_Catalog.rptCrystal Report • Description: Learner Assessment • Filter: Create a new filter file using the available report parameters • Create a sub-folder under the SCR Class Reports folder with name “Folder_<n>,” where <n>is your student number and add your new report to “Folder_<n>” • Locate your report output. • Create an Export Package called “LA<n>_Assessment_<n>,” where <n>is your student number. Be sure the Export Package contains your report.

18. Corresponding Test Items • Multiple-Choice Item:NOTE: This is also performance-based… • 1) The Stored Procedure used by the model Dietary Report is • ORE_0100 • DIETARY_REPORT_SP • ORE_0105 • ORE_0030 • The Dietary Report does not use a Stored Procedure

19. MindMap Template

20. Sample Instructional Objectives MindMap

21. Terminal and Enabling Objectives

22. Considerations for Performance-Based CRTD Implementation at Siemens

23. Benefits of Performance-Based CRTD Implementation to Siemens….and Possibly to YOU!