1 / 42

Testing in the time of crisis

BILC PROFESSIONAL SEMINAR Stockholm , October 14 - 17, 2013 INNOVATIVE TEST DESIGNS AND FORMATS. Testing in the time of crisis. Lt.Col . F. Gratton. course of action. Summary :. Past situation (up to Sept 2013) New Course Of Actions adopted A do- it - all software Proposals.

alka
Download Presentation

Testing in the time of crisis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. BILC PROFESSIONAL SEMINAR Stockholm , October 14 - 17, 2013 INNOVATIVE TEST DESIGNS AND FORMATS Testing in the time of crisis Lt.Col. F. Gratton

  2. course of action Summary: • Past situation (up to Sept 2013) • New Course Of Actionsadopted • A do-it-all software • Proposals

  3. Stanag Proficiency Test 1.0 RECEPTIVE SKILLS (Listening & Reading) 1: Multilevel test (level 1 to 4) 2: Multiple choice questions (60 for L & R) 3: No penalties for wrong answers 4: Duration: R 105’ / L 90’ 5:Separate Sections (& levels) 6: # of correctanswersmultiplied for a coefficient 7: Potentialuse of “F” factor

  4. How STANAG levels wereawarded(Stanag Proficiency Test 1.0) RECEPTIVE SKILLS # of correctanswers Multipliedby fixedcoefficient (1,66)

  5. PRODUCTIVE SKILLS Proficiency Test 1.0 SPEAKING(holisticallyassessed) 1: Functionallanguageassessed in a global manner 2: Structured interview 3: Tailored to the candidate 4: Checks & probes 5: 1 to 2 roleplays

  6. JFLT 1.0 PRODUCTIVE SKILLS WRITING (holisticallyassessed) Three tasks (one for each level)

  7. Summary: • Past situation • New COAsadopted • A do-it-all software • Proposals

  8. New COAsadopted • Specifications • Cut-off score • Joint Database

  9. PURPOSE • LEVELS OF LINGUISTIC KNOWLEDGE • TEST CONTENT • TEST FORMAT • VALIDATION PROCEDURES • ADMINISTRATION PROCEDURES

  10. Stanag Proficiency Test 1.0 TWAS HIGH TIME!!! RECEPTIVE SKILLS (Listening & Reading) 1: Multilevel test (level 1 to 4) 2: Multiple choice questions (60 for L & R) 3: No penalties for wrong answers 4: Duration: R 105’ / L 90’ 5:Separate Sections (& levels) 6: # of correctanswersmultiplied by a coefficient 7: Potentialuse of “F” factor

  11. Stanag Proficiency Test 2.0 New Key-factors • Eachsection is a mini-test (L & R) • Plus levels • Percentages

  12. New COAsadopted • Specifications • Cut-off score • Joint Database

  13. JFLT 2.0: RDS (Listening & Reading)

  14. EXAMPLE • Level 1 (15 questions) • Level 2 (15 questions) • Level 3 (15 questions) • Max score: 45 Allcandidatesanswercorrectly to 30 questions With Old test theywouldallget the same score

  15. New COAsadopted • Specifications • Cut-off score • Joint Database

  16. NEW LISTENING & READING ITEMS JDB (JOINT DATA BASE) JOINT EFFORT HOWNOT TOMAKE JOINT EFFORTS

  17. Test-writersinvolved • Accustomed to militaryenvironment • Language Testing Seminar • Qualified • Norming Sessions

  18. THE JDB FLOW AIR FORCE ARMY NAVY CARABINIERI

  19. 2012 2013 SEP OCT NOV DEC JAN FEB MAR APR MAJ JUN JUL AUG WG GdL GdL GdL Phase3 Phase1 Phase2 Phase4 MAJ 2013 40 NEW ITEMS. INTO JDB • PREPARATION OF FIRST BATCH OF ITEMS • SENT TO OTHER GROUP AUGUST 2013 60 NEW ITEMS. INTO JDB FEBRUARY 2013 20 NEW ITEMS. INTO JDB NOVEMBER 2013 80 NEW ITEMS. INTO JDB TIMINGS PRE-TESTING REVISION TRIALLIN# 50 ITEMS (-30%) 2nd Group 1st Group 4th Group 3rd Group FINAL MODIFICATION APPROVAZIONE TESTERS MEET FOR 1° REVISION Phase1 Phase1 Phase2 Phase3 Phase4 PRE-TESTING REVISION TRIALLIN# 50 ITEMS (-30%) • REPERIMENTO • PREPARAZIONE # 120 TTIVAZIONI • INVIO AD ALTRO GRUPPO • PREPARATION OF FIRST BATCH OF ITEMS • SENT TO OTHER GROUP FINAL MODIFICATION APPROVAZIONE TESTERS MEET FOR 1° REVISION • PREPARATION OF FIRST BATCH OF ITEMS • SENT TO OTHER GROUP Phase1 Phase2 Phase3 PRE-TESTING REVISION TRIALLIN# 50 ITEMS (-30%) TESTERS MEET FOR 1° REVISION

  20. Summary: • Past situation • COA (specs, JDB, cut-off score) • A do-it-all software • Proposals

  21. WHAT’S THE DIFFERENCE ?

  22. PC-assessment-related terminology

  23. the software is used for: • Needs analysis:(surveys) • Placement test • Any training activity • Assessment: • First level survey • Post-course • Pre-certification • Certification

  24. Software’s system Schema Create questions And organize them in tests using a windows based PC

  25. Assessment …… … allows to choose: • Time limits • Feedback to test-taker • Styles (Template) • Jumps • Question shuffling • Instructions to test-takers On Windows PC Questions via Browser Assessment Definitions

  26. Assessments su PC Windows • … also created with authoring manager by selecting Qs previously created • Any question can be chosen from the database via Browser Questions Assessment Definitions

  27. Software’s system Schema Result reports, CIA, graphs, gimmicks, you name it ….. Create questions And organize them in tests using a windows based PC Set security parameters, schedule assessment and link to other (Learning Management Systems assessment published using any browser, secure browsers, or a PC/MAC

  28. Types of Questions: Multiple Choice

  29. LikertScale (for questionnaires)

  30. Essay Question • Candidate can write free text in the space provided • Testers will evaluate later

  31. Summary: • Past situation • COA (specs, JDB, cut-off score) • A do-it-all software • Proposals s ing

  32. Testing in the time of crisis • Wide project • Sharingexperience & capabilities • Optimization of resources • No alternative to B.A.T.

  33. COMBINED DATABASE or … A multilateral-based CDB or … A Bilateral-based CDB

  34. 1 2 SPECS FLOW 3 TIME SCHEDULE

  35. canutesting@sclingue.esercito.difesa.it Thank you francesco.gratton@gmail.com

More Related