1 / 63

Topics Today

AT-SAT: A Test Battery for Screening Air Traffic Controller Applicants Gordon Waugh Laurie Wise Human Resources Research Organization (HumRRO). Topics Today. Background of AT-SAT Overview of AT-SAT tests Air Traffic Scenarios Test Score weighting methods Setting the cut score.

adelle
Download Presentation

Topics Today

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. AT-SAT:A Test Battery for ScreeningAir Traffic Controller ApplicantsGordon WaughLaurie WiseHuman Resources Research Organization (HumRRO)

  2. Topics Today • Background of AT-SAT • Overview of AT-SAT tests • Air Traffic Scenarios Test • Score weighting methods • Setting the cut score

  3. Background of AT-SAT

  4. ATCS Selection background • Previous Selection System • Written aptitude test battery • Academy screening (9 weeks) • On-the-job training • Issues • Test utility after 12-15 years use • Academy screening cost & fairness

  5. Air Traffic Selection and Training (AT-SAT) Validation Research • Integrated validation study • Job-performance criterion measures • 8 new computerized predictor tests (6 hrs) • Concurrent validation study on en route controllers

  6. Overview of AT-SAT Tests

  7. Dial Reading Letter Factory AT Scenario Angles EQ Applied Math Analogies Scan AT-SAT FAA V1.05

  8. Dial Reading Test Temp. 1) 22 2) 24 3) 28 4) 30 5) 45

  9. Belt A Belt B Belt C Belt D Conveyor Belt Box Storage Area Available Zone Quality Control Order Boxes Quality Control Area Loading Area Order Boxes Area Letter Factory Test

  10. Belt A Belt B Belt C Belt D A P C D Quality Control B Order Boxes

  11. Letter Factory Test - Situational Awareness Questions Which boxes should be in the loading area in order to correctly place all the letters on the belts? 1. One purple box and one orange box 2. One green box, one purple box, and one orange box 3. Two purple boxes, one green box, and one orange box 4. Two green boxes, one orange box, and one purple box

  12. Air Traffic Scenarios Test

  13. Angles Test • This test includes two different types of questions: The first presents a picture of an angle and asks you the measure of that angle in degrees (From 1 to 360) What is the measure of this angle? 1) 90° 2) 10° 3) 125° 4) 190°

  14. This test includes two different types of questions: The second provides you with the measure of an angle (from 1 to 360 degrees). It then asks you to select an angle that represents that measure. Which of the following represents a 10 degree angle? 1) 3) 2) 4)

  15. Experience Questionnaire Your emotions have sometimes prevented you from solving a difficult problem. 1- Definitely True 2- Somewhat True 3- Neither True nor False 4- Somewhat False 5- Definitely False

  16. AppliedMath • The distance from point A to point B is 560 miles. If the aircraft left point A at 8:00 “Zulu”, and flew at 400 kts, what time would the aircraft cross point B? • A. 8:56 B. 9:02 C. 9:24 D. 10:02

  17. Analogies Test Water: Liquid Ice: ? Gas Cube Solid Oxygen Freeze (1) (2) (3) (4) (5)

  18. Visual Analogies: : : (1) (2) (3) (4) (5)

  19. Scan Test V41 610 K23 250 B12 810 T65 120 C20 390 F75 560 Y85 710 P45 580 Type the identification numbers contained in the data block with lower line numbers falling beyond the range (360-710):

  20. Air Traffic Scenarios Test

  21. Simulation Overview • Description of simulation • Design considerations • Scoring • Instructions and practice

  22. ATST: Description of Simulation

  23. AT Scenarios in a Nutshell • Display of airspace sector • Control several planes • Land at two airports • Exit at four locations

  24. F3f F2e F4D M4C F4B S1A

  25. Plane Icon Heading M2B Exit/Airport B Altitude Level 2 Speed Medium

  26. Design Elements • Environment: airspace sector with exits, airports, planes, and controls • Events: planes appear, move, and disappear • Actions: user clicks controls to control planes. • Rules: eight rules related to speed, altitude, separation, etc.

  27. Environment • Display of air sector: • four exits • two airports • planes • Plane controls • Display of time remaining

  28. Actions: Plane Controls • Direction – 8 headings • Speed - 3 levels • Altitude - 4 levels • Accept handoff

  29. Events • New planes appear • Planes move • Planes disappear: • land, exit, or crash

  30. Eight Simple Rules • Land at slowest speed • Land at lowest altitude • Exit at highest altitude • Land/exit at right place • Land in correct direction • Don’t fly over airports • Keep planes separated • Don’t crash!

  31. ATST: Design Considerations

  32. Possible Approaches • Measure KSAs • Simulate wide range of job tasks • Simulate a key job activity

  33. Design Goals • Maximize score variance • No disadvantage for computer novices • Short testing time

  34. Difficulty Factors: Planes • Number of planes on the screen at one time • Initial speed, level, heading, & location • N of actions per plane

  35. Difficulty Factors: Other • Speed of plane movement • N and complexity of rules • N of exits/airports • N of controls • N of control levels • N of things to remember

  36. ATST: Scoring

  37. Record the Outcomes • Rule violations • Separation errors • Crashes • Elapsed Time

  38. Á Priori Rational Scales • N of crashes & separation errors • N of procedural errors • Percent of successful flights • Total delay time (handoff & en route)

  39. Adjust Scales and Scores • Moved “flyovers” based on data • Standardize scores before combining and weighting • Rescale scores if useful: • reverse • fix skewness • make scores more sensitive to differences at higher levels

  40. ATST:Instructions&Practice

  41. Computer Skill Effects • Will computer experience affect scores? • If so, can you minimize its effect? • If computer skill is job relevant, it’s not as serious.

  42. Simulation Practice • Reduces effects of: • learning ability • computer game skills Several mini-trials Instructions with examples Practice a few things in mini-trial Give feedback and tips

  43. Post-Instruction Practice • Add practice trials until more trials add little to: • reliability • validity

  44. Which Trials to Score • Pilot test with lots of trials • Compute trial-total r • Use last one or more trials with high rs • Trial-criterion rs are even better if available

  45. Score Weighting Methods

  46. Final AT-SAT Validity • Validity, corrected for shrinkage, range restriction, and unreliability in criterion: N 1029 1032 1053 .76 with composite criterion .78 with CBPM .38 with Ratings

  47. Criteria • Reliability • .80 - CBPM (test-retest) • .71 - Ratings (interrater reliability) • .76 - Composite (mean of CBPM & Ratings) • Composite = .6 * CBPM + .4 * Ratings • r CBPM J Ratings = .24

  48. Uncorrected Test Validities • .41 Applied Math • .38 Analogies • .33 Angles • .33 Letter Factory • .32 Air Traffic Scenarios • .27 Time Wall • .25 Planes

  49. Uncorrected Test Validities (cont’d) • .24 Dials • .23 Memory Retest • .19 Memory • .19 Scan • .17 Experiences Questionnaire • .14 Sound

  50. Goals • Find best set of tests for the final battery • Determine scale weights • Measure validity: How well test battery predicts job performance

More Related