1 / 17

Program Review

Program Review. State Board for Certification of Water and Wastewater Systems Operators July 2007. Overview. Examination Results Training Workforce Development Next Steps. Summary of Progress. Version 1 -- “Beta” Test March through October, 2004 (10 sessions)

steven-floyd
Download Presentation

Program Review

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Program Review State Board for Certification of Water and Wastewater Systems Operators July 2007

  2. Overview • Examination Results • Training • Workforce Development • Next Steps

  3. Summary of Progress • Version 1 -- “Beta” Test • March through October, 2004 (10 sessions) • Version 2 (Expansion of Version 1) • January, 2005 through July, 2006 • Questions Reviewed and Finalized • Version 3 • September, 2006 to Now • Rotators Used Where Possible • Need to Review Questions • Finalized a Complete Set of Rotator Questions for all Examinations

  4. Statistical Results • Data does have a “normal”, bell shaped, distribution • Statistical relationship exists between test scores and level of education. • Limited, or no, relationship between test scores and experience. • Some correlation between training and test scores (not conclusive yet): • General Exams • Collection and Distribution Exams • Activated Sludge

  5. Pass Rates • See Attached Summary • Tests Most Taken (>75 applicants): • General Exams • Distribution and Collection System • Subclasses: • Conventional Filtration • Corrosion Control • Chemical Addition • Gaseous Chlorine • Chemical Disinfection • Activated Sludge

  6. Examination Analysis • Problems Found: • Original Percentages • Duplicate Questions Selected • Inconsistent Ranking: • Essential • Important • Appropriate • Exams • Subclasses 7, 8, 11 and 12 finished • Subclasses 9 and 10 • Activated Sludge • Recommendation – do for all examinations as we have time

  7. STEP #1 Eliminate duplicate KSA STEP # 2 Question evaluation Eliminate any duplicate KSA questions (only one question per KSA) Choose question with the highest KSA value. If multiple questions have the same KSA value, select the question with the highest relative ranking. If the ranking is the same, select the one with the lower p-value. Evaluate for correctness, clarity and effective detractor content. Incorporate question content into training material need Identify all questions with a p-value less than 0.33 or greater than 0.92 Make corrections to question content and answers if necessary Determine if Approved Training Providers need to be notified of a training need Determine if KSA value has changed Re-calculate Anchor Number Make any necessary KSA value changes

  8. STEP #3 Re-define priority ratings Considering two factors: 1. Percentage of time an operator uses these competencies on a regular basis 2. Importance in terms of protecting the environment, public health and safety Re-evaluate the priority rating percentages STEP #4 Finalize test content Remove or add questions based on priority categories percentages. Determine final test length and number of questions for each priority competency STEP #5 Re-evaluate Essential Questions Group all essential questions in specific categories of content regardless of their competency. Subject matter experts re-confirm essential question results. Review all designated non-essential questions to assure their content does not qualify them as an essential question. If content is the same as other essential questions reclassify these non-essential questions as essential. STEP #6 Re-establish cut score Re-define cut score using the Fosnick Model Revise the Ebel Procedure Analysis to reflect results of new ranking (if necessary), elimination of duplicate questions, re-evaluation of essential questions and re-calculation of cut score. Compare resultant cut scores between Fosnick and Ebel Models. Re-define final cut score if necessary.

  9. Proposed Revisions

  10. Training • 169 Approved Training Providers • 1711 Approved Courses • 58 Denials Since October 2005 • Post-Presentation Credit Applications (through March 2007): • 244 Applications Processed • 37 Denials Because: • Not Relevant to the Industry • Seminars, not courses • Not in Renewal Cycle • Do Not Meet Guidelines • 5 In Process

  11. We are Loosing WATER Operators ! 2006 6,000 600 2005 5,000 500 4,000 400 3,000 300 2,000 200 1,000 100 Operators Tested Licenses Issued Total Number of Operators License Expirations Denials

  12. We are Loosing WASTEWATER Operators ! 2006 6,000 600 2005 5,000 500 4,000 400 3,000 300 2,000 200 1,000 100 Operators Tested Licenses Issued Total Number of Operators License Expirations Denials

  13. Water Systems Complying with the Certified Operator Requirement 60% - 79% Compliance 40% - 59% Compliance 80% - 100% Compliance Less than 39% Compliance *(Systems with an operator/Total systems per county)

  14. Wastewater Systems Complying with the Certified Operator Requirement 60-79% Compliance 40-59% Compliance 80-100% Compliance Less than 39% Compliance *(Systems with an operator/Total systems per county)

  15. and … • Over 70% of the operators today are over the age of 50. • New regulations and requirements are being promulgated. Protect Public Health & Environment

  16. Utilities, Water and Wastewater Systems • Training Providers • High Schools • Voc-Tech Schools • Community Colleges • Industry Associations • State Agencies • Environmental Protection • Education • Labor and Industry • Job Placement Agencies Game Plan • Video and Brochure • Working to Build Partnerships • Pilot Projects • Workforce Development Webpage

  17. Next Steps – 2007/2008 • ABC Online Pilot • Convert to LXR Software • “Credit Card” Format for Licenses • Nutrient Removal Subclass • Finalize Board Guidelines • Develop External Program Review Process • Promulgate Regulations

More Related