School Performance Variation on COMLEX Level 1

1. School Performance Variation on COMLEX Level 1 Linjun Shen, Ph.D., MPH National Board of Osteopathic Medical Examiners

2. Purposes • Provide more information to school administrators and educators • Help school administrators and educators to interpret and analyze COMLEX data • Demonstrate analytical approaches individual schools can adopt in digesting their own COMLEX data

3. Limitations • Analysis is limited to Level 1 June 2002 administration • Variables available for this analysis is limited, so as the results

4. A model of the medical school-licensing examination relationship

5. Scope of This Analysis • Interest of the analysis: school mean scores and school passing rates on Level 1. • Potentially influential factors: gender of students, sector of school, school mean MCAT. • Analytical approach: mainly descriptive and graphical with a brief statistical modeling.

6. Summary of June 2002 Level 1

7. June 2002 Level 1

8. Level 1 School Mean

9. Highest and Lowest on Level 1

10. School Mean Score Distribution

11. School Failing Rate Distribution

12. Failing Rate by Ranking of Means

13. Passing Rate by Ranking of Means

14. School Mean Performance • Two ways to rank schools: by mean score and passing rate • Numerical ranking is not as important as qualitative grouping • Variation among school mean scores is substantial (53 point between the highest and lowest, 40 point between above and below average groups)

15. Significant Gender Differences on June 2002 Level 1

16. Gender Difference in Mean Scores by School

17. Gender Difference in Failing Percentage by School

18. Gender Difference in Osteopathic Medical Schools • There are substantial gender differences in terms of individual student Level 1 performance • For a few schools, the differences in both mean scores and passing rates are alarming • Gender differences can also be an opportunity for schools to improve if efforts are made to narrow the gap

19. Sector Differences on June 2002 Level 1

20. Gender and Sector Interactionon Mean Score

21. Gender and Sector Interactionon Failing Rate

22. Effects of Sector of Osteopathic Medical Schools • There is no evidence that sector significantly affects Level 1 performance • According to a study 10 years ago, sector was a powerful factor in determining osteopathic Board exam performance • It is a very positive improvement of osteopathic medical education • More formal study is needed

23. School Mean MCAT Distribution

24. School Mean MCAT of Class 2000

25. Mean MCAT by Rank

26. Mean MCAT and Mean Level 1(r = .30, not significant)

27. Mean MCAT and Mean Passing Rate (r = .27)

28. MCAT and Level 1 Performance • The implication of correlation of school means of MCAT and a single Level 1 administration is highly limited • Individual student MCAT and Level 1 score relationship can be very different from school mean MCAT and school mean Level 1

29. Conventional Regression

30. Results of Conventional Regression • The model is significant. Both gender and sector are significant predictor. • This regression model with two variables only explained 3% of the student. performance variation. Most of the variation remains unexplained. • More information is needed to explain student performance.

31. Hierarchical Linear Regression

32. Results of HLM Analysis • Difference of students Level 1 performance across school is significant • Gender is a predictor for Level 1 scores but the difference of gender is not significant across schools • Neither Sector and school mean MCAT predict student Level 1 performance • More data will help: student MCAT, race, curriculum type, etc.

33. Thank You!