Download
1 / 25
250 likes | 350 Views
Remarking of Examination Answer Scripts – Finding a Standard for Quality Assurance. by Joe Cesare and Coert Loock. Introduction. The initial question was “How do we improve on the quality of marking?” In order to improve a process one needs to know where you are and where you want to be.
E N D
Remarking of Examination Answer Scripts – Finding a Standard for Quality Assurance by Joe Cesare and Coert Loock
Introduction • The initial question was “How do we improve on the quality of marking?” • In order to improve a process one needs to know where you are and where you want to be. • Remarking was selected as it shows the “mistakes” made with initial marking.
Rationale for Research • There is very little information on remarking in the public domain • How far do we stretch the principle that there will be differences in marking by two markers? • This is research in progress • Initial stages are steering away from formal statistical analysis • Needs to be understandable to the markers and they need to buy into the process
Remarking Policy • Policies or guidelines on remarking govern the process and are available in just about all institutions and organizations that conduct examinations. • “Where a student is not satisfied with the result” is a phrase common to all. • Remarking stands separate from the moderation process
Nature of Remarking Policies • Usually a monetary fee attached which varies widely. • Most institutions refund the fee if the remark result in a higher symbol/grade • Many institutions have qualifying criteria, e.g. minimum score, within a defined range • Candidate usually awarded the higher of the scores • More senior person does remarking
The Rechecking Concept • Rechecking and Remarking mostly two separate processes. • Remark changes are very often the result of clerical issues and not poor marking per se • Candidates may opt for remarking rather than rechecking, while the latter could solve their problem – tendency has major implications in terms of remark logistics
Rechecking Errors • High volume of work, short time and extreme deadlines lead to mental fatigue – human error starts to play a bigger role • Fast marking pace required lead to reading errors, resulting from poor digit and letter forming • Role of mother tongue when doing simple arithmetic cannot be discarded • Inability to do mental arithmetic (technology dependence) and even undiagnosed partial dyscalculia could have an effect
The Remarking Process • Remarking by more senior marker • Gain or loss for candidate • Effect of the marker’s ability is more pronounced if the candidate’s answer has to be interpreted (Essays, Analysis & Synthesis) • Partial correct answers – re-markers tend to give candidates the benefit • Markers as educators have a vested interest
Viewing and Access to scripts • Viewing prior to a remark application will reduce remarking • Logistical issues and preserving of security and integrity are obstacles. • Constitutional rights of access to information have never been really tested
Reasons for Remarking • Candidates apply for three reasons: • Require a mark or two to get into the next symbol range • Do not trust the system • Lately: Specifically require higher scores or symbols for HE admission; for bursaries and financial assistance • Original intention of remarking concept changed into a “second chance”
Remarking as Quality Assurance • Candidates must perceive the marking process as reliable, fair, accurate and consistent • Any process will allow for small variations and levels of tolerance are usually built in • Need to reduce remark changes to acceptable levels to be able to quality assure the process.
Finding Acceptable Levels • Use raw scores to eliminate effect of standardization and have a secure baseline • Use percentage changes to prevent hiding in the large volume. • Understandable and not statistically complex
Determining the standard • While the downward trends are gratifying, we will only have a standard once the trend stabilizes. • The trend in some subjects have already stabilized. • Applying the concept on a larger database, e.g. on a national level, will give assessment bodies a standard with which to evaluate the provincial marking
Interventions • Changing the trend did not happen automatically – deliberate and active intervention strategies had to be implemented • Valuable contribution was making the remark information known to marking teams • Chief Markers and Internal Moderators are on the lookout for contributing factors (positive and negative) • Factors informs structure of marker training • Used for Target setting and a competition element
Analysis of Physical Science • Remarks scores were analyzed per question • Attempted to identify the origin of the change – many variables to interrogate • Major “Culprits” • Errors in carry-through in incorrect/partially correct answers • Remark marker does not see a fresh script and is “influenced” by original marker in partially correct answers. Tendency also observed with moderated work
Candidates answer in bits and pieces – carry through and double dipping is prevalent • Deteriorating handwriting – letters and digits have to be interpreted rather than read (Apply equally to candidates and markers) • Many alternative answers/partially correct answers increases the complexity of the memorandum
Problem Areas - Physics • Problem areas not so much related to content as to question type • Various alternatives, requiring a complex memorandum, carry-through of errors. Required higher thinking skills (2,3,5) • Expressing a concept in words are testing language skills – both markers and candidates get it wrong!(7) • Inconsistency in approach year to year (8)
Problem Areas - Chemistry • Language ability of both markers and candidates are questionable – candidates use correct words/phrases in an incorrect context and vice versa and markers miss it – effect of haste and work pressure?(2,3,5) • Candidates writing balanced equations and getting it partially correct confuses markers, particularly if marking instruction and mark allocation varies within the paper/between years • Candidates “combine” methods and get it partially correct.(6)
Concluding Remarks • A measurable quantity can be obtained from remarking analysis that could be an indicator of quality • Need to broaden the database to determine accurate criteria. • Rid the system of using remarking as a method to gain – only real cases of difference of opinion that qualifies against norm-criteria need to be remarked and then for a final score.
