TF – SP Capability Development Programme for Competency-Based Assessment

1. TF – SP Capability Development Programme for Competency-Based Assessment Facilitator: Dennis Sale

2. Programme Objectives • Identify the main components of a competency-based qualification system (e.g., Singapore Workforce Skills) • Explain the process and principles of good assessment practice • Analyse the standards for developing and conducting competency-based assessment • Produce an assessment plan for a element of competence • Produce assessment methods and tools for an element of competence • Validate an assessment plan and accompanying assessment tools • Produce a portfolio of evidence for meeting the competency standards for assessment • Identify the importance of verification in the quality assurance of assessment practices • Identify the role and duties of an Internal Verifier

3. Competency Based Approach Key features: • Meeting specific work related functions • Meeting a defined standard (at different levels) of competence • Assessment is mainly through direct performance in real work activity

4. Singapore Workforce Skills Qualifications (WSQ) - Competency-Based System Underpinning Knowledge Performance Criteria Range and Context Evidence Sources Competency Element 1 Underpinning Knowledge Performance Criteria Range and Context Evidence Sources Competency Unit Competency Element 2 Underpinning Knowledge Performance Criteria Range and Context Evidence Sources Competency Element 3

5. Competency Unit 1: Develop A Competency-Based Assessment Competency Elements • Prepare an assessment plan • Develop competency-based assessment tools • Validate assessment plan

6. Competency Unit 2: Conduct Competency-Based Assessment Competency Elements • Prepare for conduct of assessment • Prepare candidate for assessment • Carry out conduct of assessment

7. Conduct Competency-Based Assessment Competency Element Title 1. Prepare for Conduct of Assessment Underpinning Knowledge A competent individual needs to know and understand: • How to prepare for conduct of assessment Performance Criteria A competent individual must be able to successfully perform the following: 1.1 Confirm details of assessment plan with relevant stakeholders 1.2 Ensure resources stipulated in the assessment plan are available, safe and in proper working conditions 1.3 Ensure the set up of assessment site is in accordance with assessment plan and workplace health & safety guidelines 1.4 Ensure the assessment tools required for assessment are available for use

8. Conduct Competency-Based Assessment Competency Element Title 1. Prepare for Conduct of Assessment • Range & Context • Confirm details of assessment plan with relevant stakeholder (e.g., evidence • requirements, resources needed for assessment, organisational requirements) • Ensure availability of resources ( e.g., tools equipment , technical support, etc) • and assessment tools (assessment instruments, tools, etc) • Evidence Sources • Examples of evidence needed to meet the standards required for this element • Process • Observation of work, which includes • confirming details of assessment plan with stakeholders • checking assessment site is appropriately set up and assessment instruments • are available for use • Product • Assessment site is set up according to assessment plan

9. A Structured Assessment Process

10. Technical Principles of Assessment Assessment must comply with the following Technical Principles of assessment: • Validity • Reliability • Flexibility • Fairness These Technical Principles of Assessment must be fully addressed in the design and conduct of the assessment process as well as the development of assessment items and instruments

11. In basic terms, validity refers to the ‘truth’ of the assessments made. A valid assessment is one that measures accurately what it is we intend to measure. For example, a valid driving test is one in which driving skills (Performance) is measured in typical traffic (Conditions) against the criteria established by the Motoring Authority (Standard) Validity

12. Reliability Reliability refers to the capability of a test to produce the same scores with different assessors (persons scoring the test) Grade A Grade A Grade A Assessor 3 Assessor 1 Assessor 2

13. Flexibility Flexibility is concerned with the process of assessment, not the standard of the assessment. For example, candidates can display the competence in a range of situations (e.g., on the job, simulated conditions) at mutually convenient times, etc), provided the competence is validly and sufficiently demonstrated.

14. Fairness Fairness relates to a number of considerations in assessment. However, they are all concerned with ensuring that candidates, when being assessed, are provided with appropriate access to the assessment activities and are not unfairly discriminated against in the assessment process. Unfair discrimination typically means discrimination based on criteria unrelated to the assessment activity itself, for example, gender or racial characteristics. Fairness is a general concern throughout assessment, relating as much to providing candidates with sufficient knowledge and time for assessment, to non-discriminatory processes in marking their work.

15. Rules of Evidence Evidence of competence must be: • Valid • Sufficient • Current • Authentic

16. Sufficient Sufficient refers to ‘how much assessment evidence’ do we need in order to feel confident that a candidate is competent in the area assessed?’

17. Current Currency of evidence refers to how recent the evidence is generated and whether it fully relates to the most up-to-date knowledge, skills and practices for the work function being assessed. This needs to be checked against the industry standard and any specific policy guidelines stated.

18. Authentic Quite simply this refers to how sure we are that the work produced has been done by the candidate In an examination, we can be more confident of authenticity However, with assignments done by candidates in their own time, authenticity becomes a concern

19. Efficiency Assessment can be time consuming. It is important to: • Identify assessment opportunities that enable the assessment of as wide a range of performance criteria as possible • Use methods that are efficient for what we are assessing • Avoid over-assessing – be clear about what is Sufficient evidence

20. Identifying relevant evidence Assessors and candidates need to be clear on: • What types of evidence are needed to meet the specific performance criteria • What underpinning knowledgecan be assessed either through performance or questioning (written or oral) • Arrangements for the assessment (e.g., equipment, materials required)

21. Collecting Evidence of Competency 1 • Assessment of competence must involve a full demonstration of the competence – and should involve a variety of evidence types and contexts where possible • At least one form of direct evidence should be used in judging practical performance (e.g., observation of simulated performance) • Supplementary (indirect evidence) should be used in support of direct evidence to: • Extend range of evidence and possible performance in rare situations (e.g., theft, breakdown) • Facilitate transfer to new situations/contexts • Assess underpinning knowledge

22. Collecting Evidence of Competency 2 • Evidence gathering methods should be culturally inclusive and take into account the language, literacy and numeracy skills of the participant • Reasonable adjustments may be considered for participants with physical disabilities or special learning needs • Evidence of competency must include occupational safety and health and other legislative aspects of the job • Prior learning can be recognized providing it can be validated by accepted certification.

23. Assessing Underpinning Knowledge • While to some extent underpinning knowledge is implicit in performance – it should be separately assessed to ensure understanding (the why behind the what and how) • The use of verbal questioning (especially immediately after demonstrating competence) is an effectively way to assess underpinning knowledge • Use oral or written work related questions in assessing underpinning knowledge (avoid MCQ’s)

24. Assessment Context The assessment context is the participant’s current place of work and job role. A judgement of competence denotes that the participant is able to produce the desired outcomes reflected in the competency element within the requirements of the work role and environment. Assessment may be undertaken on and off the job – the essential reference for the collection of evidence is the actual Competency Unit

25. Competency Unit 1: Develop A Competency-Based Assessment Competency Element 1: Prepare An Assessment Plan

26. Purposes of the Assessment Plan • Show how the Unit is to be assessed - detailing actual assessment process, methods and instruments • Calibrating assessment to actual competency elements and performance criteria, etc • Provides explicit guide to trainers, trainees and other stakeholders as to how the assessment is to be carried out in relation to Principles & Rules of Assessment

27. Prepare An Assessment Plan Meeting this competency requires you to: • Establish the purpose and context of assessment in consultation with relevant people • Obtain information on the characteristics of candidates • Identify the types of evidence to be gathered based on the competency standard • Determine what competences will be assessed (e.g., elements, performance criteria, etc) • Select appropriate assessment methods and tools to collect the required evidence • Identify assessment location, physical resources and specific materials required for conduct of assessment

28. Competency Unit 1: Develop A Competency-Based Assessment Competency Element 2: Develop Competency- Based Assessment Tools

29. Develop competency-based assessment tools Meeting this competency requires you to: • Develop holistic and integrated assessment activities based on the assessment plan • Develop specific assessment tools that are appropriately contextualized for the target audience • Develop clear instructions for assessor and candidate on the conduct of assessment • Develop assessment tools that meet the Principles of Assessment and the Rules of Evidence

30. Competency Unit 1: Develop A Competency-Based Assessment Competency Element 3: Validate Assessment Plan

31. Validate Assessment Plan Meeting this competency requires you to: • Identify the purpose, focus and approach to validation • Conduct the validation activities and address the critical findings • Produce a report recommending improvements to the assessment strategies, process and tools

32. Competency Unit 2: Conduct Competency-Based Assessment Competency Element 1: Prepare For Conduct of Assessment

33. Prepare For Conduct Of Assessment Meeting this competency requires you to: • Confirm details of assessment plan with relevant stakeholders • Ensure resources stipulated in the assessment plan are available, safe and in proper working conditions • Ensure the set up of assessment site is in accordance with assessment plan and workplace health & safety guidelines • Ensure the assessment tools required for assessment are available for use

34. Competency Unit 2: Conduct Competency-Based Assessment Competency Element 2: Prepare Candidate for Assessment

35. Prepare Candidate for Assessment Meeting this competency requires you to: • Explain the purpose and context of assessment to candidate • Brief candidate on the assessment requirements and process, including clear instructions on the assessment to be taken • Brief candidate on his rights and process of appeal to the assessment outcome • Communicate in a clear and supportive manner to create a conducive assessment environment • Confirm with candidates on any special needs he/she may have and explain how those needs will be addressed during the assessment • Seek feedback from candidate and clarify his/her understanding of the assessment requirements

36. Competency Unit 2: Conduct Competency-Based Assessment Competency Element 3: Carry Out Conduct of Assessment

37. Carry Out Conduct of Assessment Meeting this competency requires you to: • Use assessment plan and tools to carry out the conduct of assessment • Use evidence gathered to decide if the relevant criteria are met and make assessment decision • Deal with limitations of evidence required, where necessary • Comply with the Code of Practice for Assessors when conducting assessment and making assessment decisions • Record assessment outcomes promptly and accurately in accordance with assessment system, policies and procedures • Provide clear and constructive feedback to candidate regarding assessment decision and advice on how performance gaps can be overcome • Review assessment plan and process and provide feedback to appropriate personnel for modification, if required

38. When giving feedback to candidates • Let candidates give their opinion on the assessment first • Identify positives before areas for improvement • Limit what you cover • Be specific, not general • Focus on what can be changed/improved • Give the candidate time to think and respond • Keep to the standards • Check for understanding • Listen to how the feedback is received (including observation of non-verbal communication) • End on a positive (with encouragement)

39. Verification: Quality Assurance in Assessment Verification is the main quality assurance process associated with assessment systems and practice - whether competence-based or otherwise Quality Assurance is a vital part of any credible qualification system. It must ensure that everyone involved in the assessment of candidates is assessing correctly, working to agreed procedures and to an agreed standard of performance

40. Verification: The Range of Roles & their Relationships External Verifier Awarding Body Internal Verifier Coordinator Centre Internal Verifiers (programme level) Centre Centre Assessors Workplace

41. Internal Verifier (IV): The key Role in Quality Assurance The internal verifier(IV) is the key person(s) for ensuring quality of assessment practices at the institutional level (Centre) The IV conducts internal quality assurance of the assessment process by: • Carrying out and evaluating internal assessment and quality assurance systems • Supporting assessors • Monitoring the quality of assessors’ performance • Meeting external quality assurance requirements

42. Carrying out and evaluating internal assessment and quality assurance systems This will involve: • Comparing own institution requirements with those of the external awarding body • Identifying the outcomes needed by the agreed standards and their consequences for internal auditing • Auditing existing administrative and recording arrangements - modify/ change if necessary - to meet external audit requirements • Carrying out assessment standardization arrangements (e.g., sampling strategy) • Ensuring a procedure for complaints and appeals is in place (note: consider this in local context) • Identifying problems, developing improvement plans

43. Sampling strategy This will vary from institution, according to needs (e.g., size, programmes offered, resources, etc) but it must be agreed with the External Verifier. The following need to be taken into account: • Size of sample to ensure reliability • Assessment methods used by the centre (e.g., to ensure good principles of assessment are practice, etc) • Assessors – number, experience, workload, location • Candidate cohorts (e.g., full-time/part-time, different employees, etc) • The programmes offered – particularly if new/revised • All units must be included, but there may be a greater focus on any problem unit(s)

44. Standardizing assessment decisions It is essential to ensure, as far as is possible, that: • Assessors consistently make valid decisions • Assessors make the same decision on the same evidence base • All candidates are assessed fairly This involves: • conducting moderation activities (e.g., a number of assessors judging the same candidate’s evidence and comparing their perceptions and decisions)

45. Supporting assessors This will involve: • Ensuring assessors have appropriate technical and vocational experience • Ensuring that assessors are competent to assess • Identifying the developmental needs of assessors and facilitating any necessary training • Monitoring progress and development of assessors

46. Monitor the quality of assessors’ performance This will involve: • Ensuring that assessors are preparing and planning assessment effectively and efficiently (e.g., understand the standards and applying principles of good assessment, etc) • Observing and questioning individual assessors to ensure good practice and consistency over time • Checking records of the assessment process (from assessment planning to summative decisions) • Giving assessors accurate and helpful feedback

47. Meet external quality assurance requirements This will involve: • Identifying how internal assessment will be checked externally and ensuring these requirements can be met • Planning, collecting and analyzing information on internal assessment decisions (note: this involves the secure keeping of all records relating to the assessment of candidates) • Maintaining contact and meeting with the external verifier(s) • Providing feedback on the outcomes of meetings with the external verifier to centre assessors • Acting on the feedback given by external verifiers

48. Checklist of the range of information needed for purposes of External Verification It is the Internal Verifiers responsibility to have the following information quickly accessible: • Number of registered candidates per qualification level • Assessor details (e.g., CV’s, Assessor qualifications, development plans) • Candidate progress reviews and achievements • Assessment records and plans • Candidate support resources available (e.g., guide for candidate assessment) • Assessments sites used – where the assessments are carried out • Internal verification sampling strategy • Internal Verifier records, including feedback to Assessors and feedback from Candidates • Record of claims for certification • Candidate assessment evidence (e.g., portfolios)

49. Dennis Sale can be contacted at: Email: dennis_sale@sp.edu.sg Telephone: (65) 6772 1490 Blog: dennissale.com