1 / 28

Mary Keating and Cathie Jasper

What you see is what you get – or is it? Improving the quality and consistency of Direct Observations of Practice. Mary Keating and Cathie Jasper. NOPT 7 th July 2016. Skills for Care Pilot - sites. Why are we doing this?.

thomasjenkins
Download Presentation

Mary Keating and Cathie Jasper

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. What you see is what you get – or is it? Improving the quality and consistency of Direct Observations of Practice Mary Keating and Cathie Jasper NOPT 7th July 2016

  2. Skills for Care Pilot - sites

  3. Why are we doing this? Direct observation - integral part of social work education in UK – pre and post qualifying Conduct and assessment of practice is variable (Ruch, 2015; Bogo et al., 2014) Major issue is assessor capability and bias (Humphrey, 2007; Finch and Taylor, 2013) PEPS/PDE have no specific learning outcomes for observation Desirable to moderate/standardise assessment of practice

  4. What is different? Pilot project to test the following: Tightened assessment criteria Two stage assessment • Direct observation. • Post observation structured written review by NQSW. Assessor training Development of learning outcomes for assessors, focus on observation skills. Development of training curriculum. Video recording of live practice to support learning and internal moderation

  5. Outputs from the project What are we doing / what have we done / what do we expect to achieve? Development of improved criteria for the assessment of Direct Observations based on the Knowledge and Skills Statement (adult) 2015 and the PCF at ASYE level Identification of Learning Standards for Assessors with a focus on observation skills Development of a training programme for Assessors A video solution for recording and secure storage of live practice observations

  6. Adaptation 1….’keep/change’ rule (Doel et al.,1996) Marion Bogo work on OSCEs adapted for social work (Bogo et al.2014; Bogo, et al., 2011) Objective: Skills directly observed Structured: A set scenario and format Clinical: Assesses practice skill implementation. Exam: Performance is evaluated on a set scale.

  7. Understanding meta competency Procedural competence – “refers to performance and the ability to use procedures in various stages of the helping process and includes the ability…in direct practice, to form a collaborative relationship, to carry out an assessment, and to implement interventions with clients and systems” (Bogo et al., 2014, p.6). Meta competence–“refers to higher order, overarching qualities and abilities of a conceptual, interpersonal and person/professional nature. This includes students’ cognitive, critical and self-reflective capacities” (Bogo et al., 2014, p.6). Holistic competence – involves two dimensions – procedural and meta competence (Bogo et al., 2014)

  8. Simulation and simulated practice scenarios Direct observation – allows for demonstration of doing and being Sources: Miller,(1990); Ruch, (2015); Rushforth, (2007) Miller (1990) Pyramid of professional competence Knowledge demonstrated / assessment methods: Exams, portfolios, written reflections, work products, supervision discussion, OSCE (shows how)

  9. Adaptation 2…SfC project - what have we kept /changed? Observation of live practice (home visit; assessment, review interview etc) “Direct observation is the optimal method for assessing professional skills because it is closest to ‘real practice’ “(Ruch, 2015, p.38) Development of structured assessment criteria –focus on Adults KSS / PCF level descriptor Observation and reflective activity that: captures procedural and meta competency assesses NQSW cognitive understandings of process, content, conceptualisation of practice and affective domains

  10. Adaptation 2…SfC project - what have we kept /changed? NQSW – completes Post Observation Structured Written Review, immediately after observation, max 45 minutes Both – post observation reflective discussion Assessor – overarching holistic assessment

  11. Example: . n Observation - Assessor criteria

  12. Example: NQSW Post observation – structured written review

  13. Assessor training- Learning objectives Development of new learning objectives to supplement the existing Practice Development Educator learning objectives

  14. Assessor training- Programme and resources Skills audit, pre and post training Focus on observation skills Major vehicle for the training is the use of video material Training video - social worker and service user. Practice and discussion of observation assessment Linked example of structured written review – commented on and discussed

  15. Preliminary reflections on the training Positive response to  the project Like the standardised assessment criteria Positive if nervous about the learning potential of use of video Difficulties in finding observation opportunities where Service User has capacity Value of Assessor reviewing the video before finalising the observed element of Assessment Training programme worked well

  16. Video : Storage Secure Library All recordings will be transferred into a centrally held storage library accessible via an Internet / WiFi connection. The videos will be transferred from the Surface device, once there is an adequate internet connection. They will then be automatically wiped from the surface device. The videos will be accessible from the Surface or any PC / Laptop providing the user has the login credentials to the secure library.

  17. Who views the video?

  18. Evaluation and Conclusions Evaluating the project • Use of video • New criteria for observations • Moderation Guidance – aim to be available January 2017 Informing the Chief Social Workers thinking on ASYE

  19. What is observer bias? • A form of cognitive bias, this is something hard wired into us. • The process by which we interpret what we observe in a way which corroborates our existing biases, either positive or negative. • Evidence suggests that once we reach a conclusion about something it is difficult for us to change our minds even when evidence to the contrary is presented.

  20. Cognitive biases… • Confirmation bias Become attached to a judgement (e.g student not very .good); do not notice disconfirming evidence; accept only evidence that fits with original beliefs (‘anchor’ hypothesis) • Bandwagon effect, “group think” Effect of others opinions on your assessment/judgements; pressure not to disagree or stand out • Availability heuristic /recall bias / relative judgements How many previous examples can be brought to mind (“this student is like another one we had where something similar happened…”); more recent examples predominate. Also, student being judged ‘relatively’ against another current or recent student • Decision avoidance Making a decision invokes action ; can be time consuming and emotionally challenging

  21. Direct observations In undertaking direct observations observer bias can manifest in different ways • Seeing only what we expect to see • Looking for only the things that confirm our bias • Dismissing or minimising things that do not confirm our bias

  22. Exercise Consider in pairs for 5 minutes, please be ready to reflect back to the whole group if required: • What things in life do you take for granted as ‘facts’? • How often are these ‘facts’ challenged? • How easy do you find it to change your mind and what needs to happen for you to do this?

  23. How aware are we? Please watch the clip below and then discuss what you think influenced what you saw and how might these things impact on your direct observation of practice? http://www.youtube.com/watch?v=Ahg6qcgoay4

  24. Addressing observer bias • Where possible, try not to form bias in the first place. • Objective observation • Structure approach to recording observations • Reflection and awareness • Challenging bias- taking the opposite position. • Moderation

  25. Discussion Please take some to reflect on direct observations of practice you have undertaken in the past. Do you think observer bias played a role in this, were you aware of this at the time and if so what steps did you take to address it? What strategies will you employ to minimise the impact of observer bias on your Direct Observations?

  26. References Ruch,G (2015), Evidence scope regarding the use of practice observation methods as part of the assessment of social work practice. RIP: Dartington. https://www.rip.org.uk/resources/publications/evidence-scopes/regarding-the-use-of-practice-observation-methods-as-part-of-the-assessment-of-social-work-practice-evidence-scope-2015/ Humphrey, C. (2007) Observing Students Practice( Through the Looking Glass and Beyond. Social work education Vol. 26, No 7, October 2007 pp. 723- 736 Kowproska, J., Hicks, L., McCluskey, U., Fisher, T., and Wishart, J. (1999) Facts,Feelings and Feedback: A Collaborative Model for Direct Observations. Video.Heslington: University of York Department of Social Policy and Social Work. Doel, M, Shardlow, S, Sawdon, C and Sawdon, D (1996) Teaching Social Work Practice, Arena:Aldershot Finch, J & Taylor , I (2013) Failure to Fail? Practice Educators' Emotional Experiences of Assessing Failing Social Work Students, Social Work Education, 32:2, 244-258 Bogo, M; Rawlings, M; Katz, E and Logie, C (2014) Using Simulation in Assessment and Teaching, CSWE Press: Virginia Rushforth, H.E(2007) Objective structured clinical examination (OSCE): review of literature and implications for nursing education, Nurse Education Today, 27,481-490. Miller, G (1990) The assessment of clinical skills /competence/performance. Academic Medicine, 65 (Suppl.9), s63-s67 Bogo, M., Regehr, C., Logie, C., Katz, E., Mylopoulos, M., & Regehr, G. (2011). Adapting objective structured clinical examinations to assess social work students’ performance and reflections. Journal of Social Work Education (47)1, 5-18. Logie, C., Bogo, M., Regehr, C., & Regehr, G. (2013). A critical appraisal of the use of standardized client simulations in social work education. Journal of Social Work Education, 49(1): 66-80.

  27. References Bog, M; Regehr, C; Logie, C, Katz, E, Mylopooulos, M and Regehr, G (2011) Adapting Objective Structured Clinical Examinations to assess social work students performance and reflections. Journal of Social Work Education, 14, 1,p 5-18 Irwin, M and McGlade, A (2011) You talkin’ to me? (2011) Journal of Practice Teaching and Learning, 11,2, p.92-108. Irwin, M (2014) Back to Basics: A skill based approach to Assessing Social Work Students within Directly-observed Practice. Field Educator, vol 4.2

  28. Contact Mary.keating@skillsforcare.org.uk Cathie Jasper C.Jasper@mmu.ac.uk

More Related