Developing a feedback questionnaire: Principles and steps

1. Developing a feedback questionnaire:Principles and steps Workshop for NHS staff 28 Dec 1999 (Tuesday) Kam-Por Kwan, EDU 2766 6287 etkpkwan@polyu.edu.hk

2. Workshop outline • Why use a feedback questionnaire? • How to develop a feedback questionnaire that give useful and truthful information? • What are the common problems? • How to write good evaluation items? • How to determine if the questionnaire is valid and reliable? • How to interpret student feedback in a meaningful way? • Developing a student feedback questionnaire on clinical experience

3. Why use a feedback questionnaire? • Economical to administer to the whole group, both in terms of time and effort • Allow anonymity of responses • Allow respondents to control own pace of response but • Less chance to probe for clarification • Emphasis on evaluator’s rather than respondent’s perspectives • Reliability, validity, and usefulness depends on items included

4. Common problems • Feedback questionnaires often fail to provides true and useful information because: • the items are constructed in an ad hoc basis without any theoretical framework behind • they ask about things that the (student) raters cannot validly comment on • the items are ambiguous to (student) raters and/or difficult to understand / interpret • the interpretations of the items are not clear • the items are too standardised to be useful • respondents are not motivated to complete it seriously

5. A systematic approach • 7 steps to developing a questionnaire: • determining the focus of the evaluation • identifying all underlying dimensions and sub-dimensions involved • drafting questionnaire items • designing questionnaire: instructions and sequencing, etc. • pilot testing the questionnaire • revising questionnaire and items • implementing the questionnaire

6. Identifying focus and dimensions Clear guidelines Providing useful comments on what and how to improve Feedback Clinical supervision Providing regular feedback on students’ clinical performance Support Learning

7. Examining the dimensions • Task 1 • Examine the draft questionnaire and identify for each item: • the underlying dimension that it pertains to measure, and • what kind of variable (presage, process, or product) is being measured. • What other dimensions or variables do you think should be included in the questionnaire?

8. Problematic items • Task 2: What’s wrong with the items? In groups, discuss the problems of including the following items in a student feedback questionnaire. • The teacher seemed to have an up-to-date knowledge of evidence-based nursing practice. • The teacher worked hard to demonstrate clearly to me the proper skills of history-taking. • My progress was a major concern of the teacher.

9. More problematic items • I was provided with informed and constructive feedback on my performance by the teacher immediately after my clinical practice. • I found it difficult to apply the theory I learned at university to my clinical practice. • Every student was encouraged to participate in class discussions. • The teacher did not discourage me from using techniques that are not evidence-based. • Appropriate computer technology and AV aids were used by the teacher to facilitate learning.

10. Principles of item writing (1) • Use simple English and simple sentence structure • Avoid questions that the intended respondents do not have the knowledge to comment on • Avoid ambiguous questions or wordings that may have alternative interpretations • Avoid double-barreled questions (items containing more than one ideas)

11. Principles of item writing (2) • Avoid unnecessary jargons that may not be understood by the intended respondents • Avoid words like “all the time”, “never”, “every”, ... • Avoid double negatives • Avoid questions with unwarranted underlying assumptions

12. Using open-ended questions • Limitations of ratings: • reflect evaluator’s rather than respondents’ perspectives • suggest whether improvements are needed, but not why or how • give a false sense of objectivity and precision • Open-ended questions • allow respondents’ perspectives to emerge • offer chances for respondents to clarify personal meanings and suggest changes

13. Optional questions • Standardised questions: • allow comparisons across units or teachers BUT • cannot cater for individual needs • Optional questions: • allow users to collect information on aspects specific to the individual units or teachers • useful for improvement purposes

14. Revising the draft questionnaire • Task 3In groups, suggest how the draft questionnaire might be further modified to make it more useful and valid. You might consider: • adding new items /deleting redundant items • rewording the items as needed to make their meaning clearer and less ambiguous • inserting open-ended questions • the possibility of allowing optional questions to be included

15. Good design • as short and sharp as possible (a few short questionnaires at different points may be better than a long one at the end) • appeals to the intended respondents • with clear instruction • questions arranged in good psychological order, from general to more specific • attractive and neat in appearance • clearly duplicated / printed • easy to code and interpret

16. Validity and Reliability (1) • Validity • Does the questionnaire measure what it is supposed to measure? • Do the items together measure the most significant aspects of the evaluation question? • Improving validity by: • judgment of a panel of experts • pilot testing on a sample of intended respondents • relating to theory of teaching

17. Validity and Reliability (2) • Reliability • Does the questionnaire give a consistent results of it is measuring? • Do the items yield results that agree with each other, and are they consistent over time? • Improving reliability by establishing the: • internal reliability of the instrument, scales, and sub-scales • test-retest reliability

18. Student feedback: what research says • Quite reliable and consistent • Reasonably valid • Relatively uncontaminated by variables seen as sources of potential bias • Useful for a number of purposes BUT • An ‘imperfect’ measure of teaching • Must be interpreted in contexts • Useful insofar as one source of information • Can be abused if not interpreted properly

19. Nature of student feedback • Subjective perceptions • Based on what students have directly experienced • Influenced by their own characteristics such as prior knowledge, motivation, interest, etc. • Reflected students’ implicit theory of teaching and learning

20. Some pitfalls in interpretation • Treating student feedback as if it were a totally objective, precise, and truthful indicator of teaching • Over-interpreting small differences in ratings • Comparing ratings across units or teachers without considering the context • Ranking units/teachers by their total scores

21. Interpretation guidelines • Avoid over-interpreting small differences: only ‘crude’ judgements can be made • Focus on the relative strengths and weaknesses as reflect in the profile of ratings rather than the total scores • Interpret feedback in context: need to take into consideration the features of the centre and the students • Consider ratings from different classes, and over a number of years • Need to check student feedback against other sources of evidence

22. Some final words • Student feedback is a useful source of information, not a verdict • Student feedback cannot replace professional judgment of the teacher • Teacher’s self reflection on the feedback collected is the key to improvement