Effects of high level prompts and peer assessment on online learners’ reflection levels

1. Effects of high level prompts and peer assessment on online learners’reflection levels Presenter: Zong-Lin TsaiAdvisor: Ming-Puu Chen Date: January 19, 2009 Chen, N. S., Wei, C. W., Wu, K. T. & Uden, L. (2009). Effects of high level prompts and peer assessment on online learners’ reflection levels. Computers & Education, 52(2), 283-291.

2. Introduction • Reflection plays a very important role in learning processes and is very helpful for promoting learning performance (Chi, de Leeuw, Chiu,& Lavancher, 1994; Lee & Hutchison, 1998; McNamara, 2004; McNamara, O’Reilly, Best, & Ozuru, 2006). • Knowledge is much easier to memorize and understand if it is being constructedafter a learner has gone through the internal thinking and comprehension process. This is because learners can review, test,and modify their own ideas while engaging in reflective practice in a learning process (Wolfe & Goldman, 2005); and this is why reflectionis very important to learning. • Dewey considered reflection as a mental processing ofone’s internal problem solving activity. • Some researchers considered reflection is a kind of meta-cognitive ability to which a person can adequately apply differentstrategies, and this ability can be acquired by proper learning and training (Borkowski, Carr, Rellinger, & Pressley, 1990). • The focus of this study is to explore how to better utilize online learning to cultivatelearners’ reflection ability.

3. Introduction • Researchhas shown that good learning performance, whether through humans or computers, can be achieved by providing prompts to learners(Aleven & Koedinger, 2002; Chi et al., 1994; Davis, 2000). • These pre-designed prompts will appropriately display to learnerswhile they are reading the materials. This should prompt learners to engage in reflective practice. We call this kind of prompt strategy‘‘reflection prompt”. • King (1994) classified questions into high level and low level questions. High level questions are comprehension and integrationquestions, and low level questions are merely memorizationquestions.High level questions are very helpfulfor learners to comprehend learning material. It is our belief that providing high level prompts is a key factor for promoting reflection. • Humphreys, Greenan, and McIlveen (1997) have reported that getting learners involved in assessment would improve learning motivation.Peer assessment has positive effects both on learning attitude and learning performance (Sung, Chang, Chiou, & Hou, 2005; Topping, 1998;Tsai, Lin, & Yuan, 2002; Tseng & Tsai, 2007). In view of this, we believe that peer assessment is another factor we identified as promotingreflection. • The aim of this study is to explore how high levelprompts and peer assessment can affect a learner’s reflection levels in an online learning environment.

4. Method • Variables • Both high level prompts and peer assessment are independent variables and reflection levels is dependent variable. • High level promptsKing (1994) classified prompt questions into three different types: memorization questions, comprehension questions, and integration questions.It is generally accepted that high level prompts such as comprehensive and integration questions are more helpful to learners for constructing new knowledge compared to the memorized question prompts.Two settings for this independent variable are: with high level prompts and without high level prompts. We carefully selected a scientific article related to human hearing as an online reading material and adopted the definitions proposed by King (1994) that high level prompts are comprehension questions or integration questions.

5. Method • Variables • Peer assessmentPeer assessment means a group of learners with similar background and ability not only play the learner role, but also act as an instructor for assessing peers’ learning (Topping, 1998; Topping & Ehly, 2001). There are two types of peer assessment, peer observation and peer feedback.There are twoforms of peer assessment, peer observation and peer feedback. We would like to investigate how peer observation and peer feedback canaffect learners’ reflection levels. There are three manipulations for this independent variable: • Without peer assessment(neither peer observation nor peer feedback) • Peer observation without peer feedback • Peer observation with peer feedback

6. Method • Variables • Peer observationThe concept of peer observation is the same as Observation Learning proposed by Bandura (1986) in his social learning theory. It means learning by observing others’ behaviors and performance.We classified the two types of reflection content observed by learners into high quality and low quality. By adopting the classification of Bain, Ballantyne, Packer, and Mills (1999) on reflection contents, we considered reasoning and reconstructing as high quality reflection contents and reporting and responding as low quality reflection contents. • Peer feedbackAs there is no empirical study on how positive and negative feedback affects reflection levels, we want to explore this issue in this study.We classified peer feedback into two types, positive and negative feedback.

7. Method • Variables • Reflection levelsBain et al. (1999) classified reflection into five different levels, reporting, responding, relating, reasoning, and reconstructing, according to learners’ reflection contents.Because in a real learning practice, some learners’ reflection contents would be too simple, incomplete, or have even misunderstood the question, which cannot be classified to the lowest level of reporting, so we extended the criteria into eightreflection levels.

8. Method • Design • ParticipantsThe participants in the experiment were 157 on-campus college students from different major disciplines, aged between 21 and 26.Participants were randomly assigned into 14 different learning conditions as shown in Table 4.

9. Method • MaterialThe learning material used in this experiment is a biology article related to the human ear.Aquestion mark represents that learners will be guided to engage in reflectionwhile they are reading the sentence, this can be seen in the left hand boxof Fig. 2. • Procedure

10. Results • Testing between high level prompts and peer observation on reflection levels • In the first stage, high level prompts and peer observation are independent variables and reflection levels is the dependent variable. • The mean of reflection levels was above five in thegroups with-high-level-prompts. It showed that their reflection levels were generally higher than responding level. • However, the reflectionlevels were generally lower than responding level in the other groups.

11. Results • Testing between high level prompts and peer observation on reflection levels • A two-way ANOVA analysis was conducted on the reflection levels for high level prompts and peer observation. There were significantmain effects for high level prompts (F(1, 151) = 68.58, p-value = 0.000) and peer observation (F(2, 151) = 11.33, p-value = 0.000). Meanwhile,the interaction effect between them was also significant (F(2, 151) = 3.58, p-value = 0.030). • This result indicated that reflection levelswill not be affected by just high level prompts or peer observation alone. It implies that learner’s reflection levels can be greatly improvedby applying both high level prompts and peer observation teaching strategies.

12. Results • Testing between high level prompts and peer observation on reflection levels • A post-hoc analysis using Fisher’s least significant difference (LSD) test . • In thewithout-high-level-prompts condition, the reflection levels of the group with high quality observation was significantly higher than thegroups without peer observation and with low quality observation. • In the with-high-level-prompts condition, there were nosignificant differences among their reflection levels in the groups. • Learners who were provided with high level prompts as well as highquality observation, their reflection levels were significantly higher than the groups without high level prompts.

13. Results • Testing between peer observation and peer feedback on reflection levels • In the second stage, peer observation and peer feedback are independent variables and reflection levels is the dependent variable. • There was a significant main effect for peer observation (F(1, 113) = 14.78, p-value = 0.000). However, no significant main effect for peerfeedback (F(2, 113) = 1.43, p-value = 0.244) was found. Theinteraction effect between them was also not significant (F(2, 113) = 2.53, p-value = 0.084). • The results showed that peer observation has a statistically significant influence on reflection levels and peer feedback has no significant influence on reflection levels.

14. Discussion • There are several important implications from this study. • Firstly, for with-high-level-prompts groups, their reflection levels are significantly higher than the without-high-level-prompts andwithout-high-quality-observationgroups as shown in Fig. 4.This implies that learners often donot know how to carry out good reflection if high level prompts were not provided.

15. Discussion • Secondly, for the with-high-quality-observation groups, we found that their reflection levels are significantly higher than the withouthigh-level-promptsand without-high-quality-observation groups as shown in Fig. 5. This implies that instructors could provide high quality reflection contents to use as examples in an online learning environment.

16. Discussion • Thirdly, giving learners’ positive feedback, negative feedback, or no feedback did not significantly affect learners’ reflection levels. Thisimplies instructors can expose learners to peer feedback and not worry about the effects on learners’ reflection levels.Although positive feedback or negative feedback has no significant effect on learners’ reflection levels, but peer feedback could givelearners an opportunity to play the role as an instructor: from the peer learning perspective, this is good for learners to improve theirown critical thinking abilities by giving comments to their peers. • Finally, instructors can integrate e-portfolio to realize reflection education. Instructors can use e-portfolio as a reflection venue for learners and consider the inclusion of reflection portfolios as one of the assessment items. In addition, how to create a good atmosphere such that learners will be more willing to express their own opinions is very important. END