21-Tech Formative Evaluation Phase II Brief

1. 21-Tech Formative Evaluation Phase II Brief CECILIA GARIBAY & JANE SCHAEFER

2. Table of Contents Phase II Study Overview 3 Results: 8 Engagement 9 Group Dynamics 12 Switch between PMT & Physical Exhibit14 Questions and Challenges 16 Enjoyment 17 Emotional Response 18 Impact on the Visitor Experience 20 Likelihood of Searching out Facilitators with Tablets 23 Conclusions 25 References 28 Appendices 30

3. Phase II StudyOverview 21-Tech, an IMLS-funded project, is focused on developing strategies to increase exhibit facilitators’ capacity to use personal mobile technologies (PMT) to engage visitor learning at their museums. A major aspect of the project focuses on testing strategies for using PMTs in facilitation. This brief presents the main findings from Phase II testing. • which exhibits to test PMTs. (See Phase I evaluation presentation document [Garibay & Schaefer, 2011] for more detail.) • Phase II testing • Whereas Phase I was intentionally broad and exploratory, the second phase aimed to narrow the focus based on earlier findings. The specific goals of this phase were to: • Narrow the testing questions. • Test facilitation strategies with visitors at common types of exhibit experiences. • Test a facilitation experience at more than one partner site. • Test and inventory App categories. • Partner institutions paired up based on the availability of similar exhibits at their institutions (see “Exhibit Themes and Test Sites” table). These institution pairs • Staff from the four partner museums of this project (Children’s Museum of Houston, New York Hall of Science, Oregon Museum of Science and Industry, and Sciencenter) conducted Phase II Testing in December 2011 and January 2012. Phase II was intended to build on findings from Phase I testing. • Phase I background • Phase I testing, exploratory in nature, was designed to give staff the opportunity to pursue a range of questions about how facilitators might use PMTs to positively engage visitors with various exhibits. • 21-Tech partner museum staff prototyped facilitation strategies with PMTs in one or more exhibits using a range of content (applications, games, videos, photo-graphs, etc.) with visitors. Partners made their own choices about what to try and in • Phase II Evaluation Questions • What is the nature of the facilitator-visitor interaction? • In what ways do exhibits benefit from PMT-based facilitation? • In what ways does PMT-based facilitation impact the quality of visitor interactions?

4. Phase II Study Overview, cont’d. Typically, one staff member facilitated the interactions at the exhibit using an iPad while a second staff person observed and collected data. Throughout Phase II, the 21-Tech team met monthly by phone to share progress, discuss findings, and reflect on the process. Methods The partners collected observational data (Bobbie, 1998) using a standardized instrument that captured both qualitative and quantitative data about visitors’ interactions with the PMT, the exhibit, and facilitators; engagement within the group; and responses to worked together to develop and test a facilitation strategy at the identified exhibits at both sites. Test site one developed, tested, and revised the facilitation strategy. Test site two then tested the strategy and recommended further revisions based on their findings. This method allowed the partners to test facilitation at four exhibits in an iterative fashion. (See Appendix A and B for more information about the exhibits and PMT apps used by each institution.) Garibay Group worked with the project team to develop testing protocols and instruments. Partner staff collected data. PMT-facilitated interaction. The four partner institutions documented 94 such interactions. Visitor feedback was also collected via surveys. After an interaction, facilitators asked one adult in each visitor group to complete the brief, written survey. Fifty visitor surveys were collected (a 52% response rate). • Exhibit Themes and Test Sites

5. Phase II Study Overview, cont’d. understanding of PMT-facilitated interactions. In analyzing the data, however, we found that the descriptions of interactions and visitor behavior were often brief and missing detailsand therefore did not easily lend themselves to deep levels of analysis. Other limitations to the study included the low response rate to the visitor survey and the even lower response rate to its one open-ended question. The low response rate seemed to be due, in part, to not consistently asking each participating visitor group to complete the survey. Additionally, in an ideal situation, surveys would have been orally administered, which would have increased the number of responses to open-ended questions. The testing situation, however, did not allow observers to collect survey data; they were typically occupied with writing up observation notes and it was not appropriate for facilitators to orally administer feedback surveys on their own. • Data analysis • Garibay Group conducted an independent analysis of Phase II testing data. • Quantitative data were analyzed and summary statistics calculated, Data were cross-tabulated on several variables to identify major differences between respondent characteristics (e.g., age, gender). Differences are reported as appropriate. Also note that percentages in results are rounded up and, therefore, in some cases, may not add up to 100. • For qualitative data, data were coded using thematic analysis. In thematic analysis, the research identifies patterns (themes) within the data. A theme “captures something important about the data in relation to the research question and represents some level of patterned response or meaning within the data set.” (Braun and Clarke, 2006) • Limitations • The goal of the qualitative data was to capture rich descriptions and stories of the interactions in order to deepen our The diagram above was developed in Phase I to illustrate factors affecting the visitor-facilitator interaction. Phase II data collection focused on understanding the interaction from the visitor’s perspective.

6. Phase II Study Overview: Visitor Profile 65% of interactions came with groups of two (33%) or three (32%) visitors. Of visitor group members, 86 were adult females and 52 were adult males. Visitor groups included children from ages 1-17, with the majority of children in the 4- to 8-year-old range. N=148

7. Phase II Study Overview: Visitor Profile, cont’d. • It is likely that many respondents were comfortable with iPads or smart phone apps. More than half of visitors surveyed reported owning a smart phone, while over a third owned an iPad or other tablet.

8. Results

9. Engagement • Visitor groups engaged highly with the physical exhibit, the PMT, and the facilitator during their interactions. There was also evidence of interest and engagement in visitor-facilitator interactions. During 73% of interactions, at least one person was classified as intently engaged. During 14% of interactions, at least one visitor was classified as someone who “didn’t want to leave.” Length of interactions (in minutes)

10. Engagement, cont’d. Overall, children took a more active role in the visitor-facilitator interactions than adults. This included both engaging with the exhibit and interacting with the PMT. When it came to listening during facilitated interactions, however, children and adults took part fairly equally. • Children: Gender and Age • We disaggregated data for children by gender to look for differences. Overall, we found no significant behavioral differences during interactions. • An equal percentage of male and female children (76%) engaged with the physical exhibits during interactions. • A slightly higher percentage of male children engaged with the PMT (65%) than female children (60%). • When the ages of children who engaged the PMT were charted, the distribution followed the overall age distribution of child visitors and did not indicate that any one age was more likely to engage with the PMTs.

11. Engagement, cont’d.

12. Group Dynamics • Interactions were almost evenly divided between those led by adult visitors, those led by children, and those led by facilitators. (Nearly one-third of the interactions were missing descriptive data; therefore, further analysis was not possible.) In 15 cases, the interaction fit two categories (e.g., started facilitator-led but switched to child-led). These were removed from the data set.

13. Group Dynamics: Effect of Group Size In addition to noting behaviors that indicated engagement during visitor-facilitator interactions, observers also saw group members who appeared to not be engaged during the interaction. While at least one visitor who did not engage was present in 27% of all interactions, these visitors represented only 15% of total visitors. Group size affected the chance that at least one person in the group might not engage. Almost all interactions in which someone was not engaged involved visitor groups of three or more. Adults were slightly more likely than children to be disengaged. 57% of visitors categorized in observation data as “not engaged” were adults. No significant differences emerged between groups based on age or gender. Descriptions of visitor groups with adults who were not engaged indicated that they were often watching smaller children in the space or stood to the side observing his or her children interacting with the facilitator, PMT, or exhibit. The percentages for child males and child females were slightly lower than adults at 12% and 14% respectively. Mom was off to the side monitoring a younger child. The two boys were fully engaged with the Giant Arch and the iPad. One boy was watching [the facilitator] talk about bridges, but [the facilitator] could tell that he was losing interest as he watched his peer build. [The facilitator] decided to switch to the timer so that he could time them build[ing]. [The boys became] Much more excited…began building quickly and the parents began to help hold the parts of the arch. Man watched from the side while observing the two smaller children, but then joined in to use the microscope later. The three-year-old male child was doing his own thing. The 13-year-old and seven-year-old male children actively listened and participated. 

14. Switch Between PMT and Physical Exhibit In the majority of interactions, groups were observed switching between the PMT experience and the physical exhibit with minimal difficulty. N=82 In 12 instances, the switch between virtual and physical experience was classified into more than one category. These data have been removed from this analysis because the categories are mutually exclusive.

15. Switch Between PMT and Physical by Exhibit, cont’d. The Colored Shadows exhibit had the highest percentage of switches classified as “seamless.” Switch Between PMT and Exhibit The app used on the PMT allowed visitors to mix colors in much the same way as the physical exhibit, so this may have accounted for the seamlessness of the switch. The Structures-themed exhibits had the highest percentage of switches categorized as “some difficulty.” Based on descriptive data, this may have been because of the physical nature of the exhibit activity and because the PMT was used only to show photos.

16. Questions and Challenges • In 51% of interactions, someone in the visitor group asked a question. Both children (37) and adults (29) asked questions in these groups. In contrast, far fewer interactions (10%) involved visitors posing challenges to their groups. Only eight adults and three children posed challenges during interactions. Questions When coded, questions asked by visitors fell into two major categories: those related to the content of the exhibit and those related to the PMT itself. The following are example of questions related to the PMT: Can you move [the lights in the exhibit] with that [the iPad]? Is this an app I can get? The following are examples of questions related to the content of the given exhibit: Why can’t we get it [the arch] to stand? If I punch out the keystone, everything will fall down? What keeps the disease from infecting white blood cells? Is that the nucleus? Why does it [the wind over the tube] sound like that? It’s important to note that overall, more content-related questions were asked by visitors than PMT-related questions. This suggests that the PMT-facilitated interactions encouraged visitors to think about the exhibits, not just about what they were doing on the PMT. When data were disaggregated by exhibit, Microscopy and Pitch tended to generate deeper lines of questioning than the other exhibits (i.e., visitors asked simple questions that progressed to more sophisticated ones). Because of the level of documentation of these questions, however, it’s unclear whether deeper questions about these exhibits were due to their content (i.e., blood cells, onion cells, sound), the particular PMT app used, the skill of an individual facilitator, or some other factor. Challenges Challenges during interactions were primarily posed by facilitators (74% of engagements). Descriptive data on challenges posed by visitors was not robust enough for qualitative analysis. Below are a few examples of visitors posing challenges: The first boy built the arch one time by himself with [the facilitator’s] help and then again with two other boys' help. Then he ran to get his cousin and there they set a challenge to complete the arch as quickly as possible. They kept this up for many builds. The arch was built six times during this interaction...Each time they laughed and had a good time. They worked well together utilizing a teamwork approach to building the arch. [The man] asked about the read-outs on the Tuner+ app and then prompted kids to watch the notes change and they slid the plunger on the pipes.

17. Enjoyment The majority of survey respondents (80%) rated their experience using the tablet “enjoyable” or “very enjoyable.”

18. Emotional Response • Most emotional expressions by visitors during interactions were positive. Smiles and/or laughter were present in over half the interactions. Examples of typical verbal exclamations were “Cool” and “Wow!” Visitors expressing confusion or frustration were observed in only a few interactions. The woman and girl were the group members initially interested in “Zoom In” and seeing the iPad images. They were having fun using the microscope on their own clothes, hair, and skin. The woman was definitely more vocal and kept saying, “Ooh, look at this and that.” The man joined in later and then woman and man both stayed at “Zoom In” while the three girls played with the “Stuff Sorter.” The 9-year-old was very engaged. She smiled upon success with color matches on iPad. Left saying “Cool.”

19. Emotional Response, cont’d. Frustration or ConfusionThe two documented cases of frustration involved visitors who had to wait on their group or leave an exhibit with their group when they wanted to stay. All cases in which confusion was noted (N=4) were not sufficiently documented for in-depth analysis. In one case, though, the PMT app helped a confused visitor to better understand the exhibit. The six-year-old girl was confused about how the colored shadows were created in the exhibit. The iPadwas very helpful in clarifying relationship between how to create cyan, for example, which is different from the blue light in the exhibit.

20. Impact on the Visitor Experience There was evidence that the PMT-based interactions facilitated connections for visitors between the content at hand and prior experiences. During 47% of interactions, at least one visitor was observed making a connection between the exhibit and PMT. During 29% of interactions, at least one visitor was observed making a connection to his or her previous experiences. [The man] talked about using digital tuners to tune his violin. Both mom and daughter had previously discussed how black is made using pigments. [They were] interested to see how white is made with light. [The visitor] connected colds and white blood cells. Connection to school. In a couple of instances, children mentioned that they had “studied [cells] in school” or had used microscopes to look at blood cells in school. Connection to everyday things. This category included connections that visitors made to things they encounter in everyday life. For example, a child compared the Giant Arch to a rainbow. Another compared blood cells to a polka-dotted shirt. [The eight-year-old girl] said "I think it's [the colored shadows are] because of [the exhibit lights]." …She was then able to recreate the colors of the shadows using the PMT. After seeing pictures on the iPadand building the arch, the child realized that the keystone holds the entire structure together and that if it is removed, the arch will fall. Connections to prior experiences In 17 interactions, adults made connections to their previous knowledge or experience while in 14 instances, children made this type of connection. Three categories of connections emerged: Connection to previous knowledge. This category included connections visitors could make to their professions, hobbies or previous discussions. For example: • Connections between exhibit and PMT • In most cases (N= 38), children made connections between the exhibit and the PMT. In 14 interactions, adults made the connection between the exhibit and PMT. Though descriptions of visitors’ connections were not detailed enough to allow in-depth analysis to determine whether patterns existed in the types of connections, the following are examples of connections visitors made between the physical exhibits and the PMTs: • The 3-year-old realized that the colors on the wall can be made on the iPad. • The [girl] was able to make the connection of the keystone from the picture to the actual Giant Arch and describe why it was able to stand without help. • [The five-year-old boy] could see “dots” (nuclei) in both PMT and the microscope.

21. Impact on the Visitor Experience, cont’d. • Evidence suggests that in 19% of interactions (18 of 94), the visitor group wished to extend the experience of the PMT-facilitated interaction. This category was created to capture data related to visitor groups’ interest in extending the experience in specific ways (i.e., asking if the facilitator could provide a PMT-facilitated interaction at a nearby exhibit or asking if the app was available for the visitor to use at home), but the data collected captured broader ways in which visitors “extended” their initial experience. These fell either into those who extended the experience during their museum visit or those expressing a desire to extend the experience post-museum visit. • Extension after the visit • The following are the types of behavior that indicated a visitor wanted to extend experience after museum visit: • Visitor asked about the app or PMT • Visitor expressed a desire to use app at home • Visitor expressed a desire to use app at school • Based on the brief descriptions of the documented evidence for extending the experience, it is difficult to attribute visitors’ desires to extend their experiences to any one factor. Options might include visitors’ engagement with the PMT and apps, visitors’ overall comfort with a facilitator, visitors’ enjoyment of the experience in general, or something else altogether. • Extension during the visit • The following are the types of behavior that suggested a visitor extended the experience during the museum visit: • Visitor asked about other exhibits • Visitor engaged with the exhibit multiple times • Visitor tried another app on the PMT • Visitor tried a suggested activity at the exhibit • Visitor engaged with another, nearby exhibit

22. Impact on the Visitor Experience, cont’d. Respondents indicated that their interactions enhanced their overall experience. The majority of respondents (90%) agreed or strongly agreed with the statement “My interaction with the exhibit was enhanced by the use of the tablet.”

23. Likelihood of Searching Out Facilitators with Tablets Over 80% of respondents said they would be “somewhat” or “very likely” to look for museum facilitators who had tablets with them.

24. Likelihood of Searching Out Facilitators with Tablets, cont’d. • Twenty six respondents who indicated they were “somewhat” or “very likely” to look for museum facilitators who had tablets provided reasons for their ratings. When responses were categorized, three primary reasons emerged. (Note that some respondents gave more than one reason.) • Learning: Visitors noted that the PMT-facilitated experience helped them better understand the exhibit or that they learned something new (N=11). • Fun: Respondents felt the PMT-facilitated interactions they had were fun or good experiences for their group (N=8). • iPad Technology: Parents commented that, in general, their children were attracted to the Tablets (N=9). One parent further noted that they liked that it provided opportunities for children to spend time with the technology as well as with a facilitator. • Three respondents qualified their ratings by noting that they it would depend on the age of the children because they did not think the iPad was particularly useful to younger children. On the other hand, one parent at the Colored Shadows exhibit commented that he was not sure the exhibit would have made sense for his children without the iPad because they were young. Reasons respondents were unlikely or very unlikely to look for facilitators with tablets Of the respondents who gave “unlikely” or “very unlikely” ratings, the primary reasonswere that they saw the iPad as useful in providing particular types of information, but not something that could be used widely (N=2) and that they did not feel the activity was relevant. The iPad is good for showing very specific things, but isn’t exactly as handy as a pocket book with that information. The timer doesn’t show anything relevant. Seeing the explanation on an app gave a better understanding of what I was experiencing. Very excited about you guys using the iPad! They really seemed to enhance the knowledge imparted to my kids! Thanks! If the tablet doesn’t take away from the activity, then it’s more likely I will look for a person with a tablet. It made viewing the image very easy to zoom in and out and it focused easily! I think it was very fun and interesting adding this additional tool into the interaction. My child really enjoys playing with my iPhone, so it was nice to see that on the museum floor. My four-year-old was having a great time learning while sitting still for a whole ten minutes. Thank you. I think my children benefitted from spending time both with an educator and with the technology. It was nice to have the opportunity to use the iPad with my children.

25. Conclusions

26. Conclusions • Based on Phase II data, the use of PMTs during visitor-facilitator interactions seems to result in a positive visitor experience. • The majority of visitors (80%) rated their experience as “enjoyable” or “very enjoyable” and 90% agreed that the tablet enhanced their experience with the exhibit itself. Observations noted solid positive affective responses. In over half the interactions, at least one visitor in the group smiled or laughed. Expressions of frustration or confusion were not common. • Overall, visitor groups engaged with both the physical exhibits and virtual apps during their interaction with the facilitators. The PMT-facilitated interactions did not seem to distract visitors significantly from interacting with the physical exhibit. A mix of visitors worked with the physical exhibit, using the PMT, and listened during the visitor-facilitator interaction. The percentage of visitors classified as not engaged in interactions was low. Furthermore, there is evidence that children of all ages and both genders participate in these interactions. • There was also evidence that the PMT-based interactions positively impacted visitors’ engagement with the content. In about half the visitor-facilitator interactions, someone in the visitor group asked a question. Most of the questions documen-ted concerned the content of the exhibit. Visitors were also able to make connections between the exhibit and their prior experiences. In surveys, some respondents also commented on their experience, noting that the PMT-based interactions helped them better understand exhibit content. This suggests that visitors are thinking about the exhibit content during these interactions. • Visitors also made connections between the exhibit and the PMT apps in over half the observed interactions. This, in part, suggests that visitors were engaged with the exhibit—not just the PMT—during their interactions.This finding is another indication that facilitation is successfully integrating the PMT with the physical exhibit. Furthermore, in the majority of interactions, the visitor group was able to switch between the physical and virtual experience without much difficulty. For discussion Some Phase II findings also raise important questions for the 21-Tech team. Visitor Questions The 21-Tech team outlined visitors asking questions as an important indicator of successful engagement. Just over half (51%) of interactions, had someone in the visitor group ask a question. The team needs to determine whether this percentage is on target or if, ideally, it should be higher. (If so, the team will want to explore strategies for increasing the percentage of interactions in which visitors ask questions.) Visitor Challenges Having visitors pose a challenge was also identified by the 21-Tech team as an indicator of successful engagement. Only 10% of interactions included a visitor who posed a challenge to his or her group. This may be an acceptable percentage, especially given that in 74% of interactions the facilitator posed a

27. Conclusions, cont’d. • challenge, but it suggests that appropriate targets with accompanying strategies may need to be discussed. • Group dynamics • 82% of the interactions were classified as adult-, “near-peer”, or child-led (vs. facilitator-led) at some point. While this finding suggests that at some level, visitor groups took ownership of the interaction, more qualitative descriptive data would need to be gathered to fully understand the extent to which interactions are facilitator-led vs. visitor-led. • Switch between PMT and physical exhibit • While overall the switches between PMTs and physical exhibits by visitors were classified as seamless or fairly smooth, of the four exhibit themes, the switches for Structures and Pitch were more likely to be classified as “fairly smoothly” or “with some difficulty.” This finding suggests a number of questions for discussion including: • Are more physically engaging exhibits, such as Structures, less appropriate for PMT-assisted facilitation? • Do facilitators need to use more interactive PMT apps instead of just photos? • How can teams ensure that apps relate closely to the exhibit so that visitors can switch easily between the two? • Connections to previous knowledge • In 29% of interactions, someone made a connection to his or her previous • knowledge and/or experience. This type of • connection is an important indicator of visitors integrating their experiences into their existing knowledge. Because of its importance, it might be worthwhile to determine the extent to which facilitators should attempt to help visitors make more explicit connections to their own personal experiences and existing knowledge. • Learning • There were indications that visitors perceived that the PMT-based interactions helped them deepen their understanding of the exhibit content. This finding merits further investigation about the process and ways these interactions facilitate deeper experiences with the content. Extension of the experience Phase II data contained little evidence that visitors desired to extend their experience beyond their single interaction with the facilitator. This finding, however, contrasts with the visitor survey data that suggested that over 80% of respondents were somewhat or very likely to look for facilitators with tablets in the future. This finding merits further investigation.

28. References

29. References Bobbie, E. (1998). The Practice of Social Research. Albany: Wadsworth Publishing. Braun, V. & Clarke, V. (2006). Using Thematic Analysis in Psychology. In Qualitative Research in Psychology, 3: 77-101. Garibay, C. and Schaefer, J. (2011) . 21-Tech Formative Evaluation: Phase I Results Presentation. Available at: http://www.21-tech.org/2012/03/formative-evaluation-phase-i-results/ Miles, M. B. & Huber man, A. M. (1994). Qualitative Data Analysis (2nd ed.). London:  Sage Publications.

30. Appendices

31. Appendix A: Exhibits and Selected Apps Used During Testing Colored Shadows Exhibit Theme Colored Shadows exhibition at Sciencenter Colored Shadows exhibition at NYSCI NYSCI facilitators used Kaleidoscope and Color Mix HD with their Colored Shadows exhibition. Sciencenter facilitators used Color Uncovered and Bobo Explores Light with their Colored Shadows exhibition.

32. Appendix A: Exhibits and Selected Apps Used During Testing, cont’d. Microscopy Exhibit Theme Microscopy Lab at OMSI Zoom In exhibition at CMH CMH facilitators used photos with the Zoom In exhibition. OMSI facilitators used Objective and Anatomy 3D with the Microscopy Lab.

33. Appendix A: Exhibits and Selected Apps Used During Testing, cont’d. Pitch Exhibit Theme Pipes of Pan exhibition at NYSCI Slap Organ and Pipes of Pan exhibition at Sciencenter Sciencenter facilitators used Tuner+ and WI Orchestra with their Slap Organ and Pipes of Pan exhibitions. NYSCI facilitators used Tuning Fork and Tuner+ with their Pipes of Pan exhibition.

34. Appendix A: Exhibits and Selected Apps Used During Testing, cont’d. Structures Exhibit Theme Catenary Arch exhibition at OMSI Giant Arch exhibition at CMH OMSI facilitators used Largest Bridges and Giant Timer with the Catenary Arch exhibition. CMH facilitators used Largest Bridges and Giant Timer with the Giant Arch exhibition.

35. Appendix B: PMT Apps Used During Testing

36. Appendix B: PMT Apps Used During Testing, cont’d.

37. Appendix B: PMT Apps Used During Testing, cont’d.