Empirical Studies of Design Ideation: Alignment of Design Experiments with Lab Experiments

1. Empirical Studies of Design Ideation: Alignment of Design Experiments with Lab Experiments JamiJ. Shah Noe Vargas-Hernandez Mechanical and Aerospace Engineering Arizona State University, Tempe, AZ Steve M. Smith David R. Gerkens Department of Psychology Texas A&M University, College Station, TX Muqi Wulan Department of Industrial and Manufacturing Systems Engineering Beijing University of Aeronautics and Astronautics, Beijing, China NSF Grant Number: DMI-0115447 ASME 2003 International Conference on Design Theory and Methodology September 5, 2003, Chicago Il

2. PSYCHOLOGY VS ENGINEERING • For many years, Psychologists and Designologists have studied IG. Their experiments have different “ecological” validity (i.e. realism captured) • Lab Experiments done by Psychologists • Low level – use few and simpler variables • Design Experiments done by Engineers • High level – use more and complex variables INTRODUCTION MOTIVATION There are many methods for design synthesis How useful are these? Which ones are better? There isn’t much empirical data on specific effectiveness of Idea Generation (IG) methods

3. SCOPE At this time we do not consider human variables as experiment variables (e.g. experience, creativity of designers) Our focus is on Intuitive IG methods INTRODUCTION OBJECTIVE If our experiments at different levels have the same results, we can establish a connection and we can run more of the simpler Lab Experiments More Experiments means more empirical data on specific IG methods Our alignment approach is still WIP and is part of a bigger project Our ultimate objective is to develop a theoretical model of design ideation Such a model would help us better understand IG methods

4. INTRODUCTION DESIGN IDEA GENERATION METHODS Figure 1. Classification of Idea Generation Methods (Shah et al., 2000)

5. PRELIMINARY RESEARCH PAST EXPERIMENTS For several years we have been doing experimental studies to define the effectiveness of ideation • We used two distinct approaches to conducting experiments: • Direct Method – IG methods are studied as a whole • Indirect Method – Ideation Components are studied • And two distinct approaches to assessing the effectiveness: • Process – Assess the ideation process (e.g. protocol studies) • Outcome – Assess the ideas produced (e.g. sketches)

6. PRELIMINARY RESEARCH IDEATION COMPONENTS Ideation Components are mechanisms that are believed to intrinsically promote IG or to help designers overcome mental blocks. • Examples of Ideation Components: • Provocative Stimuli • Deferred Judgment • Flexible Representation • Frame of Reference Shifting • Incubation • Example Exposure • Examples of Blocks: • Being Judgmental • Emphasis on Quality • Lack of Motivation • Having a tight grip on problem specs. • Rigid Problem Representation • Design Fixation • Imposing Fictitious Constraints These are “common”: Known in Engineering Design Research and acknowledged by Cognitive Psychology

7. PRELIMINARY RESEARCH OUTCOME EFFECTIVENESS Four measures were defined in our previous projects (Shah, Kulkarni, and Vargas-Hernandez, 2000). Table 1. Effectiveness Measures for Idea Generation Outcome.

8. RESEARCH APPROACH Alignment: Agree on the Ideation Components to study and the Effectiveness Metrics for assessment RESEARCH APPROACH FUNDAMENTAL ISSUES How can we compare results from experiments at different levels? • DESIGN EXPERIMENTS • (Done by ASU Engineers) • Simulate real world better • Incorporate more and complex variables • Require prohibitive number of experiments • Unable to explain the performance of methods under different conditions. • LAB EXPERIMENTS • (Done A&M Psychologists) • Focus on “atomic” cognitive processes • Little similarity between the condition of these experiments and design concept generation in the real world. How can we align these two?

9. Design Experiments Done By Engineers More Lab Exp.  Better Understanding of IG  Theoretical Model of Design Ideation Results Comparison Alignment: Ideation Components Effectiveness Metrics Lab Experiments Done by Psychologists Results RESEARCH APPROACH RESEARCH APPROACH How can we compare results from experiments at different levels? Figure 2. Research Strategy

10. RESEARCH APPROACH IDEATION COMPONENTS The number of components identified is more than a dozen. Because of limited resources and the prohibitive number of experiments required to study all possible interactions, only the most relevant were selected. Table 2. Selected Components

11. Table 3. Experiments in a 23 Full-Factorial Design Table 4. Simple Comparative Experiments All 6 Ideation Components were tested at ASU and TAMU simultaneously RESEARCH APPROACH DESIGN OF EXPERIMENTS (DOE) Two levels were considered for each of the selected components. Although more levels could be defined, it is recommended to run experiments initially with few levels. Full Factorial Experiments A Frame of Reference Shifting B Incubation C Example Exposure • Simple Comparative Experiments • Provocative Stimuli • Suspend Judgment • Flexible Representation

12. RESULTS DESIGN EXPERIMENTS DONE BY ASU ENGINEERS Experiment Variables Subjects Undergraduate Engineering students. Comparable expertise/knowledge between subjects is assumed. Task One design problem was used for all experiments. The objective was to design a device for throwing a ping-pong ball the farthest distance. A list of allowed materials was given; this to improve the quality of sketches. Idea Recording Subjects were asked to generate ideas individually using sketches Nuisance Variables Similar environmental settings procured for each run (classroom, noise, light, etc.)

13. RESULTS DESIGN EXPERIMENTS DONE BY ASU ENGINEERS Figure 3. Sample Sketches from Design Experiments

14. RESULTS LAB EXPERIMENTS DONE BY TAMU PSYCHOLOGISTS • Experiment Variables • Subjects • Undergraduate Psychology students. Comparable expertise/knowledge between subjects is assumed. • Task • Listing members of large taxonomic categories, sense impression categories, and ad-hoc categories. • Divergent thinking, unusual uses of common objects. • Idea Recording • Subjects were asked to generate ideas individually using text for member listing tasks and sketches for divergent thinking tasks • Nuisance Variables • Similar environmental settings procured for each run (classroom, noise, light, etc.)

15. RESULTS LAB EXPERIMENTS DONE BY TAMU PSYCHOLOGISTS Figure 4. Sample Sketches from Lab Experiments

16. HIGH HIGH HIGH HIGH HIGH HIGH HIGH HIGH RESULTS IDEATION COMPONENT: INCUBATION Table 5. Lab Experiments Done by TAMU Psychologists: Mean Ideation Effectiveness Scores Table 6. Design Experiments Done by ASU Engineers: Mean Ideation Effectiveness Scores

17. RESULTS IDEATION COMPONENT: INCUBATION Table 7. Two Sample t-test

18. RESULTS IDEATION COMPONENT: INCUBATION Table 8. Correlation Between Lab and Design Experiments

19. CONCLUSIONS IDEATION COMPONENT: INCUBATION • Based on the results from (TAMU Psychologists) Lab and (ASU Engineers) Design Experiments, Incubation increases the effectiveness of ideas generated. Results correlate at both levels and show a satisfactory confidence level. • Incubation’s positive impact on Design Ideation is substantiated by concrete Engineering evidence (from ASU Design Experiment results) and has a theoretical basis (from TAMU Lab Experiment results).

20. More empirical data Better understanding of Ideation Theoretical Model of Design Ideation. Run more of the simpler Lab Experiments and less of the more complex Design Experiments CONCLUSIONS OVERALL • More experiments needed to prove connection • The alignment procedure provides a framework for comparison between both levels. Results for Incubation exemplify how the alignment works. Connection Proven Experiments on other components have been completed at ASU and TAMU.

21. FURTHER CONCLUSIONS • According to our results we found Frame of Reference Shifting (FORS), Incubation (I), and Example Exposure (EE) to have similar main effects. • Interaction effects weren't that clear, probably because some components are much alike (specially FORS and EE) and hence aren't independent. • This generates a question: Maybe these Ideation Components have the same effect on ideation ? • Two or more Ideation Components sharing the same effect could belong to the same higher level Ideation Principle

22. FUTURE WORK IDEATION PRINCIPLES Table 9. Comparison of Ideation Principles with Cognitive Components

23. FUTURE WORK IDEATION PRINCIPLES • Refine Ideation Principles and its Implementations (Ideation Components) • Run more exercises, collect and analyze more data to prove/disprove our theory about Ideation Principles • We still continue experimenting on Ideation Components, but with a better understanding of Ideation Principles, experiments can be better targeted (e.g. less redundant) and more efficient.

24. REFERENCES • Altshuller, G., 1984, Creativity as an Exact Science, Gordon and Breach, New York. • Dennehy, E.B., Bulow, P., Wong, F., Smith, S.M., and Aronoff, J.B. (April, 1992). A test of cognitive fixation in brainstorming groups. Paper presented at the meeting of the Eastern Psych. Association, Boston, MA. • Dodds, R.A., and Smith, S.M., 1999, Fixation. In M.A. Runco & Pritzker (Eds.) Encyclopedia of Creativity, San Diego, CA: Academic Press. • Ericsson, K., and Simon, H., 1984- “Protocol Analysis - verbal reports as data”, MIT Press. • Finke, R.A., Ward, T.B., and Smith, S.M., 1992, Creative Cognition: Theory, Research, and Applications, Cambridge, MA: MIT Press. • Hale, C., “Analysis of the Engineer Design Process in an Industrial Context”, Grant Hill Pubs, Cambridge, 1987. • Jansson, D. G., and Smith, S. M., 1991, “Design Fixation,” Design Studies, Vol. 12, pp. 3-11. • Koestler, A., 1964, “The art of Creation”, Hutchinson and Co., London. • Langley, P., and Jones, R., 1988, “Computational model of scientific insight,” in R. J. Sternberg, ed., The nature of creativity – contemporary psychological perspectives, Cambridge University Press, NY. • McKoy, F., 2000, “Experimental Evaluation of Engineering Design Representations for Idea Generation”, MS Thesis, Arizona State University, Tempe, AZ. • Mckoy, F., Vargas-Hernandez, N., Summers, J. D., Shah, J., 2001, “Experimental Evaluation of Engineering Design Representation on Effectiveness of Idea Generation”, Proceedings, ASME Design Theory and Methodology Conference, Pittsburgh, PA. • Koestler, A., 1964, “The art of Creation”, Hutchinson and Co., London. • Martindale, C., 1995, “Creativity and Connectionism,” in S. M. Smith et al., eds., The creative cognition approach, MIT Press, Cambridge, MA. • Schön, D., 1991,“Teaching and learning as a design transaction”, in Research in Design Thinking Delft Press, 1991.

25. REFERENCES • Schwartz, B.L. and Smith, S.M., 1997, The retrieval of related information influences tip-of-the-tongue states. Journal of Memory & Language, 36, 68-86. • Shah, J., 1998, “Experimental Investigation of Collaborative Techniques for Progressive Idea Generation,” Proceedings, ASME Design Theory and Methodology Conference, Atlanta, GA. • Shah, J., Kulkarni, S., Vargas-Hernandez, N., 2000, “Guidelines for Experimental Evaluation of Idea Generation Methods in Conceptual Design”, Journal of Mechanical Design, vol. 122, no. 4, pp. 377-384. • Shah, J. J., Vargas-Hernandez N., Summers, J. D., Kulkarni, S., 2001, “Evaluation of Collaborative Sketching as an Idea Generation Technique for Engineering Design”, Journal of Creative Behavior, 35:3, pp.1-31. • Smith, D. K., Paradice, D. B., and Smith, S. M. (2000). Prepare your mind for creativity. Communications of the Association for Computing Machinery , 43, 110-116. • Smith, S. M., 1995, “Creative Cognition: Demystifying Creativity,” in C. N. Hedley et al., eds., Thinking and literacy – the mind at work, Lawrence Erlbaum Associates, Hillsdale, NJ. • Smith, S. M. and Blankenship, S. E., 1991, “Incubation and the persistence of fixation in problem solving”, American Journal of Psychology, 104, 61-87. • Smith, S. M., Carr, J. A., and Tindell, D. R., 1993, April, Fixation and incubation in word fragment completion. Paper presented at the meeting of the Midwestern Psychological Association, Chicago, IL. • Smith, S. M., and Vela, E., 1991. Incubated reminiscence effects. Memory & Cognition, 19 (2), 168-176. • Smith, S. M., Sifonis, C. M., and Tindell, D. R., 1998, Hints do not evoke solutions via passive spreading activation. Proceedings of the Twentieth Annual Meeting of the Cognitive Science Society, Madison, WI. • Vargas-Hernandez, N. and Shah, J. J., 2002, Inventory of Creativity Exercises: 1995-2002, Tech. Report ASU/DAL/IG/02-1, Arizona State University. • Wallas, G., 1926, “The Art of Thought”, Harcourt, New York.