Understanding Early Stage Design Processes

1. Understanding Early Stage Design Processes Maria Yang Robert N. Noyce Career Development Assistant Professor Engineering Systems Division and Dept. of Mechanical Engineering November 25, 2009 ESD.83 Doctoral Seminar in Engineering Systems

2. Early stage of design for products and systems High-impact phase within design and development process Good design process leads to good design outcome Challenge: Early stage of design fluid, ambiguous, difficult to measure Goal: Understand (and measure) informal design activities through their outputs

3. General approach • Descriptive rather than predictive • Many models about what goes on in design that are based on intuition and experience • Is that what really happens? • Instrument the design process (Leifer) • What are artifacts of design process? • How to capture their evolution? (Yang)

4. Engineering vs. psychology approach • NSF creativity workshops (with Frey) • Psychologists use controlled studies (Paulus) • Pros – Min. confounding factors, individuals • Cons – Short exercises (not realistic design), need many participants (500 psych students – ltd domain knowledge) • Engineers coarser grain (Leifer, Agogino) • Pros – More complex design activities, longer projects, groups • Cons – Confounding factors such as group dynamics

5. Who to observe • Ideal: Real world projects, access to process and project data • Do you think this is always possible? • Often, companies do not like to open themselves to scrutiny • Confidentiality • Embedded (Owens) • Students in the classroom • Novices • Assessment inherent part of education process

6. Design Preferences • Design consists of 2 distinguishing activities • Idea generation (synthesis) • Idea selection • Idea selection assumes set of preferences • Formal design synthesis approaches require formal weightings for a preferences (Antonsson) • Populate design space given sets of preference weightings • Reality – • Design preferences given informally (“I like this better than that” vs. “weight1 = .2, weight2 = .3”) • Design preferences often aggregate of group opinion. Not easy to do explicitly.

8. Research Metrics for design process Design data Designers Early stage design process Design problem Design outcome Clarify Generate Select

9. Research Metrics for design process Design data Designers Early stage design process Design problem Design outcome Clarify Generate Select Sketching and Concepts (Yang 09; Yang 03) Preference Information (Ji, et al 07; Yang & Ji 07) Designers and teams (Yang & Jin 07, 08) Users & needs (Lai, et al 09) Sketching Skill (Yang & Cham 07; Cham & Yang 05) Prototyping (Yang 04; Yang 05) Design Information Retrieval (Yang, et al 05; Yang and Cutkosky 97, 98; Wood, Yang, et al 98; Yang, et al 98)

10. Sketching skill

11. Sketching in Design • Sketches capture & communicate[Ullman 90; Verstijnen 98; McKim 80; Schön & Wiggins 92] • Sketching process linked with design cognition[Nagai & Noguchi 03; Suwa & Tversky 97; Goel 95] • Sketching is “dialogue”[Cross 99; Shah, et al 01; Goldschmidt 91; Tovey, et al 03] • If sketching is language of design, is sketching proficiency linked to design process or performance? [Yang & Cham 07; Cham & Yang 05]

12. Research questions • What is the nature of sketching skill in design? • Is drawing a generic ability? • How are different drawing skills related? • Research in mental imagery [Kosslyn 84; Kosslyn 94] • Comprehensive, generic “trait” • Task-based skill • Somewhere between 1) and 2) • Hypothesis • Sketching ability similar to (3)

13. Research questions • How is sketching ability linked to fluency? • Hypothesis: Those who draw better also draw more • How is skill related to design outcome? • Hypothesis: Can sketching skill serve as an indicator of outcome?

14. Related work • For conceptual design, sketching preserves ambiguity[Goel 95; Kavakli, et al 98] • Sketch classification • Function [Ullman 90; Ferguson 92; van der Lugt 05; Goel 95] • Elements [McGown 98; Rodgers 00] • Sketching and outcome • Teams who sketch vs. those who don’t [Schütze 03] • 3D sketching & outcome [Song & Agogino 04] • What about sketching skill?

15. Survey to assess drawing skill(do try this at home) • In 3 minutes, draw a bicycle with as much detail as possible. • Hold out the items given to you in your non-dominant hand (left-hand for right-handed persons). In 3 minutes, make a drawing of your hand and the items [two small candy bars]. • Visualize and draw the following in 2 minutes: A rectangular box that is open at the top. Inside the box is a rubber ball. The front of the box has a large button, and each side of the box has a large “X” painted on it.

16. Survey goals & assessment • Engineering sketches may utilize many elements • Bike task - Mechanical recall • Recall and sketch familiar mechanical object • Structure, function (“Look like a bike? Could you ride it?”) • Hand task - Drawing facility • Realistic, well-composed drawings from a still • Proportions, realism (“Does this look like a hand?”) • Box task - Novel visualization • Visualize specific features • Proportions, 3D perspective, realism

17. Level 1 Level 3 Level 5 Mechanical Recall Task Drawing Facility Task Novel Visualization Task

18. Design outcomes • Sketch fluency • Paper design logbooks; relatively objective • Perspective drawings; more skill required • Grades for class and for final project • Rankings by external judges • Spearman Correlations

19. Results: Types of sketching skill • Possible results • Comprehensive skill: Strong correlations between tasks • Task-based skill: No correlation • Skill lies between the two: Range of correlations • Results suggest option 3 Correlation between sketch tasks. N = 32, Rs >= 0.296 for  = 0.10.

20. Sketching ability and fluency • Total: Drawing “well” correlates positively • 3D: Bike task correlates negatively • Drawing skill vs. other means of visualization? N = 32, Rs >= 0.296 for  = 0.10

21. Sketching and Design Outcome • Sketch fluency: Positive but no sig. correlation • Sketching skill: No clear trends • Design process depends on many skills/factors • Project type, outcome measures • More studies needed N = 32, Rs >= 0.296 for = 0.10 N = 33, Rs >= 0.291 for  = 0.10

22. Conclusions 1. Is sketching ability generic? • Sketching skills not created equal • Possible reason: Different cognitive skills required (gearhead and artist) 2. Is sketching ability linked to sketch fluency? • Hand and box task correlate, but not bike • Sketch fluency (partly) influenced by how much a designer can design without drawing. • Possible reasons • Mechanical recall = visualization in head • Common complaint: “I don’t need to keep a logbook”

23. Conclusions 3. Sketch ability linked to design performance? • No relationship between sketch tasks and outcome • “Good” sketchers did not necessarily do well (or vice versa) • Possible reasons • Engineering design complex, requires many skills; sketching is only one • Sketching may be behavioral rather than a necessary element of design activity (doodler)

24. Extraction of preferential probabilities from design team discussion

25. Overview • Making choices is one key activity in design • Designers express "design preferences" by assigning priorities to a set of possible choices • Assigning preferences can be complex for a team • Elicitation of preferences from a group (surveys, voting) • Aggregation of preferences among the group [Mark 02]

26. Research Questions • How can preferential probabilities of a design team be extracted? (Ji, et al 07; Yang & Ji 07) • Obtained implicitly, not explicitly • How to address aggregation? • Do preferential probabilities evolve over time? • Way to describe how a team selects alternatives throughout the design process • How does extracted information compare with that obtained explicitly? • Consistent with preference information captured via surveys? • Preferential probability: Likelihood one alternative will be selected as “most preferred” over all others

27. Approach • Extract preferential probabilities from transcripts of design team discussion • Design alternatives known a priori • Assume preference-related information embedded • No formal aggregation of individual information • Simple collection of words • Assumptions • What designers think in one time interval relates to what they thought in the previous interval • Designers tend to speak positively about the design alternative they prefer more and negatively about those they prefer less

28. Related Work • Preference Extraction • Surveys: The lottery method [Hazelrigg, 99; Otto & Antonsson, 93] • Pair-wise comparison: AHP [Saaty 00], fuzzy outranking [Wang 97] • Multi-criteria overall aggregation function using MoI [Scott & Antonsson 98] • Conjoint Analysis [Green 90] and Discrete choice analysis [Hensher & Johnson 81; Ben-Akiva & Lerman 85] • Collaborative filtering [Kohrs & Merialdo 00] • Group Preference Aggregation • Cardinal utility functions for accumulating group preferences [Keeney 76] • Structured pair-wise comparison chart [Dym, Wood & Scott 02] • Aggregation with equal weights [Bask & Saaty 93] • Aggregation with unequal weights [Jabeur, et al. 99; See & Lewis 05] • Arrow’s Theorem: no guarantee of consistency in a group [Arrow 70, 86] • Design Process Evolution • Surveys [Brockman 96], Coding of design journals [Jain & Sobek 06] • Team cohesion analysis (“Story telling”) [Song, et al 03]

29. Models • Preference Transition Model (PTM): relationship between preferences in 2 consecutive time intervals • Utterance-Preference Model (UPM): relationship between preferences and utterances in one interval Most-preferred alternative in i+1 Probability designers won’t change most-preferred alternative Most-preferred alternative in interval i Alternative uttered in time interval i Probability designer will utter their most-preferred alternative

30. Case Study 1: Large Scale Space System Design [http://history.nasa.gov] • Highly concurrent, real-world design team working on concept stage of space system architecture • 17 experienced scientists and engineers; range of disciplines • Focused on group of 4 working on single subsystem • Three 3-hour sessions of discussion • ~28,000 words • Primary team member talked nearly 85% of the time • Two component selection problems

31. Case study 1: Results from Large Scale Space System Design View of how probability of preference changes over time for a re-design problem a2 and a3 alternate with each other as most-preferred choice Alternative a1 is least preferred [Ji, Yang & Honda, 2007, ASME IDETC 2007]

32. Case Study 2: Coffee Maker Design • Small engineering team with 3 graduate students

33. Case study 2: Preferential Probability Results From Transcript Analysis

34. Case study 2: Comparison of Preferential Probabilities from Transcripts and Surveys Glass Carafe Stainless Steel Carafe Plastic Carafe

35. Conclusions • Approach capable of extracting preferential probabilities • Preferential probabilities extracted from transcripts changed over the course of the design process • In this work, preference-related information extracted from the transcripts was consistent over time with those from surveys

36. Future work

37. Sketches Prototypes Text Integrated view of design activities • Design thinking manifests itself in different forms at different points of the design process • What are these forms? • How do they collectively evolve over time? • What is their relationship to outcome? Prototypes Sketches Text Time

38. System model Complex interactions among subsystems Thermal-hydraulic subsystem Structures subsystem Controls subsystem Other subsystems… System modeling • Formulate better system level models to improve system design and reliability • Consider emergent properties: nonlinear, complex interactions between subsystems • Draws on existing subsystem models and empirical system data • Allows prediction of future states, balancing of design trade-offs

39. Language of RiskPreference, Choice and , Uncertainty & Preference Mathematical Models Engineering Design Validation Modeling the language of design • Understand how designers express preference in natural language • Linguistically and mathematically model preference as expressed in engineering design texts • Advance basic knowledge of the “language of design” • Challenge: Model uncertainties in preference and convert into mathematical models applied to formal design decision-making • Recommended for NSF Award

40. Teaching • ESD.40 Product Design & Development • 2.009 Product Engineering Processes • IAP 2.97 Design-A-Palooza (new, mostly ugrad) • Focus on defining problems

41. Acknowledgments • Thoughtful support of MIT Engineering Systems Division and Department of Mechanical Engineering • 2006 NSF CAREER Award DMI-0547629 • NASA Cooperative Agreement NNA04CL15A