Designing Instructions

1. Designing Effective Step-By-Step Assembly InstructionsManeesh Agrawala, Doantam Phan, Julie Heiser, John Haymaker, Jeff Klingner, Pat Hanrahan and Barbara TverskyMicrosoft Research and Stanford University

3. 1 2 3 4 5 6

4. Designing Instructions • Planning • Choose sequence of assembly operations • Robotics / AI / Mechanical Engineering[Wolter 89], [de Mello 91], [Wilson 92], [Romney 95] • Presentation • Visually convey assembly operations • Visualization / Computer Graphics[Seligmann 91], [Rist 94], [Butz 97], [Strothotte 98] We jointly optimize plan and presentation

5. B A A A B B A C B A C A C A C B B B Geometric Analysis [Romney 95] B blocked by A both parts free to move A blocked by B Input Parts Blocking Graph

6. Valid Valid Invalid Geometric Assembly Planning

7. Valid Valid Valid Valid Valid How do we choose the best sequence? Many Geometrically Valid Sequences

8. Our Approach • Identify cognitive design principles • How people conceive of 3D assemblies • How people comprehend visual instructions • Encode principles as constraints within automated design system

9. Identifying Design Principles • Experiment 1: Assemble and draw instructions • Experiment 2: Rate effectiveness • Experiment 3: Validate effectiveness

10. Sequence the Assembly Operations • Step-by-step instructions preferable Single exploded view diagram Step-by-step diagrams

11. Illustrate the Assembly Operations • Action diagrams preferable Structural diagrams Action diagrams

12. Ensure Visibility of Parts • Essential • Parts being attached • Context (show earlier parts) • Less important • All parts in a symmetric group • Repetitive operations

13. Automated Instruction Design • Step-by-Step • Action diagrams • Good visibility TV stand instructions generated by our system

14. Input • Geometry • Parts in assembled positions • Orientations • Default viewpoint / orientation • Preferred orientation for each part • Groupings • fasteners, significant parts, symmetry,similar-actions • Ordering constraints • Force one part to attach before another required optional

15. TV Stand Input • Geometry: Parts in assembled positions • Orientations: Default viewpoint / orientation • Groupings: fasteners, significant parts, symmetry required optional

16. Structural Diagrams Action Diagrams All parts Search Leftover parts Best subset of parts Sequence Parts Reorientation Sequence of assembly steps

17. All parts Search Leftover parts Best subset of parts Sequence Parts Reorientation Sequence of assembly steps

18. All parts Part subsets Sequence Parts Reorientation ...… Sequence of assembly steps All parts Search Leftover parts Best subset of parts

19. Sequence Parts Reorientation Sequence of assembly steps All parts Search Choose part subset Search Interference Leftover parts Best subset of parts Attachment Ordering Grouping Visibility

20. Computing Visibility Area(P) = # red pixels Area(P,Q) = # red pixels Area of top alone Area of top not occluded by sides Vis(P,Q) = Area(P,Q) / Area(P) % pixels that remain visible

21. Visibility Constraint • Current parts R min ( Vis(r, R-r) ) • Check that each part in current subset is visible • Attached parts A Vis(A, R) • Check that context is visible • Unattached parts U min ( Vis(u, R) ) • Check that future parts will be visible r Î R u Î U

22. Input model Lego Car

23. Bookcase Input model

24. All parts Search Leftover parts Reorientation Sequence of assembly steps • Add significant parts one by one • Visibility • Distance to viewer • Add all remaining parts • Omit repetitive operations • Skip if 2 similar-action parts already added Best subset of parts Sequence Parts

25. Bookcase – After Sequencing

26. Bookcase – Omitting Repetition

27. All parts Search Leftover parts Best subset of parts Sequence Parts Reorientation Sequence of assembly steps • Set preferred orientation for significant parts • If visibility of current parts is lowtry alternate oblique views

28. Bookcase – With Reorientation

29. All parts Search Leftover parts Best subset of parts Sequence Parts Reorientation Sequence of assembly steps

30. All parts Action Diagrams Search Choose Direction Leftover parts Best subset of parts Build Stacks Sequence Parts Place Guidelines Reorientation Sequence of assembly steps Sequence of assembly diagrams

31. 4th stackpart 3rd stackpart 2nd stackpart 1st stackpart Base Building Stacks • Stack: set of parts that • Share separation direction • Lie on a stabbing line • Are in sequential contact

32. 1st stack part 1st stack part 1st stackpart Base Base Base Building Stacks • Stack: set of parts that • Share separation direction • Lie on a stabbing line • Are in sequential contact

33. Placing Guidelines • Expand stacks • Form lines between stack parts

34. Placing Guidelines • Expand stacks • Form lines between stack parts

35. Bookcase • 9 parts Planning: 48s

36. Table • 13 parts Planning: 28s

37. Test Object • 25 parts Planning: 53s

38. Exploded View

39. Future Work • Experiment 4: Evaluation • Time/errors as they use our instructions • Assemblies with more subparts • Take advantage of hierarchy • Apply system recursively • Physical and functional constraints • Part size and mass • Gravitational stability

40. Summary • Cognitive design principles • Step-by-step • Action diagrams • Good visibility • Automated instruction design system • Integrate planning and presentation

41. Acknowledgements • Boris Yamrom • Christina Vincent • ONR grantsN000140210534, N000140110717 and N000140010649