Design and Evaluation

Design and Evaluation Human Factors Psychology Dr. Steve

Human Factors Design Activities • Design product/system interfaces • Modify designs/correct design flaws • Safety activities (warnings, analyses) • Develop training programs/materials

Cost/Benefit Analysis • Costs • HF personnel salaries • Extended time to product release • Analysis tools (software, video) • Prototyping • Increased development cost • Benefits • More effective product • Safer product • Cheaper to change design early in development • Higher sales • Fewer liability suits What other costs and benefits of HF activity in product/system design can you think of?

System Development and HF Activities • Stage 1. Front-End Analysis* • user analysis • function analysis • preliminary task analysis • environmental analysis • system specifications • Stage 2. Conceptual Design* • function allocation • support conceptual design • Stage 3. Iterative Design and Testing* • task analysis • interface design & prototyping • heuristic evaluation • cost/benefit analysis of alternatives • workload analysis (simulation & modeling) • safety analysis • usability testing • Stage 4. Design of Support Materials • develop manuals • Stage 5. System Production • Stage 6. Implementation and Evaluation • experiments to evaluate system • Stage 7. System Operation/Maintenance • monitor system performance • Stage 8. System Disposal * Topics covered in class

User-Centered Design • Early focus on the user and tasks • Empirical measurements • questionnaires, usability studies, data collection • Iterative design • prototyping of interface • Participatory design • user input to design, preferences, SLUGs

Sources for Design Data • Data Compendiums • Engineering Data Compendium: Human Perception and Performance • Human Factors Design Standards • MIL-STD-1472D (military anthropometry, ergonomic specs) • ANSI/HFES-100 (VDT specs) • ANSI/HFES-200 (software specs) • OSHA standards • General HF Principles/Guidelines • Textbooks (by topic: interface design, aviation, software) • Research Journals (Human Factors, Ergonomics, Applied Ergonomics) • Handbooks (Handbook of Human Factors) • Conferences (CHI, HFES, IEA) • Workshops (U of Mich)

Front-End Analysis • Answers the following questions: • Who are the users? • Consumers, trainers, maintainers, etc. • What functions are performed by system? • Allocate functions to machine or person • What are the environmental conditions where system will be used? • Noise, temperature, pressures, etc. • What are users’ preferences/requirements? • Color, functions, shape, etc. • What are constraints to design of the system? • Space, regulations, costs, resources • What are the HF criteria for design solutions? • Shorter training time, less accidents, higher performance

Front-End AnalysisUser Analysis • Who are the intended users? • Age • Experience level • Intellectual level • Anthropometrics (size & shape) • Strength • Culture & Language • Disabilities (sight, hearing, mobility) Why does the drive- thru ATM have Braille on its controls? What are the characteristics of DVD player consumers?

Front-End AnalysisFunction/Preliminary Task Analysis Function Analysis: What are the basic functions of the system? Example: The function of a wheel barrow is to move objects to the work site. Preliminary Task Analysis: What are the jobs, duties, tasks, and actions that user will be performing? Example: Tasks to be performed with a wheel barrow include loading it with dirt/building materials, transporting, mixing, and dumping materials, cleaning, maintaining, and storing wheelbarrow.

Front-End AnalysisPreliminary Task Analysis • Preliminary Task Analysis Methods: • Interviews • Focus Groups • Observation • Questionnaires Used prior to design to learn how intended users perform their jobs. What method(s) would you use if you were designing a new library cataloguing system?

Front-End AnalysisEnvironment Analysis • Where will the tasks be completed and under what conditions? • Indoors/outdoors • Temperature/humidity • Lighting levels • Noise levels • Vibration • Gravity • Special clothing/gear requirements • How would the design of a radio transmitter differ if the intended users were researchers in Antarctica? • Would a standard vending machine work on the space station? If not, how would you design a vending machine for that purpose?

Front-End AnalysisUser Preferences & Req’s • When different design options are available, must determine the intended users preferences and needs (perhaps determined through focus groups). • User preference issues include: • What components are automated • Color features/aesthetics • Balance between performance and preference • What makes other products popular/unpopular • What can be made adjustable and what cannot • Feasibility of user’s demands

Fitt’s List (1951) • Things Humans Do Well • Ability to: • detect small amounts of visual or acoustic energy • perceive patterns of light or sound • improvise and use flexible procedures • store very large amounts of information for long periods and to recall relevant facts at the appropriate time • reason inductively • exercise judgment • Things Machines Do Well • Ability to: • respond quickly to control signals, and to apply great force smoothly and precisely • perform repetitive, routine tasks • store information briefly and then to erase it completely • reason deductively, including computational ability • handle highly complex operations, I.e., to do many different things at once Conceptual Design ActivitiesFunction Allocation Function Allocation: the assigning of tasks to humans or machine taking into account human vs. machine capabilities and limitations Is Fitt’s list still appropriate 50+ years later?

Example: Programming your VCR to record a show Thursday night at 7:30 on Fox Network Determine When show airs “use up & down arrows to indicate selection” Access input menu Press “Program” Hit arrow to indicate day Press “Enter” button VCR accepts Input, moves to next field Hit arrow to indicate time Press “Enter” button VCR accepts Input, moves to next field Yes or No “Program complete Select another program or press End to finish” No Hit arrow to indicate channel Press “Enter” button VCR accepts Input Press “End” Yes (See homework assignment) Conceptual Design ActivitiesDecision/Action & Functional Flow Analyses Decision/Action and Functional Flow Analyses: Procedures for identifying and depicting the sequence of functions, actions, and decisions made by the system (user included as part of system).

Conceptual Design ActivitiesTypical Design Meeting Brainstorming Story-Boarding

Iterative Design and Testing

Iterative Design and TestingTask Analysis (detailed) • Purpose of Detailed Task Analysis is to identify: • User goals and activities • Tasks required to achieve goals • Conditions under which task is performed • Expected outcomes of tasks • KSA’s needed to perform tasks • Equipment needed to perform tasks • Complex decision making, problem solving, diagnoses* • Complex rule structures dependent on situation* * Part of Cognitive Task Analysis

Task Analysis MethodsInterviews • Interviews: Questioning subject matter experts to get them to explain what they do on the job. • Tries to answer questions such as: • How do you perform the task? • Why do you perform the task? • Under what conditions do you perform the task? • What happens before and after performing the task? • What are the consequences of performing or not performing the task? • Advantage: simple to use • Disadvantage: SMEs may not be good at describing tasks

Task Analysis MethodsObservation • Observation: Watch and take notes of people performing the task • Suggestions: • Be unobtrusive • Use video if possible and with approval • Review observations with SMEs • Advantages: See what workers do, not what they say they do. • Disadvantages: Does not capture cognitive processes.

Think, think, think! Task Analysis MethodsThink-Aloud Verbal Protocol • Think-Aloud Verbal Protocol: Worker state what they are thinking as they perform the task • Concurrent – verbalize as performing the task • Retrospective – verbalize what they were doing as they view a videotape of themselves • Prospective – verbalize as they imagine performing a hypothetical task Advantage: Can get info regarding thought processes and decisions of experts performing the task. Disadvantage: Doing concurrently can interfere with the task, and doing retrospectively relies too heavily on memory

Example: Table 3.5 from Wickens, Gordon, & Liu Hierarchical task analysis for using a lawnmower Step 1. Examine Lawn a. Make sure grass is dry b. Look for any objects laying in the grass Step 2. Inspect Lawnmower a. Check components for tightness 1. Make sure grass bag handle is securely fastened to grass bag support 2. Make sure grass bag connector is securely fastened to bag adapter 3. Make sure the deck cover is in place 4. Check for any loose parts (such as oil cap) and blade Task Analysis MethodsHierarchical Task Analysis Hierarchical Task Analysis: Graphical (chart or diagram) depiction of task analysis Advantage: Allows to conceptualize task sequences Disadvantage: May not accurately represent true parallel nature of task

Link Analysis of CIC of USS Louisville Iterative Design and TestingLink Analysis Link Analysis: Shows the relationships between components of a system (e.g., communication, control, movements) • For determining: • layout of related workstations • placement of controls • communication patterns • frequency of equipment use Advantage: Graphic depiction of relationships among system features Disadvantage: Shows frequency of use, not importance, and may vary by situation

Diagram for a collision avoidance system Iterative Design and TestingFlow Diagrams Operational Sequence Diagram: Graphically depicts the sequence and timing of activity, and shows the interactions between individuals and equipment Advantage: Connects actions to time and component Disadvantage: Does not lend itself easily to interface design

Iterative Design and TestingInterface Design Principles • Provide a good conceptual model • Make things visible • Use natural mappings • Provide feedback

Iterative Design and TestingInterface Design Principles Provide a good conceptual model: when things are designed to be consistent with our mental models of how they should work (expectancies) then it is easier to predict what will happen when we use them. In this metronome, the tempos are arranged around a circle with a knob and arrow in the middle. Most people would expect to twist the knob until the arrow points to the desired tempo. Actually the arrow is a button that you push if you want the tempo to increase and the knob is not a knob at all.

Iterative Design and TestingInterface Design Principles Make things visible: It is easier to operate a system when the features are clearly visible Starting this gas pump is more difficult than it sounds because the start button is hidden among the other sea of information and does not “pop out” at you as a start button either figuratively or literally. Additionally, the use of red usually signals something other than start.

Iterative Design and TestingInterface Design Principles Use natural mappings: Actions should correspond to the system’s state. To make something go up, the action should be up, to make it go right, the action should be to the right This stovetop makes good use of natural mappings. The burners are arranged in the same pattern as the controls.

Iterative Design and TestingInterface Design Principles Provide feedback: Feedback is important because it allows one to see whether the intended actions have taken place. (like hourglass icon on computer software) In this example the feedback is confusing. The light on the bottom signifies that the smaller quantity of coffee was requested. The light does not go on when the larger quantity is requested. (read “Set Phasers to Stun” in Casey book for another example).

Iterative Design and TestingInterface Design Principles • Other Design Principles (Norman, 1992): • Simplify the structure of tasks • Make alternative actions and knowledge of results visible • Make it easy to determine what actions are possible • Make it easy to evaluate current system state • Exploit the power of constraints (remove possibility of error) • Design for easy error recovery (undo) • When all else fails, standardize.

Example of Rapid Prototyping Software used to model different phone interfaces Iterative Design and TestingPrototyping Prototypes or Mockups: Used to test features of the interface prior to production - may be anything from crude cardboard cut-outs to realistic models

Iterative Design and TestingEvaluative Studies • Types of evaluative studies: • Cost/Benefit Analysis for Design Alternatives • Trade-Off Analyses • Safety Analysis • Workload Analysis • Simulation & Modeling

Cognitive Modeling Biomechanical Modeling Mathematical Modeling Evaluative Studies Simulation & Modeling • Modeling programs: • MicroSAINT • Mannequin • COMBIMAN • HUMANCAD

Evaluative Studies Usability Testing • Usability Testing: • Ease of use (user-friendly) • May test for: • Learnability • Efficiency • Memorability • Errors • Satisfaction

Final Test and EvaluationField Studies • Design:between vs. within subjects • Test Participants:user population • Proximal Measures: • satisfaction • usability • performance • errors • Distal Measures: • manufacturing costs • personnel costs • number of accidents/disability claims

Alternatives to HF Design and Development Training To use system capabilities Methods to make task safer Modifying Current Equipment Equipment Replacement Ergonomic Aids Wrist pads Foot stools Anti-fatigue mats Job Rotation (and other admin approaches) Vary tasks Rest periods

Cow Factors in Design This is what happens when you fail to consider the user in the design

Design and Evaluation