Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download class3
Document related concepts
Transcript
January 20 Topics (Design and Evaluation Part 2) Administrivia next week’s readings on web (focus: technology); name tag usability; reading report feedback Usability Study Guidelines Contextual Design discussion Guest speaker: Domenick Dellino, “What Usability Can Borrow from Anthropological Methos” More paper discussion: Ethnography Heuristic Evaluation; Discount Usability Engineering McGrath - Methodology Project Pitches or Group meetings CSE 595, Winter 2000 1 Quote of the Week “You can observe a lot just by watching.” -- Yogi Berra CSE 595, Winter 2000 2 Another Quote of the Week “Le mieux est l’ennemi du bien” -- Voltaire CSE 595, Winter 2000 3 Guidelines for User Testing with Thinking Aloud Practical study design Written materials Carrying out the study Improving the study (this material is also linked from the class web page) CSE 595, Winter 2000 4 Practical Study Design Reflect on the participants' backgrounds and how they might affect the study Be aware of problems that arise when experimenters know the users personally Prepare for the study carefully (avoid last minute panic) Select the tasks carefully to be representative and to fit the allotted time In general, start with an easier (but not frivolous) task Write down features of the system that are not being tested as well as those that are! CSE 595, Winter 2000 5 Practical Study Design (2) Define the start-up state for the study precisely Define precise rules for when and how users can be helped during the study Plan the timing and cut-off procedure (if subject gets stuck) for each part of the study Include reasonable provisions for data collection (e.g., notes, tape or video recorder, keystroke capture where appropriate) Plan data analysis techniques in advance Carry out a pilot study (important but often overlooked) CSE 595, Winter 2000 6 Written materials Participant release form (if needed) Questionnaire covering prior experience etc. (if relevant) Introduction to the study for users, including scenario of use Checklist for experimenters Evaluation survey (if relevant) CSE 595, Winter 2000 7 Carrying out the study Let users know that complete anonymity will be preserved Let them know that they may quit at any time Stress that the system is being tested, not the participant Indicate that you are only interested in their thoughts relevant to the system Demonstrate the thinking-aloud method by acting it out for a simple task, such as figuring out how to load a stapler, and a computerrelated task Hand out instructions for each part of the study individually, not all at once Maintain a relaxed environment free of interruptions Encourage users to keep talking using unobtrusive comments that don't point the user in a particular direction. Debrief each user after the experiment CSE 595, Winter 2000 8 Improving the study The pilot study should "debug" the study. This minimizes changes during the study, allowing quantitative data analysis. But improvements may be warranted. Experimenters' role can be improved Tasks given to participant can be improved Written materials can be improved CSE 595, Winter 2000 9 Notes on “Contextual Design” Widely praised in industry The class rated it highly (average 5.1) An eclectic customer-centered design methodology — includes ideas from ethnography, participatory design, much empirical work, etc. Data gathered from customers is the basis for making design decisions Team and organizational considerations explicitly dealt with CSE 595, Winter 2000 10 Principles of Contextual Inquiry The initial data gathering stage Principles: Context: summary vs ongoing experience; abstract vs concrete data Partnership: master/apprentice model (avoid interviewer/interviewee, expert/novice, guest/host) Interpretation : check interpretations on the spot (“but won’t it bias the data?”) Focus: interviewer’s point of view while studying the work CSE 595, Winter 2000 11 Stages in Contextual Design Methodology contextual inquiry interpretation sessions work models affinity diagram work model consolidation vision storyboards user environment design paper prototyping CSE 595, Winter 2000 12 Notes on Ethnography Dictionary definition: “a branch of anthropology dealing with the scientific description of individual cultures” Many ethnographers reside in the field a year or more, learning the local language, and participating in everyday life, while maintaining some objective detachment (participant observation). CSE 595, Winter 2000 13 Ethnography — Issues identification with culture being studied informants cultural change resulting from ethnographer’s presence technology: field notes, video, audio recordings CSE 595, Winter 2000 14 Relevance for Design Designers create artifacts for work settings need to understand those settings. Technology shapes practice designer's world view should not be imposed inappropriately on the users. Allows us to gain broader understanding of technology in use (see traditional methods) Joint exploration of technology and work allows both users and designers to participate in new designs. CSE 595, Winter 2000 15 How Usability Borrows from Anthropological Methods Usability Research: Anthropology of the Workplace Domenick J. Dellino Overview Traditional anthropological methods Verbal methods Non-verbal methods Anthropological methods in usability Examples and comparisons Confounds of each method CSE 595, Winter 2000 17 Traditional Anthropological Methods — Verbal Participant observation Key-informant interviewing Collection of life histories Structured interviews and surveys Questionnaires Rankings and Ratings CSE 595, Winter 2000 18 Traditional Anthropological Methods—Non-verbal Proxemics Kinesics Videotape Research* Content Analysis* Myths Folktales Erosion and Accretion CSE 595, Winter 2000 19 Participant Observation Trad. Anthropology Anthropologist lives with the tribe as a member of the community until barely noticed by the natives Observes and records the culture while participating as a native CSE 595, Winter 2000 Usability Research Contextual Inquiry: Usability Specialist infiltrates school computer lab as a computer nerd needing remedial training Observes student’s questions/behavior 20 Key Informant Interviewing Trad. Anthropology Anthropologist finds one native who is willing to tell everything Informant enjoys special treatment (e.g.: Hershey bars) CSE 595, Winter 2000 Usability Research Participants volunteer to be recruited for usability studies Participants are rewarded (e.g.:software) 21 Collection of Life Histories Trad. Anthropology Anthropologist develops rapport with a few individuals Collects extensive materials about these individuals CSE 595, Winter 2000 Usability Research Heuristic Evaluation: Usability specialist finds other specialists who understand interface design Asks them to report all “usability issues” they can find 22 Structured Interviews & Surveys Trad. Anthropology Fieldworker administers formal interview about lifestyle, earnings, expenditures, etc. by going from house to house. CSE 595, Winter 2000 Usability Research Specialist asks specific questions in the workplace (or lab) about users’ work processes, policies, and practices Contextual Interview 23 Questionnaires Trad. Anthropology Researcher hands out questionnaires to an informant and returns to collect the answers E.g.: Informants are asked to record what they purchase during the week CSE 595, Winter 2000 Usability Research Specialist distributes questionnaires to a group in a lab setting or e-mails survey Recipients are asked attitudinal and behavioral questions. 24 Ranking and Ratings Trad. Anthropology Commonly used to determine the hierarchical structure of caste, class, or kinship systems CSE 595, Winter 2000 Usability Research Cluster Analysis Participants are asked to sort cards of menu commands into stacks 25 Proxemics Trad. Anthropology Unobtrusive observations of how close people stand to each other CSE 595, Winter 2000 Usability Research Eye tracker software Tells us where the user is looking on the screen 26 Kinesics Trad. Anthropology Lurking Watching what people do Observing the people one touches, grooms, hits, holds CSE 595, Winter 2000 Usability Research Counts of clicks and keystrokes Counts of “mistakes” Recording of success rate 27 Erosion & Accretion Trad. Anthropology Archaeology Study of paths Studies of middens, fireplaces, burial grounds, & tool making sites CSE 595, Winter 2000 Usability Research Use of “instrumented versions” Investigation of query & error logs, Web cache, trash The “Lame” button F1 key wear 28 Confounds of Each Method-1 Participant Observation Heisenberg Uncertainty Principle Key Informant Interviewing/Life Histories Sampling—Informants may not be “representative” Structured Interviews & Surveys “Performance anxiety” Memory may be flawed, suspicion CSE 595, Winter 2000 29 Confounds of Each Method-2 Questionnaires Literacy, focus, desire to show favor Ranking and Ratings Task requires sophisticated thought Data analysis is aggregate (non-parametric) Validity of interpretation (are we measuring what we think we’re measuring?) CSE 595, Winter 2000 30 Confounds of Each Method-3 Proxemics Ethnocentric/design-centric interpretation Kinesics Question of learning style: “I like to try everything first.” Erosion & Accretion “Whose data is it, anyway?” CSE 595, Winter 2000 31 …for Reflection What value does ethnography add? When should it be conducted? Is it cost effective? CSE 595, Winter 2000 32 Ethnography — Ethics anthropologists must be open about the purpose, potential impacts, and source of support primary responsibility to people and animals with whom researchers work responsibility to scholarship and science responsibility to the public American Anthropological Association link: http://www.ameranthassn.org/committees/ethics/ethics.htm CSE 595, Winter 2000 33 Notes on J. Blomberg et al, “Ethnographic Field Methods” Rating: 4.3 Why Observe? Why not just ask? ideal vs. manifest behavior "What people say and do are not the same thing." People have a great deal of tacit knowledge that they cannot verbalize. Observational role: Unobtrusive Observer (aka Observer Participant). This is difficult or inappropriate in many settings. Participant Observer +access, +first-hand experience, +-point-of-view issues, -logistical problems of recording CSE 595, Winter 2000 34 Blomberg (2) In reality, the ethnographer doesn't assume either of the above extremes, but moves back and forth along the continuum between pure observer and pure participant. CSE 595, Winter 2000 35 Blomberg (3) What do we focus on? Event focus - meetings, seminars, ceremonies. Person focus - "a day/week/year in the life of..." Place focus - receptionist's desk, printer room, etc. Object focus - life history of a document, transaction, etc. How do you know you are finished? When you're no longer suprised by what you're seeing! CSE 595, Winter 2000 36 Blomberg (4) — Techniques Notes: + inexpensive, interpretive, flexible (sketches, etc), holistic - interpretive, can be hard to reconstruct, "low bandwidth," lots of work for observer Audiotape: + inexpensive, "medium bandwidth," less work for observer - limited slice of activity (audible/verbal only), requires some audible actions to be useful, difficult to reconstruct CSE 595, Winter 2000 37 Blomberg — More Techniques Videotape: + "high bandwidth," can make detailed reconstructions (content log), advantages of audio, less work for observer, unobstrusive, somewhat holistic - limited slice of activity, expensive, building content logs can be exhausting, coordinating multiple videos difficult, activity space/lighting might not be suitable for filming Event Logs: (computer capture of input events) + "high bandwidth," can make detailed reconstructions, can get good statistics about failures/difficulties, time sensitive - computer only, emphasizes human-machine dyad view – human is viewed as mere source of input events anti-holistic CSE 595, Winter 2000 38 Linking ethnography and design Ethnographer reports findings to designers Ethnographic study is undertaken by team of ethnographers and designers Full participation in designing by ethnographers, designers, and users Issues: Whose "side" is the ethnographer on? Who "owns" the result (knowledge) of ethnographic research? CSE 595, Winter 2000 39 Notes on Jacob Nielsen, “Guerilla HCI” average rating: 5.1 Techniques: scenarios simplified thinking aloud heuristic evaluation CSE 595, Winter 2000 40 Notes on Jacob Nielsen, “Heuristic Evaluation” average rating: 4.3 Heuristic evaluation: have a small set of evaluators examine an interface and judge its compliance with recognized usability principles (the “heuristics”). Evaluators work independently. (Can compare results afterwards.) Output: a list of usability problems with reference to the heuristics. Comment: doesn’t require working with eventual real users. CSE 595, Winter 2000 41 # of Evaluators vs Problems CSE 595, Winter 2000 42 Cost-Benefit Ratio CSE 595, Winter 2000 43 Ten Usability Heuristics (from Nielsen) Visibility of system status Match between system and the real world User control and freedom Consistency and standards Error prevention Recognition rather than recall Flexibility and efficiency of use Aesthetic and minimalist design Help users recognize, diagnose, and recover from errors Help and documentation CSE 595, Winter 2000 44 Usability Heuristics Applied to the Web Highly recommended: Keith Instone’s discussion of applying these to the web (http://webreview.com/97/10/10/usability/sidebar. html) CSE 595, Winter 2000 45 Notes on Joseph McGrath, “Methodology Matters” Ratings: binary (five 1’s, three 6’s). Average 3.5 I’ve received very positive comments on this paper from social scientists science vs. design and engineering; discount usability engineering CSE 595, Winter 2000 46 McGrath: 3 domains content that is of interest (e.g. grad students using Powerpoint to prepare a job talk). This is the substantive domain. ideas that give meaning to the content (e.g. "able to prepare slides, frustrated" — i.e. task-related descriptions, affect-related descriptions, etc). This is the conceptual domain. techniques or procedures for studying the ideas and content (ethnography, laboratory experiment, field study, heuristic evaluation). This is the methodological domain. CSE 595, Winter 2000 47 McGrath: techniques Techniques for manipulating features of the research situation: giving instruction Imposing constraints Selecting materials Giving feedback Using experimental confederates CSE 595, Winter 2000 48 McGrath: desirable features of a research strategy generalizibility precision realism We can't maximize all of these at the same time! We can have multiple studies, and ask "does study A support study B?" CSE 595, Winter 2000 49 McGrath: Some key concepts Baserates Correlations Causality CSE 595, Winter 2000 50
