Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Evaluating Visualizations cs5764: Information Visualization Chris North Evaluating Visualizations • Usability Test • Observation, problem identification • Controlled Experiment • Formal controlled scientific experiment • Comparisons, statistical analysis • Expert Review • Examination by visualization expert • Heuristic Evaluation • Principles, Guidelines • Algorithmic Projects • Implementation projects: • Small usability test of implementation • Short usability report • Experiment projects: • Main controlled experiment • Experiment materials and raw data • Then data analysis Usability test vs. Controlled Expm. • Usability test: • • • • • Formative: helps guide design Single UI, early in design process Few users Usability problems, incidents Qualitative feedback from users • Controlled experiment: • • • • • Summative: measure final result Compare multiple UIs Many users, strict protocol Independent & dependent variables Quantitative results, statistical significance Controlled Experiments What is Science? • Measurement • Modeling Scientific Method 1. 2. 3. 4. Form Hypothesis Collect data Analyze Accept/reject hypothesis • How to “prove” a hypothesis in science? • • • • Easier to disprove things, by counterexample Null hypothesis = opposite of hypothesis Disprove null hypothesis Hence, hypothesis is proved Empirical Experiment • Typical question: • Which visualization is better in which situations? Spotfire vs. TableLens Cause and Effect • Goal: determine “cause and effect” • Cause = visualization tool (Spotfire vs. TableLens) • Effect = user performance time on task T • Procedure: • Vary cause • Measure effect • Problem: random variation random variation Real world Collected data uncertain conclusions • Cause = vis tool OR random variation? Stats to the Rescue • Goal: • Measured effect unlikely to result by random variation • Hypothesis: • Cause = visualization tool (e.g. Spotfire ≠ TableLens) • Null hypothesis: • Visualization tool has no effect (e.g. Spotfire = TableLens) • Hence: Cause = random variation • Stats: • If null hypothesis true, then measured effect occurs with probability < 5% • But measured effect did occur! (e.g. measured effect >> random variation) • Hence: • Null hypothesis unlikely to be true • Hence, hypothesis likely to be true Variables • Independent Variables (what you vary), and treatments (the variable values): • Visualization tool » Spotfire, TableLens, Excel • Task type » Find, count, pattern, compare • Data size (# of items) » 100, 1000, 1000000 • Dependent Variables (what you measure) • • • • User performance time Errors Subjective satisfaction (survey) HCI metrics Example: 2 x 3 design Ind Var 2: Task Type Task1 Task2 Task3 SpotInd Var 1: fire Vis. Tool TableLens • n users per cell Measured user performance times (dep var) Groups • “Between subjects” variable • • • • 1 group of users for each variable treatment Group 1: 20 users, Spotfire Group 2: 20 users, TableLens Total: 40 users, 20 per cell • “With-in subjects” (repeated) variable • • • • • All users perform all treatments Counter-balancing order effect Group 1: 20 users, Spotfire then TableLens Group 2: 20 users, TableLens then Spotfire Total: 40 users, 40 per cell Issues • Eliminate or measure extraneous factors • Randomized • Fairness • Identical procedures, … • Bias • User privacy, data security • IRB (internal review board) Procedure • For each user: • Sign legal forms • Pre-Survey: demographics • Instructions » Do not reveal true purpose of experiment • Training runs • Actual runs » Give task, measure performance • Post-Survey: subjective measures • * n users Data • Measured dependent variables • Spreadsheet: User Spotfire TableLens task task task task task task 1 2 3 1 2 3 Step 1: Visualize it • • • • Dig out interesting facts Qualitative conclusions Guide stats Guide future experiments Step 2: Stats Ind Var 2: Task Type SpotInd Var 1: fire Vis. Tool TableLens Task1 Task2 Task3 37.2 54.5 103.7 29.8 53.2 145.4 Average user performance times (dep var) TableLens better than Spotfire? Avg Perf time (secs) Spotfire TableLens • Problem with Averages: lossy • Compares only 2 numbers • What about the 40 data values? (Show me the data!) The real picture Avg Perf time (secs) Spotfire TableLens • Need stats that compare all data Statistics • t-test • Compares 1 dep var on 2 treatments of 1 ind var • ANOVA: Analysis of Variance • Compares 1 dep var on n treatments of m ind vars • Result: • p = probability that difference between treatments is random (null hypothesis) • “statistical significance” level • typical cut-off: p < 0.05 • Hypothesis confidence = 1 - p Excel p < 0.05 • • • • Woohoo! Found a “statistically significant” difference Averages determine which is ‘better’ Conclusion: • • • • • • Cause = visualization tool (e.g. Spotfire ≠ TableLens) Vis Tool has an effect on user performance for task T … “95% confident that TableLens better than Spotfire …” NOT “TableLens beats Spotfire 95% of time” 5% chance of being wrong! Be careful about generalizing p > 0.05 • Hence, no difference? • Vis Tool has no effect on user performance for task T…? • Spotfire = TableLens ? • NOT! • • • • Did not detect a difference, but could still be different Potential real effect did not overcome random variation Provides evidence for Spotfire = TableLens, but not proof Boring, basically found nothing • How? • Not enough users • Need better tasks, data, … Data Mountain • Robertson, “Data Mountain” • (Microsoft) Comparison of Info Vis Systems • Kobsa Cleveland’s Rules for Secondary Tasks • Chewar et al. Usability Testing Usability test vs. Controlled Expm. • Usability test: • • • • • Formative: helps guide design Single UI, early in design process Few users Usability problems, incidents Qualitative feedback from users • Controlled experiment: • • • • • Summative: measure final result Compare multiple UIs Many users, strict protocol Independent & dependent variables Quantitative results, statistical significance Usability Specification Table Scenario task Worst case Planned Target Best case (expert) Observed Find most expensive house for sale? 1 min. 3 sec. ??? sec … 10 sec. Usability Test Setup • Set of benchmark tasks • Easy to hard, specific to open-ended • Coverage of different UI features • E.g. “find the 5 most expensive houses for sale” • Consent forms • Not needed unless video-taping user’s face (new rule) • Experimenters: • Facilitator: instructs user • Observers: take notes, collect data, video tape screen • Executor: run the prototype if faked • Users • 3-5 users, quality not quantity Usability Test Procedure • Goal: mimic real life • Do not cheat by showing them how to use the UI! • Initial instructions • “We are evaluating the system, not you.” • Repeat: • • • • Give user a task Ask user to “think aloud” Observe, note mistakes and problems Avoid interfering, hint only if completely stuck • Interview • Verbal feedback • Questionnaire • ~1 hour / user Usability Lab • E.g McBryde 102 Data • Note taking • E.g. “&%$#@ user keeps clicking on the wrong button…” • Verbal protocol: think aloud • E.g. user expects that button to do something else… • Rough quantitative measures • HCI metrics: e.g. task completion time, .. • Interview feedback and surveys • Video-tape screen & mouse • Eye tracking, biometrics? Analyze • Initial reaction: • “stupid user!”, “that’s developer X’s fault!”, “this sucks” • Mature reaction: • “how can we redesign UI to solve that usability problem?” • the user is always right • Identify usability problems • Learning issues: e.g. can’t figure out or didn’t notice feature • Performance issues: e.g. arduous, tiring to solve tasks • Subjective issues: e.g. annoying, ugly • Problem severity: critical vs. minor Cost-Importance Analysis Problem Importance Solutions Cost Ratio I/C • Importance 1-5: (task effect, frequency) • 5 = critical, major impact on user, frequent occurance • 3 = user can complete task, but with difficulty • 1 = minor problem, small speed bump, infrequent • Ratio = importance / cost • Sort by this • 3 categories: Must fix, next version, ignored Refine UI • Simple solutions vs. major redesigns • Solve problems in order of: importance/cost • Example: • Problem: user didn’t know he could zoom in to see more… • Potential solutions: – – – – Better zoom button icon, tooltip Add a zoom bar slider (like moosburg) Icons for different zoom levels: boundaries, roads, buildings NOT: more more “help” documentation!!! You can do better. • Iterate • Test, refine, test, refine, test, refine, … • Until? Meets usability specification Project revisited • For implementation projects: • Informal test • A few users – Not (tainted) info vis students • 102 lab not required • Simple data collection – Biometrics optional! • 1 iteration • Exploit this opportunity to improve your design • For experiment projects: • See controlled experiments