Download MANY ROADS LEAD TO ROME. MAPPING USERS` PROBLEM

Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
MANY ROADS LEAD TO ROME.
MAPPING USERS’ PROBLEM SOLVING STRATEGIES
Mayr Eva1
Michael Smuc1
Hanna Risku2
1 Research Center KnowComm
Danube University Krems (Austria)
2 Department of Translation Studies
Unversity of Graz (Austria)
Contact information:
[email protected]
[email protected]
[email protected]
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
ABSTRACT
There is more than one path to a solution, especially when it comes to ill-defined problems like
complex, real-world tasks. Until now, the evaluation of information visualizations has often been
restricted simply to a measuring of outcomes (time and error) or insights into the data set. A
more detailed look into the processes which facilitate or hinder task completion is provided by
analyzing user problem solving strategies. The study presented in this paper illustrates how such
processes can be assessed and how the resulting knowledge can be used in participatory design to
improve a visual analytics tool. To equip users with tools which really do function as a problem
solving scaffold, these tools should allow them to choose their own path to the solution - their
own route to Rome. We also discuss how the evaluation of problem solving strategies can shed
more light on the “exploratory minds” of users.
KEYWORDS
problem solving strategies, participatory design, visual analytics, evaluation, methodology
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
1. Introduction
“The goal of visual analytics is to create software systems that will support the analytical
reasoning process”1. Following this rationale, we are currently engaged in a research project
which aims to support the daily work processes of business consultants and time scheduling
experts by means of novel visual analytics tools. To ensure that the tools successfully support
data exploration, prototypes are iteratively evaluated in real-world settings with real users and
refined based on the evaluation results.
A successful information visualization enables users to generate insights into the data and
supports exploratory data analysis. But evaluation techniques building on task completion time
and number of errors cannot unveil a tool’s quality and utility and, therefore, were criticized as
restricted in the past2. In more recent evaluations, researchers code and count the users’
insights3,4. This shift in thinking from outcome to process measures can be compared to the
cognitive revolution in psychology5: Instead of observing the outcomes of cognitive processes
only (like time and errors in information visualization evaluation), researchers analyzed the
cognitive processes themselves, that is, information processing, memory6, and problem solving7.
From this view, insights – in contrast to time and errors – illuminate parts of the “Black Box”
symbolizing the human mind (see figure 1).
[insert figure 1 here]
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
However, insights can only partially uncover the black box. Research on processes that lead to
insight generation or successful task completion might further unveil the black box of the
“exploratory mind”8. But they were studied to a lesser extent in information visualization until
now (for approaches to enlighten the black box see
9,10
). Therefore, we propose to look more
closely into these processes by relying on one of the research topics which mark the beginning of
the cognitive revolution: problem solving7.
It makes sense to focus on these processes, as in cognitive psychology problem solving processes
are closely linked to insights11. Therefore, we conducted an evaluation study to identify whether
problem solving processes are relevant during the exploration of information visualizations and
worth further research. Before we present this study, we shortly introduce the theoretical
background of problem solving and how it might connect to information visualizations.
2. Problem Solving
Simon and Newell were among the first who conducted research on problem solving7,12. They
differentiate between an objectively defined task and a subjectively defined problem. When users
proceed to fulfill externally given tasks, they are solving subjective problems. That is why the
problem space (the subjective representation of the problem) and also the paths to a task’s
solution differ between different users – depending on their experience, domain knowledge,
available information, and the tools at hand.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
2.1 Types of Problems
It is important to distinguish between different types of problems. “Research in situated and
everyday problem solving (e.g., Lave, 1988) makes clear distinctions between convergent
problem-solving thinking and the thinking required to solve everyday problems”13. Similarly in
cognitive psychology, two major types of problems are distinguished13: Well-defined problems
have one correct solution and provide all information needed to solve them. Typical visualization
tasks of locating (e.g. finding a date) or identifying (e.g. finding the maximum)14 can be
associated with this kind of problems. In contrast, ill-defined problems have more than one
solution (sometimes there even does not exist an optimal solution at all) and often include only
fragmentary information. This is typically the case in real-world contexts13. Exploratory data
analysis also only seldom converges in one single correct solution; therefore, it can be classified
as ill-defined.
In the context of data visualizations and visual analytics, a necessary prerequisite for successfully
solving problems is graph comprehension. Three levels of graph comprehension can be
differentiated15: reading the data (i.e. extracting data, locating), reading between the data (i.e.
finding relationships, integrating), and reading beyond the data (i.e. extrapolating from the data,
generating hypotheses). To solve well-defined problems an analyst has to reach level 1 and
sometimes level 2, whereas to solve ill-defined problems an analyst often has to reach all three
levels. Current task taxonomies for information visualizations or visual analytics16 are often
restricted to tasks on level 1 and 2 and do not convey the complex real-world problems usually
tackled on level 3 (see the taxonomies of Amar and Stasko17 and Valiati et al.14 which dealt with
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
higher level processing or real world tasks to some extent). Thus, most taxonomies do not
include higher levels of graph comprehension, and the role of ill-defined problem solving has not
been examined so far. This is problematic, as the three levels of graph comprehension and the
two types of problems do require different processes and strategies to be mastered successfully.
Therefore, an information visualization or visual analytics tool which is intended for real-world
use should support both kinds of problem solving and their evaluation should ensure that it
actually does so.
2.2 Problem Solving Strategies
Well-defined and ill-defined problems not only differ in the number of correct, respectively
plausible solutions and the required level of graph comprehension, but also in the processes
needed and the strategies applied to reach a solution. A problem solving strategy is “a technique
that may not guarantee solution, but serves as a guide in the problem solving process”18.
To solve well-defined problems, one has to know rules, strategies, and when to apply which. For
ill-defined problems one has to generate different solutions and evaluate them based on one’s
own knowledge and opinions18. Ill-defined problems can be solved in multiple ways, probably
leading to different solutions; this is a very creative process13. Therefore, it is difficult to predict
either solutions or strategies applied for such problems. People who are able to successfully solve
well-defined problems cannot necessarily solve ill-defined problems, too19.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
On a more detailed level, schema- and search-based problem solving strategies can be
distinguished18: A schema includes knowledge related to a problem type, including its goal,
constraints, applicability, and solution procedures; it can be domain-specific or general. Schemadriven problem solving is activated if certain features of a problem resemble those of a schema
stored in memory. It steers the process as an initial hypothesis of how to solve the problem;
however, the problem solving process itself unfolds and forms itself in a situated manner hereand-now, depending on the interaction process and the situational cues appearing during the
activity. If no appropriate schema is available, problem solvers engage in less direct and more
effortful search-based problem solving. There, they have to gather further information,
decompose the problem into subproblems (which might again allow the application of schemas
or not), use analogies, etc.
Depending on the type of problem and the situation, but also on the problem solver’s expertise
level, different problem solving strategies are applied: Schema-based problem solving strategies
depend on the availability of a appropriate schema, which is more likely for well-defined than for
ill-defined problems. But domain experts might also have schemata for ill-defined problems, as
they have a richer repertoire of problem solving strategies within their domain18. They have
familiarised themselves with relevant methods, defined or adapted these to suit their own activity
and combined them with their own experience20. In this way, experts create flexible models that
sensitise them to different types of problems and enable them to act in a more flexible, creative
way. They can more easily and more effectively search through the problem spaces and select
more appropriate schemas21 and deviate from prior strategies in order to take the specific case
into account.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
2.3 Scaffolding Problem Solving
The aim of visual analytics is to support the problem solving process1. From the view of situated,
embedded cognition, the visual representation of information serves as a scaffold22 for the
problem solving process. By pre-processing and visualizing information, visual analytics tools
reduce the need to process and store data in the analyst’s memory.
As discussed above, experts are often better able to solve problems, as they can faster and better
identify the type of problem at hand, have a bigger repertoire of problem solving strategies13,18,21
and act in a more flexible way than laypersons and beginners. A visual analytics tool should
consequently allow for multiple problem solving strategies and support the creative process of
solving ill-defined problems in real-world contexts and thus to really serve as a scaffold.
Let us exemplify our point with an example from everyday life: You want to tighten a screw, but
you do not have a screwdriver at hand. With good skills and strength, you might be able to
tighten it with a simple coin, a key, or a pocket knife. But if you are provided a Swiss army knife,
you will solve this problem more easily – regardless of the screw’s shape and condition.
From our point of view, it is important to ensure in the development process of visual analytics
tools that a visual analytics tool is such a flexible scaffold. This can be reached by evaluating
how many different problem solving strategies a tool supports and which strategies it impedes.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
3. Research Questions
During the participatory design process of two visual analytics tools, we observed that users
apply many different strategies to solve tasks when exploring information visualizations. There
was not only one single strategy that led to a correct solution; as many ways lead to Rome, users
reached a solution via different paths. Still, some of the strategies applied did not yield a
sufficient solution. Interestingly, the question of how problem solving strategies interact with the
characteristics of the tool or visualization and task completion was not addressed in prior
evaluations. This study aims to close this gap.
Our assumption is that by analyzing users’ problem solving strategies we can understand how a
visual analytics tool supports or impedes the problem solving process, how it can act as a
scaffold – better than by coding and counting insights alone. In addition, we can generate ideas
on how the tool should be improved to allow for multiple problem solving strategies. By looking
more deeply into the processes while working on a task, the evaluation produces results beyond
time and errors or number of insights.
More specifically, our aims are (1) to identify which additional knowledge about the analysis
processes can be gained by looking at problem solving processes in comparison to time and
errors or insights, (2) to examine the relationship between time, errors, insights, and problem
solving processes, and (3) to show how the results of our research can be applied during the
participatory design process to strengthen the scaffolding function of visual analytic tools.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
4. Method: Identifying Problem Solving Strategies
In a study within the research project DisCō we compared two different prototypes, the Groove23
and a Multiscale variant24. Whereas the Groove allows users to interactively fold and unfold time
scales and overlays the temporal granularities (see figure 2), the Multiscale (see figure 3) shows
all temporal granularities one below the other.
[insert figures 2 & 3 here]
Twelve experts (5 female, 7 male) with many years of theoretical and practical expertise in the
exploration and analysis of time-oriented data participated in the study. In addition to their
common expertise in data exploration, each of them had a specific domain expertise, for example
in the fields of business intelligence, management, or education. To deal with the mixture of
expertise in the sample, each participant worked with five datasets from different domains that
suited each of the experts’ domains at least once (see Smuc, Mayr, & Risku26 for more details on
this approach of domain-independent testing). Additionally, to decrease the difference between
the experts’ knowledge, we presented a short introduction about the datasets, similar to Whiting
et al.25. These “dataset stories” consisted of basic information (a short domain and data
description) and quirks and peculiarities in the data set (special events, anomalies and exceptions
as well as data acquisition errors).
To gain meaningful results from the evaluation of information visualizations, it was proposed
that users have to solve ecologically valid tasks during the study26. Therefore, we did not only
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
collect real-world data sets, but also associated real-world tasks of different complexity for each
data set. Our participants had to solve five to seven well- and ill-defined problems for each data
set with the aid of the two different prototypes (see table 1). The five data sets differed in the
amount of data points, in the time range, and in the presented temporal granularities (some were
only on a weekly base, some included a monthly base as well).
[insert table 1 about here]
As the problem solving process includes different cognitive13 and perceptual28 processes, we
used multiple measures to study the participants’ analysis processes. We logged their interaction
with the tool, tracked their viewing behaviour, and asked them to think aloud during the
experiment. The think-aloud-protocols were then enriched with interaction and viewing data to
make sense of deictic references. The resulting annotated protocol was segmented according to
the data sets and tasks. The resulting data were analysed on three levels: (1) task completion time
and errors, (2) insights, and (3) problem solving strategies. For the first level of analysis, we
measured the time needed for each task from reading the task description to the final solution and
coded the solution as either correct (in the case of an ill-defined problem: plausible), incorrect (or
implausible), or not solved. For the second level of analysis, we coded the insights in the thinkaloud protocols following the procedure proposed by Saraiya, North, and Duca3 with a few
noteworthy modifications: While they only concentrated on deep, complex, and non-trivial
insights, we also counted more shallow, elementary, and expectable discoveries as insights.
Similar to the concept of knowledge building insights29, we define insights as the understanding
gained by an individual using a visualization tool (or parts thereof) for the purpose of data
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
analysis, which is a gradual process towards discovering new knowledge (see
4,26
for examples
and further discussion). We coded and counted the insights during solving specific tasks only,
not during free exploration. Though this is not the intended way of application since Saraiya and
her colleagues3 coded and counted the insights during the whole interaction with a tool, we chose
this procedure to be able to compare insights to our other levels of analysis. For the third level of
analysis we decomposed the task into subproblems and analyzed the users’ think-aloudprotocols, interaction logs, and viewing behaviour to determine which strategies were applied.
These strategies include which data and patterns they extracted from the tool, how they interacted
with the tool (e.g., which temporal granularity they used), and how they thought (aloud) during
this process. From the methodological point of view, coding the data and identifying the
strategies in that way proved easy and provided clear, meaningful descriptions of the problem
solving strategies.
5. Results
To show the specifics, benefits and pitfalls of the three methods of data analysis, we present
empirical results for two exemplary problems – one well-defined problem from graphcomprehension level 1, reading the data, and one ill-defined problem from graph-comprehension
level 3, reading beyond the data.
5.1 Example 1: Extracting a Concrete Value
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
For each data set and tool, our users had the same task to solve: to name the value of Christmas
day in a concrete year. This is a rather narrow and well-defined task, as it has a single correct
solution. But despite this fact, we observed a variety of different strategies that were applied.
5.1.1 Time & Errors
An analysis of time and errors for the two tools was conducted first. For each data set, the time
needed to solve the problem was measured. In addition, the solution was coded as correct,
incorrect, or not solved.
There are no profound differences between the two tools in the time needed to solve the task.
Only for the first dataset (climate) there is a tendency towards a faster solution with the Groove
prototype (MG = 122.4 seconds, SD = 65.8, MMS = 188.3 seconds, SD = 52.6; t = -1.85, df = 9,
p < .10, see figure 4), which might be an indicator that it is easier to grasp for first-time-users
than the Multiscale is. On the other hand, after changing to the other tool with the education data
set (see table 1) no such differences are found (MG = 124.0 seconds, SD = 78.3, MMS = 132.7
seconds, SD = 89.6; t = 0.86, df = 10, p > .10, see figure 4).
[insert figure 4 about here]
However, clear differences emerge in the error rates across all data sets when using the two tools.
When working with the Groove, participants are more likely to solve the task correctly than when
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
working with the Multiscale, where they more often gain a wrong solution or no solution at all
(χ² = 6.17, df = 2, p < .05).
5.1.2 Insights
In a second step the participants’ insights were coded and counted. We distinguish insights into
the data and insights into the tool. Looking specifically into the second kind of insights (insights
into the tool) is very useful to understand how users make sense of the tool (e.g., “It seems that
above are data on an overview level, below they get more specific”), which misconceptions they
develop and how (these can be useful in explaining unintended usage and errors, e.g., “no,
December is found on top and not at the bottom as I would have assumed”), how they use the
tool to solve a problem (e.g., “now I would need to know which weekday Christmas was in 2005
… or I could count from 31st”), and which improvements they require (e.g., “It would be a lot
better with a mouse-over function showing the date and the exact value”). But also analyzing
data insights for such a basic well-defined task does make sense, as on average our users had two
to three insights into the data (Does the year exist? Where is the concrete date? What is the exact
value?).
On a global level, no differences exist between the two tool prototypes in the average number of
data insights (MG = 2.6, SD = 1.0, MMS = 2.2, SD = 0.8; t = 1.33, df = 11, p > .10) or tool
insights (MG = 2.9, SD = 2.0, MMS = 2.8, SD = 1.7; t = 0.39, df = 11, p > .10, see figure 5).
[insert figure 5 about here]
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
When looking at the data insights at a more differentiated level, we can specify whether they are
on a detailed level (identifying single data points) or on an overview level (identifying patterns of
multiple data points) or whether participants generate a new hypothesis from what they saw in
the data. As can be seen in figure 6, most insights are gained on a detail level, only some insights
are obtained on an overview level, and hardly any hypotheses are generated. This result can be
explained by the nature of the task, being very specific on a detailed data level as well.
Comparisons between the tools reveal no differences for the number of insights on an overview
level or for the generation of new hypotheses, but more insights are generated on a detail level
with the Groove than with the Multiscale (MG = 1.8 detail insights, SD = 0.8, MMS = 1.3 detail
insights, SD = 0.6; t = 2.3, df = 11, p < .05).
[insert figure 6 about here]
A more differentiated analysis of the tool insights shows a higher number of how-insights
(insights on how the tool functions) and meta-insights (insights on how the tool can be used to
solve the task) than improvement-insights (insights on how the tool could be made better). There
are no profound differences between the two tools, Groove and Multiscale (see figure 7).
[insert figure 7 about here]
However, the number of tool insights shows an interesting pattern across the data sets which
might be explained by the change of the tool: Participants worked with the first tool for datasets 1
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
to 3 and with the second tool for datasets 4 and 5 (see table 1). To analyze this effect more clearly
a 2x2 ANOVA was conducted for tool insights into the datasets 2 to 5. The order of the tool (1st,
2nd) and the data set (1st, 2nd) serve as within factors. There is a significant main effect of both
factors, the order of the tool F(1, 9) = 5.02, MSE = 11.84, p = .05, η² = .36, and the order of the
data set F(1, 9) = 45.77, MSE = 73.27, p < .001, η² = .84. Participants generate less tool insights
with the first tool than with the second. Also more tool insights are generated for the first data set
than for the second with the same tool. This result shows on the one hand the cost of switching
the tool, resulting in the need for more tool insights, and on the other hand it demonstrates the
good learnability of the two tools: Both result in a significant drop of tool insights when used the
second time with the same task but a different data set. This pattern is also resembled in the time
needed to solve the task with the different data sets (see figure 4).
5.1.3 Problem Solving Strategies
To name the value on Christmas day in a specific year, a set of subproblems have to be solved:
First, users have to identify the location of this date, second they have to extract the colour of the
data point, and third, they have to associate a value to the colour of a data point. For the first
subproblem, we observed seven different problem solving strategies which are applied either
individually or in combination with each other: (1) counting days from the beginning of
December or (2) from the end of December; (3) mapping specific data characteristics (e.g., shop
closes earlier on 1 day, less activity) onto the characteristics of Christmas day; (4) using an
external scaffold (e.g., calendar on mobile phone) to determine the associated day of week; (5)
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
remembering the correct day of week from a prior dataset; (6) approximating the location by
searching for week 51; or (7) estimating roughly.
The applied strategies differ highly between participants, but also within participants. Nobody
uses one single problem solving strategy consistently. A more detailed look at the variations
showed that participants apply problem solving strategies differently in dependence of the tool
and the data set at hand (see figure 8).
[insert figure 8 here]
Obviously some data sets suggest specific strategies: For example, the financial data set has only
little variance within weeks. Therefore, approximating the location and roughly estimating the
correct value is a highly efficient strategy, resulting in a correct solution in 82 % of all cases.
With the economic turnover data set, on the other hand, the problem is solved correctly only by
17 % of the participants. It has high variance within the data and is visualized on a weekly rather
than on a monthly base. Therefore, only participants who count from the end of the year
successfully solve this problem. Three out of twelve participants are not able to generate any
solution at all.
A clear difference also exists between the two tools, Groove and Multiscale, in the problem
solving strategies applied. This difference in the problem solving strategies used might explain
the higher error rates of the Multiscale users (see chapter 5.1.1): They more frequently use
approximation and estimation strategies. With the Multiscale, users often experience problems to
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
find the data point (33 %), but also for the second step to solve the problem: to differentiate
between colours (10 %) and to read the scale (10 %).
5.1.4 Design Implications
How can these results be used to improve the two tools? The first level of analysis shows that the
Groove users solve the problem more often than the Multiscale users (55 % vs. 25 %). But even
the Groove users’ success rate is disappointingly low, showing a need for improvement for both
tools. Despite this fact, no further ideas on how to improve the tools can be gained from the first
level of analysis.
On the second level of analysis we observe that the Groove users generate more insights on the
detail level than the Multiscale users. So we know that something hinders the Multiscale users
from generating more detailed insights. But what? Questions like this, which can be derived from
the results of the first two levels, open the gate widely for speculation: Does the huge number of
data points in the Multiscale hinder the users to find a way to a detailed analysis? Does it depend
on the interaction with the tool – where more directed analysis is facilitated by the self selected
detail level in the Groove? Or does it depend on orientation issues – is there simply too much
distracting clutter in the Multiscale? Are there any conceptual or representational problems in
handling the huge amount of granularities displayed in Multiscale? And, in case one or more of
these factors are decisive, are they influential for all users in the same way? To conclude, this
level of analysis does not seem to give us many concrete answers, but instead a bunch of open
questions which are difficult to answer based on the quantitative results.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
We suggested analysing the insights more qualitatively instead of providing insight counters
only4. Especially the tool improvement insights can be turned into a better design directly:
•
Label the figure on both sides, instead of the left side only. As can be seen in figure 3,
labelling the visualization is a challenge due to the amount of data displayed.
•
Show date and value on mouse over to make locating a date and associating a value to a
data point easier.
•
Facilitate making the association of data point colours with the legend.
But still, we do not know whether and how these improvements will support the exploration
process or have only aesthetic value. We have to look at the processes more directly in order to
redesign a tool to act as a scaffold for data exploration.
Let us, therefore, turn to the third level of analysis. The amount of different problem solving
strategies employed shows that locating a specific date is very difficult with both applications.
Many users experience problems to locate the Christmas day and use approximation and
estimation strategies. Therefore, for both tools, the possibility to find a location should be
improved; in the easiest way by providing a tooltip with date and value or even by providing a
search function for specific dates (e.g., with a calendar overlay).
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
We observed a high variance between the strategies applied for different data sets. One
observation was that for data sets with a weekly instead of a monthly time structure (economy,
education) the task is more difficult to solve: Participants often employ approximation strategies,
which does hardly ever lead to a correct solution. Therefore, a temporal re-organization of the
data should be enabled by the tool to allow participants to switch between a weekly and a
monthly representation of the data.
The analysis of the users’ viewing behaviour also showed us why the users’ improvement
insights include important design enhancements: Finding a date in the right corner of the data set
(like Christmas day) requires a lot of effort in visual search, as it has a high distance to the scales
on the top and the left. Users’ gazes frequently switched between the top and the bottom and the
right and the left until they were able to identify the date. Similarly their gazes switched backand forward from the data point to the legend indicating their difficulty in matching the colours.
Therefore, these improvements can also help users to solve the problems more easily and
flexibly.
With the Multiscale, users experienced even more problems in identifying a specific data point
and in differentiating between the colours used. These problems could be solved by providing an
optical zoom function and a (user-customized) colour scale to increase the contrast for specific
scale segments.
5.2 Example 2: Drawing Inferences from Data
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
For the economic turnover data set, participants had to find out from which gastronomic business
these data are from (e.g., lunch restaurant, bakery, or coffee house). To make this conclusion,
they had to build on the different features of the visualization and draw inferences from their
insights. Though there is one single business where the data stem from, different businesses can
be seen as plausible data sources. In addition, users had to rely on their own knowledge on what
sales patterns are typical for which kind of business. Therefore, this task goes beyond the data
provided and can be classified as an ill-defined, real-world problem.
5.2.1 Time & Errors
Overall, participants need longer to solve the ill-defined problem than they need on average to
solve
defined
the
first,
well-defined
problem
(Mill-defined = 219.0 seconds;
SD = 184.2,
Mwell-
= 116.2 seconds; SD = 40.4; t = 2.16, df = 11, p = .05). For the ill-defined problem there is
no significant difference between the two tools, neither in the time needed (t = 0.53, df = 10,
p > .05) nor in the solution quality (χ² = 2.40, df = 2, p > .05, see figure 9). Overall 42 % of the
participants gain a plausible solution, another 42 % gain an implausible solution, and the other
16 % gain no solution at all. There tends to be a difference in the time needed to solve the
problem in dependence of the solution quality (χ² = 5.59, df = 2, p > .10). Those who do not gain
any solution explore the visualization longer than those who gain a solution – whether it is
plausible or not.
[insert figure 9 about here]
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
5.2.2 Insights
Overall, participants generate more data insights than tool insights. As this task is solved after the
first, well-defined task, users are already accustomed with the tool. Therefore, the tool insights
will not be analyzed.
In comparison to the first, well-defined problem they generate a lot more data insights (Mwelldefined
= 2.43, SD = 0.18, Mill-defined = 12.82, SD = 2.30, t = -4.62, df = 10, p < .01). The higher
number of data insights shows that solving the ill-defined problem requires more mental effort
than solving the well-defined problem. There are no differences between the two tools in the
number of data insights for the ill-defined task. Looking more closely at the kinds of data insights
which were gained, most are on an overview level (detecting patterns), some are the generation
of hypotheses, and only few are on a detail level (see figure 10).
[insert figure 10 about here]
5.2.3 Problem Solving Strategies
The ill-defined problem is composed of a number of sub-problems: Next to graph comprehension
level 1 activities of locating dates and assigning values to them, users have to identify patterns in
time and in the sales, compare these patterns with their own knowledge on possible patterns,
raise hypotheses, and test these hypotheses. For this very open task, many different information
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
resources could be used for the identification of relevant patterns: the sales data (circadian,
weekly, and annual sales patterns), the temporal data (weekly, daily opening hours), and the
range of the data scale (amount of turnover).
[insert figure 11 here]
In our study, users relied on six different information resources (see figure 11). Whereas different
sales patterns were identified by most participants, only some also took the temporal patterns into
account. Only one single participant additionally considered the amount of turnover as a relevant
information source.
No difference exists between the two tools, Groove and Multiscale, in the kind and number of
information sources used and the quality of the solution gained. Overall, 17 % of the users are
not able to generate any solution for this task. Half of the other participants generate a plausible,
near-to-correct solution (42 %), the other half an implausible solution (42 %).
We compared the problem solving strategies used by these three groups and found that the
quality of the solution correlates with the number of information sources the participants took
into account (see figure 12): If they consider only two or three different kinds of information they
are likely to generate a wrong solution. If they attend to a medium number of information sources
they do not generate any solution (“I give up. I’ve no idea what this could be.”). Only if they pay
attention to a higher number of four to five different information sources, are they likely to
generate a plausible, near-to-correct solution.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
[insert figure 12 here]
When we look at the kinds of information sources more qualitatively, we see a tendency that
those participants who consider the temporal patterns, that is, weekly and daily opening hours,
are more likely to come to a plausible solution. A frequent wrong solution is that the data stem
from a dinner restaurant, but it neglects the information that the business closes before 8 pm.
The difference in the information sources used cannot be explained by a motivational deficit as
participants take a similar amount of time, independent from the quality of their solution
(plausible: 3.1 min, implausible: 2.1 min). Only those who do not come to a solution at all take
more time (8.3 min, see chapter 5.2.1).
5.2.4 Design Implications
Again, we asked ourselves what these results tell us for improving the two tools. The first level
of analysis shows that there is no significant difference in the time needed or in the solution
quality between the Groove and the Multiscale. The number of participants generating a plausible
solution is quite low, showing a need for improvement. Those users who did not generate any
solution at all took more time than the others. From these first level results no concrete ideas on
how to improve the tools can be gained.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
On the second level we observed that users generate a lot of data insights (especially in
comparison to the well-defined problem), most on an overview level, but also hypotheses and
some on a detail level. It seems that supporting the generation of novel insights is especially
important for ill-defined problems. But how can we build a scaffolding visual analytics tool?
Again, we do not obtain much information to improve the tool from this level of analysis.
Let us look at the third level, the analysis of problem-solving strategies. We found that a crucial
factor in generating a plausible solution for this task is to take into account not only the sales
patterns in the data set, but also the temporal boundaries of the visualized data. Many participants
failed because they did not consider the daily and weekly opening hours as a relevant information
source. Consequently the design should be improved in a way that makes the daily opening hours
more salient. One possibility would be to highlight the closing hours by showing not only labels
for those hours of the day where data exist, but also for those where no data exist. Another
possibility would be to increase the label size.
Participants who gained an implausible solution often aborted hypotheses testing early – after
they found first evidence in favor of their hypothesis. Therefore, a second challenge is how
participants can be encouraged to test their hypotheses against more information sources and
thereby become more likely to discover wrong assumptions. A possibility within the Groove
would be to lead users through all granularities step by step and thereby make it easier to check
assumptions against all temporal granularities. A disadvantage of such a solution is that the user
looses freedom of action and would possibly click through the granularities without exploration.
A second possibility would be to allow for a more flexible arrangement of the temporal
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
granularities: With the prototype, users were only able to aggregate the data on the next higher
temporal granularity. By allowing users to aggregate the data also on other temporal granularities
(e.g., annual means of each hour of the day, weekly means of each weekday) novel cyclic
patterns can be discovered and further exploration of the data can be encouraged.
We also observed that only one participant took into account the amount of turnover as a relevant
information source. Consequently, a design element worth improvement is the display of the
scale. The display of values as colours seems to be powerful for the identification of patterns, but
less powerful for the extraction of data boundaries and single values. Therefore, design
improvements should make the data values more salient: For our visualizations, this could be
achieved by applying a visual filter which greys out values of a lower range or by using smart
legends30 to highlight data points based on their value. In addition, the already suggested tool-tip
would be a solution, but also a calculation function which allows displaying sums and means for
selected areas.
6. Discussion & Conclusion: Mapping the Users’ Path to Rome
What did we learn from analysing problem solving strategies for the evaluation of a visual
analytics tool? Overall, we can contend that knowledge on problem solving strategies can
provide useful insights on how to improve a visual analytics tool during participatory design. In
contrast to analysing time and errors or counting insights, we not only discovered that there is a
design problem, but we also found out which parts of the problem solving process and which
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
problem solving strategies are affected, and we were able to generate suggestions on how to
overcome the design problem.
For the first task – extracting a concrete value – we found differences between the strategies
applied for the two tools. This finding suggests that the problem solving strategies do indeed
depend on the tool a user has at hand. By analysing the successful (and the less successful)
problem solving strategies we were able to make suggestions on how our two tools should be
improved concretely during the next development phase: Due to the large amount of data points
displayed it is important to provide easier means to find a concrete date; for example, by a
calendar function or a calendar overlay. In addition, a better linkage between the coloured data
points and the associated areas on the scale is needed. For the Multiscale, problems to
differentiate between the small data points were found. It is therefore important to provide a
zoom function, but maybe also a filtering function which allows displaying less data.
Some of these suggestions for improvements might be called trivial. But the analysis of concrete,
real-world tasks and of the strategies applied to solve them did on the one hand provide us with a
way to set priorities within the different possible further developments of the tool. On the other
hand, it allowed us to rule out some developments as not relevant that would have seemed
necessary without this study (e.g., an optical zoom does not seem to be necessary for the
Groove).
The problem solving strategies applied do not only depend on the tool, but also on the context,
that is, the task and the data set: For a data set structured on a weekly base, users applied
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
different strategies than for a data set structured on a monthly base. Our results confirmed that
the best problem solving strategy varied from data set to data set and from tool to tool. As prior
research shows, expert users are more likely to select the most appropriate strategies for the
situation at hand21. Correspondingly we observed that our expert users flexibly use different
problem solving strategies.
These results support our claim that an information visualization or visual analytics tool should
allow for multiple ways to solve a problem. In the field of information visualization and visual
analytics, different assumptions exist concerning the effectiveness of a visualization1,2,3,26. We
content that an effective visualization is a flexible one which affords and supports multiple
problem solving strategies. This is especially important for tools designed for expert users in a
domain. As a consequence, it is important to develop the visual analytics tool for a concrete realworld context and also evaluate it in such a context, as suggested in the grounded evaluation
approach31 and in situated, embedded cognition32: The prototype of a visual analytics tool should
be tested within the context of its intended use; that is, with real users (be it experts or
laypersons, seniors or youngsters) and a set of realistic data sets and tasks (that quite often
happen to be ill-defined13) consistent with the purpose of the visualization.
We found that users need more time and effort to generate insights for the ill-defined problem of
determining the data source of a visualization by comparing found patterns in the data to already
known patterns. However, the time taken is not a predictor of the quality of the solution; rather,
participants who take more time do not gain any solution at all. Rather, the choice of information
sources that are used for pattern identification proved to serve as a predictor of the quality of the
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
solution. Only those who find the patterns in the temporal data (e.g., daily opening hours)
generate a plausible solution. Because the information needed to come to a solution is less
obvious in ill-defined problems, it is important to take multiple sources of information into
account. This is especially difficult when some relevant information resources are less salient
than others, as was the case in our study. A visual analytics tool which aims to support ill-defined
problem solving therefore has to be highly flexible to scaffold the exploration of less obvious
patterns and the application of different strategies to identify them.
At the beginning of this paper we proposed that analysing problem solving strategies can shed
light on the black box of the exploratory mind (see figure 1) by identifying which cognitive
processes support or impede solving a task. Indeed in our study we found a correlation of
problem solving strategies with the task’s solution quality in both examples. We found it useful
not only to analyse problem solving strategies that led to successful task completion, but also
strategies that led to near-to-correct, false, or no solutions at all. The latter can be used to analyze
at which subproblems these users fail and also to find solutions on how solving these
subproblems can be scaffolded by an improved visual analytics tool.
Overall, we can content that the problem solving process indeed further illuminates the black box
by showing how a task is solved – or not (see figure 13).
[insert figure 13 here]
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
What we did not address in our study is the relationship between problem solving processes and
insights. In our opinion, it is plausible that insights are solutions to the subcomponents of a task.
As displayed in figure 13, we assume that solving these subcomponents lead to insight
generation. The user’s expertise and the complexity of a task (and consequently the number and
nature of its subcomponents) determine the number of insights needed to solve a task. This
question is definitely an area in need of further research. However, if we can show a relationship
between insights and task-related problem solving processes, we at the same time gain further
knowledge on the relationship between tasks and insights as well. Thereby, we could also better
classify insights (e.g., in accordance to their task relevance) and could shed further light on the
exploratory mind.
Still, we are aware that we cannot understand the exploratory mind fully by looking at the
problem solving processes alone (see the remaining black parts of the box in figure 13). Further
processes remain in the dark and might be uncovered in the future. In addition, small black boxes
at the level of the sub-processes remain, as we are not yet on the smallest possible level of
analysis and cannot say for sure what the most practical level of analysis might be.
A further restriction of our study is that we mainly relied on the think-aloud protocols for these
process analyses. As problem solving strongly depends on perceptual processes28 and the
interaction with the tool10 we maintain that in order to illuminate the black box even further,
future research should therefore address the question of how knowledge processes, perceptual
processes, and interaction concur during problem solving.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
We are aware that the methodology we used in our study will not be transferable to every
situation, every visualization, every domain, and every task, because problem solving strategies
differ in dependence of the situation. Still, we want to encourage other researchers to follow our
procedure for evaluation of information visualizations during participatory design. We found this
approach easy to apply: By observing the users’ problem solving process and analyzing the thinkaloud protocols different strategies became apparent quite quickly. Independent from the task’s
complexity, whether it was well- or ill-defined, strategies could be identified which hinder or
support the problem solving process in the context of the existing visualization.
Indeed, many ways lead to Rome in information visualizations. It is worth mapping them to shed
light on the Eternal City and the exploratory minds walking within. Let your users cut their own
path to Rome – and make sure that your visualization allows them to do so, by adjusting your
tool to act as a scaffold for multiple problem solving strategies.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
7. References
1.
Thomas, J. J. and Cook, K. A. (2005) Illuminating the Path: The Research and Development
Agenda for Visual Analytics. Los Alamitos, CA: IEEE Computer Society Press.
2.
Bertini, E., Perer, A., Plaisant, C. and Santucci, G. (2008) BELIV’08: Beyond time and
errors – novel evaluation methods for information visualization. CHI '08 extended abstracts on
Human factors in computing systems; 5-10 April, Florence, Italy. New York: ACM Press,
pp.3913-3916.
3.
Sarayaia, P. B., North, C. and Duca, K. (2005) An insight-based methodology for evaluating
bioinformatics visualizations. IEEE Transactions on Visualization and Computer Graphics 11(4):
443-456.
4.
Smuc, M., Mayr, E., Lammarsch, T., Aigner, W., Miksch, S. and Gärtner, J. (2009) To score
or not to score? Tripling insights for participatory design. IEEE Computer Graphics and
Applications 29(3): 29-38.
5.
Proctor, R.W. and Vu, K.P. (2006) The cognitive revolution at age 50: Has the promise of
the human information-processing approach been fulfilled? International Journal of HumanComputer Interaction 21(3): 253-284.
6.
Miller, G.A. (1956) The magical number seven, plus or minus two: Some limits on our
capacity for processing information. Psychological Review 63(2): 81-97.
7.
Newell, A., Shaw, J.C. and Simon, H.A. (1958) Elements of a theory of human problem
solving. Psychological Review 65(3):151-166
8.
Smuc, M. (forthcoming) Unveiling the exploratory mind. A cognitive approach to Human-
Computer Interaction in visual data analysis. PhD thesis, University of Vienna, Austria.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
9.
Card, S.K., Moran, T.P. and Newell, A. (1983) The Psychology of Human-Computer
Interaction. Hilldale, NJ: Lawrence Erlbaum Associates.
10. Pike, W.A., Stasko, J., Chang, R. and O′Connell, T.A. (2009). The science of interaction.
Information Visualization 8(4): 263-274.
11. Jones, G. (2003) Testing two cognitive theories of insights. J. Exp. Psychol. Learn. 29(5):
1017-1027.
12. Simon, H.A. and Newell, A. (1971) Human problem solving: The state of the theory in
1970. American Psychologist 26(2): 145-159.
13. Jonassen, D.H. (1997) Instructional design models for well-structured and ill-structured
problem solving learning. Educational Technology Research and Development 45(1): 65-94.
14. Valiati, E.R.A., Pimenta, M.S. and Freitas, C.M.D.S. (2006) A taxonomy of tasks for
guiding the evaluation of multidimensional visualizations. In: E. Bertini, C. Plaisant and G.
Santuppi, (eds.) Proceedings of the 2006 AVI workshop on BEyond time and errors: novel
evaLuation methods for Information Visualization; 23 May, Venice, Italy. New York: ACM
Press, pp.1-6.
15. Friel, S.N., Curcio, F.R. and Bright, G.W. (2001) Making sense of graphs: Critical factors
influencing comprehension and instructional implications. Journal for Research in Mathematics
Education 32(2): 124-158.
16. Amar, R., Eagan, J. and Stasko, J. (2005) Low-level components of analytic activity in
information visualization. In: J. Stasko and M. Ward, (eds.) Proceedings of the 2005 IEEE
Symposium on Information Visualization; 23-25 October, Minneapolis, MN. IEEE Computer
Society, pp.111-117.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
17. Amar, R. and Stasko, J. (2004) A knowledge task-based framework for design and
evaluation of information visualizations. In: M. Ward and T. Munzner, (eds.) Proceedings of the
IEEE Symposium on Information Visualization; 10-12 October, Austin, TX. IEEE Computer
Society, pp. 143-150.
18. Gick, M.L. (1986) Problem-solving strategies. Educational Psychologist 21(1/2): 99-120.
19. Schunn, C.D., McGregor, M.U. and Saner, L.D. (2005) Expertise in ill-defined problem
solving domains as effective strategy use. Memory and Cognition 33(8): 1377-1387.
20. Risku, H. (2003) Situatedness and artefacts in expert knowledge and activity. In: J. Gadner,
R. Buber, and L. Richards, (eds.) Organizing everyday knowledge: methods and case studies.
Houndmills, UK: Palgrave, pp. 28-42.
21. Schraw, G., Dunkle, M.E. and Bendixen, L.D. (1995) Cognitive processes in well-defined
and ill-defined problem solving. Applied Cognitive Psychology 9(6): 523-538.
22. Clark, A. (1997) Being There. Putting Brain, Body, and World Together Again. Cambridge,
MA: The MIT Press.
23. Lammarsch, T., Aigner, W., Bertone, A., Gärtner, J., Mayr, E., Miksch, S. and Smuc, M.
(2009) Hierarchical temporal patterns and interactive aggregated views for pixel-based
visualizations. In: E. Banissi, et al., (eds.) Proceedings of 13th International Conference on
Information Visualisation; 15-17 July, Barcelona, Spain. Washington, DC: IEEE Computer
Society Press, pp.44-49.
24. Shimabukuro, M., Flores, E.F., de Oliveira, M.C.F. and Levkowitz, H. (2004) Coordinated
views to assist exploration of spatio-temporal data: A case study. In: J. Roberts, (eds.)
Proceedings of the 2nd International Conference on Coordinated and Multiple Views in
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
Exploratory Visualization; 13 July, London, England. Los Alamitos, CA: IEEE Computer
Society Press, pp.107-117.
25. Whiting, M.A., Haack, J. and Varley, C. (2008) Creating realistic, scenario-based synthetic
data for test and evaluation of information analytics software. In: E. Bertini, A. Perer, C. Plaisant
and G. Santucci, (eds.) Proceedings of the 2008 conference on BEyond time and errors: novel
evaLuation methods for Information Visualization; 5 April, Florence, Italy. New York: ACM
Press, pp.8-15.
26. Smuc, M., Mayr, E. and Risku, H. (2010) Is your user hunting or gathering insights?
Identifying insight drivers across domains. Paper presented at BELIV 10; 10-11 April, Atlanta,
GA.
27. Robertson, G. (2008) Beyond time and errors – position statement. Paper presented at
BELIV 08; 5 April, Florence, Italy.
28. Grant, E.R. and Spivey, M.J. (2003) Eye movements and problem solving: Guiding attention
guides thought. Psychological Science 14(5): 462-466.
29. Chang, R., Ziemkiewicz, C., Green, T.M. and Ribarsky, W. (2009) Defining insight for
visual analytics. IEEE Computer Graphics and Applications 29(2): 14-17.
30. Sieber, R., Schmid, C., and Wiesmann, S. (2005) Smart legend–smart atlas. In Proceedings
of the 22th International Cartographic Conference, 9-16 July, La Coruña, Spain. International
Cartographic Association.
31. Isenberg, P., Zuk, T., Collins, C. and Carpendale, S. (2008) Grounded evaluation of
information visualizations. In: E. Bertini, A. Perer, C. Plaisant and G. Santucci, (eds.)
Proceedings of the 2008 conference on BEyond time and errors: novel evaLuation methods for
Information Visualization; 5 April, Florence, Italy. New York: ACM Press, pp.56-63.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
32. Risku, H., Mayr, E. and Smuc, M. (2009) Situated interaction and cognition in the wild, wild
world: Unleashing the power of users as innovators. Journal of Mobile Multimedia 5(4):
287-300.
Figure 1. Illuminating the black box of data exploration by studying insights.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
Figure 2. Close-ups illustrating the Groove’s functionality. Left-click displays a lower temporal
granularity (first row), right-click a higher temporal granularity (second row).
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
Figure 3. The Multiscale variant.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
Table 1. Balanced research design. Each expert worked with five datasets and with both tools.
The order of the tools was balanced.
Datasets
climate
economic
traffic
educational
turnover
accidents
logfiles
finance
Experts 1-6
Groove
Groove
Groove
Multiscale
Multiscale
Experts 7-12
Multiscale
Multiscale
Multiscale
Groove
Groove
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
Figure 4. Time (minutes) and errors (% correct, incorrect, or not solved) for the two tools,
Groove and Multiscale.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
Figure 5. Amount of data insights (left) and tool insights (right) for each data set in comparison
between the two tools, Groove and Multiscale.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
Figure 6. Amount of data insights in the categories overview, detail and hypothesis for each data
set: Groove (left) and Multiscale (right).
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
Figure 7. Amount of tool insights in the categories how, improvement, and meta for each data set:
Groove (left) and Multiscale (right).
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
Figure 8. Problem solving strategies (%) for 5 data sets.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
Figure 9. Time (minutes, left) and errors (% plausible, implausible, or not solved, middle) for the
two tools, Groove and Multiscale. Time in dependence of solution quality (minutes, right).
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
Figure 10. Amount of data insights overall and in the categories overview, detail, and hypotheses
(left) in comparison between the Groove and the Multiscale.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
Figure 11. Number of participants (%) who used a specific information source.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
Figure 12. Number of information sources used in dependence of the solution’s quality.
Pre-print version – for citation please use: Mayr, E., Smuc, M. & Risku, H. (2011). Many roads lead to
Rome: Mapping users' problem-solving strategies. Information Visualization, 10, 232-247.
Figure 13. Illuminating the black box of data exploration by studying problem solving processes.