Download 2 knowledge pump: overview and related systems

Making Recommender Systems Work for Organizations
Natalie S. Glance
Damián Arregui
Manfred Dardenne
Xerox Research Centre Europe
6 chemin de Maupertuis
38240 Meylan, France
+33 476 615 022/99
{glance, arregui, dardenne}@xrce.xerox.com
ABSTRACT
For the past two years, we have been investigating the use of recommender systems as a
technology in support of knowledge sharing in organizations. Recommender systems are a way
of extending the natural process of recommendation by word-of-mouth to networked groups of
people. They are able to provide personalized recommendations that take into account
similarities between people based on their user profiles. The community around recommender
systems that has emerged in the past five or so years has focused on methods for constructing and
learning user profiles, the exploration and testing of various recommendation algorithms, and the
design of user interfaces, with applications primarily in the domains of electronic commerce and
leisure/entertainment.
Thus far, we have focused our research in two areas: adapting recommendation algorithms and
user profile construction methods to take into account prior information regarding the existing
organizational social network; and addressing the incentive issues surrounding the use of a
recommender system for knowledge sharing in an organization. In this paper, we describe
principally the incentive issues that we have identified and how we have attempted to alleviate
them. We also report and analyze results from an internal year-long trial of our recommender
tool, the Knowledge Pump.
1
INTRODUCTION
It is not enough to know that the 100 items returned by your search contain both of the words
“knowledge” and “discovery.” It’s not even enough to know which ones are more or less relevant
to the domain of “knowledge discovery.” What people really need to know is which papers are
the ground-breaking ones, which contain important new results, which are the provocative ones,
as well as who are the experts. Current search techniques based on indexing, retrieval and
relevance feedback can’t give this kind of information, but recommender systems promise the
ability to augment relevance judgements with these kinds of quality judgements.
Recommender systems are intelligent agents that provide a way to filter items by personalized
measures of quality: users receive only those recommendations from their colleagues that they are
most likely to perceive to be of high quality. Conversely, the person doing the recommending is
ensured that only those people likely to be interested in what s/he has to share will receive the
item. Actually, since measuring quality is highly subjective, it’s more accurate to say that
recommender systems work by filtering by taste. Recommender systems learn their users’ tastes
and recommend items to users by first matching users to each other. How recommender systems
To appear in Proceedings of PAAM’99
learn users’ tastes and how they perform the matching is to a large extent what differentiates
current systems (as well as their domains and genres of application).
A natural evolution of recommender systems has been to also provide matchmaking services by
bringing together people with similar tastes and interests. In this sense, many recommender
systems have morphed into “communityware,” a new term for software which supports
communities. However, most of these systems are oriented towards the needs and requirements
of Internet users engaging primarily in leisure activities, and not towards the workplace and work
activities. In the workplace, these matchmaking services can become ways to extend one’s
professional network and to locate experts in a domain.
In this paper, we report on our work which focuses on re-orienting recommender systems towards
the workplace. In work settings, the primary foci shift from sharing recommendations to sharing
knowledge and from community-building to community support. Moving recommender systems
from the Internet to the workplace also means turning “leisure-ware” into groupware, creating
both new challenges and new opportunities.
When we think about putting recommender systems to work for organizations, we need to rethink a number of assumptions underlying Internet-based recommender systems: for example, the
potential size of the user base (huge and still rising exponentially) and the raison d'être
(recommendations oriented towards leisure and personal interests). In contrast, in work
organizations, the potential user base will be relatively fixed and of more modest size. Also, the
main impetus, or at least, a major impetus for usage, will be sharing information pertinent to
workplace objectives, competencies, and interests.
On the Internet, a useful service can hope for a user base of thousands, if not millions, of users.
Recommender systems that depend on statistical algorithms thrive on extensive usage. In
contrast, recommender systems in primarily closed organizations of more modest size must
implement new ways to make recommendations for a smaller user base. Many Internet-based
systems using collaborative filtering techniques have already discovered the benefit of filtering
first by content and then by taste. This two-tiered approach is likely to be even more important in
work environments where differences in preferences cannot be accounted for simply by taking
into account the opinions of yet more people rating yet more items.
However, filtering first by content and then by taste means calculating correlations based on far
fewer rating pairs, a problem which is likely to be far worse in an organizational setting where
interest domains may be populated by mere handfuls of people. These problems highlight the
importance of going beyond automated collaborative filtering for making recommendation
predictions. To make good predictions, organization-based recommender systems will have to
take into account usage data across many systems, potentially including search engines, document
management systems, possibly e-mail, and will have to finesse methods for combining the many
kinds of evaluations into one form.
Recommender systems as groupware instead of leisure-ware also suffer even more from the
classical critical mass problem. Within a work organization, a recommender system can be a
valuable tool only if most (as opposed to many) people are using it. Like the telephone or the fax
or e-mail, this kind of technology will be used at work only if most others use it as well. People
in the workplace are likely to be more discouraged from sharing knowledge via a recommender
system if only a fraction of their colleagues are signed up to receive recommendations, because
then they will have to supplement their effort by using other communication media, such as email. The critical mass problem exists in Internet-based recommender systems, as well, but
arguably to a lesser extent, as the Internet can make up in numbers what it lacks in density.
2
Thus, it is vital in a work setting to investigate the incentive issues that arise in achieving
acceptance and usage of the technology. These incentives include ease-of-use, integration with
users’ software environment, and perhaps most importantly, immediate and sustained perceived
benefit. This last issue is particularly difficult: how to overcome the well-known cold-start
problem of recommender systems to provide useful and high-quality recommendations
immediately?
In addition, we can expect the dynamics of group participation and usage to be qualitatively
different in a work situation where people know each other by name and reputation. How can
work-based recommender systems leverage the notions of trust, reputation and reciprocity to best
serve working communities and help them serve themselves? Here, we can expect to learn both
from other peer-based models for collaborative work and evaluation and from market
mechanisms for privatizing public goods (evaluations being a public good). However, the true
test will be in the usage: in this sense, work-centric recommender systems are just like their
Internet cousins; they serve as platforms for novel social experiments and institutions.
Finally, while Internet-based recommender systems focus on creating communities by bringing
people together, Intranet-based recommender systems should focus on supporting communities
that already exist. The power of recommender systems to help people find each other is
multiplied many times over in an organization: locating experts; reducing re-work by bringing
together people working in the same area in different geographic locations; and identifying
competencies, both established ones and emerging ones.
To study both the utility and usability of recommender systems for the workplace, we have
implemented and deployed a research prototype we call the Knowledge Pump. Our objectives in
designing the Pump were two-fold. The first was to help automate the process of sharing
recommendations. The second was to support workplace communities of interest. With these
ends in mind, we designed ways to provide user feedback on collective behavior and
implemented market mechanisms intended to regulate the flow of recommendations. Below, we
first give an overview of the current prototype and compare it with the current breed of
recommender systems. Then, we discuss what we mean by community in a networked
organization, since the notion of community is central to our work. Next, we take a closer look at
issues of utility and usability of recommender systems in the workplace that we have encountered
in the development and usage of the Knowledge Pump. Finally, we present usage analysis results
and summarize user feedback from the first year of the Pump’s operation within our research
centre. In the discussion, we describe the next steps we are taking, both in the directions of
commercialization and on-going research.
2
KNOWLEDGE PUMP: OVERVIEW AND RELATED SYSTEMS
The first prototype of the Knowledge Pump has been implemented in Java as a client-server
system. The client runs as an applet off of a WWW browser. By riding on top of the Web and
coding in Java we can get most of the way towards fulfilling one requirement for acceptance of
any groupware tool with only one version of the code: cross-platform availability.
When a user first becomes a member of the Pump, s/he chooses a set of advisors from the list of
current members and a set of communities, or domains of interest, from the list of communities
suggested. Users are suggested to think of advisors as people whose opinions they most value
and trust. What we mean by community and how the list of communities is created (and how it
evolves) is explained in Section 2.1.
3
Figure 1: An abridged example of “What’s Recommended?” by the Knowledge Pump today.
The Pump provides its members with two principal functionalities on top of a WWW browser.
These are (1) a shared bookmarking system and (2) a daily-updated list of recommended items
(for example, articles, case studies, customer solutions, and, of course, WWW pages) classified
by community of interest. The shared bookmarking system allows the user to bookmark items and
save them as either private (for the user’s eyes only) or public. A public bookmark can be thought
of as a review or a recommendation that the user shares with other members of the Pump. The
Pump also helps users search and browse the set of shared bookmarks. In addition, there is a
“What’s new?” button that generates a page of all the new submissions and reviews of previously
submitted items for the past week which spans across all communities. The shared bookmarking
facility and the search function as well as additional functionality will be discussed further in later
sections. The software architecture and implementation of the system are described in [4].
Each day, the Pump sends users a new set of shared bookmarks likely to interest them. An
example of “What’s Recommended?” by the Knowledge Pump today is shown in Figure 1. For
each community to which the user belongs, the Pump delivers every day (by automatically reloading the WWW page) a set of recommendations. Each recommendation is preceded by the
Pump’s prediction of the user’s interest, shown as a number of stars. The recommendation itself
is a hyperlink pointer to the item recommended. The next line after the item shows the list of
reviewers who have rated and commented upon the item. Many of these names are likely to be
known to the person receiving the recommendations from day-to-day work interactions. Finally,
there is also a hyperlink to the detailed review page for the item: each reviewer’s rating,
comments and the time of evaluation. The review page functions as a kind of “conversation”
around the recommended item, and will be described further in Section 4.1.
4
The Knowledge Pump uses what we call community-centered collaborative filtering [4] to predict
a user’s level of interest for unread items in each of the users’ domains of interest. This
mechanism combines elements of social and content-based filtering. With respect to the latter, we
currently rely on recommenders to classify items into a commonly agreed upon classification
scheme. (The process for defining the classification scheme is discussed in the next section.) A
different, or perhaps complementary, approach would be to use statistical techniques for content
filtering as well, as do Balabanovic and Shoam [2] in Fab, a Web-page recommender system.
The second layer of social filtering – matching items to people by first matching people to each
other – lies on top of the initial classification by community.1 Community-centered collaborative
filtering extends the technique of automated collaborative filtering presented by, for example,
Shardanand and Maes [20] and Resnick et. al. [17] in the context of movie and music
recommendations and Netnews recommendations respectively. These systems automate
collaborative filtering using statistical algorithms to make recommendations based on correlations
between personal preferences. Recommendations usually consist of numerical ratings, but could
be also deduced from implicit ratings, such as time spent reading a text [12].
We’ve modified the standard approach in order to address the cold-start problem faced by
recommender systems. The cold-start problem refers to the poor performance of automated
collaborative filters when usage data is sparse, which is the case when the system first starts up,
or, possibly, when a new community or domain of interest is created. Unfortunately, poor
performance is likely to discourage the usage that would overcome the lack of usage data. The
cold-start problem is likely to be even more significant in the workplace, where the goodwill of
users is more rapidly consumed.
In extending automated collaborative filtering, we have tried to ensure that recommendations will
be of immediate value for new members. One technique we employ is to bootstrap the
collaborative filter by a partial view of the social network constructed from user’s lists of
advisors. When the usage data is too sparse to make automated collaborative filtering feasible, the
Pump instead relies first on a member’s advisors to make predictions. Over time as more usage
data is collected, the automated collaborative filter kicks in to contribute to the Pump’s prediction
of the user’s interest in an item. Of course, bootstrapping the Pump in this way only works in a
setting where many members of the social network already know each other.
A number of other approaches to bootstrapping recommender systems have also been taken,
which could be usefully combined with ours. For example, Shahabi et. al. [19] determine
similarities between users from the WWW navigation paths using a server-side agent. Kautz
et. al. [10] have developed methods for reconstructing social networks from the text of Web
pages and hyperlinks between them. Nichols et. al. [14] describe the different kinds of implicit
recommendations and associations that can be mined from usage data in the context of digital
libraries. Pirolli and his colleagues [15, 16] examine how patterns of usage at a server site can be
used to inform the subsequent Web navigations of other users and deduce properties of
documents (faddish vs. sustaining interest, for example). Foner [3] has designed and
implemented a multi-agent system for deducing common interests among people using
matchmaking agents.
1
It's important to filter by content first and by social relationships second because similarities
among people tend to vary greatly across different domains. For example, the authors of this
chapter have similar evaluations of existing recommender systems, but wildly different opinions
concerning the best guitar players alive. Social filtering over all domains at once tends to wash
out the differences in people's similarities to each other.
5
It’s a natural move to juxtapose recommendation services and matchmaking services; a number
of recommender systems have extended their services in this direction. A major reason why
Firefly2 has become so popular is because it provides chat facilities alongside its movie and video
recommendation service: the recommendations provide the seed for new relationships. Similarly,
another movie recommendation system described by Hill et. al. [9] lets users know who has
similar tastes to them and offers joint recommendations for more than one person. In a workplace
setting, the power of recommendations to link people together has the potential to be extremely
valuable. We are exploring a number of techniques for enhancing users’ sense of community,
which we will discuss in Section 4. In particular, our use of market mechanisms for regulating
the flow of recommendations, the several kinds of user feedback on collective behavior, and the
support for community awareness are, to our knowledge, novel extensions of recommender
systems.
2.1 What does ‘community’ mean?
If recommender system-like technology is to succeed in the workplace, it will be for some of the
same reasons that e-mail and Internet newsgroups have proved to be so useful: because it
provides a new medium for communication and information sharing; and because it provides a
new way to support communities.
‘Community’ is an over-used word currently, so it’s worthwhile to state what we mean more
precisely. Specifically, a community is more than a domain of interest/activity or a collection of
people. Mynatt et. al. [13] introduce the term network community and define a set of properties
that hold for community in both “virtual” and “real” worlds. The properties they list are: “shared
spatial relations, social conventions, a sense of membership and boundaries, and an ongoing
rhythm of social interaction.” Their view is consistent with Lave and Wenger’s concept of
community of practice [11], who emphasize that community “does not imply necessarily copresence, a well-defined, identifiable group, or socially visible boundaries. It does imply
participation in an activity system about which participants share understandings concerning what
they are doing and what that means in their lives and for their communities.”
What’s interesting about this definition is that it makes precise what a community is not. A
community is something else than a team or a workgroup. A community is also something else
than a collection of people (such as a crowd at the fair, or the set of users of a form of information
technology, or everyone with an interest in Java programming).
Mynatt et. al. go on to discuss the implications of designing technology for network communities,
illustrating their arguments with anecdotal evidence from behavior observed in MUDs and media
spaces. They outline some design dimensions for technology that enables and supports network
communities. These dimensions stress the importance of maintaining tight links between real and
virtual worlds. For example, in order to integrate the two worlds, it’s important to draw upon
pre-existing social conventions, such as social identity and protocols of interaction. The authors
also stress that design must be strongly and continuously informed by ongoing social interaction.
Our objective with Knowledge Pump is certainly not to provide all the affordances required to
sustain a community of practice. Rather, our objective is to provide a new form of support for
existing communities of practice whose design is informed by the criteria discussed above.
While one of the strengths of recommender systems has been their ability to tease apart
recommendations from social relationships using statistical techniques [9], it’s clear that these
social relationships must be re-introduced in other ways in order to support community use.
Thus, in discussing the utility and usability of recommender systems in the workplace below, we
2
http://www.firefly.com/
6
will stress the preservation of social identity and of social conventions such as trust and
reciprocity. We will describe several technical mechanisms we have introduced to enhance preexisting social conventions, although clearly what we propose and have implemented is just a
starting point. We will also indicate how peripheral awareness and permeable boundaries
between communities can be supported.
2.2 Defining (and re-defining) community boundaries
For the Knowledge Pump to function in a given organization, the set of communities supported
by the Pump must somehow be decided; i.e., there must be a methodology for defining this set
and for re-defining it as the organization evolves. There are a number of possible methods for
managing the set of communities, varying along two dimensions: automatic to manual and
centralized (top-down) to decentralized (bottom-up). An example of automatic, decentralized
management is inferring communities from usage data. An example of manual and decentralized
is the voting mechanism used in Netnews to create a new newsgroup.
In deciding how to manage the set of communities, we had several criteria. First, we decided that
each member of a community should call that community by the same name. The name actually
chosen is not as important as the meaning attributed to the name by users. Secondly, the
community naming scheme must well represent the actual set of communities of interest of the
organization. Thus, creating it should be an organization-wide iterative and evolving effort. In
this sense, defining and re-defining the set of communities to be supported is itself another
process that falls within the tight sociotechnical loop of designing for network communities: both
defining the set of communities and participation in the set of communities are interdependent
processes.
These criteria point to some level of centralized and (initially) manual management of the set of
communities, complemented by a continuous dialogue between Pump administrators and users.
Currently, the authors play both the roles of designers and administrators, although at some point
we expect the latter role to be fulfilled by actors whose background is more appropriate. To play
the role of administrators, we have relied on our intuitions and a set of rules-of-thumb that have
come from interacting with early users.
Starting with an initial community classification scheme intended to reflect the different interests
within the targeted organization, we canvas current and potential users to help refine it. We tell
them that each community was included because we felt that it corresponded to a common
interest of the organization, general enough to cut across project teams and functional boundaries,
but specific enough to provide a particular focus. We also acknowledge that the list of
communities will evolve over time in accordance with observed usage patterns and in response to
their feedback: we ask users to let us know if they feel a community is not represented, or is
misrepresented. In addition, as enough usage data is accumulated over time, complementary
statistical analysis of patterns of behavior can be used to refine the definition of the community
set.
We’ve found that it’s important that this exercise be carefully undertaken before initial
deployment of the Knowledge Pump. Once in place, it becomes much harder to modify the
existing classification scheme. In order for the Pump to provide community awareness it must
also serve as a kind of community memory. Thus, members of a digital library community
should always know that past recommendations are archived under the digital library
classification, even if the community itself has in practice since split into two separate
communities.
The need to support community memory tightly constrains (in a technical sense) how
communities can be split, merged, dissolved and created, without, however, limiting what can be
7
done in a practical sense. For example, when splitting a community in two, the original one
remains, but is inactivated. All recommendations classified into the original community are also
filed into the two new ones. The delineation between the two communities can become apparent
only from recommendations made after the split. The dynamics of community participation will
lead to the natural rise and decline of individual communities. A community with negligible
activity levels has effectively removed itself.
3
UTILITY: END-USER FUNCTIONALITY
In considering both the utility and usability of recommender system-like functionality in the
workplace, a critical user incentive issue arises often. This issue was succinctly articulated by
Grudin [7] in the context of CSCW systems with his question: “Who does the work and who gets
the benefit?” Another way to characterize this incentive problem is to think of the provision of
recommendations as a common good [8]: pretty much everyone is happy to receive pertinent
recommendations, but few may be motivated to actually submit them. As with any common
good, recommendations are likely to be underprovided.
There are a number of ways to alleviate the common good problem. One way is to reduce costs.
Thus, recommending an item should be as easy as possible. This suggests that a recommender
system should not be a stand-alone application or tool, but instead should be embedded in the
constellation of tools that users employ in their everyday work. In this section concerning utility,
we propose a number of easy entry points for recommendations: a shared bookmarking utility, a
digital library system and a shared document repository. If recommending an item is as easy as
creating a bookmark or printing a paper, the likelihood that the public good will be served
increases.
The common good problem is also reduced if the benefits of providing recommendations are
increased.3 One way to increase benefits is to provide additional services along-side
recommendations. Several of these will be discussed in the section below on usability, such as
context-sensitive search and feedback mechanisms to support community awareness. Another
possible service is collaborative searching, which can be thought of as the integration of
recommender systems with an information retrieval facility [6]: the user profile information
accumulated by the recommender system can be used to rank results returned by a query. Finally,
community mapping and content repository mapping can be provided by mining the usage data,
helping people to more easily find both the items they need and the domain experts to contact.
3.1 Shared bookmarking system
The current Knowledge Pump prototype consists of two main modules: a shared bookmarking
system for the WWW and a recommender system (along with their respective user interfaces).
We developed a basic shared bookmarking system as a first functionality to draw users in,
allowing the recommender module to function as a side effect. We also provide a new way to
search through bookmarks described later. The shared bookmarking facility, provided as an
applet4, allows users to save a pointer to an item as private (akin to a bookmark) or public (akin to
a review), as shown in Figure 2. Users classify their bookmarks into any of a number of their
3
Part of the beauty of automated collaborative filtering is that contributing provides its own
reward: because of its statistical nature, the performance of the algorithm improves for a user
with each contribution s/he makes.
4
While directly piggy-backing off a Web browser’s built-in bookmarking facilities would have
been the most elegant solution, this would have interfered with our goals of portability and easy
installation.
8
organization’s commonly agreed upon set of communities (see Section 2.1). To be useful as a
private bookmarking system, users should be able to create their own personal categories in
addition to the public ones, a functionality that the current system does not yet support. Perhaps
most useful would be tying together the Web browser’s internal bookmarking system with that of
the Pump, synchronizing the user’s bookmark hierarchy among the two (or more) tools. Inside
the Pump, users would also be able to classify public bookmarks recommended by others into
their own personal categories.
Figure 2: Making a recommendation is like saving a bookmark.
In addition, incorporating personal domain hierarchies opens up possibilities for extending the
recommender module. For instance, new emergent communities could be deduced from personal
hierarchies, an avenue that has been explored in [21] and [18].
The bookmarking system stays faithful to Knowledge Pump’s philosophy to encourage sharing.
Thus, users are not permitted to save as private a pointer already known to the system as public.
The utility of private bookmarks is two-fold: (1) to save bookmarks in domains of personal
interest; (2) to save bookmarks of items to be evaluated and perhaps shared at a later date.
Because of the way the shared bookmarking process is designed, the Pump helps serve as an
organizational memory. To assure a coherent common view, users are not allowed to delete
shared public bookmarks. (However, it’s necessary that users be able to delete their own private
bookmarks, as in any other bookmarking utility.) Although users cannot delete public
bookmarks, the recommender system module ensures that highly-valued and relevant bookmarks
naturally become more accessible while low-valued, less relevant bookmarks become less
accessible. For example, if a bookmark is classified into a community that becomes less active
over time, then the bookmark is likely to be accessed less frequently over time. Thus, a kind of
“organizational forgetting” is built-in.
3.2 Integration with document repositories
The shared bookmarking tool in some sense treats the entire WWW as one huge repository of
information. In fact, the WWW also serves as a standard interface to many individual document
repositories and databases. For many of these, it would be useful to have a direct interface to the
recommender module of the Pump, a kind of “recommend me” button. Although this involves
modifying the user interface (be it WWW-accessible or otherwise), the added value to the user
can be significant. Tight integration is especially valuable in the workplace where users spend
less time surfing and more time working with repositories and document management systems.
9
To increase the utility of the Knowledge Pump for our first set of users, we have modified the
interfaces to a digital library server called Calliope5. Calliope was developed as part of a digital
library project being conducted by XRCE Grenoble and two French research laboratories, IMAG
and INRIA. It grants access to the combined set of journals available at a number of participating
sites and allows users to print copies of an article on demand, regardless of the actual physical
location of the journal. The Calliope system is already widely used, giving users access to journal
articles, which (in contrast to typical WWW pages) are of comparatively uniformly high quality.
What Calliope lacks is social context, which the recommender module of the Pump attempts to
provide: who is reading what article and how they evaluated it; which new articles are relevant to
which common domains of interest. As shown in Figure 3, the Calliope user interface has been
modified to include, on the one hand, a way to review articles and, on the other hand, a way to
browse reviews for articles. Thus, members of the Pump can read other people’s reviews while
using the Calliope system, as well receive recommendations of articles to read directly from their
“What’s recommended” Knowledge Pump page.
The integration uses two hooks into Knowledge Pump: one to allow users to review the article
within Calliope and one which allows users to directly view comments about the article stored in
the Pump’s database. Both functions are implemented using servlets. For example, clicking on
the review link calls up a Java servlet that contacts the review service of Knowledge Pump. The
review service saves the item’s attributes in the KP database – its URL, title, author, source
(provided by Calliope), as well as the user’s rating, classification and comments.
Figure 3: Abridged TOC page from Calliope, with access to available scanned
articles and recommendation facilities (latter, courtesy of Knowledge Pump).
We also plan to integrate Knowledge Pump with DocuShare6, a document management system
commercialized by Xerox, which is oriented towards workgroup collaboration and organizationwide usage. It allows groups and teams of people to more easily share access to collections of
5
http://www.xrce.xerox.com/ats/digilib/calliope/
6
http://www.xerox.com/products/docushare/index.html
10
documents. Its primary focus is on the team, as opposed to the community, in contrast to
Knowledge Pump. However, it can also be seen as a tool in support of communities and
integrating it with the Pump is another step towards providing better support for communities in
DocuShare. By adding a “recommend me” button to its interface, a member of a workgroup can
choose to share an item with the greater community at large. DocuShare already has a “What’s
new” function; connecting it to the Pump will allow users to benefit from a more sophisticated
“What’s new” that is informed by the recommender module of the Pump.
4
USABILITY: MECHANISMS SUPPORTING FLOW, FEEDBACK AND
COMMUNITY AWARENESS
The integration of Knowledge Pump into a larger set of tools such as digital libraries and
repositories as well as document management systems is intended to help establish and sustain a
critical mass of users. In conjunction with this basic set of services, however, the Knowledge
Pump aims to provide more particular support for communities of users. To this end, we have
designed and implemented a number of features intended to enhance users’ sense of community.
4.1 Evaluations as “conversations”
For each item in a user’s “What’s Recommended” page (Figure 1), the Pump attaches a
hyperlinked review followed by the names of the reviewers. These reviewers are often members’
colleagues and co-workers; thus, the reviewers’ names will be of people connected to members
via the work setting, people whose reputation is already well-established outside of the Pump.
However, our intention is that the Pump serve as an additional avenue for establishing one’s
identity in a community and building a reputation. For example, we expect that over time users
will form an impression of whose reviews are most useful.
By following the link to the set of reviews, the user can jump into the conversation that has
started around the item. An example review page for a document is shown in Figure 4. The user
quickly gets a better sense of not only how strongly each reviewer recommends the item, but also
why it was recommended. Equally important, the user has a better frame of reference for
deciding whether or not the recommended item is worth more of his/her time. The user can later
add to the conversation by making a review of his/her own. There’s also the option to make
one’s comments anonymous (to avoid publicly confronting one’s boss, for example). Comments
are optional, of course; even ratings by themselves are useful to help the recommender system
learn more about its users.
Treating evaluations as conversations means that the Pump mimics somewhat the functionality of
on-line discussion groups, such as newsgroups or mailing lists. However, in this case, the
conversations are focused around a particular recommended item. Also, since the conversations
are saved, we expect that each review is more likely to build upon previous ones, avoiding rehashing of old commentary.
11
Figure 4: Review page for a recommended item.
4.2 Context-sensitive search
In conjunction with the shared bookmarking facility described in the previous section, the Pump
incorporates some specialized search capabilities. Like other bookmark search facilities, the
Pump’s search function can retrieve bookmarks based on keywords in title, author, etc. It can
also return, for example, all bookmarks classified into a particular community, making it easy to
construct an up-to-date reference collection of bookmarks for a given domain.
Of more interest is the ability to construct a query that takes into account which community, what
reviewer, which dates, and how interesting. This is what we mean by context-sensitive search.
Using the search interface, members of the Pump can ask questions like: “What has such-andsuch well-known expert in Knowledge Management submitted to that community recently?”
“How have other people reacted to that article I recommended to the Information Retrieval
community last month?” “What was that WWW page on Java libraries that so-and-so said so
highly recommended a couple of months ago?”
For each query, the Pump returns a list of results, not unlike a concatenated set of review pages.
Each result, apart from containing a hyperlink to the actual item, includes the set of reviews
submitted for that item (the “conversations” discussed above) and also the set of communities
into which its reviewers have classified the item. This last piece of information serves the
additional function of further promoting cross-fertilization between communities. For example,
if a user asks for all items classified into the CSCW community and is returned a list where many
items are cross-classified into Knowledge Management and WWW programming, s/he may well
be motivated to explore what else these communities and its members have to offer.
12
4.3 Community feedback: hits and gauges
We have experimented with a number of different kinds of visual feedback for providing social
context. The most basic of these is similar to the visitor count shown on many Web pages: after
each item listed (either as a recommendation or as a search result), the number of hits the item has
received via the Pump is shown, as well as the date it was first submitted. We count as a hit
whenever a person follows the link to the recommended item (which we call a visit). Currently,
we only allow a maximum of one hit per person per item. The number of hits shown gives the
user another measure of its popularity and indirectly of its perceived value. In a future release, we
plan to replace or augment this feedback on visits with a time profile of aggregate user access to
the item.
On the right-hand side of the “What’s Recommended” page of Figure 1, there is a panel of
gauges, one gauge for each community for which the user is a member. These gauges are
intended to reflect the level of activity in the community. The INFLOW half of the dial indicates
how many recommendations are flowing in per person per week for the community. In black (the
larger semi-circle) is the community average; in gray is the individual inflow. The OUTFLOW
part of the dial indicates the number of visits being made to items in the community, on average.
Once again, the community average is shown in black, the individual average in gray.
In providing users with a sense of the amount of activity in each community, we hope to stimulate
the dynamics of community growth and decline. We expect that users will be more likely to
actively participate in communities that already demonstrate high or increasing activity levels,
which will further stimulate growth. Likewise, we expect that users will be less willing to
participate in communities whose activity levels are low or have declined, thus leading to further
decline. Most significantly, we hypothesize that feedback mechanisms such as the gauges will
encourage users to participate and to contribute more frequently to the more active communities.
By providing aggregate information on both user and community activity levels, we hope to
encourage alignment between user and average community behavior: users of active communities
will be incited to participate more and users of inactive communities will be incited to shift their
efforts. See Glance and Huberman [5] for the theoretical discussion that grounds our hypotheses
concerning the role of user expectations in group dynamics.
4.4 Community feedback: market fluctuations
The simplest way to handle common goods is to privatize them. Thus, to encourage the provision
of recommendations, we could simply pay people for them, as Avery and Zechauser [1] have also
noted in this context. In the workplace, remuneration might be hard currency, but is more likely to
take the form of an improved performance appraisal or a reward.7 In addition to alleviating the
common good dilemma, market mechanisms provide the necessary feedback to regulate the
optimal provision of goods. Sure, recommendations are a good thing, but how many? what kind?
for which communities?
Based on these premises, we’ve introduced some very simple market mechanisms, both to help
privatize the production of recommendations and to provide the feedback that promotes the
optimal level of production. Given that our marketplace is electronic, we have a lot of freedom to
experiment with new kinds of transactions. We’ve put in place some simple economic rules: for
each visit to a recommended item, the visitor pays one chit which is redistributed as royalties
7
In fact, one of the tenets of the current knowledge management trend in business is that
employees be remunerated in proportion to their knowledge sharing activities. The difficulty
lies in measuring the amount of knowledge sharing that goes on and its value to the
organization.
13
Figure 5: Account transactions history.
shared among the item’s reviewers. If the visitor later reviews the item, s/he is reimbursed in
proportion to the item’s actual value to him/her, and, in return, is eligible to receive future
royalties.8 Reviewers receive royalties whenever another user in turn visits an item they have
reviewed. The amount of the royalty depends on the number of reviewers for that item.
Currently, the recommendation market we have put in place approximates a zero-sum game.
These kinds of microtransactions (both payments and reimbursements) are made possible by the
electronic nature of the marketplace.
As a feedback mechanism, the history of account transactions appears to be very useful (see
Figure 5). User comments indicate that they look forward to seeing the red royalty increments in
their account history window (shown in gray in the figure) which indicate that people are visiting
their recommendations. Also, the account balance makes users aware that the recommendations
are not free and that for each recommendation visited, they in some sense “owe” the system a
review of their own. However, as we will argue in the next section from the collected usage data,
the payment mechanisms we have put in place appear to overly discourage so-called “passive”
users of the system – those users who visit and read recommended items, but do not contribute
themselves.
5
USAGE ANALYSIS
In this section we report on results drawn from the first trial of Knowledge Pump which brought
together users from our research centre as well as users brought in by word-of-mouth across
Xerox and Xerox-related entities. The trial covers three phases. In the first phase, only we three
developers of Knowledge Pump used it plus a few other volunteers, principally for testing
purposes, and also to partially populate the recommendations database so that future users would
not face an empty playing field. In addition, the user feedback from our other testers was used to
modify and augment the functionality. This first phase lasted from July 1997 to January 1998. In
January 1998 we opened it up to our research group of 10-15 people, asking their help in further
testing the prototype. Then gradually, as the software stabilized and interest in Knowledge Pump
increased by word-of-mouth, we began to accept new members. In addition to XRCE employees,
these new members included: a group of people across Xerox who wanted to start a new
community in the Pump to share information around a particular domain of their choice; and a
number of Xerox visitors who were shown Knowledge Pump during their visit and requested to
become members.
8
At first sight, reimbursing users based on their ratings may seem like a perverse incentive that
will encourage people to cheat, rating everything as irrelevant and receiving full
reimbursements, effectively paying nothing for recommendations. However, users that cheat in
this way misrepresent themselves to the system. Knowledge Pump, over time, will learn that
such a user judges everything irrelevant and will stop finding items of value to recommend to
the user. More minor forms of cheating will similarly affect the Pump’s ability to accurately
predict users’ preferences.
14
The third phase started in October 1998 with a presentation of the system to the entire research
centre, plus a co-located Xerox business entity, about 80 people in all. At this point, we decided
to experiment with a tangible incentive – the audience was told that the 10 most active members
of the Pump during the next three months would be given PalmPilots (a device seen as attractive
at our workplace).
Here are some overall statistics, which will be further broken down and elucidated afterwards:

As of January 1, 1999, the Knowledge Pump has 66 members, of which about 10 are currently
fairly active, and about 15 have shown sustained activity sometime in the past year. Among
the top 10 most active members currently, only one of these is a Knowledge Pump developer.
By fairly active, we mean several interactions with the Pump per week, on average, where an
interaction is either a review, a visit to a recommended item, or one chit’s worth of royalties.

51 of the 66 members are XRCE employees. Nine of the top 10 most active users are XRCE
employees. 15-20 users are almost completely inactive – many of these were non-XRCE
users who requested membership after seeing a demo, but then never interacted with the
Pump.

Phase 3 (with the PalmPilot bribe) generated 14 new users within XRCE, 2 of whom are now
in the top 10 most active.

Since its inception, 496 documents have been submitted across 38 communities. These
documents have received a total of 1050 reviews (including the original submitter’s reviews)
and 894 unique visits (we count at most one visit per member, and do not count any visits by
the original submitter). Thus, the average submission receives one review in addition to the
submitter’s review.

69% of the submitted documents have one or more visits. 64% of the submitted documents
have one or more reviews in addition to the submitter’s review. Thus a little over 30% of the
recommended items are never viewed by a user other than the original submitter.

Of the 496 documents submitted, only 12 were marked ‘private.’

Of the 894 reviews, only 34 were marked ‘anonymous.’ On average, each review was
classified into 1.8 communities.

All together, Pump users deleted 649 recommended items without visiting them or reviewing
them (but we did not have a way to track if they read the reviews first). They visited 165
items linked from either a “What’s New in the past week” page or from a search results page.

Very little management participation (despite strong backing of the project and the concept).
It is interesting to speculate on the reasons for some of these observations, but more substantiated
conclusions will have to await the results of a user survey. We can hazard that the reason so few
documents were marked as either private or anonymous were at least partly for user interface
reasons. However, we also suspect that the Pump is not currently conducive to being used as a
private bookmarking system – better support, such as integration with WWW browser
bookmarks, is required. That people delete many recommended items without visiting them can
be explained in two possible ways: (1) the filtering process is insufficiently accurate and users are
being recommended many items of little interest to them; and/or (2) the reviews act as additional
filter which helps people to decide to delete items without reading them.
To better understand the usage of Knowledge Pump, we analyzed data patterns in a number of
different ways to elucidate user activity, community activity, and document access and review
distributions.
15
Title:
(S-PLUS Graphic s)
Creator:
S-PLUS
Prev iew :
This EPS picture w as not s av ed
w ith a preview inc luded in it.
Comment:
This EPS picture w ill print to a
Pos tSc ript printer, but not to
other ty pes of printers.
Figure 6: Scatter plot of reviews vs. visits for each user during 15/9/98 –15/12/98.
Figure 6 shows a scatter plot of the number of reviews vs. the number of visits for each member
of the Pump (discarding members who have been completely inactive). This plot was constructed
from data collected over the three month period of September 15, 1998 to December 15, 1998.
We chose a subset of the entire trial in order to capture a period of time in which the user base
would be fairly stable. Overlaid on the scatter plot is a diagonal line representing hypothetical
users who visits others’ recommended items as often as they review. This figure makes it easy to
pinpoint the more active users, of which there are ten during this period of time. The remaining
users are clustered in the bottom left corner of the graph and show very little activity. What’s
particularly interesting is that 6 of the 10 active users write more reviews than make visits of
recommended items. It’s difficult to draw a definitive conclusion for this as there could be many
reasons for this result (for example, in some cases users may decide not to visit a recommended
item because they have already seen the item outside of Pump). However, it does seem to
indicate that the majority of active members contribute more than they receive from the Pump. In
contrast, the literature on knowledge sharing indicates that we can expect the majority of the
population to be recipients rather than contributors to a system of this kind. This seems to
indicate that we have failed to support recipients to the same extent as contributors. Indeed, the
feedback we have put in place, such as royalties and payments as well putting the spotlight on
reviewers and their comments, does privilege contributors and penalize visitors. A more positive
interpretation is that we successfully communicated to our users the basic tenet of automated
collaborative filtering, i.e., that rating items (as part of a review) improves the filtering process
for the user. Still, these results will lead us to re-think the incentives we have put in place. The
scatter plot of reviews vs. visits for the entire trial period shows an even greater bias towards
contributors vs. recipients, with 9 of 13 active members reviewing more than they visit.
Figure 7 shows the distribution of activity levels across communities for the year-long period
from January-December 1998. For the purpose of our analysis we defined activity as inflow +
outflow for the community, i.e., the number of reviews into and visits from the community. If we
set a threshold of 50 as the threshold separating inactive from active communities, we see that 17
of the communities were somewhat active.
Admittedly, the threshold of 50 reviews plus visits still represents a fairly low amount of activity
over a year. However, the distribution has a long tail, with the most active receiving close to 400
reviews and visits within a year, or more than one a day. Interestingly enough, the most active
communities included only one non-work related community (of eight), which, in fact, underlines
16
Title:
(S-PLUS Graphic s)
Creator:
S-PLUS
Prev iew :
This EPS picture w as not s av ed
w ith a preview inc luded in it.
Comment:
This EPS picture w ill print to a
Pos tSc ript printer, but not to
other ty pes of printers.
Figure 7: Distribution of community activity levels (activity = reviews+visits).
a trend we observed, which is that the Pump was used mostly for work-related sharing of
information.
Earlier, we observed that on average, each submission was reviewed once in addition to the
original submitter’s review and visited about 1.5 times. Figure 8 gives the detailed distribution of
the number of visits and reviews per submitted document for the entire period of the trial. Recall
that each document has at least one review (the submitter’s review), but will have zero visits
unless at least one other person besides the submitter visits the document. Again, it is interesting
to note that both of these distributions have long tails and that for a good proportion of documents
the Pump is eliciting something like a ‘conversation’ around the document, if we take multiple
reviews as evidence of a conversation.
From these results, it does not seem possible to say whether this first trial of Knowledge Pump
was a clear success or a clear failure. Some conclusions can be drawn. First of all, an important
Title:
(S-PLUS Graphic s)
Creator:
S-PLUS
Prev iew :
This EPS picture w as not s av ed
w ith a preview inc luded in it.
Comment:
This EPS picture w ill print to a
Pos tSc ript printer, but not to
other ty pes of printers.
Title:
(S-PLUS Graphic s)
Creator:
S-PLUS
Prev iew :
This EPS picture w as not s av ed
w ith a preview inc luded in it.
Comment:
This EPS picture w ill print to a
Pos tSc ript printer, but not to
other ty pes of printers.
Figure 8: (Left) Distribution of #visits per doc; (Right) Distribution of #reviews per doc
17
development effort will be required to make such a system easy enough to use and well enough
integrated with other information-related tools. For example, some users have requested an email interface, which would require modifying a number of e-mail clients. Equally (or more)
importantly, the activity of knowledge sharing must be highly valued by the participants and their
managers. Again, a user survey will help clarify this point, but among the research community
here, it is not clear that there was a very strong perceived need for a knowledge sharing tool. In
such a case it is difficult to convince people to divert time from their other activities towards
sharing knowledge. If knowledge sharing is indeed given a high priority within a given
organization, than it is possible that appropriate incentives can be find to encourage knowledge
sharing activities. Finally, we can only extrapolate so far from the results of this first trial.
Increased potential benefits and risks will arise when usage of the Pump is more greatly spread
over different geographic sites and functional roles.
6
DISCUSSION
Making recommender systems work for organizations introduces the usual difficulties
experienced by developers of groupware, be they scientists in pursuit of new research results or
software engineers in pursuit of new markets. The groupware system (including technical
support) has to be useful enough and robust enough to both elicit and sustain usage in
communities of practitioners already ensconced in set patterns of work.
Thus, the immediate utility of a new groupware tool must be apparent. In a research project, this
leads to a lot of development overhead. For example, it’s clear that we will need to significantly
improve the bookmarking module of the Knowledge Pump in order to satisfy our users. To this
end, we plan to extend the shared bookmarking facility to allow users to add private categories
alongside the public ones and to synchronize with their Web browser’s bookmarking system.
In addition, we plan to make it easy to Pump-enable other commonly used collaborative systems,
as well as document repositories and document management systems. One part of this work is
integration (as well as the development of modules to make integration easier), as described
earlier. However, there are also some interesting research questions; integration with other tools
will allow us to study how to gather and analyze implicit ratings from people’s usage of digital
library systems, e-mail, etc. without impinging upon their privacy.
Moving beyond utility, there are a number of directions in which to improve the usability of our
system, again focussing on our objective to provide improved community support in the
workplace. As mentioned earlier, by mining usage data, community maps can be automatically
inferred from usage data. Views onto these community maps will help members of the Pump
locate experts in the field, as well as track the dynamics of community participation (for example,
the movement of new members from the periphery towards the center). Once again, this direction
of research will have to deal carefully with privacy concerns, and, in addition, find solutions for
presenting what will necessarily be more or less incomplete mappings.
We also plan to further test our approach to collaborative filtering. We have elaborated a notion
of confidence that will allow users to filter recommendations along a dimension of
trustworthiness vs. timeliness. “Gatekeepers” – those people who, self-appointed or by
managerial mandate, separate the information gems from the dross, can opt for the extreme of
timeliness. On the other hand, managers who need simply to track highlights can opt to receive
recommendations whose quality can be predicted with high confidence.
Additionally, we have been refining the concept of recommendation to cover different genres of
recommendations. These genres could include news, reviews, references, events, and so on. The
genre of an item could either be specified by the user or inferred automatically from the time
18
pattern of access to recommended items. For example, a sharp spike may well correlate with a
news item; a high, but level curve with a classic reference.
Lastly, we plan to investigate how the simple market for evaluations that we’ve introduced in the
Knowledge Pump can be used to define metrics for valuing information and knowledge sharing.
Such metrics would go a long way to helping organizations recognize the true value of “word-ofmouth.”
The results of the first trial of Knowledge Pump are promising, but not conclusive. It’s clear to
us that recommender systems have high potential for supporting knowledge sharing in
organizations, and the race is on to get a successful tool onto the market. The advanced
development group here is now at work developing a stable release of Knowledge Pump in order
to set up trials with early adopters in the second half of 1999.
7
REFERENCES
1. Avery, C. and Zeckhauser, R. Recommender systems for evaluating computer messages.
Communications of the ACM 40, 3 (March 1997), 88-89.
2. Balabanovic, M. and Shoham Y. Fab: Content-based, collaborative recommendation.
Communications of the ACM 40, 3 (March 1997), 66-72.
3. Foner, L.N. Yenta: a multi-agent, referral-based matchmaking system, in Proceedings of
Agents’97 (Marina del Rey CA, February 1997), ACM Press.
4. Glance, N., Arregui, D. and Dardenne M. Knowledge Pump: Supporting the flow and use of
knowledge, in Information Technology for Knowledge Management. Eds. U. Borghoff and
R. Pareschi, New York: Springer-Verlag, 1998.
5. Glance N.S. and Huberman B.A. The outbreak of cooperation. Journal of Mathematical
Sociology 17, 4 (1993), 281-302.
6. Grasso, A., Borghoff, U., Glance, N. and Willamowski, J. Collaborative information
gathering, in Proceedings of EuroMedia/WEBTEC’98 (Leicester UK, January 1998).
7. Grudin, J. Groupware and social dynamics: Eight challenges for developers. Communications
of the ACM 37, 1 (January 1994), 92-105.
8. Hardin, G. The tragedy of the commons. Science 162 (1968), 1243-1248.
9. Hill, W.C., Stead, L., Rosenstein, M. and Furnas G. Recommending and evaluating choices in
a virtual community of use, in Proceedings of CHI’95 (Denver CO, May 1995), ACM Press,
194-201.
10. Kautz, H., Selman, B. and Shah, M. The hidden web. AI Magazine 18, 2 (February 1997),
27-36.
11. Lave, J. and Wenger, E. Situated Learning: Legitimate peripheral participation. Cambridge
University Press, New York NY, 1991.
12. Morita, M. and Shinoda, Y. Information filtering based on user behavior analysis and best
match text retrieval, in Proceedings of SIGIR’94 (Dublin Ireland, July 1994), 272-281.
13. Mynatt, E.D., Adler A., Ito, M. and O’Day, V.L. Design for network communities, in
Proceedings of CHI’97 (Atlanta GA, March 1997), ACM Press, 210-217.
14. Nichols, D.M., Twidale, M.B. and Paice, C.D. Recommendation and usage in the digital
library. Technical Report, Computing Department, Lancaster Univ., UK, 1997. Available at
http://www.comp.lancs.ac.uk/computing/research/cseg/projects/ariadne/docs/recommend.html.
19
15. Pirolli, P. and Card, S. Information foraging in information access environments, in
Proceedings of CHI’95 (Denver CO, May 1995), ACM Press, 51-58.
16. Pitkow, J. and Pirolli, P. Life, death, and lawfulness on the electronic frontier, in Proceedings
of CHI’97 (Atlanta GA, March 1997), ACM Press, 383-390.
17. Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P. and Riedl, J. GroupLens: An open
architecture for collaborative filtering of netnews, in Proceedings of CSCW’94 (Chapel Hill
NC, October 1994), ACM Press, 175-186.
18. Rucker, J. and Polanco, M.J.
Siteseer:
Personalized navigation for the Web.
Communications of the ACM 40, 3 (March 1997), 73-75.
19. Shahabi, C., Zarkesh, A., Adibi, J. and Shah, V. Knowledge Discovery from Users Web-Page
Navigation, in Proceedings of RIDE’97 (Birmingham UK, April 1997). Available at
http://www.usc.edu /dept/cs/technical_ reports.html
20. Shardanand, U. and Maes, P. Social information filtering: Algorithms for automating word of
mouth, in Proceedings of CHI’95 (Denver CO, May 1995), ACM Press, 210-217.
21. Wittenburg, K., Das, D., Hill, W. and Stead, L. Group asynchronous browsing on the World
Wide Web, in Proceedings of WWW’95 (Boston MA, December 1995). Available at
http://www.w3.org /Conferences/WW4/Papers /98/.
Acknowledgements
The authors are very grateful to our early users and reviewers of the Knowledge Pump, notably
Stefania Castellani, Christer Fernström, Antonietta Grasso, David Hull, Jean-Luc Meunier, Dave
Snowdon, and Ian Soboroff. We are also indebted to François Pacull for lending his not
inconsiderable technical expertise. Lastly, we thank Daniele Pagani and Dan Holtshouse for
inspiring this work.
20