Download Human Computation and Crowdsroucing

Human Computation and
Crowdsourcing
Uichin Lee
May 8, 2011
Content Networking
•
Human intelligence:
– Distributed human computation, crowdsourcing
•
Mobile device intelligence:
– Sensing (camera, GPS)
•
Network intelligence:
– Internet, mobile networks (w/ advanced services)
•
Application intelligence:
– Agent, processing, mining
Human
Device
Content
Fixed
access
Crowd
Network
Internet
Smart
home/office
Content provider
Application
Radio
access
On the move
Networking
Applications
Contents
• Overview
• Genres of distributed human computation
– Games with a purpose, mechanized labor, wisdom of
crowds, crowdsourcing, dual-purpose work, grand
search, human-based genetic algorithms, knowledge
collection from volunteer contributors
• Dimensions
– Motivation, quality, human skill, participation time,
cognitive load
• Analyzing Amazon Mechanical Turk Marketplace
Overview
• Distributed human computation (DHC) aims at solving rich
computation problems through collaboration between
humans and computers
– Particularly, in some problem domains where humans could be
much better than machines
– Examples: artificial intelligence, natural language processing,
and computer visions
• Well artificial intelligence has been trying hard to solve
these problems using machines
– But its quality may be not satisfactory..
• DHC offers the possibility of combining humans and
computers: faster than individual human efforts, and
quality is as good as human efforts (or even better)
Taxonomy of Distributed Human Computation
Alexander J. Quinn, Benjamin B. Bederson, 2009
Overview
• How? The system has global knowledge of the problem
and forms small sub-problems that take advantage of
humans’ special abilities
– Delegating sub-problems to a large number of people
connected via Internet (could be geographically dispersed)
• Examples:
– Searching for a person in a large number of satellite photos
covering thousands of square miles of ocean (e.g., Jim
Gray)
– Image labeling (e.g., ESP game)
– Human-computer interaction, cryptograph, business,
genetic algorithms, etc. (and many others!)
DHC Genres
•
•
•
•
•
•
•
•
•
Games with a purpose
Mechanized labor
Wisdom of crowds
Crowdsourcing
Dual-purpose work
Grand search
Human-based genetic algorithms
Knowledge collection from volunteer contributors
People sensing
Games with a purpose
• Game that requires the player to perform some
computation to gain points or to succeed
• Defining factor: people are motivated by the fun of a game
Mechanized labor
• Crowdsourcing with monetary rewards
• Amazon’s Mechanical Turk, ChaCha (paid per micro task)
– Cf) Mturk was lunched in 2005 by the needs of Amazon; they
wanted to eliminate all the duplicate pages as much as possible
which couldn’t be done using automated algorithms
Wisdom of crowds
• Crowd intelligence: very difficult when done individually,
but very easy when aggregated (asking opinions of crowds)
• Example services: online polling, prediction markets
Crowdsourcing
• Coined by Jeff Howe in a Wired magazine article
– Displacement of usual internal labor by soliciting unpaid help from the general
public
– Motivated by curiosity or serendipity while browsing the web (e.g., online
product reviews)
• Examples:
– Question answering services: Naver KiN, Yahoo Answer, Askville, Aardvark
– Stardust@Home (finding elusive particles from space images)
Dual-purpose work
• Translating a computation into an activity that many
people were already doing frequently
Grand search
• Finding a solution (instead of aggregation)
• Examples: finding an image that contains
something (e.g., search for a missing person, or
for elusive particles as in Stardust@home)
Human-based genetic algorithms
• Humans contribute solutions to problems and
subsequent participants by performing
functions such as initialization, mutation, and
recombinant crossover
• Defining factor is that solutions consists of a
sequence of small parts and that they evolve
in a way that is controlled by human
evaluation
Knowledge collection from
volunteer contributors
• Aims to advance artificial intelligence
research by using humans to build
large databases of common sense
facts
– E.g., “people cannot brush their hair with
a table”
• Common methods have been using
data mining, e.g., Cyc
• Human-based methods could help,
e.g., FACTory, Verbosity, 1001
Paraphrases, etc.
People sensing
• Community awareness (participatory sensing)
• Emergency/rescue operations
Geiger counter;
방사능측정기
Safecast.org seeks to aggregate worldwide sensor information
Pictures from http://news.cnet.com/japan-radiation-monitoring-goes-crowd-open-source/8301-17938_105-20060639-1.html
Dimensions
• Motivation
– Pay (e.g., Mturk), altruism (e.g., Naver KiN, Wikipedia), fun (e.g., games),
implicit (e.g., embedded in regular activities)
• Quality
– Mechanisms: forced agreement (e.g., games), economic models (when money
is involved), defensive task design, redundancy
– Checking: statistical, redundant work, multilevel review, expert review, forced
agreement, automatic check, reputation systems
• Aggregation
– Knowledge base, statistical, grand search, unit tasks (ChaCha, Mturk)
• Human skill
– Language understanding, vision, communications, reasoning, common
knowledge/sense
• Participation time: <2min, 2-10min, >10min
• Cognitive load (affecting contributor’s willingness to help)
Analyzing the Amazon
Mechanical Turk Marketplace
Panagiotis G. Ipeirotis (NYU)
AMT Screenshot
Screenshot
AMT questions
• Who are the workers that complete these
tasks?
• What type of tasks can be completed in the
marketplace?
• How much does it cost?
• How fast can I get results back?
• How big is the AMT market place?
Demographics
• Countries: 46.80% US, India: 34%, Misc: 19.2%
(from 66 different countries)
1
http://behind-the-enemy-lines.blogspot.com/2010/03/new-demographics-of-mechanical-turk.html
Demographics
• Why do you complete tasks in Mechanical Turk? Please check any of the
following that applies:
– [1] Fruitful way to spend free time and get some cash (e.g., instead of
watching TV)
– [2] I find the tasks to be fun
– [3] To kill time
– [4] For "primary" income purposes (e.g., gas, bills, groceries, credit cards)
– [5] For "secondary" income purposes, pocket change (for hobbies, gadgets,
going out)
– [6] I am currently unemployed, or have only a part time job
1
2
3
Demographics
• Why do you complete tasks in Mechanical Turk? Please check any of the
following that applies:
– [1] Fruitful way to spend free time and get some cash (e.g., instead of watching
TV)
– [2] I find the tasks to be fun
– [3] To kill time
– [4] For "primary" income purposes (e.g., gas, bills, groceries, credit cards)
– [5] For "secondary" income purposes, pocket change (for hobbies, gadgets,
going out)
– [6] I am currently unemployed, or have only a part time job
4
5
6
Type of tasks
Requester distribution
Price distribution
Keywords vs. Ranks
Posting vs. completion rate
Completion time