Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Human Computation and Crowdsourcing Uichin Lee May 8, 2011 Content Networking • Human intelligence: – Distributed human computation, crowdsourcing • Mobile device intelligence: – Sensing (camera, GPS) • Network intelligence: – Internet, mobile networks (w/ advanced services) • Application intelligence: – Agent, processing, mining Human Device Content Fixed access Crowd Network Internet Smart home/office Content provider Application Radio access On the move Networking Applications Contents • Overview • Genres of distributed human computation – Games with a purpose, mechanized labor, wisdom of crowds, crowdsourcing, dual-purpose work, grand search, human-based genetic algorithms, knowledge collection from volunteer contributors • Dimensions – Motivation, quality, human skill, participation time, cognitive load • Analyzing Amazon Mechanical Turk Marketplace Overview • Distributed human computation (DHC) aims at solving rich computation problems through collaboration between humans and computers – Particularly, in some problem domains where humans could be much better than machines – Examples: artificial intelligence, natural language processing, and computer visions • Well artificial intelligence has been trying hard to solve these problems using machines – But its quality may be not satisfactory.. • DHC offers the possibility of combining humans and computers: faster than individual human efforts, and quality is as good as human efforts (or even better) Taxonomy of Distributed Human Computation Alexander J. Quinn, Benjamin B. Bederson, 2009 Overview • How? The system has global knowledge of the problem and forms small sub-problems that take advantage of humans’ special abilities – Delegating sub-problems to a large number of people connected via Internet (could be geographically dispersed) • Examples: – Searching for a person in a large number of satellite photos covering thousands of square miles of ocean (e.g., Jim Gray) – Image labeling (e.g., ESP game) – Human-computer interaction, cryptograph, business, genetic algorithms, etc. (and many others!) DHC Genres • • • • • • • • • Games with a purpose Mechanized labor Wisdom of crowds Crowdsourcing Dual-purpose work Grand search Human-based genetic algorithms Knowledge collection from volunteer contributors People sensing Games with a purpose • Game that requires the player to perform some computation to gain points or to succeed • Defining factor: people are motivated by the fun of a game Mechanized labor • Crowdsourcing with monetary rewards • Amazon’s Mechanical Turk, ChaCha (paid per micro task) – Cf) Mturk was lunched in 2005 by the needs of Amazon; they wanted to eliminate all the duplicate pages as much as possible which couldn’t be done using automated algorithms Wisdom of crowds • Crowd intelligence: very difficult when done individually, but very easy when aggregated (asking opinions of crowds) • Example services: online polling, prediction markets Crowdsourcing • Coined by Jeff Howe in a Wired magazine article – Displacement of usual internal labor by soliciting unpaid help from the general public – Motivated by curiosity or serendipity while browsing the web (e.g., online product reviews) • Examples: – Question answering services: Naver KiN, Yahoo Answer, Askville, Aardvark – Stardust@Home (finding elusive particles from space images) Dual-purpose work • Translating a computation into an activity that many people were already doing frequently Grand search • Finding a solution (instead of aggregation) • Examples: finding an image that contains something (e.g., search for a missing person, or for elusive particles as in Stardust@home) Human-based genetic algorithms • Humans contribute solutions to problems and subsequent participants by performing functions such as initialization, mutation, and recombinant crossover • Defining factor is that solutions consists of a sequence of small parts and that they evolve in a way that is controlled by human evaluation Knowledge collection from volunteer contributors • Aims to advance artificial intelligence research by using humans to build large databases of common sense facts – E.g., “people cannot brush their hair with a table” • Common methods have been using data mining, e.g., Cyc • Human-based methods could help, e.g., FACTory, Verbosity, 1001 Paraphrases, etc. People sensing • Community awareness (participatory sensing) • Emergency/rescue operations Geiger counter; 방사능측정기 Safecast.org seeks to aggregate worldwide sensor information Pictures from http://news.cnet.com/japan-radiation-monitoring-goes-crowd-open-source/8301-17938_105-20060639-1.html Dimensions • Motivation – Pay (e.g., Mturk), altruism (e.g., Naver KiN, Wikipedia), fun (e.g., games), implicit (e.g., embedded in regular activities) • Quality – Mechanisms: forced agreement (e.g., games), economic models (when money is involved), defensive task design, redundancy – Checking: statistical, redundant work, multilevel review, expert review, forced agreement, automatic check, reputation systems • Aggregation – Knowledge base, statistical, grand search, unit tasks (ChaCha, Mturk) • Human skill – Language understanding, vision, communications, reasoning, common knowledge/sense • Participation time: <2min, 2-10min, >10min • Cognitive load (affecting contributor’s willingness to help) Analyzing the Amazon Mechanical Turk Marketplace Panagiotis G. Ipeirotis (NYU) AMT Screenshot Screenshot AMT questions • Who are the workers that complete these tasks? • What type of tasks can be completed in the marketplace? • How much does it cost? • How fast can I get results back? • How big is the AMT market place? Demographics • Countries: 46.80% US, India: 34%, Misc: 19.2% (from 66 different countries) 1 http://behind-the-enemy-lines.blogspot.com/2010/03/new-demographics-of-mechanical-turk.html Demographics • Why do you complete tasks in Mechanical Turk? Please check any of the following that applies: – [1] Fruitful way to spend free time and get some cash (e.g., instead of watching TV) – [2] I find the tasks to be fun – [3] To kill time – [4] For "primary" income purposes (e.g., gas, bills, groceries, credit cards) – [5] For "secondary" income purposes, pocket change (for hobbies, gadgets, going out) – [6] I am currently unemployed, or have only a part time job 1 2 3 Demographics • Why do you complete tasks in Mechanical Turk? Please check any of the following that applies: – [1] Fruitful way to spend free time and get some cash (e.g., instead of watching TV) – [2] I find the tasks to be fun – [3] To kill time – [4] For "primary" income purposes (e.g., gas, bills, groceries, credit cards) – [5] For "secondary" income purposes, pocket change (for hobbies, gadgets, going out) – [6] I am currently unemployed, or have only a part time job 4 5 6 Type of tasks Requester distribution Price distribution Keywords vs. Ranks Posting vs. completion rate Completion time