Download slides

Building Data Integration
Systems for the Web
Alon Halevy
Google
NSF Information Integration Workshop
April 22, 2010
Without (too much) Loss of Generality
Web

Enterprise, Science projects, …
Information integration ≅ data management
A Few Principles
• Data management “in situ”
– Data meaning is derived from its context
– Manipulate data in its natural location
• Pay-as-you-go data management
– Provide services before modeling is done
– Data can be about any domain
• Collaboration should be built in
– Query answering is only step the first step
Alex Labrinidis
@via Facebook
Structured Data & The Web
Hard to find structured data via search engines
Discover
Requires
infrastructure,
concerns
about losing
control
Data is
embedded in
web page,
behind forms
Publish
Extract
Manage,
Analyze,
Combine
Hard to query, visualize, combine data across organizations
Outline
• Surfacing the Deep Web
• Searching tables on the surface Web
• Fusion Tables: a platform for data
management on the Web.
What is the Deep Web?
• Deep = not accessible through general purpose search
engines
– Major gap in the coverage of search engines.
used cars
store locations
radio stations
patents
recipes
Tree Search
Amish quilts
Parking tickets in India
Horses
Solution Constraints
• Can’t design a solution that requires
domain engineering
– (unless you can make money in that
domain!)
• Boundaries between domains are fuzzy
• Solution needs to be integrated into
general web search
– Can’t assume special query syntax
Surfacing the Deep Web
[Madhavan et al. VLDB 2008]
• Surfacing:
– Find high-quality forms
– Guess good queries to submit
– Put the resulting HTML pages in the index
• ~3M sites, 50 languages, 700 domains.
• 1000 queries per-second get results from the
deep web.
• 400K forms served per day, 800K per week
• Impact mostly on the long and heavy tail of
queries
Deep Web: The Future
• Still an opportunity to go deeper into the
deep web:
– E.g., map the user query into a form
submission.
• Key challenge: given a keyword query,
map it to forms in any domain
• Understanding the meaning of forms is
still hard (e.g. content, geo constraints).
Outline
Surfacing the Deep Web
Searching tables on the surface Web
• Fusion Tables: a platform for data
management on the Web.
Bad table
Vertical Tables
Sub-Header Rows
Winners of the Boston Marathon
(but that’s nowhere in the table)
Schema Ok, but context is subtle
(year = 2006)
WebTables: Exploring the Relational Web
[Cafarella et al., VLDB 2008, WebDB 08]
• In corpus of 14B raw tables, we estimate
154M are “good” relations
– Single-table databases; Schema = attr labels + types
– Largest corpus of databases & schemas we know of
• The Webtables system:
– Recovers good relations from crawl and enables search
– Builds novel apps on the recovered data
(Web-scale) Schema Collection
With 2.6 million schemas you can do some
very interesting things.
Synonym discovery
name
e-mail|email, phone|telephone,
e-mail_address|email_address, date|last_modified
instructor
course-title|title, day|days, course|course-#,
course-name|course-title
elected
candidate|name, presiding-officer|speaker
ab
k|so, h|hits, avg|ba, name|player
sqft
bath|baths, list|list-price, bed|beds, price|rent
“KR”-Based Table Search
[Wu, Madhavan, Miao, Pasca, Shen]
• Ideally, we describe every table:
– Class of entities it contains
– Properties being modeled
– Context, quality, …
• Use Web-extracted knowledge bases
– Extract isa-hierarchy using patterns:
– “cities such as Paris and London”
– “chemical elements including hydrogen
and oxygen”
Step 1: Find “Subject” of Table
Not always the left (or first non-number column)
Step 2: associate classes with subject
Chemical elements
Most of the time, the class labels are not in attribute name
Leveraging Web-extracted
Ontologies
• Given a query, e.g., (country, GDP)
– Rank tables about countries that have
GDP somewhere in the schema.
– Very high precision (~90%)
• Next challenge: understand binary
properties and binary relationships.
• Domain specialization:
– System should improve if given ontologies
in a particular domain.
Combine Search, Extraction,
Cleaning and Integration
[Cafarella, Koussainova, H., VLDB 2009],
• Try to create a database of all
“VLDB program committee members”
25
Outline
Surfacing the Deep Web
Searching tables on the surface Web
Fusion Tables: a platform for data
management on the Web.
Data Management for the Web Era
• Integrate seamlessly with the Web:
– Search, maps, …
• Easy to use:
– Much broader user base, pay-as-you-go
– Very simple data integration
• Provide incentives for sharing data
• Facilitate collaboration
Fusion Tables – our current attempt
[Madhavan, Gonzalez, Langen, Shapley, Shen]
Incentive
We store and leverage a large collection of tables.
Incentive, Pay-..-Go
Coffee Production
Coffee Consumption
Seamless integration with other web tools
Toilet heat map…
Database functionality on map
Collaboration
Table Search
Show up in
search results!
Data Integration
Merged Table
Carries attribution from both base tables. Owners maintain
control of their own data.
Fine Grained Discussions
Example Uses of Fusion Tables
•
•
•
•
•
•
•
•
Tracking potholes in Spain
Displaying bike routes (MTBGuru)
State of California statistics
Government data from data.gov
Data about voting locations in the USA
Brazilian beaches
Chicago homicides
Most requested pop songs by year
Conclusions
• Information integration “in situ”
– Blur the boundary between structured and
unstructured data
• Combine search, extraction, cleaning
and integration into a single experience
• Pay-as-you-go: introduce complexity as
needed
– Serve enterprises without IT depth
• OpenII – an open-source platform for
information integration.
References
• Fusion Tables:
– tables.googlelabs.com
– SIGMOD, SOCC, 2010
• Deep-web crawling:
– [Madhavan et al., VLDB 08]
• WebTables:
– [Cafarella et al., VLDB 08]
• Octopus:
– [Cafarella et al., VLDB 09],
– [Elmeleegy et al, VLDB 09]