Download Slide 1

Data Warehousing from the Web
Chris Fernandes ([email protected]) and Michael Whalen ([email protected])
Department of Computer Science, Union College, Schenectady, NY, 12308
Summary
Procedure
Results
Warehouse projects provide students
with robust capstone experiences and
produce interesting results.
Meeting scheduler
Course analysis
Enrollment data
Room availability
Introduction
Data warehousing is the process of collecting information
from various repositories and combining it into a single
structured repository that can be queried for new
information such as performance trends. Many Internet
web sites contain useful but unstructured data, thus making
them ideal for student projects related to data warehouses.
We describe one such project, developed from the
registration web pages at Union College, which allows
faculty and students to get on-line access to course
enrollment trends, classroom availability, and other
pertinent information. The results of this project were so
successful in the type of information that could be obtained
that the administration became concerned about student
privacy issues, allowing the student to extend his work into
the area of warehouse security.
HTML data is automatically parsed nightly
for content and transferred to the warehouse
backend, called SCOUR (Search Contents
Of Union’s Registry)
Course
Number
Name Homepage
Term …
…
…
…
…
Query results can be displayed in a variety
of formats, including histograms and
importing results to a spreadsheet.
…
Event
BeginTime EndTime Location
…
…
…
…
…
User
Fname
Lname AdvisorID
…
…
…
…
…
Unlike traditional databases, the SCOUR
warehouse contains historical and
summarized data for use in statistical queries
Conclusions
1. Data warehousing projects yield many pedagogical
benefits. They allow students to build bridges
between many areas of computer science including
• database and data warehouse theory
• GUI design
• security and authorization
• interface usability
• privacy and ethics
2. Projects can be diverse. Raw data abounds on the
Web in many fields.
3. Projects can have flexible scope to meet time
constraints. One can easily extend a warehouse
with security to restrict access to sensitive queries.
UNION registrar web pages
contain semistructured data
Dynamic web-based front ends were created for
a variety of queries. It was essential to maintain
ease of use by non-technical operators.