Download Special Interest Activity - School of Information Technology

Special Interest Activity
Trio – A System for Integrated Management of Data,
Uncertainty, and Lineage
ITK478
Yan Cui
Agenda









Understand Trio new database system
System requirements of installing Trio system
Source codes/packages for Trio system
Procedure of Trio system installation
Configuration of Trio System
Experiment Trio DBMS Using TrioExplorer and
TrioPlus (demo)
Trio Query Language (TriQL) Structure (demo)
Advantage and Disadvantage of using Trio DBMS
Documenting Trio system report/bugs
Understand Trio new database system

Basic concept of new Trio DBMS
◦ Trio is a new kind of database system (DBMS), which was developed by
Stanford University Lab at Dec, 2006. It is based on an extended
relational model called Uncertainty Lineage Database (ULDB) [1], and
also supports Trio’s query language Called TriQL [1]. This new database
system technology handles structured data, uncertainty of data, and data
lineage together in a fully integrated manner.
Trio System architecture (components)
 Uncertainty-Lineage Database (ULDB)
 TriQL: The Trio Query Language

Trio System architecture

Four primary components of Trio DBMS
◦
◦
◦
◦

command-line client (TrioPlus)
TrioExplorer
Trio API and translator (Python)
standard relational DBMS (PostgreSQL)
Uncertainty-Lineage of Database
◦
◦
◦
◦
Encoded data table
Lineage table
Trio metadata
Trio Stored Procedures
Uncertainty-Lineage Database (ULDB)





Alternatives
‘?’ (Maybe) Annotations
Numerical Confidences
Lineage
Sample: Drives (person, color, car) and Saw (witness,
color, car) uncertainty tables with/no with confidence
Uncertainty-Lineage Database (ULDB)

Alternatives
◦ Definition: Alternatives are presenting
uncertainty about the contents of a tuple [2]
◦ ‘||’ annotation
◦ Drives(person, color, car) and Saw (witness,
color, car) uncertainty tables
◦ ‘Select * from Drives”
Uncertainty-Lineage Database (ULDB)

‘?’ (Maybe) Annotations
◦ Definition: ‘?’ annotation present the existence
of a tuple on the x-tuple, also called maybe xtuple [2]
◦ Drives(person, color, car) and Saw (witness,
color, car) uncertainty tables
◦ ‘select * from Drives’
Uncertainty-Lineage Database (ULDB)

Numerical Confidences - Numerical
confidence also was considered as probability
[2].
 Drives(person, color, car) and Saw (witness,
color, car) uncertainty tables with confidence
 ‘select * from Drives’
Uncertainty-Lineage Database (ULDB)

Lineage
◦ “recorded at the granularity of tuple
alternatives: Lineage connects as x-tuple
alternative to other x-tuple alternative.” in [2]
◦ Drives(person, color, car) and Saw (witness,
color, car) uncertainty tables with confidence
◦ ‘select person from Drives’
TriQL:The Trio Query Language

Two major parts - built-in functions and
predicates for querying confidence values
and lineage, and regular SQL syntax [2]
System requirements of installing Trio
system



operating systems - as Linux, Mac OS X, and
Win-32 (XP,Vista, and 32-bit Server)
PostgreSQL database (version 8.2.5, 8.1.10,
8.0.14, and 7.4.18) - Linux and Win32
Python API - windows, Linux/Unix, Mac OS
X, OS/2, and Amiga
Source codes/packages for Trio system













Listing source codes:
Python 2.4 can be downloaded from http://www.python.org/ .
Easy_install can be downloaded from
http://peak.telecommunity.com/DevCenter/EasyInstall and the file called
ez_setup.py.
Readline 1.7.win32 can be downloaded from http://www.python.org/ .
ctypes-1.0.2.win32-py2.4 can be download from http://www.python.org/ .
PostgreSQL 8.1 can be downloaded from http://www.postgresql.org/ .
Graphviz 2.14 is the only version compatible with Trio API. It is available in
http://infolab.stanford.edu/trio/code/graphviz-2.14.1.exe .
PyGreSQL can be downloaded from http://www.pygresql.org/ .
Pylons 0.9.5 can be downloaded from http://pylonshq.com/ .
PLY 2.2 can be downloaded from http://www.dabeaz.com/ply/ .
PyParsing can be downloaded from http://pyparsing.wikispaces.com/ .
PyDot can be downloaded from
http://code.google.com/p/pydot/downloads/list .
Trio API 1.0 can be downloaded from
http://infolab.stanford.edu/~theobald/sources/TRIO.zip .
Procedure of Trio system installation

Python
◦ Download Python 2.4 windows version (python-2.4.4.msi).
◦ Install Python in C:/Python directory
◦ Set path=c:/Python24; in environment variables
Readline - download Readline-1.7.win32-py2.4.exe and install into Python
directory
 Ctypes – download ctypes-1.0.2.win32-py2.4.exe and install into Python
directory
 PostgreSQL

◦ Download PostgreSQL 8.1 windows version (postgresql-8.1.msi )
◦ Install PostgreSQL 8.1 as following:









Language selection (Fig 10) – English
Introduction screen (Fig 11) – next
Welcome message and instructions (Fig 12) – next
Feature selection (Fig 13) – next
Service installation (Fig 14) – check install a service, input account name ‘postgres’ and password.
Initdb (Fig 15) – check initialize database cluster, superusername and password.
Procedural languages (Fig 16) – Check PL/pgsql only.
Contrib modules (Fig 17) – check Admin81 only.
Next in (Fig 18, 19, 20, 21) to complete the installation.
◦ Set path C:\Program Files\PostgreSQL\8.1\bin; after completed the installation.
Procedure of Trio system installation (cont)








Graphviz - Download Graphviz 2.14 version and install to your
workstation and set path C:\PROGRA~1\ATT\Graphviz\bin; in
environment variables after completed the installation
Easy_install - Download ez_setup.py in C:/ directory
PyGreSQL – In command line, cd\ to c: directory, and run python
ez_setup.py PyGreSQL to install components.
Pylons – In command line, cd\ to c: directory, and run python ez_setup.py
Pylons==0.9.5 to install Pylons. Set path c:\python24\Scripts in
environment variables.
PLY – In command line, cd\ to c: directory, and run python ez_setup.py
Ply==2.2 or easy_install Ply==2.2.
PyParsing – In command line, cd\ to c: directory, and run python
ez_setup.py PyParsing.
PyDot – download the source from website. Access to folder in command
line, and then install manually by running ‘python setup.py install’.
Trio API
◦ Download source code in any directory
◦ Copy Trio-1.0\spi\triospi_win32.dll to PostgreSQL’s lib directory and renamed as
triospi.dll
Procedure of Trio system installation (cont)
PostgreSQL installation
Configuration of Trio System



Windows superuser authentication to access
PostgreSQL
TrioExplorer
TrioPlus
Configuration of Trio System

Windows superuser authentication to access PostgreSQL
◦ Double click in start->all programs->PostgreSQL 8.1->pgadmin III
◦ Right click on Login Roles to create new login role
 Role name ‘myname’ (as same as windows login account)
 Set password (Password can be any)
 Check all role privileges and click ok.
◦ Right click on Database to create new database
 Database name ‘myname’ (as same as username)
 Owner is ‘myname’ and click ok
◦ Initialize Trio schema information
 In Trio-1.0\setup, open setup.py with notepad to comment out the
last three codes and put the following. (After complete the
initialization, please change back to original).
os.system("psql %s %s < setup.sql" % (pgdbname, username))
os.system("psql %s %s < setup_triospi.sql" % (pgdbname, username))
os.system("psql %s %s < trio_get_conf.sql" % (pgdbname, username))
 Save the file, and at the command line, cd \Trio-1.0\setup, and run
‘python setup.py myname myname’ in command line.
 Provide password to create schema and done.

PostgreSQL 8.1->pgadmin III
Configuration of Trio System

TrioExplorer
◦ Make sure PostgreSQL is working.
◦ Running TrioExplore – Ensure path ‘c:\python24\Scripts;’ in
environment variables. And double click ‘start_te_server.bat’ under
Trio-1.0\explorer.
◦ At the command line, you are now prompted for an admin user
login to PostgreSQL, which should have been created along with
your PostgreSQL installation and which will be used by TrioExplorer
to create new user roles and database instances.
◦ TrioExplorer should now be reachable from your browser using
http://localhost:8080/. For new users can now press ‘Create a new
user’ and create their own Trio login and database instances, which
are then managed by the PostgreSQL server.
Configuration of Trio System

TrioPlus
◦ Create new PostgreSQL user role and database instance
 Run ‘createuser demo’
 Run ‘createdb demo’, the name must be the same as username
◦ Initialize Trio schema information for new user by access as same
as windows superuser authentication to access PostgreSQL. Use
TrioExplorer will be easily just press ‘Create new role’ in Web.
◦ Connect to new Trio database using the command line clients by
running ‘python trioplus.py –u demo –d demo –p’
Demo


Experiment Trio DBMS Using TrioExplorer
And TrioPlus
Trio Query Language (TriQL) Structure
(Drop index/table, Create Trio table/index,
TriQL language) from
http://infolab.stanford.edu/~widom/triql.html
Drives(person, color, car) and Saw (witness,
color, car)
Table of TriQL contents
ULDBs
Uncertian attibutes, maybe annotations and
confidence values
SQL over ULDBs
Selection, projection, join, subqueries, duplicateelimination, grouping and aggregation, aggregate
variants, set operators, order by
Flatten and GroupAlts
Flatten is used to turn tuples with alternative
values into regular tuples, while GroupAlts is
used to create or restructure alternative values
Horizontal subqeries: The [ ] construct
[ ] in the where clause, [ ] whith joins, Syntactics
shortcuts in [ ], [ ] in the select clause, [ ] with
Self-Joins
Builit-in Functions Conf() and Maybe()
Result confidences
Multi-table conf()
Result confidence evaluation, uniform and scaled
result confidences, On-Demand confidence
computation
Built-in Predicate Lineage()
The Lineage() predicate lets queries filter joined
tuples based on whether they are related via
lineage
Options Nolineage, Noconf, and NoMaybe
Indicate lineage, confidence values, and/or?'s
should be omitted from query results
Data modificaiton
Insert statement, delete statement, update
statement
Advantage and Disadvantage

Advantage
◦ Open source and free support for any non-benefit users to
experience new Trio DBMS
◦ Advanced components in relational DBMS
◦ Computing confidences
◦ Efficient, Convenient, safe, Multi-User storage of and access to,
Massive, Persistent

Disadvantage
◦ Time cost for query
◦ Dependency
◦ On development stage
Advantage and Disadvantage

Disadvantage
◦ Time cost for query
Using ‘SELECT attr-list FROM X1, X2, ..., Xn WHERE predicate’ as a query
example in [6] for a comparison between relational database and ULDB.
Over standard relational database:
 For each tuple in cross-product of X1, X2, ..., Xn
 Evaluate the predicate
 If true, project attr-list to create result tuple
Over ULDB:
 For each tuple in cross-product of X1, X2, ..., Xn
 Create “super tuple” T from all combinations of alternatives
 Evaluate predicate on each alternative in T ; keep only the true ones
 Project attr-list on each alternative to create result tuple
 Details: ‘?’, lineage, confidences
Documenting Trio system report/bugs




The install instruction in website
http://dbpubs.stanford.edu:8011/doku.php/trio:installation ,
indicated unclearly the version of Graphviz for Trio system. In
Graphviz website http://www.graphviz.org/ only has version 2.16
but not compatable except version 2.14. Graphviz version 2.14 is
available for download in
http://infolab.stanford.edu/trio/code/graphviz-2.14.1.exe .
The windows authentication supperuser needs to be created first
in the PostgreSQL in order to connect to database. After
established the connection, TrioExplorer and TrioPlus can use the
supperuser’s login and password as windows authentiction to
access to database system. However, it doesn’t mention at all in the
installation procedure on how to create this typle of new user. The
only way to solve it is to use PostgreSQL->pgadmin III manually.
After created the supperuser, I have to modify some codes in
setup.py in Trio-1.0->setup directory in order to run ‘python
setup.py –u myname –d myname –p’.
TriQL query statements in
http://infolab.stanford.edu/~widom/triql.html#options, there are
many samples queries not working properly as desired.
References






M. Mutsuzaki, M. Theobald, A. de Keijzer, J. Widom, P. Agrawal, O.
Benjelloun, A. Das Sarma, R. Murthy, and T. Sugihara. Trio-One:
Layering Uncertainty and Lineage on a Conventional DBMS.
Proceedings of the Third Biennial Conference on Innovative Data
Systems Research (CIDR '07), Pacific Grove, California, January
2007. Demonstration description.
O. Benjelloun, A. Das Sarma, C. Hayworth, and J. Widom. An
Introduction to ULDBs and the Trio System. IEEE Data Engineering
Bulletin, Special Issue on Probabilistic Databases, 29(1):5-16, March
2006.
Trio: A System for integrated Management of Data, Uncertainty, and
Lineage. Retrieved on November, 18, 2007 from
http://infolab.stanford.edu/trio/ .
PostgreSQL. Retrieved on November, 20, 2007 from
http://www.postgresql.org/.
Python. Retrieved on November, 15, 2007 from
http://www.python.org/.
Trio: A System for Data, Uncertainty, and Lineage. given by Jennifer
at various venues, 2006-07. Ppt.