Download 4020-Lecture5_6

ITEC 4020A:Internet Client Server Systems
Professor: Marin Litoiu
Lectures 5&6: Data Persistence
© Marin Litoiu, York University, Canada
This lecture uses captions from Dennis et el., System Analysis and Design with
UML 2.0
Content
 Data Layer Design
 Relational Databases
 Referential Integrity
 Indexing
 Accessing the databases: SQL and JDBC
 Optimizing databases
 Normalization
 Denormalization
 Case Study
 Summary
© Marin Litoiu, York University, Canada
Data Layer Design
3
Objectives
Choose object-persistence format to support the
system
Problem domain drives storage design
Design of Data Storage
Must optimize processing efficiency
Data access and manipulation
Separate application from storage format
Handle all communication with the database
4
Object Persistence Formats




Files (Sequential and Random)
Relational databases
Object-relational databases
Object-oriented databases
5
Relational Databases
 Collection of tables
 Comprised of fields that define entities
 Primary key has unique values in each row of a table
 Foreign key is primary key of another table
 Tables related to each other
 Primary key field of a table is a field of another table and called a
foreign key
 Relationship established by a foreign key of one table connecting to
the primary key of another table
6
Database Management System (DBMS)





Software that creates and manipulates a database
RDBMS is a DBMS for a relational database
RDBMS usually support Referential Integrity
Commercial DBMSs: Oracle, DB2, MSSQL
Non-commercial, open source: MySQL, PostgreSQL
7
Referential Integrity
 The idea of ensuring that values linking the tables together
through the primary and foreign keys are valid and correctly
synchronized.
 Example:
 Cust. ID is a primary key for the customer table
 Cust. ID is a foreign key for the order table
 A violation of referential integrity would happen if an order was entered
in the order table for a Cust. ID that had not been entered into the
customer table first
 An RDBMS prevents such a record from being entered
8
Example of Referential Integrity
9
Indexing
 An index in data storage is like an index in the back of a
textbook;
 it is a mini table that contains values from one or more
columns in a table and the location of the values within the
table.
 A query can use an index to find the locations of only those
records that are included in the query answer, and
 a table can have an unlimited number of indexes but too
many can add overhead
10
Indexing Example
11
Designing data access
 Design data access software
 Prevent data management functionality from creeping into the problem
domain classes
 SQL, JDBC
12
Structured Query Language(SQL) - Overview
 SELECT
SELECT field1, field2,...
FROM table1, table2,...
WHERE table_i.field_j condition (constant, t_k.f_l)
 Conditions:
<, >, =, . . .
LIKE
 ORDER BY field
 Criteria
 Joining tables
• WHERE t_m.f_n = t_p.f_q
• Must use fully qualified names
SQL 2/2
 INSERT
INSERT INTO table (field_1, field_2, . . . )
VALUES (value_1, value_2, . . . )
 UPDATE
UPDATE table
SET field_1 = value_1, . . .
 DELETE
DELETE FROM table
WHERE criteria
SQL Example
create database mycommunity;
use mycommunity;
create table mailing_list ( email varchar(100) not null primary key, name
varchar(100) );
insert into mailing_list (name, email) values (‘Joe Doe',‘[email protected]');
insert into mailing_list (name, email) values (‘Mike
Smith',‘[email protected]');
SELECT * FROM mailing_list;
delete from mailing_list where email = ‘[email protected]';
delete from mailing_list;
drop table mailing_list;
15
Manipulating/querying databases with JDBC
Steps:
Load JDBC driver (is database specific)
Connect to db
Create a statement
Execute query
Retrieve result set metadata
Retrieve result set data
Close statement
Close connection
JDBC Sample: Load the Driver, Open a Connection (1)
 Import the classes that come with the standard jdk
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.sql.Statement;
import java.sql.ResultSet;
The connection is opened
with the help of Driver
Manager
 Load the driver, which is DBMS vendor specific:
Class.forName("com.mysql.jdbc.Driver");
 Open a connection to the database by using the above driver
Connection conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/mycommunity", "root", "4020");
Database name:
mycommunity
User id: root
Pswd 4020
© Marin Litoiu, York University, Canada
JDBC Sample: Create a Statement, Execute a Query(2)
 Create a statement
Statement stmt = conn.createStatement();
Execute a statement
 Use the method executeUpdate(String query) of the Statement
class
 If I want to create a table named “Users” with 3 columns: email,
firstname, lastname the SQL statement is
CREATE TABLE Users(email char(45) not null, "firstname
char(15),lastname char(15));
 And the JDBC statement is
stmt.executeUpdate ( "CREATE TABLE Users(email char(45) not null, firstname
char(15), lastname char(15)));
© Marin Litoiu, York University, Canada
JDBC Sample: Insert(3)
The whole SQL statement has to be provided
as a String to executeUpdate()
Execute an Insert
If I want to insert a row, the SQL statement is
INSERT INTO USERS (email,firstname, lastName)
VALUES (‘[email protected]‘, ‘Joe’, ‘Doe’);
The JDBC statement is
Statement stmt = conn.createStatement();
stmt.executeUpdate("INSERT INTO Users(email, firstname, lastname)”
+” VALUES(‘Joe@yahoo',“
+ " ‘Joe’, ‘Doe’ )");
Since the string can become too long, you can
use + to concatenate java strings( especially
when you split the string on multiple lines
© Marin Litoiu, York University, Canada
JDBC Sample: Retrieve the data (4)
 Data is retrieved in an object of type ResultSet
 Ex: SELECT
ResultSet rs = stmt.executeQuery("SELECT * FROM Users");
 Result Set is a collection of rows
 To get the next row, use next()
while(rs.next()){
 Now get all columns of a row with getString or getDate or getDouble etc..
 You have to know the “type” of the column: string, int, etc…
String em= rs.getString("email");
String fname = rs.getString("firstname");
String lname = rs.getString("lastname");
System.out.println("\t" + em+ ",\t" + fname+ "\t = " + lname );
}//end while loop
© Marin Litoiu, York University, Canada
JDBC full example: MySQLProgram.java(1)
/*
* this program creates a table "users", inserts a row, retrieves the row and print it to the console
* you have to provide the database name, the user id and the password
* the database resides on sit.yourku.ca
*/
public class MySQLProgram {
public static void main(String[] args) {
try {
// load the driver
Class.forName("com.mysql.jdbc.Driver");
} catch (Exception ex) {
System.out.println("Class not found exception: " + ex.getMessage());
}
Connection conn = null;
try {
// get the connection
conn = DriverManager.getConnection( "jdbc:mysql://sit.yorku.ca:3306/" + args[0], args[1],args[2]);
//see next page…..
© Marin Litoiu, York University, Canada
JDBC full example: MySQLProgram.java(2)
//declare the statement and the resultser
Statement stmt = null;
ResultSet rs = null;
// create a statement
stmt = conn.createStatement();
// executes the statement --creates the table
stmt.executeUpdate("CREATE TABLE Users(email char(45) not null,"
+ "firstname char(15)," + "lastname char(15))");
// insert a row
stmt.executeUpdate("INSERT INTO Users(email, " + "firstname, "
+ "lastname) VALUES('" + "moo@yahoo" + "'," + "'" + "Stan"
+ "','" + "Theman" + "' )");
© Marin Litoiu, York University, Canada
JDBC full example: MySQLProgram.java(3)
System.out.println("Display all results:");
rs = stmt.executeQuery("SELECT * FROM Users");
// ResultSet is a collection that can be traversed with rs.next
/*
* A ResultSet object maintains a cursor pointing to its current row
* of data. Initially the cursor is positioned before the first row.
* The next method moves the cursor to the next row, and because it
* returns false when there are no more rows in the ResultSet
* object, it can be used in a while loop to iterate through the
* result set.
*/
while (rs.next()) {
String em = rs.getString("email");
String fname = rs.getString("firstname");
String lname = rs.getString("lastname");
System.out.println("\t" + em + ",\t" + fname + "\t = " + lname);
}// end for loop
© Marin Litoiu, York University, Canada
JDBC full example: MySQLProgram.java(4)
//catch the exceptions..
} catch (SQLException ex) {
// handle any errors
System.out.println("SQLException: " + ex.getMessage());
System.out.println("SQLState: " + ex.getSQLState());
System.out.println("VendorError: " + ex.getErrorCode());
}
}
}
© Marin Litoiu, York University, Canada
MVC Example, with JSP, Servlets, Java Beans and JDBC
 No the bean Name saves its data into the database
 MyProxy class implements all the operations that have to do with the
database
Browser
2
3
UserRegistration.java
<<Servlet>>
4
5
1
ShowName.j
sp
UserReg.jsp
ShowAllUsers.java
<<Servlet>
© Marin Litoiu, York University, Canada
Name.java
<<bean>>
3’
MySQLProxy.java
<<JDBC proxy>>
3’’
“mycommunity”
<<MySql
database>>
MySQLProxy (1)
public class MySQLProxy {
static String driver = "com.mysql.jdbc.Driver";
// you have to change the database name, most likely your database on a
// production server
// has a different name.
static String jdbcURL = "jdbc:mysql://localhost:3306/mycommunity";
// you have to provide your user name and pswd
static String user = "root";
static String pswd = "4020";
© Marin Litoiu, York University, Canada
MySQLProxy.insert ( String, String, String)
// the method takes the e-mail, the first name and the second name and
// inserts a row in the table "users“ the table has 3 columns, e-mail, firstname, lastname
public static void insertUser(String email, String firstName, String lastName) {
/*
* stm.executeUpdate.. Executes the given SQL statement, which may
* be an INSERT, UPDATE, or DELETE statement or an SQL statement
* that returns nothing, such as an SQL DDL statement.
*
* Returns: either
* (1) the row count for SQL Data Manipulation Language (DML)
* statements or (2) 0 for SQL statements that return nothing
* Throws: SQLException - if a database access error occurs, this
* method is called on a closed Statement or the given SQL statement
* produces a ResultSet object
*/
stmt.executeUpdate("INSERT INTO Users(email, " + "firstname, “ + "lastname) VALUES('" + email + "',"
+ "'" + firstName” + "','" + lastName + "' )");
© Marin Litoiu, York University, Canada
MySQLProxy.getAllUsers()
/*
* returns all users in the database mycommunity ; it returns the raw data into a
* ReusultSet object A ResultSet object maintains a cursor pointing to its
* current row of data. Initially the cursor is positioned before the first
* row. The next method moves the cursor to the next row, and because it
* returns false when there are no more rows in the ResultSet object, it can
* be used in a while loop to iterate through the result set.
*
*/
public static ResultSet getAllUsers() {
// load the driver, just in case this is the first method executed
try {
Class.forName(driver);
ResultSet rs = stmt.executeQuery("SELECT * FROM Users");
} catch (SQLException ex) {}
// return the result
return rs;
}
© Marin Litoiu, York University, Canada
MVC Example, with JSP, Servlets, Java Beans and JDBC
Browser
2
3
UserRegistration.java
<<Servlet>>
4
5
1
ShowName.j
sp
UserReg.jsp
ShowAllUsers.java
<<Servlet>
© Marin Litoiu, York University, Canada
Name.java
<<bean>>
3’
MySQLProxy.java
<<JDBC proxy>>
3’’
“mycommunity”
<<MySql
database>>
Name.java
/this bean holds the values of firsName,lastName and e-mail
//it has setters and getters to set and get the data
public class Name {
private String firstName="first name missing;
private String lastName="last name missing";
private String email="email is missing";
//setters and getters
public String getEmail() {
return email;
}
//this method inserts the bean data into the database
public void saveInDB(){
//the assumption is that the e-mail is unique;
MySQLProxy.insertUser(email, firstName, lastName);
}
© Marin Litoiu, York University, Canada
UserReg.jsp
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
<title>Insert title here</title>
</head>
<body>
<body>
<p><br>
Greetings</p>
<form ACTION="UserRegistration" METHOD="POST">
<b>First Name</b> <input type="text" name="firstName" size="20"> <br>
<b>Last Name</b> <input type="text" name="lastName" size="20"> <br>
<b>email</b> <input type="text" name="email" size="20"> <br>
<b>Password</b> <input type="password" name="email" size="20"> <br>
<input type="submit" name="submit" value="Submit"></form>
</body>
</html>
© Marin Litoiu, York University, Canada
This is normal HTML taging
This is a form, when submitted to the
server, the server has to invoke the
object “UserRegisrtation” with the
method POST
Form input data: each entry has a type
( string, password) and a name( need to
remember those when implementing
the method doPost of the servlet
UserRegistration)
Submit button
The UserRegistration servlet ( controller)
protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException,
IOException {
request object was received from the
// get the session object, by calling a request method
server, it should have a session object
HttpSession session= request.getSession();
//once I have the session, I want to know if the session has an object Name --- named "NameBean"--Name nb= (Name)session.getAttribute("NameBean");
if (nameBean==null) {
// if it does not, I need to create an object Name
nameBean=new Name();
}
//now get the attributes that came from client ( see the UserReg.jsp for the form and its text fields... )
String fn= request.getParameter("firstName");
nameBean.setFirstName(fn);
//get the last name...
String ln= request.getParameter("lastName");
nameBean.setLastName(ln);
© Marin Litoiu, York University, Canada
….The UserRegistration servlet ( controller)
//tell the bean to save the data in the database
nameBean.saveInDB();
…….
//store the bean in the session and request objects so other servlets or JSP can use them
//( see ShowNames.jsp)
session.setAttribute("NameBean", nameBean);
//pass the control to a JSP...
getServletConfig(). getServletContext().getRequestDispatcher(
"/ShowNames.jsp").forward(request, response);
}
© Marin Litoiu, York University, Canada
And finally….ShowNames.jsp
</head>
<body>
<jsp:useBean id="NameBean"
type="users.Name"
scope="session"></jsp:useBean>
Retrieves the bean “NameBean” from
the session object
<jsp:getProperty name="NameBean"
property="firstName"/>
Displays the field “firstName” of the bean
<jsp:getProperty name="NameBean"
property="lastName"/>
<p>
</p>
</body>
</html>
Displays the fields “lastName” of the
bean
© Marin Litoiu, York University, Canada
MVC Example, with JSP, Servlets, Java Beans and JDBC
Browser
2
3
UserRegistration.java
<<Servlet>>
4
5
1
ShowName.j
sp
UserReg.jsp
ShowAllUsers.java
<<Servlet>
© Marin Litoiu, York University, Canada
Name.java
<<bean>>
3’
MySQLProxy.java
<<JDBC proxy>>
3’’
“mycommunity”
<<MySql
database>>
ShowAllUsers servlet, doGet method
protected void doGet(HttpServletRequest request,
HttpServletResponse response) throws ServletException, IOException {
// in this doGet method, request object is not used, but response is used
// invoke MySQLProxy to get all rows
ResultSet rs = MySQLProxy.getAllUsers();
// get a printer object from the response object
PrintWriter out = response.getWriter();
// start writing html tags to the printer
out.println("<html>");
out.println("<head>");
out.println("<title>All registered users</title>");
out.println("</head>");
out.println("<body>");
//see next slide
© Marin Litoiu, York University, Canada
….ShowAllusers…
//…..
try {
while (rs.next()) { // iterate through each row of the ResultSet
String em = rs.getString("email");
String fname = rs.getString("firstname");
String lname = rs.getString("lastname");
out.println("\t" + em + ",\t" + fname + "\t " + lname + "<br>");
}// end for loop
} catch (Exception ex) {
}
;
// final tags
out.println("</body>");
out.println("</html>");
}
© Marin Litoiu, York University, Canada
Configuring a web application: web.xml file
<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/webapp_2_5.xsd" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" id="WebApp_ID" version="2.5">
<display-name>ServletsJSPsMySQL</display-name>
<servlet>
<description></description>
<display-name>UserRegistration</display-name>
<servlet-name>UserRegistration</servlet-name>
<servlet-class>users.UserRegistration</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>UserRegistration</servlet-name>
<url-pattern>/UserRegistration</url-pattern>
</servlet-mapping>
<!—COMMENT: you have to configure all servlets here
</web-app>
© Marin Litoiu, York University, Canada
Discussion about User registration and management
 What other information do you store about the user?
 One table or many tables for users (fat versus skiny tables)
 What type of data do you store ( varchar, blobls, etc..)?
 What do you encript? How?
 UserGroups
 How do you represent the relationship between users and groups
© Marin Litoiu, York University, Canada
Page flow design
© Marin Litoiu, York University, Canada
User authorization and access control
© Marin Litoiu, York University, Canada
Optimize RDBMS Object Storage: Normalization
 No redundant data
 Wastes space
 Allow more room for error
 Few null values in tables
 Difficult to interpret
42
Example of Non-normalized Data
Redundant data
NULL columns
43
Normalization
44
Normalization Example
Original Model
45
3NF Normalized Model
46
Denormalization
Problem
To access data in multiple tables, the tables must be
joined
This can result in many database operations and lead to
huge tables and slow processing
Solution
Denormalization – Adds data from one table to another in
order to speed processing and eliminate a join operation
Example: Add customer last name to order table to avoid
joining order to customer to get just last name
47
Example
48
More on data management in client server systems...
 Review of online community requirements
 Content management issues
 Workflows
© Marin Litoiu, York University, Canada
Case Study- Content management in online communities
 Requirements:
 magnet content authored by experts
 means of collaboration
 powerful facilities for browsing and searching both magnet content and
contributed content
 means of delegation of moderation
 means of identifying members who do not behave and/or excluding them from
the community
 means of software extension by community members themselves
© Marin Litoiu, York University, Canada
Case Study: Issues
 Issues
 Storing and retrieving data from a content repository
 News, comments on news, articles, comments on articles, questions, answers
 Users: guest, registered users, administrators
 Support the workflow of people contributing to the website
 Authors: author articles, comments, etc...
 Publishers: decide what content goes live
 Information designers: navigation flow: what links on each page, how the
information is organized
 Graphic designers: design GUI artefacts :logos, drawings
 Programmers: programs, templates
© Marin Litoiu, York University, Canada
Case Study- A flat data table
 Do we need a table for each type of content (news, articles, comments)?
 What data type do we store the content in?
content_raw
content_id integer primary key
content_type varchar(100) not null ----this can be news, articles, comments...
refers_to references content_raw------if is a comment, is comment to what?
creation_user not null references users—who created this?
creation_date not null date
release_time date
expiration_time date
zip_code varchar(5) -- some of our content is geographically specific
language char(2) -------------------------------can be english, spanish, etc..
mime_type varchar(100) not null -- could be text/html or text/plain or some sort of XML
one_line_summary varchar(200) not null -- will hold the title in most cases
body blob------------------------------------------the content of the article/news/comment...
© Marin Litoiu, York University, Canada
Supporting multiple versions...Add a new column “version”
Content_id
Content_type
ZIP
Creation_date
Body
1
news
1010
10/10/08
Dow Jones loses 300
points..
2
comment
1111
10/10/08
The news 1 is wrong
1
news
1010
10/10/08
Dow Jones loses 800
points..
Primary key
1st NF
Content_id
Content_type
ZIP
Creation_date
Body
version
1
news
1010
10/10/08
Dow Jones loses
300 points..
1
2
comment
1111
10/10/08
The news 1 is
wrong
1
1
news
1010
Dow Jones loses
800 points..
2
Primary key
© Marin Litoiu, York University, Canada
2nd NF –Creating new tables
 In 2nd NF, each table should talk about one thing: in previous table we
mix content & versions in the same
 We need another table, to store each version
create table content_raw (
content_id integer primary key,
content_type varchar(100) not null,
refers_to references content_raw,
...
)
create table content_versions (
version_id integer primary key,
content_id not null references content_raw,
version_date date not null,
language char(2) ,
body blob
)
© Marin Litoiu, York University, Canada
2nd NF ( “each table should talk about one thing”)
Content_id
Content_type
ZIP
Creation_date
1
news
1010
10/10/08
2
comment
1111
10/10/08
Content_id
version_date
Body
version
Current
_version
1
10/10/08
Dow Jones
loses 300
points..
1
false
2
10/10/08
The news 1 is
wrong
1
True
1
10/10/08
Dow Jones
loses 800
points..
2
true
Primary key
© Marin Litoiu, York University, Canada
3rd NF ( “each non-key column should depend on all key
columns”)
 In 3rd NF, all columns should depend on the whole key
 Current_version in previous slide, does depends on the key but on the
editorial_status
Content_id
Content_type
ZIP
Creation_date
1
news
1010
10/10/08
2
comment
1111
10/10/08
Content_id
Creation_date
Body
version
1
10/10/08
Dow Jones
loses 300
points..
1
2
10/10/08
The news 1 is
wrong
1
1
10/10/08
Dow Jones
loses 800
points..
2
Primary key
© Marin Litoiu, York University, Canada
Content_id
version
2
1
1
2
Primary key
Content workflow
 How does the information flow on the web site?
 A field editorial_status keeps track of each version status
content_versions
version_id integer primary key
content_type varchar(100) not null ----this can be news, articles, comments...
------
body blob
-----------------------------------the content of the article/news/comment...
editorial_status varchar(30) check (editorial_status in ('submitted','rejected','approved','expired'))--------workflow support
© Marin Litoiu, York University, Canada
Designing the content workflow (activity diagrams)
Author
System
Editor
Publisher
roles
Logs in
Uploads article
Updates table c
”content_raw”
Author notified
Updates tables
E-mail author
Edit the article
“approves”
Publishes article
activities
Control and data flow
© Marin Litoiu, York University, Canada
Summary
 Choose an object-persistent format
 Files (sequential or Random Access)
 Databases (RDBMS, ORDBMS, OODBMS)
 Optimizing object storage and performance
 Normalization
 Denormalization, Indexes
59