Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Microsoft Jet Database Engine wikipedia , lookup
Ingres (database) wikipedia , lookup
Microsoft SQL Server wikipedia , lookup
Entity–attribute–value model wikipedia , lookup
Extensible Storage Engine wikipedia , lookup
Functional Database Model wikipedia , lookup
Clusterpoint wikipedia , lookup
Open Database Connectivity wikipedia , lookup
ITEC 4020A:Internet Client Server Systems Professor: Marin Litoiu Lectures 5&6: Data Persistence © Marin Litoiu, York University, Canada This lecture uses captions from Dennis et el., System Analysis and Design with UML 2.0 Content Data Layer Design Relational Databases Referential Integrity Indexing Accessing the databases: SQL and JDBC Optimizing databases Normalization Denormalization Case Study Summary © Marin Litoiu, York University, Canada Data Layer Design 3 Objectives Choose object-persistence format to support the system Problem domain drives storage design Design of Data Storage Must optimize processing efficiency Data access and manipulation Separate application from storage format Handle all communication with the database 4 Object Persistence Formats Files (Sequential and Random) Relational databases Object-relational databases Object-oriented databases 5 Relational Databases Collection of tables Comprised of fields that define entities Primary key has unique values in each row of a table Foreign key is primary key of another table Tables related to each other Primary key field of a table is a field of another table and called a foreign key Relationship established by a foreign key of one table connecting to the primary key of another table 6 Database Management System (DBMS) Software that creates and manipulates a database RDBMS is a DBMS for a relational database RDBMS usually support Referential Integrity Commercial DBMSs: Oracle, DB2, MSSQL Non-commercial, open source: MySQL, PostgreSQL 7 Referential Integrity The idea of ensuring that values linking the tables together through the primary and foreign keys are valid and correctly synchronized. Example: Cust. ID is a primary key for the customer table Cust. ID is a foreign key for the order table A violation of referential integrity would happen if an order was entered in the order table for a Cust. ID that had not been entered into the customer table first An RDBMS prevents such a record from being entered 8 Example of Referential Integrity 9 Indexing An index in data storage is like an index in the back of a textbook; it is a mini table that contains values from one or more columns in a table and the location of the values within the table. A query can use an index to find the locations of only those records that are included in the query answer, and a table can have an unlimited number of indexes but too many can add overhead 10 Indexing Example 11 Designing data access Design data access software Prevent data management functionality from creeping into the problem domain classes SQL, JDBC 12 Structured Query Language(SQL) - Overview SELECT SELECT field1, field2,... FROM table1, table2,... WHERE table_i.field_j condition (constant, t_k.f_l) Conditions: <, >, =, . . . LIKE ORDER BY field Criteria Joining tables • WHERE t_m.f_n = t_p.f_q • Must use fully qualified names SQL 2/2 INSERT INSERT INTO table (field_1, field_2, . . . ) VALUES (value_1, value_2, . . . ) UPDATE UPDATE table SET field_1 = value_1, . . . DELETE DELETE FROM table WHERE criteria SQL Example create database mycommunity; use mycommunity; create table mailing_list ( email varchar(100) not null primary key, name varchar(100) ); insert into mailing_list (name, email) values (‘Joe Doe',‘[email protected]'); insert into mailing_list (name, email) values (‘Mike Smith',‘[email protected]'); SELECT * FROM mailing_list; delete from mailing_list where email = ‘[email protected]'; delete from mailing_list; drop table mailing_list; 15 Manipulating/querying databases with JDBC Steps: Load JDBC driver (is database specific) Connect to db Create a statement Execute query Retrieve result set metadata Retrieve result set data Close statement Close connection JDBC Sample: Load the Driver, Open a Connection (1) Import the classes that come with the standard jdk import java.sql.Connection; import java.sql.DriverManager; import java.sql.SQLException; import java.sql.Statement; import java.sql.ResultSet; The connection is opened with the help of Driver Manager Load the driver, which is DBMS vendor specific: Class.forName("com.mysql.jdbc.Driver"); Open a connection to the database by using the above driver Connection conn = DriverManager.getConnection("jdbc:mysql://localhost:3306/mycommunity", "root", "4020"); Database name: mycommunity User id: root Pswd 4020 © Marin Litoiu, York University, Canada JDBC Sample: Create a Statement, Execute a Query(2) Create a statement Statement stmt = conn.createStatement(); Execute a statement Use the method executeUpdate(String query) of the Statement class If I want to create a table named “Users” with 3 columns: email, firstname, lastname the SQL statement is CREATE TABLE Users(email char(45) not null, "firstname char(15),lastname char(15)); And the JDBC statement is stmt.executeUpdate ( "CREATE TABLE Users(email char(45) not null, firstname char(15), lastname char(15))); © Marin Litoiu, York University, Canada JDBC Sample: Insert(3) The whole SQL statement has to be provided as a String to executeUpdate() Execute an Insert If I want to insert a row, the SQL statement is INSERT INTO USERS (email,firstname, lastName) VALUES (‘[email protected]‘, ‘Joe’, ‘Doe’); The JDBC statement is Statement stmt = conn.createStatement(); stmt.executeUpdate("INSERT INTO Users(email, firstname, lastname)” +” VALUES(‘Joe@yahoo',“ + " ‘Joe’, ‘Doe’ )"); Since the string can become too long, you can use + to concatenate java strings( especially when you split the string on multiple lines © Marin Litoiu, York University, Canada JDBC Sample: Retrieve the data (4) Data is retrieved in an object of type ResultSet Ex: SELECT ResultSet rs = stmt.executeQuery("SELECT * FROM Users"); Result Set is a collection of rows To get the next row, use next() while(rs.next()){ Now get all columns of a row with getString or getDate or getDouble etc.. You have to know the “type” of the column: string, int, etc… String em= rs.getString("email"); String fname = rs.getString("firstname"); String lname = rs.getString("lastname"); System.out.println("\t" + em+ ",\t" + fname+ "\t = " + lname ); }//end while loop © Marin Litoiu, York University, Canada JDBC full example: MySQLProgram.java(1) /* * this program creates a table "users", inserts a row, retrieves the row and print it to the console * you have to provide the database name, the user id and the password * the database resides on sit.yourku.ca */ public class MySQLProgram { public static void main(String[] args) { try { // load the driver Class.forName("com.mysql.jdbc.Driver"); } catch (Exception ex) { System.out.println("Class not found exception: " + ex.getMessage()); } Connection conn = null; try { // get the connection conn = DriverManager.getConnection( "jdbc:mysql://sit.yorku.ca:3306/" + args[0], args[1],args[2]); //see next page….. © Marin Litoiu, York University, Canada JDBC full example: MySQLProgram.java(2) //declare the statement and the resultser Statement stmt = null; ResultSet rs = null; // create a statement stmt = conn.createStatement(); // executes the statement --creates the table stmt.executeUpdate("CREATE TABLE Users(email char(45) not null," + "firstname char(15)," + "lastname char(15))"); // insert a row stmt.executeUpdate("INSERT INTO Users(email, " + "firstname, " + "lastname) VALUES('" + "moo@yahoo" + "'," + "'" + "Stan" + "','" + "Theman" + "' )"); © Marin Litoiu, York University, Canada JDBC full example: MySQLProgram.java(3) System.out.println("Display all results:"); rs = stmt.executeQuery("SELECT * FROM Users"); // ResultSet is a collection that can be traversed with rs.next /* * A ResultSet object maintains a cursor pointing to its current row * of data. Initially the cursor is positioned before the first row. * The next method moves the cursor to the next row, and because it * returns false when there are no more rows in the ResultSet * object, it can be used in a while loop to iterate through the * result set. */ while (rs.next()) { String em = rs.getString("email"); String fname = rs.getString("firstname"); String lname = rs.getString("lastname"); System.out.println("\t" + em + ",\t" + fname + "\t = " + lname); }// end for loop © Marin Litoiu, York University, Canada JDBC full example: MySQLProgram.java(4) //catch the exceptions.. } catch (SQLException ex) { // handle any errors System.out.println("SQLException: " + ex.getMessage()); System.out.println("SQLState: " + ex.getSQLState()); System.out.println("VendorError: " + ex.getErrorCode()); } } } © Marin Litoiu, York University, Canada MVC Example, with JSP, Servlets, Java Beans and JDBC No the bean Name saves its data into the database MyProxy class implements all the operations that have to do with the database Browser 2 3 UserRegistration.java <<Servlet>> 4 5 1 ShowName.j sp UserReg.jsp ShowAllUsers.java <<Servlet> © Marin Litoiu, York University, Canada Name.java <<bean>> 3’ MySQLProxy.java <<JDBC proxy>> 3’’ “mycommunity” <<MySql database>> MySQLProxy (1) public class MySQLProxy { static String driver = "com.mysql.jdbc.Driver"; // you have to change the database name, most likely your database on a // production server // has a different name. static String jdbcURL = "jdbc:mysql://localhost:3306/mycommunity"; // you have to provide your user name and pswd static String user = "root"; static String pswd = "4020"; © Marin Litoiu, York University, Canada MySQLProxy.insert ( String, String, String) // the method takes the e-mail, the first name and the second name and // inserts a row in the table "users“ the table has 3 columns, e-mail, firstname, lastname public static void insertUser(String email, String firstName, String lastName) { /* * stm.executeUpdate.. Executes the given SQL statement, which may * be an INSERT, UPDATE, or DELETE statement or an SQL statement * that returns nothing, such as an SQL DDL statement. * * Returns: either * (1) the row count for SQL Data Manipulation Language (DML) * statements or (2) 0 for SQL statements that return nothing * Throws: SQLException - if a database access error occurs, this * method is called on a closed Statement or the given SQL statement * produces a ResultSet object */ stmt.executeUpdate("INSERT INTO Users(email, " + "firstname, “ + "lastname) VALUES('" + email + "'," + "'" + firstName” + "','" + lastName + "' )"); © Marin Litoiu, York University, Canada MySQLProxy.getAllUsers() /* * returns all users in the database mycommunity ; it returns the raw data into a * ReusultSet object A ResultSet object maintains a cursor pointing to its * current row of data. Initially the cursor is positioned before the first * row. The next method moves the cursor to the next row, and because it * returns false when there are no more rows in the ResultSet object, it can * be used in a while loop to iterate through the result set. * */ public static ResultSet getAllUsers() { // load the driver, just in case this is the first method executed try { Class.forName(driver); ResultSet rs = stmt.executeQuery("SELECT * FROM Users"); } catch (SQLException ex) {} // return the result return rs; } © Marin Litoiu, York University, Canada MVC Example, with JSP, Servlets, Java Beans and JDBC Browser 2 3 UserRegistration.java <<Servlet>> 4 5 1 ShowName.j sp UserReg.jsp ShowAllUsers.java <<Servlet> © Marin Litoiu, York University, Canada Name.java <<bean>> 3’ MySQLProxy.java <<JDBC proxy>> 3’’ “mycommunity” <<MySql database>> Name.java /this bean holds the values of firsName,lastName and e-mail //it has setters and getters to set and get the data public class Name { private String firstName="first name missing; private String lastName="last name missing"; private String email="email is missing"; //setters and getters public String getEmail() { return email; } //this method inserts the bean data into the database public void saveInDB(){ //the assumption is that the e-mail is unique; MySQLProxy.insertUser(email, firstName, lastName); } © Marin Litoiu, York University, Canada UserReg.jsp <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"> <title>Insert title here</title> </head> <body> <body> <p><br> Greetings</p> <form ACTION="UserRegistration" METHOD="POST"> <b>First Name</b> <input type="text" name="firstName" size="20"> <br> <b>Last Name</b> <input type="text" name="lastName" size="20"> <br> <b>email</b> <input type="text" name="email" size="20"> <br> <b>Password</b> <input type="password" name="email" size="20"> <br> <input type="submit" name="submit" value="Submit"></form> </body> </html> © Marin Litoiu, York University, Canada This is normal HTML taging This is a form, when submitted to the server, the server has to invoke the object “UserRegisrtation” with the method POST Form input data: each entry has a type ( string, password) and a name( need to remember those when implementing the method doPost of the servlet UserRegistration) Submit button The UserRegistration servlet ( controller) protected void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { request object was received from the // get the session object, by calling a request method server, it should have a session object HttpSession session= request.getSession(); //once I have the session, I want to know if the session has an object Name --- named "NameBean"--Name nb= (Name)session.getAttribute("NameBean"); if (nameBean==null) { // if it does not, I need to create an object Name nameBean=new Name(); } //now get the attributes that came from client ( see the UserReg.jsp for the form and its text fields... ) String fn= request.getParameter("firstName"); nameBean.setFirstName(fn); //get the last name... String ln= request.getParameter("lastName"); nameBean.setLastName(ln); © Marin Litoiu, York University, Canada ….The UserRegistration servlet ( controller) //tell the bean to save the data in the database nameBean.saveInDB(); ……. //store the bean in the session and request objects so other servlets or JSP can use them //( see ShowNames.jsp) session.setAttribute("NameBean", nameBean); //pass the control to a JSP... getServletConfig(). getServletContext().getRequestDispatcher( "/ShowNames.jsp").forward(request, response); } © Marin Litoiu, York University, Canada And finally….ShowNames.jsp </head> <body> <jsp:useBean id="NameBean" type="users.Name" scope="session"></jsp:useBean> Retrieves the bean “NameBean” from the session object <jsp:getProperty name="NameBean" property="firstName"/> Displays the field “firstName” of the bean <jsp:getProperty name="NameBean" property="lastName"/> <p> </p> </body> </html> Displays the fields “lastName” of the bean © Marin Litoiu, York University, Canada MVC Example, with JSP, Servlets, Java Beans and JDBC Browser 2 3 UserRegistration.java <<Servlet>> 4 5 1 ShowName.j sp UserReg.jsp ShowAllUsers.java <<Servlet> © Marin Litoiu, York University, Canada Name.java <<bean>> 3’ MySQLProxy.java <<JDBC proxy>> 3’’ “mycommunity” <<MySql database>> ShowAllUsers servlet, doGet method protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { // in this doGet method, request object is not used, but response is used // invoke MySQLProxy to get all rows ResultSet rs = MySQLProxy.getAllUsers(); // get a printer object from the response object PrintWriter out = response.getWriter(); // start writing html tags to the printer out.println("<html>"); out.println("<head>"); out.println("<title>All registered users</title>"); out.println("</head>"); out.println("<body>"); //see next slide © Marin Litoiu, York University, Canada ….ShowAllusers… //….. try { while (rs.next()) { // iterate through each row of the ResultSet String em = rs.getString("email"); String fname = rs.getString("firstname"); String lname = rs.getString("lastname"); out.println("\t" + em + ",\t" + fname + "\t " + lname + "<br>"); }// end for loop } catch (Exception ex) { } ; // final tags out.println("</body>"); out.println("</html>"); } © Marin Litoiu, York University, Canada Configuring a web application: web.xml file <?xml version="1.0" encoding="UTF-8"?> <web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/webapp_2_5.xsd" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" id="WebApp_ID" version="2.5"> <display-name>ServletsJSPsMySQL</display-name> <servlet> <description></description> <display-name>UserRegistration</display-name> <servlet-name>UserRegistration</servlet-name> <servlet-class>users.UserRegistration</servlet-class> </servlet> <servlet-mapping> <servlet-name>UserRegistration</servlet-name> <url-pattern>/UserRegistration</url-pattern> </servlet-mapping> <!—COMMENT: you have to configure all servlets here </web-app> © Marin Litoiu, York University, Canada Discussion about User registration and management What other information do you store about the user? One table or many tables for users (fat versus skiny tables) What type of data do you store ( varchar, blobls, etc..)? What do you encript? How? UserGroups How do you represent the relationship between users and groups © Marin Litoiu, York University, Canada Page flow design © Marin Litoiu, York University, Canada User authorization and access control © Marin Litoiu, York University, Canada Optimize RDBMS Object Storage: Normalization No redundant data Wastes space Allow more room for error Few null values in tables Difficult to interpret 42 Example of Non-normalized Data Redundant data NULL columns 43 Normalization 44 Normalization Example Original Model 45 3NF Normalized Model 46 Denormalization Problem To access data in multiple tables, the tables must be joined This can result in many database operations and lead to huge tables and slow processing Solution Denormalization – Adds data from one table to another in order to speed processing and eliminate a join operation Example: Add customer last name to order table to avoid joining order to customer to get just last name 47 Example 48 More on data management in client server systems... Review of online community requirements Content management issues Workflows © Marin Litoiu, York University, Canada Case Study- Content management in online communities Requirements: magnet content authored by experts means of collaboration powerful facilities for browsing and searching both magnet content and contributed content means of delegation of moderation means of identifying members who do not behave and/or excluding them from the community means of software extension by community members themselves © Marin Litoiu, York University, Canada Case Study: Issues Issues Storing and retrieving data from a content repository News, comments on news, articles, comments on articles, questions, answers Users: guest, registered users, administrators Support the workflow of people contributing to the website Authors: author articles, comments, etc... Publishers: decide what content goes live Information designers: navigation flow: what links on each page, how the information is organized Graphic designers: design GUI artefacts :logos, drawings Programmers: programs, templates © Marin Litoiu, York University, Canada Case Study- A flat data table Do we need a table for each type of content (news, articles, comments)? What data type do we store the content in? content_raw content_id integer primary key content_type varchar(100) not null ----this can be news, articles, comments... refers_to references content_raw------if is a comment, is comment to what? creation_user not null references users—who created this? creation_date not null date release_time date expiration_time date zip_code varchar(5) -- some of our content is geographically specific language char(2) -------------------------------can be english, spanish, etc.. mime_type varchar(100) not null -- could be text/html or text/plain or some sort of XML one_line_summary varchar(200) not null -- will hold the title in most cases body blob------------------------------------------the content of the article/news/comment... © Marin Litoiu, York University, Canada Supporting multiple versions...Add a new column “version” Content_id Content_type ZIP Creation_date Body 1 news 1010 10/10/08 Dow Jones loses 300 points.. 2 comment 1111 10/10/08 The news 1 is wrong 1 news 1010 10/10/08 Dow Jones loses 800 points.. Primary key 1st NF Content_id Content_type ZIP Creation_date Body version 1 news 1010 10/10/08 Dow Jones loses 300 points.. 1 2 comment 1111 10/10/08 The news 1 is wrong 1 1 news 1010 Dow Jones loses 800 points.. 2 Primary key © Marin Litoiu, York University, Canada 2nd NF –Creating new tables In 2nd NF, each table should talk about one thing: in previous table we mix content & versions in the same We need another table, to store each version create table content_raw ( content_id integer primary key, content_type varchar(100) not null, refers_to references content_raw, ... ) create table content_versions ( version_id integer primary key, content_id not null references content_raw, version_date date not null, language char(2) , body blob ) © Marin Litoiu, York University, Canada 2nd NF ( “each table should talk about one thing”) Content_id Content_type ZIP Creation_date 1 news 1010 10/10/08 2 comment 1111 10/10/08 Content_id version_date Body version Current _version 1 10/10/08 Dow Jones loses 300 points.. 1 false 2 10/10/08 The news 1 is wrong 1 True 1 10/10/08 Dow Jones loses 800 points.. 2 true Primary key © Marin Litoiu, York University, Canada 3rd NF ( “each non-key column should depend on all key columns”) In 3rd NF, all columns should depend on the whole key Current_version in previous slide, does depends on the key but on the editorial_status Content_id Content_type ZIP Creation_date 1 news 1010 10/10/08 2 comment 1111 10/10/08 Content_id Creation_date Body version 1 10/10/08 Dow Jones loses 300 points.. 1 2 10/10/08 The news 1 is wrong 1 1 10/10/08 Dow Jones loses 800 points.. 2 Primary key © Marin Litoiu, York University, Canada Content_id version 2 1 1 2 Primary key Content workflow How does the information flow on the web site? A field editorial_status keeps track of each version status content_versions version_id integer primary key content_type varchar(100) not null ----this can be news, articles, comments... ------ body blob -----------------------------------the content of the article/news/comment... editorial_status varchar(30) check (editorial_status in ('submitted','rejected','approved','expired'))--------workflow support © Marin Litoiu, York University, Canada Designing the content workflow (activity diagrams) Author System Editor Publisher roles Logs in Uploads article Updates table c ”content_raw” Author notified Updates tables E-mail author Edit the article “approves” Publishes article activities Control and data flow © Marin Litoiu, York University, Canada Summary Choose an object-persistent format Files (sequential or Random Access) Databases (RDBMS, ORDBMS, OODBMS) Optimizing object storage and performance Normalization Denormalization, Indexes 59