Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
ICS 224: Database Management Systems Spring 2011 Professor Sharad Mehrotra Information and Computer Science Department University of California, Irvine 1 Course General Info • URL: http://www.ics.uci.edu/~cs224/ – All course info will be posted online • Lecture times: Tue-Thurs 5 – 6.30 • Instructor: Sharad Mehrotra, BH 2082, [email protected] • Office Hours: on request ICS214A Notes 01 2 Prerequisites • Basic Data Management Concepts: – DB design, relational model, SQL, database programming CS 122 or equivalent – Database system implementation Indexing, query optimization, query processing, storage management, etc. ICS 222 or equivalent • Basic Computer Science Concepts: – Depth-first search, directed/undirected graphs, “big O” notation, computational complexity, NP completeness … ICS214A Notes 01 3 Course Requirements • Class Participation: 50% – Attendance, presentations, comments, interaction, enthusiasm, etc. • Class Projects: 50% – Implementation Oriented: Take a idea/topic, identify a project, get it okayed by instructor, develop a demonstration – Survey of an area In depth survey in the style of computing survey articles. Provide your own perspective in a subarea. – MUST commit to project at end of 2nd week. ICS214A Notes 01 4 Class Structure • Each week we will – Pick a topic – identify 1 paper per student/group of 2 students – 2 papers as lead papers for presentation (one for each class), others presented as short presentations • Each week – – – – ICS214A Start with overview Lead paper presentation short presentation of other papers (main idea) Discussions Notes 01 5 This course … Most important ideas in data management (instructor’s pick) But with the eye towards an end application … Sentient spaces ICS214A Notes 01 6 Sentient Spaces … • Spaces in which sensors are used to capture the dynamic evolving state which is then analyzed for implementing adaptations. • Numerous examples … – intelligent transportation systems – reconnaissance – surveillance systems – smart buildings – smart grid ... 7 Example:Smart Video Surveillance Query Query Analysis CS Building in UC Irvine Event Database Semantic Extraction Surveillance Video Database Video collection 8 Implications of Sentient Space focus .. • Class focuses on topics which you might need to know if you wanted to explore application in sentient space … • Projects should target something about sentient spaces … – E.g., data cleaning of sentient data, data model to represent sentient spaces, … ICS214A Notes 01 9 Data Models (2 weeks) – Representing time - TSQL2 – Representing space – Querying streaming data – CQL, ASQL – Semi-structured data –OEM, Lore ICS214A Notes 01 10 New Ideas in Storage & Indexing (2 weeks) • New storage models – Key-Value store – Bigtable – Column Stores • New database system architecture – Data outsourcing – Multitenant databases • New Indexing techniques – Correlation maps ICS214A Notes 01 11 Data Quality (2 weeks) • Data quality issues – Inaccuracy, incompleteness, ambiguity, errors, … • Two aspects: – Techniques to improve quality Exploiting contextual knowledge, issues of efficiency – Techniques to tolerate poor quality of data in applications. ICS214A Notes 01 12 New Computing Architecture (2 weeks) • • • • • • Map Reduce framework Hive Pig latin Join processing HadoopDB Hyrax? ICS214A Notes 01 13 Data Privacy (2 weeks) • Use cases – Data publishing, queries, sharing, data outsourcing. • Diverse criteria – Differential privacy, Anonymity, l-diversity, .. • Mechanisms to implement ICS214A Notes 01 14 A walk down the history of data models … Two papers (MUST READ) •Inclusion of New types in relational databases, Stonebraker •Postgrest Next Generation databsase, Stonebraker. 16 The Paleolithic Period … • There were no general purpose tools for managing large volumes of data… – OS provided resource management – Data was stored in files – Applications performed data management functionalities Fault-tolerance Concurrency control Reliability Optimizations … – Such functionalities had to be re-implemented for each application ICS214A Notes 01 17 The Neolithic Period… • Early file systems evolve into general-purpose data management tools. • DBMS Goals: – Efficiency and scalability (faster than files) – Management of large heterogeneous types of structured data – High reliability – Information sharing (multiple users) • DBMS Users: – E-commerce companies, banks, airlines, transportation companies, corporate databases, government agencies, … – Anyone you can think of! ICS214A Notes 01 18 The Dark Ages …. • Network & hierarchical data models – Resulted in data spaghetti – Applications needed to chase pointers – There was little data abstraction or separation of concerns little difference between physical data representation and logical data representation – optimization was entirely left to application writers – There were no clean data management languages Unless you are a Cobol fan! ICS214A Notes 01 19 The Relational Era.. • Relational model proposed by Codd – Everything is a relation – Query consists of algebraic composition of a few powerful operators – Equivalent to a first-order relational calculus • Primary features – Simple clean data representation solid mathematical basis – data abstraction Users did not need to be concerned about how data is stored physically – simple declarative query language User’s specify what to compute not how to do it. – ICS214A optimization by the system Notes 01 20 Data Wars (1) • Codasyl versus relational debates began… – Heated arguments during early SIGMODS – Codasyl: relational model is too simple, applications built using it will never scale in performance. – Relational: network/hierarchical models have no formal basis, are too complex, and unmanageable as application complexity increases. • Relational model found many supporters – Specially at universities – Its simplicity was enticing ICS214A Notes 01 21 Data Wars (2) • Many projects started off trying to implement a relational DBMS – System R @ IBM Almaden – Ingres @ Berkeley – These early systems led to the technologies that drive modern data management • Early prototypes became products – DB2 & Ingres • Principle designers from both the System R teams & Ingres left to start companies – Oracle, Sybase • Early relational companies went door to door converting industry to the relational model – Industry got hooked on to the simplicity of writing complex applications in relational model – Boeing among the first converts ICS214A Notes 01 22 Pointer’s Strike Back… Application data structures Relational Copy and Transparent translation ODBMS representation data transfer RDBMS • Complex objects in emerging DBMS applications cannot be effectively represented as records in relational model. • Representing information in RDBMSs requires complex and inefficient conversion into and from the relational model to the application programming language • ODBMSs provide a direct representation of objects to DBMSs overcoming the impedance mismatch problem ICS214A Notes 01 23 Object Model • Object: – observable entity in the world being modeled – similar to concept to entity in the E/R model • An object consists of: – attributes: properties built in from primitive types – relationships: properties whose type is a reference to some other object or a collection of references – methods: functions that may be applied to the object. ICS214A Notes 01 24 Object Oriented Databases • Evolved as persistent Object Oriented Programming Languages: • Start with an OO language (e.g., C++, Java, SMALLTALK) which has a rich type system • Add persistence to the objects in programming language where persistent objects stored in databases ICS214A Notes 01 29 Persistent Programming Languages • Single programming language for application and data management ii2 a[ j] a[ j 1] 3 EmployeeSpousebenefit_levelbenefitlevel1 • Update to persistent variable results in automatic update to database. • Persistent data could be types such as sets and lists and arrays. • Application can follow pointers (OID) to navigate through data. ICS214A Notes 01 30 Persistence • Objects created may have different lifetimes: – transient: allocated memory managed by the programming language run-time system. E.g., local variables in procedures have a lifetime of a procedure execution global variables have a lifetime of a program execution – persistent: allocated memory and stored managed by ODBMS runtime system. • Classes are declared to be persistence-capable or transient. • Different languages have different mechanisms to make objects persistent: – creation time: Object declared persistent at creation time (e.g., in C++ binding) (class must be persistent-capable) – persistence by reachability: object is persistent if it can be reached from a persistent object (e.g., in Java binding) (class must be persistent-capable). ICS214A Notes 01 31 Persistent Object-Oriented Programming Languages • Persistent objects are stored in the database and accessed from the programming language. • Single programming language for applications as well as data management. – Avoid having to translate data to and from application programming language and DBMS efficient implementation less code – Programmer does not need to write explicit code to fetch data to and from database persistent objects to programmer looks exactly the same as transient objects. System automatically brings the objects to and from memory to storage device. (pointer swizzling). ICS214A Notes 01 32 Approaches To Persistent Programming • Persistent Virtual Memory – disk representation and memory representation of data is identical. – No cost to translate data from one representation to another— efficient! – DB size limited to address space 32bit processor 2^32 byte addressability (4 GBytes) – Differentiating persistent objects and non-persistent objects is difficult. – Difficult to optimize disk layout and locality of access. – Example system using approach: OBJECT STORE. ICS214A Notes 01 33 Approaches To Persistent Programming Languages • Store persistent objects in files – Objects brought to memory on demand. – Implementation of OID complex since pointers do not suffice in general. If object in memory pointer can be used for OID if object on disk a disk address still not good as OID since storage can be reorganized. A separate mechanism needed. Pointer swizzling for efficiency. ICS214A Notes 01 34 Challenges In Building Persistent Languages • Efficient caching of objects in client address space. – Cache coherence. • In OODB data migrates to clients unlike relational client server systems where query migrates to server. • Given a large number of clients each with the cache of objects ensuring consistency of object across multiple clients is a challenge. ICS214A Notes 01 35 Disadvantages of ODBMS Approach • Low protection – since persistent objects manipulated from applications directly, more changes that errors in applications can violate data integrity. • Non-declarative interface: – difficult to optimize queries – difficult to express queries • But ….. – Most ODBMSs offer a declarative query language OQL to overcome the problem. – OQL is very similar to SQL and can be optimized effectively. – OQL can be invoked from inside ODBMS programming language. – Objects can be manipulated both within OQL and programming language without explicitly transferring values between the two languages. – OQL embedding maintains simplicity of ODBMS programming language interface and yet provides declarative access. ICS214A Notes 01 36 The Return of the Relations … POSTGRES • Relational model evolved into ORDBMSs that include “best of” objectoriented concepts • Amongst the first ORDBMS prototype built @ Berkeley POSTGRES commercialized Illustra bought by Informix IUS • Has had major impact on major commercial DBMS which have all migrated to ORDBMS model. • SQL3 supported by modern databases adapted many of the concepts developed in Postgres ICS214A Notes 01 37 POSTGRES — Combinations • Introduced object orientation into relation DBMSs. • Fundamental Concepts. – Each record has an OID. – Access to data though: query language POSTQUEL. navigation through OIDs. – Classes: – Inheritance: – Types: rich set of types available for columns. – Functions: can be called within POSTQUEL. ICS214A Notes 01 38 Classes And Inheritance • Class analogous to relation • User can create new class create Emp (name = c12, salary = float, age = int) • Classes can inherit from others create Salesman (quota = float) inherits Emp • Multiple inheritance permitted. If new class causes ambiguity it is not created. • Classes: – real: base classes or relations – derived: views – version: maintained differentially compared to parent class ICS214A Notes 01 39 Types In POSTGRES • Standard base types – float, int, charac. Strings, etc. – Abstract data type (ADT) facility to create new base types e.g.; create type point (x = int, y = int) create type polygon • ADT’s can be used in class definitions. Create Dept( dname = c10, mgr = c12, floorspace = polygon mailstop = point ) mailstop ICS214A Notes 01 40 Functions In POSTGRES • Three types: (1) C functions (2) Operators (3) POSTQUEL functions • C-functions – any C-function over base types or composite type retrieve (Dept. name) where area (Dept. floorspace) > 500 retrieve (Emp. name) where overpaid (Emp) Function over a class or method ICS214A Notes 01 41 Operators • Arbit C-functions are not optimized by query optimizers. – Special functions - operators can utilize indexes for their evaluation. • Operator: function with 1 or 2 operand Area Greater Than retrieve (Dept. name) where Dept. floor space-AGT “(0,0), (1,1), (0,2)” • Index (e.g.; B-tree) defined properly can be used to speed up evaluation of operators such as AGT. ICS214A Notes 01 42 Other Features Of POSTGRES • Allowed creation of new indices by user. • To an extent pioneered the approach of extensible database technology which is prevalent with vendors today • Supported transitive closure in query. retrieve* into ans (parent. older) from a in answer where. Parent. younger = “John” or parent. younger = a. older • Supported rules ICS214A Notes 01 43 POSTQUEL Functions • Any collection of commands in POSTQUEL. – query = POSTQUEL function. define function high-pay returns Emp as retrieve (Emp. all) where Emp. salary > 50k • POSTQUEL function with parameters. define function Sal-lookup (c12) returns float as retrieve (Emp. Salary) where Emp. name = $1 • Usage of POSTQUEL function retrieve Emp. name where Emp. Salary = Sal-lookup (“Joe”) ICS214A Notes 01 45 Composite Types In POSTGRES • POSTQUEL: – Composite types accessed via path expressions, using nested dot notation. remove (Emp mgr age) where (Emp name = ‘joe’) • Prevents having to specify a join. ICS214A Notes 01 46 Composite Types In POSTGRES • Attributes can have a class name as a type resulting in complex objects with structure. Create Emp ( name = c12, salary = float [c12], age = int, Refers to 0 or more references of Emp class. mgr = Emp, coworker = Emp ) Could be elements of any class • A set type that can hold elements of any class. Add to Emp (hobbies = set) ICS214A Notes 01 47 Types In POSTGRES • Array type (constructor) crate Emp ( name = c12, salary = float [12], age = int Salary for each month. ) • POSTQUEL query retrieve (Emp name) where (Emp salary [4] = 1000) Array in query usage. ICS214A Notes 01 48 Database Technology Matrix Q u r y S u p p o r t Y E S RDBMSs ORDBMSs N O File System OODBMSs Simple Complex Database Types ICS214A Notes 01 49 XML & RDF - the new revolution • Just when relational model had driven out object-oriented database technology, WWW led to the proliferation of semi-structured data. • 2 approaches to supporting XML/RDF – Extend relational technology to support XML/RDF – Native XML databases ICS214A Notes 01 50 Summary of Evolution of Data Model • The Dark Ages: network & heirarchical models • Victory of simplicity and beauty over data spaghetti: The Relational DBMS: • The pointers strike back -- Object-Orientation, OODBMSs • The return of the relations -- ORDBMS -- took the best of the OO concepts and incorporated them in the relational model. • The current and near future -- support for XML & RDF • The final frontier -- anyone’s guess! ICS214A Notes 01 51 Key Data Management Technologies (quick review)… 52 Key Database Technologies • File Management – provides a file abstraction as a collection of records stored in disk • Index Management and Access Methods – implements techniques for associative access to data • Query Optimization and Processing – given a query and data storage structures, determines an efficient strategy to evaluate the query. • Transaction management – ensures consistency of the database in presence of concurrent transactions and various types of failures • Catalog Management – maintains database schema information • Authorization and Integrity Management – tests for integrity constraints and user authorization ICS214A Notes 01 53 Database Management System Architecture Application Queries Schema changes compilers Metadata and data dictionary optimizer evaluator Query processor Buffer manager Transaction Manager File system Storage manager Database and Indices ICS214A Notes 01 54 Storage Media and their Properties • Main Memory – – – – – costs $100/Mbyte -- reduces every year ‘volatile’ -- does not survive system failures random I/O very fast data can be processed by CPU directly capacity limited to orders of magnitude lower than what database needs. • Magnetic Disk – – – – costs $0.50/Mbyte -- reduces each year Non-volatile (except when disk crashes) random I/O not as fast CPU cannot directly process data. Needs to be transferred to main memory • Tape – Cheaper but slower than disks. Sequential I/O devices. Handy for backups, sometimes for archival. ICS214A Notes 01 Databases and Storage Devices • • • • Due to capacity, cost, volatility factors databases traditionally stored in disks. Data brought to main memory for processing from disks There are many ways to interface memory with disk resident data E.g., virtual memory: – VM size limited to max address generated by CPU – Existing VM does not support durability • • File system provides a more powerful mapping between memory and disk storage A bunch of tricks used ensure that high latency of secondary storage does not impact application response time and system throughput – access disks asynchronously with active applications – prefetch data before application needs it – intelligent caching techniques ICS214A Notes 01 56 Functional Abstraction of a Simplistic DBMS beginT SQL SQL endT Access plan optimizer SQL statements beginT SQL SQL endT Query Processor Read write records, scan relations Record-oriented file system Get page containing tuples Buffer manager Basic file system Read/write file pages Hardware ICS214A Notes 01 57 Basic File System • Provides the abstraction of a file where a file is an array of fixed size blocks • Hides the disk geometry -- cylinders, tracks, sectors, slots and other functional components like arms, head, etc. such that the programs do not need to deal with these complexities • Operations supported: – – – – – – – ICS214A create a file delete a file open a file close a file extend a file read (set of) file blocks into buffers in memory write (set of) file blocks Notes 01 58 Basic File System Design Issues • File allocation: how to allocate blocks on disk to a file. – Contiguous allocation: file stored in contiguous disk blocks. Blocks for storing file found using either of best-fit, worst-fit or first-fit policies. +ve: provides fast sequential scan of file -ve: fragmentation, difficult to enlarge files – Linked allocation: file is a linked list of disk blocks +ve: prevents fragmentation, easy to enlarge files -ve: slow for both sequential and random access – Index allocation: file implemented using fixed size blocks pointed to by an index (e.g., B-tree). Popularized by Unix +ve: good random access, easy enlargement, no fragmentation. -ve: poor sequential access performance – Extent based allocation: file is a collection of clusters of consecutive disk blocks (extents) where collection maintained using linked lists or index Most popular approach with vendors. • Free space management: information about which blocks are free ICS214A Notes 01 59 Buffer Management • Makes file pages addressable in memory and coordinates writing of pages to disk with other components to guarantee transactional properties • Acts as a mediator between basic file system and recordoriented file system • Buffer frames maintained in main memory. When a request for file page access comes, check if page in buffer. Else get a free frame and load file page into buffer • Operations Supported: – – – – ICS214A bufferfix bufferunfix get block flush Notes 01 60 • Database Buffer Management Design Issues DBMS buffer manager returns pointer to frame containing data instead of returning copy of requested page to caller. – Efficiency: prevents unnecessary copying of data – Allows sharing of data at finer granularity than a page 2 transactions T1 and T2. T1 and T2 update records r1 and r2 on same page if buffer manager allowed applications to copy data to their address space and rewrite updated versions, updates might be lost • • Database buffer manager participates in protocols to implement transactions (WAL, FL@C, pinning buffer slots) Novel page replacement strategies: – Traditional LRU strategy used in OS works well only under the assumption of locality of reference which may not hold for DBMSs – Since DBMS query language are declarative, system has much more information about reference patterns which it can exploit to improve caching performance of buffer manager ICS214A Notes 01 61 Record-Oriented File System • • Provides the abstraction of a file as a collection of records. Records can be: – – – – • fixed size or variable length short, long, or very long attributes can be fixed length or variable length simple or complex (e.g., containing set valued attributes) Operations supported: – create, delete, open, close, alter, drop – read, insert, update, delete record – scan all records in a file • Issues Involved: – mapping records to pages – file organization: organization of records in a file. Where to insert new records what mechanism can be used to retrieve records ICS214A Notes 01 62 Index Management and Associative Access • Associative access: accessing records based on their attribute values. • Index Files – an index file declared over a (set of) attribute of the data file provides associative access to records in the data file. – Index file contains pointers to disk blocks where the record corresponding to the value appear. • Types of an Index: (let indexing attribute be A) – – – – primary: A is a key and data file stored sorted on A clustered: A is not a key but data file stored sorted on A secondary (key): A is a key but data file not sorted on A secondary (non-key): A is neither a key and nor is data file sorted on A. ICS214A Notes 01 Organization of Index File • B-tree Index: index file is organized as a B-tree – Advantages: Supports range searches efficiently. E.g., retrieve all employees with salary between 100K and 200K – Disadvantages: Guaranteed good storage utilization searching for a given record could take around 3-4 disk I/Os • Hash Index: index file maintained as a hash file. – Advantages: Looking for a specific record very efficient -- 1 disk I/O – Disadvantages: cannot support range searches • Multdimensional Access Methods – modern databases are beginning to support novel data structures like Rtrees, grid files, inverted lists to better serve emerging application requirements ICS214A Notes 01 64 Multidimensional Indexing Motivation • Many applications of databases are geographical = 2-d data. Others involve large number of dimensions • Examples: – location of restaurants in a city. – Map data: zones, county lines, rivers, lakes, etc. (Data has spatial extent) – Sales information described by store, day, item, color, size, etc. Sale = point in multidimensional space. – Student described by age, zipcode, marital status. • Queries: – Range Query: “ find all McDonald restaurant within a given region”. – Nearest Neighbor Query: Find the nearest McDonald to my house – partial match queries ICS214A Notes 01 65 Approach: Utilize Single Dimensional Index • • • • Index on attributes independently Project query range to each attribute determine pointers. Intersect pointers go to the database and retrieve objects in the intersection. May result in very high I/O cost ICS214A Notes 01 66 R-tree Data Structure • • • • • ICS214A Notes 01 Extension of B-tree to multidimensional space. Paginated, balanced, guaranteed storage utilization. Can support both point data and data with spatial extent Groups objects into possibly overlapping clusters (rectangles in our case) Search for range query proceeds along all paths that overlap with the query. 67 Split Node • • Given a node split it into two nodes which are each atleast half full Multiple Objectives: – – • • minimize overlap minimize covered area R-tree minimizes covered area What is an optimal criteria??? Minimize overlap ICS214A Notes 01 Minimize covered area 68 Minimizing Covered Area • Group objects into 2 parts such that the covered area is minimized • NP Hard!! • Hence use heuritics • Two heuristics explored – quadratic and linear ICS214A Notes 01 69 Other Multidimensional Data Structures • Many generalizations of R-tree – different splitting criteria – different shapes of clusters (e.g., d-dimensional spheres) – adding redundancy to reduce search cost: • store objects in multiple rectangles instead of a single rectangle to reduce cost of retrieval. But now insert has to store objects in many clusters. This strategy also increases overlap causing search performance to detoriate. Space Partitioning Data Structures – unlike R-tree which group objects into possibly overlapping clusters, these methods attempt to partition space into non-overlapping regions. – E.g., KD tree, quad tree, grid files, KD-Btree, HB-tree, hybrid tree. • Space filling curves – superimpose an ordering on multidimensional space that preserves proximity in multidimensional space. (Z-ordering, hilbert ordering) – Use a B-tree as an index on that ordering ICS214A Notes 01 70 KD-tree • A main memory data structure based on binary search trees – can be adapted to block model of storage (KD-Btree) • Levels rotate among the dimensions, partitioning the space based on a value for that dimension • KD-tree is not necessarily balanced. ICS214A Notes 01 71 KD-Tree Example X=7 X=3 X=5 y=6 y=5 Y=6 x=3 x=8 x=7 Y=2 y=2 X=5 ICS214A X=8 Notes 01 72 Adapting KD Tree to Block Model • Similar to B-tree, tree nodes split many ways instead of two ways – Risk: insertion becomes quite complex and expensive. No storage utilization guarantee since when a higher level node splits, the split has to be propagated all the way to leaf level resulting in many empty blocks. • Pack many interior nodes (forming a subtree) into a block. – Risk it may not be feasible to group nodes at lower level into a block productively. Many interesting papers on how to optimally pack nodes into blocks recently published. ICS214A Notes 01 73 Quad Tree • Nodes split along all dimensions simultaneously • Division fixed: by quadrants • As with KD-tree we cannot make quadtree levels uniform ICS214A Notes 01 74 Quad Tree Example X=7 X=3 SW NW SE X=5 ICS214A NE X=8 Notes 01 75 Grid Files • Space Partitioning strategy but different from a tree. • Select dividers along each dimension. Partition space into cells • Unlike KD-tree dividers cut all the way. • Each cell corresponds to 1 disk page. • Many cells can point to the same page. • Cell directory potentially exponential in the number of dimensions ICS214A Notes 01 76 Space Filling Curve • Assumption – finite precision in representing each coordinate. B A 01 10 11 Z(A) = shuffle(x_A, y_A) = shuffle(00,11) = 0101 = 5 Z(B) = 11 = 3 00 (common prefix to all its blocks) 00 01 C 10 11 Z(C1) = 0010 = 2 Z(C2) = 1000 = 8 ICS214A Notes 01 77 Deriving Z-Values for a Region • Obtain a quad-tree decomposition of an object by recursively dividing it into blocks until blocks are homogeneous. 01 11 Objects representation 00 10 00 11 01 00 ICS214A is 0001, 0011,01 11 Notes 01 78 Generalized Search Trees • Motivation: – disparate applications require different data structures and access methods. – Requires separate code for each data structure to be integrated with the database code too much effort. Vendors will not spend time and energy unless application very important or data structure has general applicability. • Generalized search trees abstract the notion of data structure into a template. – Basic observation: most data structures are similar and a lot of book keeping and implementation details are the same. – Different data structures can be seen as refinements of basic GiST structure. Refinements specified by providing a registering a bunch of functions per data structure to the GiST. ICS214A Notes 01 79 GiST supports extensibility both in terms of data types and queries • GiST is like a “template” - it defines its interface in terms of ADT rather than physical elements (like nodes, pointers etc.) • The access method (AM) can customize GiST by defining his or her own ADT class i.e. you just define the ADT class, you have your access method implemented! • No concern about search/insertion/deletion, structural modifications like node splits etc. ICS214A Notes 01 80 Query Processing in DBMSs Internal relational algebra based representation of query Select … From … Where ... Parsing and Translation optimizer Statistics about data Sally 4000 Dick 9000 … … ... Evaluation engine Optimized execution plan Query results Data and index ICS214A Notes 01 81 Query Optimization • • Goals: to find the cheapest evaluation strategy for a query Stages of Optimization: – algebraic manipulations: heuristics used to convert query tree into an equivalent but more efficient representation. perform selections and projections as early as possible. combine selections with cartesian products to make a join combine sequence of unary operations (selections and projections). look for common subexpressions in an expression. – Cost based Analysis: given optimized representation produced after algebraic manipulation: generate all possible query plans and estimate their costs based on the statistical information and costs of each unary and binary operations. Best possible query plan chosen as an execution strategy. Number of plans considered even after heuristic are applied is exponential in the number of operators in query tree. It is important to choose a good plan since cost of generating plan amortized over multiple query executions. ICS214A Notes 01 Cost of Query Execution • Access to disk: cost of reading, writing, searching data blocks. (i/o cost) • Storage Costs: cost of storing intermediate files generated during query execution. (i/o cost) • Computation cost: cost of in memory execution of operations. (cpu cost) • Communication cost: cost of shipping the query and results from site to site or terminal where query originated. (communication cost) • Total cost = I/O cost + w1* CPU cost + w2 *Communication cost • Traditionally I/O cost considered most important ICS214A Notes 01 Transaction Management Applications in databases are modeled as transactions which provides ACID guarantees. • Atomicity: either all the effects of a transaction appear in database or none of the effects of a transaction appears in database. • Consistency: each transaction maps a database from consistent state to another consistent state • Isolation: concurrent execution of trasnactions is hidden from other concurrently executing transactions • Durability: if a transaction completes its effects are permanent and survive failures. ICS214A Notes 01 84 Transaction Model • Transactions provide a simple, powerful, and a natural programming model for writing database applications. • Transaction concept supports: – simple failure semantics: either all the effects of transaction appear in database or none do -- all or nothing – isolated view of the world: protection from partial effects of other concurrent applications. • Transactions allows applications to share data without having to explicitly deal with either fault-tolerance or synchronization • Transactions are the enabling technology for large distributed applications. ICS214A Notes 01 85 Isolation • • Isolation is implemented by using 2 phase locking protocol 2 Phase Locking Protocol: – Each transaction acquires a lock on a data item before accessing data – Locks are released when a transaction commits User 1 reads account = 1500 time User 2 reads account = 1500 User 1sets account value = 500 (withdraws 1000 dollars) User 2 sets account value = 700 (withdraws 800 dollars) The execution will be prevented by 2 phase locking since user 1’s transaction will not release the lock on account until user 1 transaction terminates ICS214A Notes 01 86 Atomicity • • • Atomicity is implemented by using a logging strategy. A transaction, before updating a data item writes a undo log record, using which its effects can be undone. If transaction aborts then undo log records used toreconstruct database state before transaction execution Old state New state Normal processing DO Undo log record New state Transaction rollback due to either user requested abort, system failure, consistency violation ICS214A Old state UNDO Undo log record Notes 01 87 Durability • • • Durability is implemented using logging strategy A transaction, before updating a data item, writes a redo log record using which its effects are redone If system fails before a committed transaction’s effects appear in database its effects are redone using redo log records on recovery. Old state New state Normal processing DO Redo Log record Old state New state Redo of committed transaction ICS214A REDO Redo log record Notes 01 88