Download index-ceng351-fall2012

Indexing
CENG 351 File Structures
1
What is an Index?
• An index on a file is an auxiliary file that
makes it more efficient to search for a
record in the data file.
– Any subset of the fields of a relation can be the
search key for an index on the relation
– Search key is not necessarily the same as a key
(minimal set of fields that uniquely indentify a
record in a relation)
• The index is called an access path on the search
field.
CENG 351 File Structures
2
What is an Index?
• Simple indexes are represented using simple
arrays of structures that contain keys and
references.
– <search key, rid of record>
– <search key, list of rids>
CENG 351 File Structures
3
Uses of an index
• An index lets us impose order on a file
without rearranging the file.
• Indexes provide multiple access paths to a
file.
– e.g. library catalog providing search for
author,book and title
• An index can provide keyed access to
variable-length record files.
CENG 351 File Structures
4
Index Classification
• Primary vs Secondary: If search key
contains primary key, then called primary
index
• Clustered vs. Unclustered: If order of data
records is same as or close to data entries
(in the index), then it is called clustered.
CENG 351 File Structures
5
A simple index for a pile file
Label ID
17
62
117
152
Title
Artist
LON|2312|Symphony No.9| Beethoven|Giulini
RCA|2626|Romeo and Juliet|Prokofiev|Maazel
WAR|23699|Nebraska| …
ANG|3795|Violin Concerto| …
Address of
record
Primary key = company label + record ID.
• Index is sorted (in main memory).
• Records appear in file in the order they entered.
CENG 351 File Structures
6
Index array:
Key
ANG3795
LON2312
Reference
152
17
RCA2626
62
WAR23699
117
• How to search for a recording with given LABEL ID?
– Binary search in the index and then seek for the record
in position given by the reference field.
CENG 351 File Structures
7
Operations to maintain an indexed file
•
•
•
•
•
•
•
Create the original empty index and data files.
Load the index file into memory before using it.
Rewrite the index file from memory after using it.
Add data records to the data file.
Delete records from the data file
Update records in the data file.
Update the index to reflect changes in the data file
CENG 351 File Structures
8
Record Addition
1. Append the new record to the end of the
data file.
2. Insert a new entry to the index in the right
position.
– needs rearrangement of the index: we have to
shift all the entries with keys that are larger
than the inserted key and then place the new
entry in the opened space.
• Note: this rearrangement is done in the
main memory.
CENG 351 File Structures
9
Record Deletion
• This should use the techniques for
reclaiming space in files when deleting
records from the data file.We must delete
the corresponding entry from the index:
– Shift all records with keys larger than the key
of the deleted record to the previous position;
or
– Mark index entry as deleted.
CENG 351 File Structures
10
Record Updating
There are two cases to consider:
1. The update changes the value of the key
field:
–
Treat this as a deletion followed by an
insertion
2. The update does not affect the key field:
–
If record size is unchanged, just modify the
data record. If record size is changed treat this
as a delete/insert sequence.
CENG 351 File Structures
11
Example
Given the following data file:
EMPLOYEE(NAME, SSN, ADDRESS, JOB, SAL, ... )
Suppose that:
record size R=150 bytes
block size B=512 bytes
r=30000 records
Then, we get:
blocking factor Bfr= B div R= 512 div 150= 3 records/block
number of file blocks b= (r/Bfr)= (30000/3)= 10000 blocks
For an index on the SSN field, assume the field size VSSN=9 bytes,
assume the record pointer size PR=7 bytes. Then:
index entry size RI=(VSSN+ PR)=(9+7)=16 bytes
index blocking factor BfrI= B div RI= 512 div 16= 32 entries/block
number of index blocks b= (r/ BfrI)= (30000/32)= 938 blocks
binary search needs log2bI= log2938= 10 block accesses
This is compared to an average linear search cost of:
(b/2)= 10000/2= 5000 block accesses
If the file records are ordered, the binary search cost would be:
log2b= log210000= 13 block accesses
Types of Single-Level Indexes
• Primary Index
– Defined on an ordered data file
– The data file is ordered on a key field
– Includes one index entry for each block in the data file; the
index entry has the key field value for the first record in the
block, which is called the block anchor
– A similar scheme can use the last record in a block.
– A primary index is a nondense (sparse) index, since it
includes an entry for each disk block of the data file and the
keys of its anchor record rather than for every search value.
CENG 351 File Structures
CENG 351 File Structures
Types of Single-Level Indexes
• Clustering Index
– Defined on an ordered data file
– The data file is ordered on a non-key field unlike primary
index, which requires that the ordering field of the data file
have a distinct value for each record.
– Includes one index entry for each distinct value of the field;
the index entry points to the first data block that contains
records with that field value.
– It is another example of nondense index where Insertion and
Deletion is relatively straightforward with a clustering index.
CENG 351 File Structures
CENG 351 File Structures
CENG 351 File Structures
Types of Single-Level Indexes
• Secondary Index
– A secondary index provides a secondary means of accessing a
file for which some primary access already exists.
– The secondary index may be on a field which is a candidate key
and has a unique value in every record, or a nonkey with
duplicate values.
– The index is an ordered file with two fields.
• The first field is of the same data type as some nonordering
field of the data file that is an indexing field.
• The second field is either a block pointer or a record
pointer. There can be many secondary indexes (and hence,
indexing fields) for the same file.
– Includes one entry for each record in the data file; hence, it is a
dense index
CENG 351 File Structures
CENG 351 File Structures
Problems with simple indexes
If index does not fit in memory:
1. Seeking the index is slow (binary search):
–
We don’t want more than 3 or 4 seeks for a search.
N
Log(N+1)
15 keys
4
1000
~10
100,000
~17
1,000,000
~20
2. Insertions and deletions take O(N) disk accesses.
–
Index rearrangement requires shifting on disk
CENG 351 File Structures
21
Alternatives
• Two main alternatives:
1. Hashed organization (when access speed is a
top priority)
2. Tree-structured (multi-level) index such as
B+trees.
CENG 351 File Structures
22
Multilevel Indexing and
B+ Trees
CENG 351 File Structures
23
Multilevel Indexing
• Tree-structured indexing can support both
equality and range queries
– ISAM (Indexed Sequential Access Method):
static tree;
– B+ tree: dynamic tree structure
CENG 351 File Structures
24
Tree indexes
• Single level index scheme is nice if the index fits
in memory.
• If index doesn’t fit in memory:
– Divide the index structure into blocks,
– Organize these blocks similarly building a tree
structure.
CENG 351 File Structures
25
Multi-Level Indexes
• Because a single-level index is an ordered file, we can
create a primary index to the index itself ; in this case,
the original index file is called the first-level index and
the index to the index is called the second-level index.
• We can repeat the process, creating a third, fourth, ...,
top level until all entries of the top level fit in one disk
block
• A multi-level index can be created for any type of firstlevel index (primary, secondary, clustering) as long as
the first-level index consists of more than one disk
block
A two-level
primary index
resembling
ISAM
organization.
Multi-Level Indexes
• Such a multi-level index is a form of search tree
; however, insertion and deletion of new index
entries is a severe problem because every level
of the index is an ordered file.
• Solution: dynamic tree structure
B Trees
• B-tree is one of the most important data structures
in computer science.
• What does B stand for? (Not binary!)
• B-tree is a multiway search tree.
• Several versions of B-trees have been proposed,
but only B+ Trees has been used with large files.
• A B+tree is a B-tree in which data records are in
leaf nodes, and faster sequential access is possible.
CENG 351 File Structures
29
Formal definition of B+ Tree Properties
• Properties of a B+ Tree of order v :
– All internal nodes (except root) has at least v keys
and at most 2v keys .
– The root has at least 2 children unless it’s a leaf..
– All leaves are on the same level.
– An internal node with k keys has k+1 children
CENG 351 File Structures
30
B+ tree: Internal/root node
structure
P0 K1 P1 K2
………………
Pn-1 Kn Pn
Each Pi is a pointer to a child node; each Ki is a search key value
# of search key values = n, # of pointers = n+1
 Requirements:
 K1 < K2 < … < K n
 For any search key value K in the subtree pointed by Pi,
If Pi = P0, we require K < K1
If Pi = Pn, Kn  K
If Pi = P1, …, Pn-1, Ki  K < Ki+1
CENG 351 File Structures
31
B+ tree: leaf node structure
L
K 1 r1 K 2
………………
K n rn R
 Pointer L points to the left neighbor; R points to
the right neighbor
 K1 < K2 < … < Kn
 v  n  2v (v is the order of this B+ tree)
 If primary index  holds data records
If secondary index  holds <Ki, ri>
We will use Ki* for the data entry/record and omit L and R for
simplicity
CENG 351 File Structures
32
Example: B+ tree with order of 1
• Each node must hold at least 1 entry, and at most
2 entries
Root
40
10*
15*
20
33
20*
27*
51
33*
37*
40*
46*
CENG 351 File Structures
51*
63
55*
63*
97*
33
Example: Search in a B+ tree order 2
• Search: how to find the records with a given search key value?
– Begin at root, and use key comparisons to go to leaf
• Examples: search for 5*, 16*, all data entries >= 24* ...
– The last one is a range search, we need to do the sequential scan, starting from
the first leaf containing a value >= 24.
Root
13
2*
3*
5*
7*
14* 15*
17
24
19* 20* 22*
30
24* 27* 29*
CENG 351 File Structures
33* 34* 38* 39*
34
Cost for searching a value in B+ tree
• Typically, a node is a page (block or cluster)
• Let H be the height of the B+ tree: we need to
read H+1 pages to reach a leaf node
• Let F be the (average) number of pointers in a
node (for internal node, called fanout )
Height
– Level 1 = 1 page = F0 page
0
– Level 2 = F pages = F1 pages
1
– Level 3 = F * F pages = F2 pages
– Level H+1 = …….. = FH pages (i.e., leaf
nodes)
– Suppose there are D data entries. So there are
D/(F-1) leaf nodes
– D/(F-1) = FH. That is, H = logF( FD 1)
CENG 351 File Structures
level
1
2
35
B+ Trees in Practice
• Typical order: 100. Typical fill-factor: 67%.
–
average fanout = 133 (i.e, # of pointers in internal
node)
• Can often hold top levels in buffer pool:
–
–
–
Level 1 =
1 page = 8 Kbytes
Level 2 = 133 pages = 1 Mbyte
Level 3 = 17,689 pages = 133 MBytes
• Suppose there are 1,000,000,000 data entries.
–
–
H = log133(1000000000/132) < 4
The cost is 5 pages read
CENG 351 File Structures
36
Basic Operations in a B+ Tree
• Insertion
• Deletion
• Bulk Loading
CENG 351 File Structures
37
Inserting 16*, 8* into Example B+ tree
Root
2*
3*
5*
7*
8*
13
3*
5*
7*
24
30
15* 16*
14*
13
2*
17
17
24
30
8*
One new child (leaf node)
generated; must add one more
pointer to its parent, thus one more
key value as well.
CENG 351 File Structures
38
Inserting 8* (cont.)
• Copy up the
middle value
(leaf split)
13
17
24
30
Entry to be inserted in parent node.
(Note that 5 is
s copied up and
continues to appear in the leaf.)
5
2*
5
3*
13
5*
17
24
7*
8*
30
CENG 351 File Structures
39
Insertion into B+ tree (cont.)
• Understand
difference
between copy-up
and push-up
• Observe how
minimum
occupancy is
guaranteed in
both leaf and
index pg splits.
5
13
17
24
30
We split this node, redistribute entries evenly,
and push up middle key.

17
5
13
24
CENG 351 File Structures
Entry to be inserted in parent node.
(Note that 17 is pushed up and only
appears once in the index. Contrast
this with a leaf split.)
30
40
Example B+ Tree After Inserting 8*
Root
17
5
2*
3*
24
13
5*
7* 8*
14* 15*
19* 20* 22*
30
24* 27* 29*
33* 34* 38* 39*
Notice that root was split, leading to increase in height.
CENG 351 File Structures
41
Inserting a Data Entry into a B+ Tree:
Summary
• Find correct leaf L.
• Put data entry onto L.
–
–
If L has enough space, done!
Else, must split L (into L and a new node L2)
• Redistribute entries evenly, put middle key in L2
• copy up middle key.
• Insert index entry pointing to L2 into parent of L.
• This can happen recursively
–
To split index node, redistribute entries evenly, but push
up middle key. (Contrast with leaf splits.)
• Splits “grow” tree; root split increases height.
–
Tree growth: gets wider or one level taller at top.
CENG 351 File Structures
42
Basic Operations in a B+ Tree
• Insertion
• Deletion
• Bulk Loading
CENG 351 File Structures
43
Delete 19* and 20*
Root
17
5
2*
3*
24
13
5*
7* 8*
14* 16*
19* 20* 22*
30
24* 27* 29*
33* 34* 38* 39*
22*
22* 24*
CENG 351 File Structures
27* 29*
Is that it?
44
Deleting 19* and 20* (cont.)
Root
17
5
2*
3*
•
•
•
•
27
13
5*
7* 8*
14* 16*
22* 24*
30
27* 29*
33* 34* 38* 39*
Notice how 27 is copied up.
But can we move it up?
Now we want to delete 24
Underflow again!CENG
But351can
we
redistribute
this
time?
File Structures
45
Deleting 24*
• Observe the two leaf
nodes are merged, and
27 is discarded from
their parent, but …
• Observe `pull down’ of
index entry (below).
New root
2*
3*
5*
5
7*
8*
13
14* 16*
30
22*
17
27*
29*
33*
34*
38*
39*
30
22* 27* 29*
CENG 351 File Structures
33* 34* 38* 39*
46
Deleting a Data Entry from a B+ Tree:
Summary
• Start at root, find leaf L where entry belongs.
• Remove the entry.
–
–
If L is at least half-full, done!
If L has only d-1 entries,
• Try to re-distribute, borrowing from sibling (adjacent node
with same parent as L).
• If re-distribution fails, merge L and sibling.
• If merge occurred, must delete entry (pointing to L or
sibling) from parent of L.
• Merge could propagate to root, decreasing height.
CENG 351 File Structures
47
Example of Non-leaf Re-distribution
• Tree is shown below during deletion of 24*. (What
could be a possible initial tree?)
• In contrast to previous example, can re-distribute entry
from left child of root to right child.
Root
22
5
2* 3*
5* 7* 8*
13
14* 16*
17
30
20
17* 18*
20* 21*
CENG 351 File Structures
22* 27* 29*
33* 34* 38* 39*
48
After Re-distribution
• Intuitively, entries are re-distributed by `pushing
through’ the splitting entry in the parent node.
• It suffices to re-distribute index entry with key 20;
we’ve re-distributed 17 as well for illustration.
Root
17
5
2* 3*
5* 7* 8*
13
14* 16*
20
17* 18*
20* 21*
CENG 351 File Structures
22
30
22* 27* 29*
33* 34* 38* 39*
49
Basic Operations in a B+ Tree
• Insertion
• Deletion
• Bulk Loading
CENG 351 File Structures
50
Bulk Loading of a B+ Tree
• If we have a large collection of records, and we
want to create a B+ tree on some field, doing so by
repeatedly inserting records is very slow.
• Bulk Loading can be done much more efficiently.
– Initialization: Sort all data entries, insert pointer
to first (leaf) page in a new (root) page
CENG 351 File Structures
51
Bulk Loading of a B+ Tree
CENG 351 File Structures
52
A Secondary B+-Tree index
Root
17
5
2
3
5
7
8
13
14 16
22* 3* 14*
8*
20
17 18
2* 16* 5* 39*
20 21
22
30
22 27 29
33 34 38 39
20* 7* 38* 29*
Actual data
buckets
Terminology
• Bucket Factor: the number of records which can
fit in a leaf node.
• Fan-out : the average number of children of an
internal node.
• A B+tree index can be used either as a primary
index or a secondary index.
– Primary index: determines the way the records are
actually stored (also called a sparse index)
– Secondary index: the records in the file are not
grouped in buckets according to keys of secondary
indexes (also called a dense index)
CENG 351 File Structures
54
Summary
• Tree-structured indexes are ideal for rangesearches, also good for equality searches.
• B+ tree is a dynamic structure.
–
–
–
–
Inserts/deletes leave tree height-balanced; High fanout (F)
means depth rarely more than 3 or 4.
Almost always better than maintaining a sorted file.
Typically, 67% occupancy on average.
If data entries are data records, splits can change rids!
• Most widely used index in database management
systems because of its versatility. One of the most
optimized components of a DBMS.
CENG 351 File Structures
55