Download NiagaraCQ - CS

NiagaraCQ
A Scalable Continuous Query System
for Internet Databases
Outline
1.
2.
3.
4.
Problem
NiagaraCQ
Selection Placement Strategies
Dynamic Regrouping Algorithm
NiagaraCQ
2
Problem
Lack of a scalable and efficient system which
supports persistent queries, that allow users to
receive new results when they become available:
Notify me whenever the price of Dell stock drops by more
than 5% and the price of Intel stock remains unchanged over
next three months.
NiagaraCQ
3
NiagaraCQ
Support continues queries
Change-based queries
Timer-based queries
Scalability
Performance
Adequate to the Internet
User Interface - high level query language
NiagaraCQ
4
Command Language
Create continuous query:
CREATE CQ_name
XML-QL query
DO action
{START start_time} {EVERY time_interval}
{EXPIRE expiration_time}
Delete continuous query:
DELETE CQ_name
NiagaraCQ
5
Expression Signature
Represent the same syntax structure, but possibly
different constant values, in different queries.
Where <Quotes> <Quote>
<Symbol>INTC</>
</> </> element_as $g
in “http://www.cs.wisc.edu/db/quotes.xml”
construct $g
Where <Quotes> <Quote>
<Symbol>MSFT</>
</> </> element_as $g
in “http://www.cs.wisc.edu/db/quotes.xml”
construct $g
NiagaraCQ
6
Expression Signature (2)
=
Quotes.Quote.Symbol
in quotes.xml
NiagaraCQ
constant
7
Query Plan
Trigger Action I
Trigger Action J
Select
Symbol=“INTC”
Select
Symbol=“MSFT”
File Scan
File Scan
quotes.xml
quotes.xml
NiagaraCQ
8
Group Signature
Common expression signature of all queries in the
group
=
Quotes.Quote.Symbol
in quotes.xml
NiagaraCQ
constant
9
Group Constant Table
Constant_value
Destination_buffer
…
…
INTC
Dest . I
MSFT
Dest . J
…
…
NiagaraCQ
10
Group Plan
……..
Trigger Action I
Trigger Action J
Split
Join
Symbol = Constant_value
File Scan
File
quotes.xml
Constant Table
NiagaraCQ
11
Incremental Grouping Algorithm
1. Group optimizer
traverses the query
plan bottom up.
2. Matches the query’s
expression
signature with the
signatures
of
existing groups.
NiagaraCQ
Trigger Action
Select
Symbol=“AOL”
File Scan
quotes.xml
12
Incremental Grouping Algorithm (2)
3. Group optimizer breaks the
query plan into two parts.
Lower – removed
Upper – added onto the
group plan.
Trigger Action
Select
Symbol=“AOL”
File Scan
4. Adds the constant to the
constant table.
NiagaraCQ
quotes.xml
13
Pipeline Approach
Tuples are pipelined from the output of one
operator into the input of the next operator.
Disadvantages
Doesn’t work for grouping timer-based queries.
Split operator may become a bottleneck.
Not all parts should be executed.
NiagaraCQ
14
Intermediate Files
NiagaraCQ
15
Intermediate Files (2)
Advantages
 Intermediate files and data sources are monitored
uniformly.
 Each query is scheduled independently.
 The potential bottleneck problem of the pipelined
approach is avoided.
Disadvantages
Extra disk I/Os.
Split operator becomes a blocking operator.
NiagaraCQ
16
Virtual Intermediate Files
Where <Quotes> <Quote>
<Change_ratio>$c</>
</> </> element_as $g
in “quotes.xml”, $c>0.05
construct $g
Overlap
Where <Quotes> <Quote>
<Change_ratio>$c</>
</> </> element_as $g
in “quotes.xml”, $c>0.15
construct $g
>
Quotes.Quote.Change_Ratio
in quotes.xml
constant
NiagaraCQ
17
Virtual Intermediate Files (2)
All outputs from split operator are stored in one
real intermediate file.
This file has index on the range attribute.
Virtual intermediate files store a value range.
Modification of virtual intermediate files can
trigger upper-level queries.
The value range is used to retrieve data from the
real intermediate file.
NiagaraCQ
18
Event Detection
Types of Events
Data-source change
Timer
Types of data sources
Push-based
Pull-based
NiagaraCQ
19
Timer-based
Timer events are stored in an event list, sorted in
time order.
Each entry stores query ids.
Query will be fired if its data source has been
modified since its last firing time.
After a timer event, the next firing times are
calculated and the queries are added into the
corresponding entries.
NiagaraCQ
20
Incremental Evaluation
Queries are been invoked only on changed data.
For each file, NiagaraCQ keeps a “delta file”.
Queries are run over delta files.
Incremental evaluation of join operators
requires complete data files.
Time stamp is added to each tuple in order to
support timer-based.
NiagaraCQ
21
Memory Caching
Query plans - using LRU policy that favors
frequently fired queries.
Data files - favors the delta files.
Event list – only a “time window”
NiagaraCQ
22
System Architecture
NiagaraCQ
23
If file changes and timer
events are satisfied, ED
provides CQM with a list
of firing CQs
CQM adds continuous
queries with file and timer
information to enable ED
to monitor the events
Continues Queries Processing
CQM
Continuous
invokes
Query Manager
QE to
(CQM)
execute
6
firing CQs
1
5
Event Detector
(ED)
2, 3
7
4
ED asks DM
to monitor
changes to
files
Data Manager DM informs
ED
of
(DM)
8
changes
to
When a timer event
pushed-based
File scan operator
DM only returns changes
happens, ED asks
data sources
calls DM to retrieve
between last fire time DM
andthe last modified
selected documents
24
current fire time NiagaraCQtime of files
Query Engine
(QE)
Selection Placement Strategies
Where <Quotes><Quote><Symbol>$s</> <Price>$p</></>
element_as $g </> in “quotes.xml”, $p > 90
<Companies><Company><Symbol>$s</></>
element_as $t</> in “profiles.xml” construct $g, $t
Where <Quotes><Quote><Symbol>$s</> <Price>$p</></>
element_as $g </> in “quotes.xml”, $p > 100
<Companies><Company><Symbol>$s</></>
element_as $t</> in “profiles.xml” construct $g, $t
NiagaraCQ
25
Expressions Signatures
>
Quotes.Quote.Price
in quotes.xml
constant
Symbol=Symbol
quotes.xml
profiles.xml
NiagaraCQ
26
Where to place the selection
operator ?
Below the join - PushDown
(σ1R S) U (σ2R S) U … U (σnR S)
Above the join – PullUp
σ1(R S) U σ2(R S) U … U σn(R S)
PullUp achieves an average 10-fold performance
improvement over PushDown.
NiagaraCQ
27
PushDown - Query Plan
Join
Select
Price>90
profiles.xml
quotes.xml
NiagaraCQ
28
PushDown - Groups Plans
NiagaraCQ
29
PullUp - Groups Plans
NiagaraCQ
30
PullUp Vs. PushDown
Only one join group and one selection group
 Maintains a single intermediate file
Irrelevant tuples being joined
Very large intermediate file
Changes in profiles.xml affect the intermediate
file (file_k) – maintenance overhead.

NiagaraCQ
31
Filtered PullUp
quotes.xml
Grouped Join Plan
Join
Selection
Price>90
profiles.xml
quotes.xml
NiagaraCQ
32
Filtered PullUp Vs. PullUp
Relevant tuples being joined
 Reduce the size of intermediate file
 Reduce the cost of PullUp by 75%
Complexity – the selection predicate may need
to be dynamically modified (query with
price>70)

NiagaraCQ
33
Dynamic Re-grouping
Let Q1 (A B C) and Q2 (B C) be two
continuous queries submitted sequentially.
Incremental grouping algorithm chooses a plan
((A B) C).
Neither of these groups can be used for Q2.
ABC
AB
ABC
BC
BC
NiagaraCQ
34
Dynamic Re-grouping (2)
Existing queries are not regrouped with new
grouping
opportunities
introduced
by
subsequent queries.
Reduction in the overall performance - queries
are continuously being added and removed.
Naive regrouping-algorithm – periodically
perform a global query optimization:
Expensive
Redundant work (already done by incremental opt.)
NiagaraCQ
35
Data Structures
A query graph – directed acyclic graph, with each node
representing an existing join expression in the group plan.
Node {
char* query;
SIG_TYPE sig;
int final_node_count;
list<Child*> children;
list<Node*> parents;
float updateFreq;
float cost;
//ASCII query plan
//signature of the query string
//number of users that require this query.
//0: non-final node; >0: final node
//children of this node, where Child={Node*, weight}
//parents of this node
//update frequency of this node
//the cost for computing this node
//Following data structures used only for dynamic regrouping
int reference_count;
//reference count
bool visited;
//a flag that records whether
//purgeSibling has performed on this node
}
NiagaraCQ
36
Data Structures (2)
A group table – array of hash tables.
i-th hash table - queries with query length
(number of joins) i.
Hash table entry - mapping from a query string
to the corresponding node in the graph.
Array
Hash
Node
NiagaraCQ
37
Data Structures (3)
A query log – array of vectors.
Stores new nodes that have been added since the
last regrouping.
Cleared after regrouping.
Array
Vector
Node
NiagaraCQ
38
Incremental Grouping Algorithm
Top-down local exhaustive search:
If the query exists, increases the final node count by 1.
Else
Enumerates all possible sub-query in a top-down
manner and probes the group table to check whether a
sub-query node exists.
Computes the minimal cost of using existing sub-query
nodes.
Computes the minimal cost without using existing subquery nodes.
The least-costly plan will be chosen.
NiagaraCQ
39
Dynamic Regrouping Algorithm
Phase 1 : constructing links among existing
nodes and new nodes.
Phase 2 : find minimal-weighted solution from
the current solution by removing redundant
nodes.
ABC
AB
BC
NiagaraCQ
40
Phase 1: constructing links among
existing nodes and new nodes
Main idea - for any pair of nodes in the graph,
if one node is a sub-query of another node, it
creates a link between them if it did not exist
before.
Relationships are only evaluated between
existing nodes and nodes added since last
regrouping.
The difference of levels between a parent and a
child is always 1.
NiagaraCQ
41
Phase 1 - Algorithm
bottom-up
for each node in level i query log
if node has parents in level i+1 group table
connect node to parent
if node has children in level i-1 group table
connect node to children
NiagaraCQ
42
Phase 2: A greedy algorithm for levelwise graph minimization
Main idea – traverse the query graph level-bylevel and attempt to remove any redundant
nodes at one level a time.
Starts from the second level from the top.
Subset of level i nodes retain if:
Nodes at level i+1 have at least one child in this set.
These nodes have a minimum total cost.
Nodes that are not selected are removed
permanently.
NiagaraCQ
43
Phase 2 - Algorithm
MinimizeGraph() {
for each level L in group-table {
// L ranging from the maximum number of join-1 to 1
for each node N in the level-L group table
InitializeSet(N)
for each node N in finalSet
PurgeSiblings(N);
while (remain set is not empty) {
scan each node R in the remain set {
if (R’s reference count == 0) {
remove R from the remain set
deleteNode(R)
}
else if (R.cost/R.reference_count <
Current_minimum) {
M=R
Current_minimum
=R.cost/R.reference_count;
}
} //scan …
remove M from the remain set
PurgeSiblings(M)
} //while…
InitializeSet(Node N) {
if N is a final node
Add N into final_set
else {
add N into the remain_set
N.reference_count =
number of parents of N
}
N.visited = false
}
purgeSiblings(Node N) {
For each parent P of N {
if (!P.visited) {
Decrease the reference count of
N’s siblings of same parent P by 1
P.visited = true
}
}
}
} //for each level …
} //MinimizeGraph
NiagaraCQ
44
Cost Analysis
N = number of queries
Number of nodes is proportional to the number
of queries = C*N
Each query contains no more then 10 joins.
Each level contain about C*N/10 nodes
NiagaraCQ
45
Cost Analysis – Phase 1
R or K*R = regrouping frequencies
In frequency R
N/R = number of regrouping
C*R = number of nodes that will be joined with
existing nodes.
m*C*R = number of nodes after m-1
regrouping.
m*(C*R)2 = number of comparisons for m-th
regrouping (ignoring a constant reduction).
NiagaraCQ
46
Cost Analysis – Phase 1 (2)
Total number of comparisons, frequency R:
(C*R)2+2*(C*R)2+…+N/R*(C*R)2 =
N(N+R)C2/2 = O(N2)
Total number of comparisons, frequency K*R:
(C*K*R)2+…+(N/(K*R))*(C*K*R)2 =
N(N+KR)C2/2
The ratio:
[N(N+KR)C2/2]/[N(N+R)C2/2] = (N+KR)/(N+R)
NiagaraCQ
47
Cost Analysis – Phase 2
Worst case – each pass remove one node.
Cost for a level:
(C*N/10) +(C*N/10-1) +…+1 =
CN(CN+10)/200 = O(N2)
Purge siblings:
(C*N/10 * C*N/10) = (CN)2/100 = O(N2)
All 9 levels: O(N2)
NiagaraCQ
48
References
NiagaraCQ: A Scalable Continuous Query System for
Internet Databases
http://www.cs.wisc.edu/niagara/papers/NiagaraCQ.pdf
Design and Evaluation of Alternative Selection
Placement Strategies in Optimizing Continuous Queries
http://www.cs.wisc.edu/niagara/papers/Icde02.pdf
Dynamic Re-grouping of Continuous Queries
http://www.cs.wisc.edu/niagara/papers/507.pdf
NiagaraCQ
49