Download Amortized Analysis - Wilfrid Laurier University

Amortized Analysis
1. The concepts of amortized analysis
2. Methods of amortized analysis
•
•
•
Aggregate method
Accounting method
Potential method
3. Problem examples by amortized analysis
•
•
•
Stack
Binary counter
Dynamic array
CP412 ADAII, Wilfrid Laurier University
Introduction # 1
1. What is Amortized Analysis
• Previous analysis focuses on time/space usage of
algorithms for arbitrary input of a problem.
– The worst analysis considers the worst cost over all possible
inputs of given size
– The average case considers the average cost over all possible
input of given size. The average is a probability expectation.
– Rough and pessimistic
• Amortized Analysis is concerned with the overall cost of
a sequence of operations
– the average performance of each operation in the worst case
over all possible sequences of the operations.
CP412 ADAII, Wilfrid Laurier University
Introduction # 2
Why concerning a sequence of operations
– data structures typically undergo a sequence of operations.
E.g. push or pop operations stack.
– For a sequence of operations, the cost of an operation at different
position are different.
E.g. for tree insertion operation, the cost of early insertion and later
insertions are different.
– amortized analysis allows a tighter bound that better reflects
performance than the worst case analysis
CP412 ADAII, Wilfrid Laurier University
Introduction # 3
Amortized Analysis vs other analysis
• Amortized analysis vs average case analysis
– Average case analysis relies on probabilistic assumptions
about the data structures and operations in order to compute
an expected running time of an algorithm.
– Amortized analysis needs no such assumptions.
– Amortized analysis offers a true upper bound on the worst case
running time of a sequence of operations.
– An average case bound does not provide a true upper bound.
That means one may not “unlucky” and encounter an input that
requires much more than the expected computation time.
• Amortized analysis vs competitive analysis
– Amortization is useful for competitive analysis's performance.
Sleator and Tarjan (1985a) offer an example of using amortized
analysis to perform competitive analysis
CP412 ADAII, Wilfrid Laurier University
Introduction # 4
History
• In finance, amortization refers to paying off a debt,
such as a loan or mortgage, by smaller payments
made over time
• Tarjan introduced amortized analysis for algorithms as
a general technique in 1985
–
–
–
–
Three methods of amortized analysis were recognized:
the banker's method (Brown and Tarjan,1980)
Aggregate method ((Aho et al. 1974)
potential method (Sleator 1985)
CP412 ADAII, Wilfrid Laurier University
Introduction # 5
2. Amortized analysis methods
•
The aggregate method, where the total running time for a sequence
of operations is analyzed.
T(n) is total cost over a worst-case sequence of n operations
T(n)
Amortized cost per operation =
n
•
The accounting (or banker's) method, where we impose an extra
charge on inexpensive operations and use it to pay for expensive
operations later on.
•
The potential (or physicist's) method, in which we derive a potential
function characterizing the amount of extra work we can do in each
step. This potential either increases or decreases with each
successive operation, but cannot be negative.
CP412 ADAII, Wilfrid Laurier University
Introduction # 6
Stack with push, pop, MultiPop operations
Consider stack data structure with three operations
Push(S, x) -- push x onto stack S, worst-case cost O(1).
Pop (S)
-- pop the top element of S, worst-case cost O(1).
MultiPop(S, k): it pops top k elements of S, worst-case cost O(k).
MultiPop (S, k)
while S is not empty ; and k > 0
Pop (S)
k=k-1
end
What is the worst-case complexity of n MultiPop Stack ops?
Because any single MultiPop can be (n), the standard worst-case
bound is O(n2).
CP412 ADAII, Wilfrid Laurier University
Introduction # 7
Aggregate method of stack
Theorem:
A sequence of n ops, on an initially empty stack, has cost O(n).
Proof:
Each element is popped at most once (either by Pop or during a
MultiPop). Total cost of Pops, including MultiPops, is ≤ number of
Pushes, which is at most n. Total cost of all operations is T(n) ≤ 2n.
Thus, amortized cost per operation for this stack is at most 2.
This bound holds in worst-case, over any sequence!
CP412 ADAII, Wilfrid Laurier University
Introduction # 8
Accounting method of Stack
The key is to choose the amortized costs carefully.
Suppose the actual cost of the ith operation is Ci ,
we use some other cost A i as its amortized cost.
Then, we have to make sure that, for any possible
n
sequence of n operations,
n
 A  C
i 1
i
i 1
i
The total credit accrued by the data structure at any point
n
n
i 1
i 1
is ( Ai  Ci )  0
CP412 ADAII, Wilfrid Laurier University
Introduction # 9
Accounting method of Stack
The actual costs (Ci) of operations in the stack example are:
1 for Push, 1 for Pop, min(k, s) for MultiPop,
where s is the stack size when the MultiPop is called.
Let us assign the following amortized costs (Ai) to these ops:
2 for Push
0 for Pop
0 for MultiPop
The amortized cost of MultiPop is a constant (in fact, 0), even
though it's real cost if variable. The intuition is that the credit
accrued through Push operations will be enough to pay for the actual
cost of MultiPop.
CP412 ADAII, Wilfrid Laurier University
Introduction # 10
Accounting method of stack
Think of stack as a bank, where each deposit and withdrawal costs
$1. When an item is pushed onto the stack, we use $1 for the Push
operation, and leave the second dollar (of its amortized cost) with the
item in the bank. This second (spare) dollar will be used to pay for the
item's Pop.
The item may be popped either through a single Pop or a MultiPop,
but since each item in the stack has a spare dollar allocated to it, the
Pop and MultiPop can be entirely paid using that credit. The
amortized cost of Pop and MultiPop is therefore 0.
Thus, for any sequence of n Push, Pop, MultiPop, the total amortized
cost is at most 2n = O(n), and it is an upper bound on the actual cost.
The average cost per operation is 2.
CP412 ADAII, Wilfrid Laurier University
Introduction # 11
Potential method of stack
Keep track of the potential energy of the data structure.
Start with initial data structure Di , and perform n updates.
Let Ci be the actual cost of ith op;
let Di be data structure state after op i.
Function  measures potential of data structure.
Define amortized cost of operation i as
Ai = Ci + ( ( Di )   ( Di 1 ))
CP412 ADAII, Wilfrid Laurier University
Introduction # 12
Potential method of stack
Since Ai =Ci + ( ( Di )   ( Di 1 )),
by adding them up them up over the sequence, we get
n
n
 A   C +  ( D )   ( D ),
i 1
i
i 1
i
n
0
Suppose  ( Di )   ( D0 )  0, for all i , then
n
n
 A  C
i 1
i
i 1
i
So, amortized cost is an upper boun d on actual total cost.
How to define  ( Di )
CP412 ADAII, Wilfrid Laurier University
Introduction # 13
Potential method of stack
L et  = n u m b er o f item s o n stack
 ( D 0 )  0,  ( D i )   ( D 0 )  0
A m o rtized co st o f P u sh :
Ai  C i  (  ( D i )   ( D i  1 ))  1  1  2
A m o rtized co st o f P o p :
Ai  C i  (  ( D i )   ( D i  1 ))  1  1  0
A m o rtized co st o f M u ltiP o p (S , k )::
Ai  C i  (  ( D i )   ( D i  1 ))  m in ( s , k )  m in ( s , k )  0
n
T h u s, am o rtized to tal co st

i 1
Ai  2 n ,
am o rtized co st p er o p eratio n is O (1 ).
CP412 ADAII, Wilfrid Laurier University
Introduction # 14
Amortized Analysis of a Binary Counter
B inary counter im plem ented as an array of bits: A [0...k - 1].
C onsider increm enting A starting from 0.
Increm ent (A )
i = 0;
w hile i  k and A [i] = 1
A [i] = 0; i = i + 1;
if i < k then A [i] = 1;
C ost of Increm ent is num ber of bits flipped.
CP412 ADAII, Wilfrid Laurier University
Introduction # 15
Analysis of a Binary Counter
Binary counter bit sequence:
Value Bits
Cost
0
0…000 0
1
0…001 1
2
0…010 2
3
0…011 1
In the worst-case, a single increment can flip k bits.
There are n increments, so by standard worst-case analysis, the cost
is O(nk).
CP412 ADAII, Wilfrid Laurier University
Introduction # 16
Amortized Analysis of Binary Counter
Theorem: starting with 0, worst-case cost of n increments is O(n).
Proof:
Bit A[0] flips each increment.
Bit A[1] flips every other increment.
Bit A[i] flips during every 1/2i -th increment.
During n increments, bit A[i] flips n/2i times.
Assume n ≤ 2k; otherwise, the counter resets, and restart analysis.
Total cost
lo g n

i0

 n 
 1 
 2 i   n   2 i   2 n
i0
Amortized cost per increment is 2
CP412 ADAII, Wilfrid Laurier University
Introduction # 17
Accounting method of binary counter
Let us assign the following amortized costs Ai = 2 for every increment
operation
Prove that up to j increments, the actual cost and amortized total coast
satisfying:
j

i 1
Ci  2 j 
CP412 ADAII, Wilfrid Laurier University
j

i 1
Ai
Introduction # 18
CP412 ADAII, Wilfrid Laurier University
Introduction # 19
Potential method of binary counter
L et  i =  ( D i ) b e th e n u m b er o f 1 's in th e co u n ter
after n th o p eratio n , th en  i  0 =  0 fo r all i.
S u p p o se ith In crem en t resets t i b its.
A ctu al co st C i = t i + 1 .
 i  1  b i , th en  i  b i  t i + 1
A m o rtized co st
Ai  C i  (  i   i  1 )  ( t i + 1)  ( b i  t i + 1  bi )  2
n
T h u s, to tal am o rtized co st

i 1
CP412 ADAII, Wilfrid Laurier University
Ai  2 n .
Introduction # 20
Dynamic Array
Description
Think of an array initialized with a fixed number of slots, and
supports APPEND and DELETE operations.
When we APPEND too many elements, the array would be
full and we need to expand the array (make the size larger).
When we DELETE too many elements, we want to shrink to
the array (make the size smaller).
Requirement: the array must be using one contiguous block
of memory all the time.
CP412 ADAII, Wilfrid Laurier University
Introduction # 21
Append algorithm
CP412 ADAII, Wilfrid Laurier University
Introduction # 22
One way to expand
•
•
•
•
If the array is full when APPEND is called
Create a new array of twice the size
Copy the all the elements from old array to new array
Append the element
CP412 ADAII, Wilfrid Laurier University
Introduction # 23
Amortized analysis of expand
• Now consider a dynamic array initialized with size 1 and a
sequence of m APPEND operations on it.
• Analyze the amortized cost per operation
• Assumption: only count array assignments, i.e., append an
element and copy an element
CP412 ADAII, Wilfrid Laurier University
Introduction # 24
Content template
• …
CP412 ADAII, Wilfrid Laurier University
Introduction # 25
Aggregate Analysis
CP412 ADAII, Wilfrid Laurier University
Introduction # 26
Accounting method
How much money do we need to earn at each operation, so
that all future costs can be paid for?
• How much money to earn for each APPEND element ?
$1 ?
$2 ?
$3 ?
$m ?…
CP412 ADAII, Wilfrid Laurier University
Introduction # 27
Earn $1 for each appended element
This $1 (the “append-dollar”) is spent when appending the
element.
But, when we need to copy this element to a new array
(when expanding the array), we don’t have any money to
pay for it -BROKE!
CP412 ADAII, Wilfrid Laurier University
Introduction # 28
Earn $2 for each appended element
$1 (the “append-dollar”) will be spent when appending the
element
$1 (the “copy-dollar”) will be spent when copying the
element to a new array
What if the element is copied for a second time (when
expanding the array for a second time)?
BROKE!…
CP412 ADAII, Wilfrid Laurier University
Introduction # 29
Earn $3 for each appended element
$1 (the “append-dollar”) will be spent when appending the
element
$1 (the “copy-dollar”) will be spent when copying the
element to a new array
$1 (the “recharge-dollar”) is used to recharge the old
elements that have spent their “copy-dollars”.
NEVER BROKE!…
Amortized cost is A(n) = O(3n)
Average cost O(1)
CP412 ADAII, Wilfrid Laurier University
Introduction # 30
Potential Method
CP412 ADAII, Wilfrid Laurier University
Introduction # 31
Potential Method
CP412 ADAII, Wilfrid Laurier University
Introduction # 32
Potential Method
CP412 ADAII, Wilfrid Laurier University
Introduction # 33
Amortized analysis provides us valuable insights into what
is the proper strategy of expanding dynamic arrays.
Shrinking strategy:
When the array is ½ full after DELETE, create a new array
of half of the size, and copy all the elements.
Consider the following sequence of operations performed
on a full array with n element…
APPEND, DELETE, APPEND, DELETE, APPEND, …
Ɵ(n) amortized cost per operation since every APPEND
or DELETE causes allocation of new array.
NO GOOD!
CP412 ADAII, Wilfrid Laurier University
Introduction # 34
The right way of shrinking
When the array is ¼ full after DELETE, create a new array
of ½ of the size, and copy all the elements.
For any sequence of APPEND or DELETE , earning $3
per APPEND and $3 per DELETE would be enough for
paying all the cost.
1 append/delete-dollar
1 copy-dollar
1 recharge-dollar
Therefore the amortized cost per operation of any
sequence is upper-bounded by 3, i.e., O(1).
CP412 ADAII, Wilfrid Laurier University
Introduction # 35
Expansion and Contraction
Let α be element count / array size. When α drops too low,
contract the table.
– Allocate a new, smaller one.
– Copy all items.
• Still want
– α bounded from below by a constant,
– amortized cost per operation = O(1).
CP412 ADAII, Wilfrid Laurier University
Introduction # 36
Expansion and Contraction
Double as before:
– When inserting with α = 1
– After doubling, α = 1/2.
• Halve size
– When deleting with α = 1/4
– After halving, α = 1/2.
• Thus, immediately after either expansion or
contraction have
– α = 1/2.
• Always have
– 1/4 ≤ α ≤ 1.
CP412 ADAII, Wilfrid Laurier University
Introduction # 37
Expansion and Contraction
CP412 ADAII, Wilfrid Laurier University
Introduction # 38
Expansion and Contraction
CP412 ADAII, Wilfrid Laurier University
Introduction # 39
Expansion and Contraction
CP412 ADAII, Wilfrid Laurier University
Introduction # 40
Expansion and Contraction
CP412 ADAII, Wilfrid Laurier University
Introduction # 41