Download Recurrence Relations

This work is licensed under a Creative Commons Attribution-Share Alike 3.0 Unported License.
CS 312: Algorithm Analysis
Lecture #7: Recurrence Relations
a.k.a. Difference Equations
Slides by: Eric Ringger, with contributions from Mike Jones, Eric Mercer, Sean Warnick
Announcements
 Looking ahead to Project #2
 Much more challenging than Project #1
– plan accordingly
 In particular, the report will take some time!
Objectives
 Goal: Analyze Divide and Conquer recursive
algorithms using recurrence relations (RRs)
 Also known as Difference Equations
 We will work up to such RRs
 Today: Focus on a special type of RRs
 Homogeneous
 Linear
 Constant Coefficients
 Leading up to a proof of the Master Theorem
Recall: Analysis Framework
Given a problem,
 Identify platform’s elementary operations
(sometimes implicit)
 Formulate solution as an algorithm
 Define the measure of the input size
 Measure time efficiency by counting the number of
times an elementary operation is executed
 Measure space efficiency by counting the number of
memory units consumed by the algorithm
 The efficiency of some algorithms may differ
significantly for inputs of the same size.
 Distinguish among worst-case, average-case, and
best-case efficiencies. Choose one.
 Plot efficiency vs. input size
 Establish order of growth. Use asymptotic notation.
c2g(n)
f(n)
c1g(n)
n0
f (n)  ( g (n))
Where is the difficulty for Analyzing
Recursive Functions?
Recurrence Relations
 Theory to aid us in analyzing recursive algorithms
 Given a sequence of (real) numbers
𝑦 0 ,𝑦 1 ,𝑦 2 ,𝑦 3 ,…,𝑦 𝑘 − 2 ,𝑦 𝑘 − 1 ,𝑦 𝑘 ,𝑦 𝑘 + 1 ,…
 A recurrence relation (RR) is an equation (a rule) expressing the value
𝑦(𝑘) in terms of earlier (usually neighboring) values.
 Examples:
𝑦 𝑘 = 4𝑦 𝑘 − 1 + 7𝑦 𝑘 − 2 + 3
𝐶 𝑁 =𝐶 𝑁−1
2
+𝑁
 The RR describes a function (e.g., 𝑦(𝑘) or 𝐶 𝑁 ) by such a rule instead
of by a list of its values!
 𝑘 is the index variable (representing, e.g., moment in time, size of an
input to an algorithm, etc.)
Recursive Algorithms
Example: Compute Factorial function N!
function Factorial(N)
if N=0 return 1
else return Factorial(N-1)*N
•
•
•
•
Assume 32-bit integers
Define input size: _____
Define elementary operations:
Focus on worst-case
Cworst ( N )  Cworst ( N  1)  4
What sequence is generated by this recurrence relation?
In order to answer that question we need an initial condition!
C (0)  1
Now we can build a table of values:
How long to compute C(1,000,000)?
Want: C(N) in closed form for quick computation.
N
C(N)
Recursive Algorithms
Example: Tower of Hanoi, move all disks to third peg
without ever placing a larger disk on a smaller one.
How to solve?
Recursive Algorithms
Example: Tower of Hanoi, move all disks to third peg
without ever placing a larger disk on a smaller one.
Recursive Algorithms
Example: Tower of Hanoi, move all disks to third peg
without ever placing a larger disk on a smaller one.
C (n)  C (n  1)  ...
Recursive Algorithms
Example: Tower of Hanoi, move all disks to third peg
without ever placing a larger disk on a smaller one.
C (n)  C (n  1)  1  ...
Recursive Algorithms
Example: Tower of Hanoi, move all disks to third peg
without ever placing a larger disk on a smaller one.
C (n)  C (n  1)  1  C (n  1)
C (n)  2C (n  1)  1
C (1)  1
Recursive Algorithms
Example: Tower of Hanoi, move all disks to third peg
without ever placing a larger disk on a smaller one.
C (n)  2C (n  1)  1
C (1)  1
Can you figure out an explicit (closed form) formula for this sequence?
• As before, build a table, recognize the pattern
OR
• Use substitution, recognize the pattern
OR
• Appeal to theory of recurrence relations
n
C(n)
Substitution
y (k )  2 y (k  1)
k  1, 2,
y (0) : initial condition
y (1)  ay (0)
y(k)
y (2)  ay (1)  a y (0)
2
y (3)  ay (2)  a y (0)
3
k
1 2 3 4 5 6 7 8 9
Recurrence Relations
The most general form for our purposes:
y(k )  f  y(k 1), y(k  2),..., y(k  n), k   x(k )
RR is linear when it can be written as:
a0 (k ) y(k )  a1 (k ) y(k  1) 
 an (k ) y(k  n)  x(k )
where:
 𝑥(𝑘) is arbitrary function of 𝑘
 Coefficients 𝑎𝑖 (𝑘) are functions of 𝑘 independent of 𝑦
for all 𝑖 in {0,1,2, … . , 𝑛}
 Order 𝑛 is finite
Constant Coefficients
a0 (k ) y(k )  a1 (k ) y(k  1) 
 an (k ) y(k  n)  x(k )
 If the coefficients 𝑎𝑖 (𝑘) (1 ≤ 𝑖 ≤ 𝑛) do not
depend on 𝑘, the equation is said
 to have constant coefficients
 to be time-invariant.
a0 y(k )  a1 y(k  1) 
 an y(k  n)  x(k )
 LTI = “linear, time-invariant”
Forcing Function
a0 y(k )  a1 y(k  1) 
 an y(k  n)  x(k )
 The function 𝑥(𝑘) is called





the forcing function
the forcing term
the driving term
the system input
or simply the right-hand side
Homogeneous
 A solution of the RR is a function 𝑦(𝑘) that satisfies
the recurrence relation:
a0 y(k )  a1 y(k  1) 
 an y(k  n)  x(k )
 A recurrence relation is said to be homogeneous if
𝑦(𝑘) = 0 is a solution of the equation (for all 𝑘).
 Consequence:
a0 y(k )  a1 y(k  1) 
 an y(k  n)  0
 We’ll focus on this type today.
What kind of RR is this?
y (k )  ay (k  1)  0
k  1, 2,
y(k)
1 2 3 4 5 6 7 8 9
k
Terminology Review
Note the alternate notations.
Existence and Uniqueness
Theorem
For an arbitrary real-valued function 𝑓, let a difference equation of the
form
𝑦(𝑘) + 𝑓 𝑦(𝑘 − 1), 𝑦(𝑘 − 2), . . . , 𝑦(𝑘 − 𝑛), 𝑘 = 0
be defined over a sequence of consecutive integer values of 𝑘 (𝑘 ∈
𝑘0, 𝑘0 + 1, 𝑘0 + 2, … ). Then the equation has one and only one
solution corresponding to each specification of the 𝑛 initial values (i.e.,
initial conditions) 𝑦(𝑘0), 𝑦(𝑘0 + 1), … , 𝑦(𝑘0 + 𝑛 − 1).
Proof (sketch): Suppose the 𝑛 initial values are specified. Then the
value of 𝑦(𝑘0 + 𝑛) can be uniquely determined simply by evaluating
the function 𝑓. The proof then proceeds by induction for each
consecutive value of 𝑘.
Starting Point
 The theory of recurrence relations (and
linear systems) gives us a basic solution:
𝑦 𝑘 = 𝑟𝑘
For some value 𝑟.
Goal: find solution tn
Example
General Solution
 For this type of RR, the general solution is
a linear combination of 𝑛 linearly
independent solutions.
Solution?
Assignment
 Read: the rest of the Recurrence Relations
Notes
 HW #5: Part I (Section 1.2) Exercises in
the RR notes
 we’ll do these in class together
End
Extra: Big Picture
 Look at the big picture: functions as
“systems”
Recurrence Relations
So why should we care about recurrence relations?
Generalizes the idea of a function to an operator or dynamic system
A function is like a table that takes a number in and gives a number out.
x
y
f
x
y
1
2
3
4
2
4
8
16
y
y
1 2 3 4 5 6 7 8 9
x
1 2 3 4 5 6 7 8 9
x
Recurrence Relations
Functions can also take in a vector of numbers or yield a vector out.
x 
x   1
 x2 
 y1 
y   y2 
 y3 
f
x1
1
1
1
1
y1
x1
y2
y3
x2
x1
x1
x2
x2
x2
1
3
2
4
y1
1
2
1
2
y2
1
2
3
4
y3
3
5
6
3
Recurrence Relations
What if the input and output vectors of a function were infinitely long?
 x1 
x 
x   2
 x3 
 

x
 y1 
y   y2 
  
f
f becomes a map from one
function to another!
We call maps that take
functions as inputs or
generate functions as outputs
operators, or systems.
y
123456789 k
Then x and y themselves are
functions!
123456789 k
An operator is like a rule,
it’s the mathematical
description of a
subroutine!
Recurrence Relations
x(k)
x
y(k)
S
y
123456789 k
123456789 k
Recurrence Relations
x(k)
S
y(k)
Unlike a simple function, an operator
can have local variables that store
intermediate calculations—just like
a subroutine.
In other words, S is a map that can
have memory!
Recurrence Relations
x(k)
y(k)
S
y(0)=yo
Subroutine Description
Math Description
static memory: w = 0
function S(x,k)
if k=0 then y f yo
else y f w + x
wfy
return y
y (k  1)  y (k )  x(k )
y (0)  yo
Recurrence Relations
x(k)
y(k)
S
y(0)=yo
Subroutine Description
static memory: w = 0
function S(x,k)
if k=0 then y f yo
else y f 5w + x
wfy
return y
Math Description
y (k  1)  5 y (k )  x(k )
y (0)  yo
Recurrence Relations
x(k)
y(k)
S
What if more than one
number is stored in
memory?
y(0)=yo, y(1) = y1
Subroutine Description
static memory: w0, w1 = 0
function S(x,k)
if k=0 then y f yo
else if k=1 then y f y1
else y f a1w1 + aow0+x
w0 f w1
w1 f y
return y
Math Description
y (k  2)  a1 y (k  1)  a0 y (k )  x(k )
y (0)  yo
y (1)  y1