Download Introduction to Probability Supplementary Notes 2 Recursion Instructor:

Introduction to Probability
Supplementary Notes 2
Recursion
MSIS 26:960:575:01
Instructor:
Farid Alizadeh
October 7, 2000
Topic 1: Recursive formulas based on one variable
In many cases you will need to reason recursively in order to solve a problem.
In general for an unknown function of n, say fn a recursive relation, instead of
giving an explicit formula for fn , gives a relation which expresses fn in terms
of fn−1 (or sometimes fn−1 and fn−2 and so on) and some other explicitly
known function. You will also need initial values for instance f0 or f1 (or
f0 and f1 if the recurrence relation depends on two previous value). With
initial values and the recurrence formulas you can completely determine fn .
Example1: How many subsets does the set An = {1, 2, · · · , n} have?
Answer: We have already solved this problem and shown that the number
is 2n . Let us get this again by recursive methods. To approach the problem,
let us list the subsets of A0 = {}, A1 = {1}, A2 = {1, 2}, and A3 = {1, 2, 3}:
A0
A1
A2
A3
= {} :
= {1} :
= {1, 2} :
= {1, 2, 3} :
{}
{}, {1}
{}, {1}, {2}, {1, 2}
{}, {1}, {2}, {1, 2}, {3}, {1, 3}, {2, 3}, {1, 2, 3}
1
If you look at the list above you will see an algorithm: To get the list of
subsets of An , first copy the list of subsets of An−1 , (because they are also
subsets of An without the new element n in them) and then copy that same
list again, except that this time add the new element n to each subset. This
reasoning also gives you an immediate recurrence relation. If an is the number
of subsets then we have shown:
an = 2an−1
from here and the fact that a0 = 1 it should be easy to show by induction
that an = 2n .
Example 2: How many subsets does the set An = {1, 2, · · · , n} have so that
no subset contains consecutive numbers?
Answer: We attack this problem just like Example 1. First let us list all
such subsets for a few values of n:
A0
A1
A2
A3
A4
= {} :
= {1} :
= {1, 2} :
= {1, 2, 3} :
= {1, 2, 3, 4} :
{}
{}, {1}
{}, {1}, {2}
{}, {1}, {2}, {3}, {1, 3}
{}, {1}, {2}, {3}, {1, 3}, {4}, {1, 4}, {2, 4}
Again if you examine the list you will see the following pattern: Obviously
the list of subsets of An−1 without consecutive elements are still valid subsets
of An ; these are exactly those subsets that don’t have the new element n in
them. To list those subsets that do have n in them observe that they cannot
possibly have n − 1 in them, so all we have to do is to list the subsets of
An−2 and add the new element n to them. This will account for all possible
subsets of An with no consecutive numbers in them. Here again, if fn is the
desired number, then we have actually proved the recurrence:
fn = fn−1 + fn−2
with the initial conditions that f0 = 1 and f1 = 2. As you may know
these numbers are called the Fibonacci numbers. Observe that the recurrence
relation along with the initial numbers for f0 and f1 are all you need to
determine the Fibonacci numbers. If I ask you the value of fn for any n you
can easily write a program to calculate it. In this case it is also possible to
2
find an explicit formula for fn , we will derive it when we discuss generating
functions. However, there does not seem to be any direct combinatorial proof
for this explicit formula.
Topic 2: Number of walks in graphs
Consider the following directed multigraph:
a
1
b
h
c
d
g
i
2
3
e
f
We want to know how many walks of length exactly equal to n are there
from vertex 1 to vertex 3. Notice that each walk is a sequence of directed
edges and if there are two or more edges between two vertices, (as in between
vertices 1 and 3 in our graph) then these edges are counted separately and
therefore walks made up of one is different from walks made up of the other.
Also loops that connect a vertex to itself, if taken, count as one step. So, for
instance “abddef” is counted as a different walk of length 6 than, say “acdedf”
even though the sequence of vertices visited are identical in both walks.
Another way of viewing this problem is to ask for number of n–letter words
over the alphabet {a, b, c, d, e, f, g, h, i} with restrictions on which letters can
follow each letter, and theses rules are encoded in the graph above.
Now to count such walks let us define
an : number of walks of length n from vertex 1 to vertex 1,
bn : number of walks of length n from vertex 1 to vertex 2,
cn : number of walks of length n from vertex 1 to vertex 3.
We are interested in cn . In order to obtain a recurrence relation, notice that
for a walk to have length n and end in vertex 3, it needs to have walked a
length of n − 1 and have ended up in vertices 1 or 2 and then have taken a
3
final step into vertex 3. Now if it was in vertex 1 before moving to vertex
3 then there are, by definition, an−1 ways of getting from 1 to 1 in n − 1
steps and then there are two ways of walking from 1 to 3. Therefore there
are 2an−1 ways of getting from 1 to 3 with one to the last stop at 1. On the
other hand if the last vertex before 3 is vertex 2, then there are, by definition,
bn−1 ways of getting there, and then there is only one way of getting from
2 to 3. We conclude that the total number of ways of getting from 1 to 3
satisfies:
cn = 2an−1 + bn−1
However, this is not enough to compute cn because we have no formulas or
recurrences for an or bn , and so we need to discover such recurrences for them
too. The logic is quite similar to the case of cn . For bn , we have an−1 ways
of going from 1 to 1 in n − 1 steps and then 2 ways of going from 1 to 2, and
so there are 2an−1 ways of getting from 1 to 2 with one to the last stop at 1.
Similarly there are 2bn ways of going from 1 to 2 with one to the last stop at
2. Thus we get
bn = 2an−1 + 2bn−1
Finally we need a recurrence for an and similar argument as before will give:
an = an−1 + cn−1
Now we set a0 = 1 and b0 = c0 = 0. (There is one way of going from 1 to 1
without taking any steps at all, and that is taking the empty walk: Simply
stay put! There is of course no way of getting from 1 to 2 or to 3 with zero
steps.) We have now all the information to obtain an , bn and cn . There is a
shorthand way of writing the three recurrences and that is by using matrix
notation:





an
an−1
1 0 1





 2 2 0   bn−1  =  bn 
cn−1
cn
2 1 0
More succinctly,
vn = Avn−1 = A2 vn−2 = · · · = An v0
where


1 0 1


A =  2 2 0 ,
2 1 0


an−1


vn =  bn−1 
cn−1
and
This formula can be generalized to any multigraph:
4


1


v0 =  0 
0
Theorem 0.1 Let G be a multigraph with vertices {1, 2, · · · , n} and zero or
more edges from each vertex i to each vertex j. Let A be an n×n matrix such
that aij is the number of edges from vertex i to vertex j. Then the number
of walks of length exactly equal to n from vertex i to vertex j is the ij entry
of matrix An .
Proof: The proof is by induction and it is essentially the adaptation of the
way we obtained the recurrence relation for the three vertex graph above.
Suppose the statement is true for all paths of length n − 1, that is the i, j
entry of the matrix An−1 counts the number of walks of length n − 1 from
vertex i to vertex j, for any pair of vertices i and j. Now, to go from vertex
i to vertex j in n steps it is necessary to go from vertex i to some vertex
in n − 1 steps and then from there to vertex j in one step. This one to
the last vertex could be any one of vertices 1, 2, · · · , n. Let’s assume it is
vertex k. Then the number of ways of getting from i to k in n − 1 steps, by
induction hypothesis, is (An−1 )ik , that is the i, k entry of matrix An−1 . Now
by definition of matrix A, the number of ways to go from vertex k to vertex
j is Akj . Thus the total number of walks from vertex i to vertex j with one
to the last stop at vertex k, by principle of multiplication, is (An−1 )ik Akj .
Summing over all possible values of k from 1 to n we count all possible walks
of length n from i to j:
(An−1 )i1 A1j + (An−1 )i2 A2j + · · · + (An−1 )in Anj
But by definition of matrix multiplication, this sum is nothing but the i, j
entry of the matrix product An−1 .A, which is equal to An . And we are done!
Topic 3: Two-variable recurrences
Recursive formulas do not only apply to formulas with one parameter like
n. One can have formulas depending on two or more parameters. In these
cases, however, instead of initial conditions, one needs boundary conditions;
let us look at a familiar example.
Example 3: Let us forget that we know an explicit formula for
n
k
n
k
and
try to find a recursive formula for it. We know that
counts the number
of k–subsets of an n–set. Each such subset has either the element n in it
5
or not. If it does
not then it is actually a k-subset of An−1 ; by definition
n−1
there are k such subsets. If the subset has the element n in it, then upon
removing n we get a subset
of k − 1 elements of a set of n − 1 elements. By
n−1
definition there are k−1 such subsets. Since these two cases account for all
possibilities, we have actually proved that
!
!
!
n
n−1
n−1
=
+
k
k
k−1
Now this formula in itself is not enough to determine the values of
need to have some initial conditions. We know that
(Why?) And that
n
n
= 1 and
can now tabulate values of
k ↓, n →
0
1
2
3
4
5
0
1
1
1
1
1
1
1
2
n
k
n
0
n
k
n
k
. We
= 0 for k > n.
= 1. With the latter two conditions we
:
3
4 5
1
2 1
3 3 1
4 6 4
5 10 10
1
5 1
Here the
boundary
conditions–literally the two sides of the triangle–are
n
numbers 0 and nn .
Example 4: Stirling numbers of the second kind. Remember that
a partition of a set An = {1, 2, · · · , n} is a collection of subsets of An , say
B1 , B2 , · · · , Bk such that
i. Bi ∩ Bj = φ,
ii. B1 ∪ B2 ∪ · · · ∪ Bn = An , and
iii. Bi 6= φ.
We are interested in the number of different partitions of a set of n elements
into k blocks. Such(numbers
are called Stirling numbers of the second kind
)
n
and are denoted by
. To count these kinds of partitions, we focus again
k
6
on the largest element of the set, that is n (or any other element, it really
does not matter.) In particular we focus on the block of the partition that
contains n. There are two possibilities:
1. The block that contains n is a singleton set {n}. Suppose we remove n
from our set. Then this block disappears and we are left (
with a partition
)
n−1
of a set of n − 1 elements into k − 1 blocks. There are
such
k−1
partitions by definition. And to each one we can add the singleton set
{n} to make a partition of An into k blocks.
2. The block that contains n contains some other elements in it. Now
if we remove n from An as before, the block that contains n will not
disappear: Here we have
a partition
of a set of n − 1 elements into still
(
)
n−1
k blocks. There are
such partitions by definition. Now the
k
new element n can get into any one of the k blocks to make a partition
of An -set into k blocks.
The argument above immediately proves the recursion:
(
n
k
)
=
(
n−1
k−1
)
+k
(
n−1
k
)
What about the boundary conditions? Well, we know that
(
n
1
)
=
(
n
n
)
= 1.
(Why?)
boundary conditions and the recurrence are sufficient to generate
( These
)
n
for all values of n and k. One can create a triangle similar to the
k
triangle we obtained for nk :
k ↓, n →
1
2
3
4
5
6
1
1
1
1
1
1
1
2
3
4
5
6
1
3 1
7 6 1
15 25 10 1
31 90 65 15 1
7
Example: Balls and bags revisited: Let us consider again the case of n
balls and r bags. This time however, the bags are non-distinct and the balls
are distinct.
1. How many arrangements are there such that no bag is empty?
2. How many arrangements are there with empty bags allowed?
Answer: None of the bags are empty and the balls are distinct, but the bags
are not distinct, so in the following picture the first and second arrangements
are considered the same, but the third arrangement is different.
1
3
2
4
6
5
7
8
First arrangement
3
1
4
5
8
7
6
2
Second arrangement
6
2
1
3
4
5
7
Third arrangement
8
8
In this case observe that the bags essentially partition
(
) the set of n distinct
n
balls into r nonempty parts. Thus the answer is
.
r
Now if some of the bags are empty, we decompose the problem into the
case where exactly one bag is nonempty, or exactly two bags are nonempty,
or exactly three bags are nonempty, ... or exactly r bags are nonempty. The
answer therefore is equal to:
(
n
1
)
+
(
n
2
)
+ ···
(
n
r
)
Topic 4: Recurrences with full memory
So far the recurrences that we have seen relate an to an−1 or a few previous
terms. There are however recurrences that need all of the values of ak for
k = 1, 2, · · · , n − 1 to obtain an . Let us see a couple of examples:
Example: Bell numbers. We are interested in the total number of partitions of a set of n elements. Let us call such numbers bn . These numbers are
sometimes called Bell numbers. Of course, based on what we know so far we
can compute these numbers:
bn =
(
n
1
)
+
(
n
2
)
+ ··· +
(
n
n
)
(Why?) Let us derive a recurrence formula directly for these numbers. Take
the set An+1 = {1, 2, · · · , n, n + 1}. We focus on the block that contains
n + 1. This particular bloc, in addition to the element
n + 1, contains k other
n
elements where k is either 0 or 1 or ... n. There are k ways of choosing the
companions of n + 1 that go with it into a block. Then we are left with n − k
other elements and we may partition these in any way we want and attach the
resulting blocks to the block that contains n + 1 and its k companions. There
are, by definition,
bn−k such partitions. Thus, by principle of multiplication,
n
we have k bn−k such partitions. But this counts only the case k; ranging
over all possible k = 0, · · · , n and adding up we obtain
bn+1
!
!
!
!
n
n
n
n
bn
bn−k + · · · +
b1 + · · · +
b0 +
=
0
k
n−1
n
9
Observing that
n
k
=
n
n−k
we get:
bn+1 =
n
X
k=0
bk
n
k
!
We also agree that b0 = 1: There is only one partition of the empty set and
that is the empty partition! Therefore we have all the necessary information
to calculate the Bell numbers using this initial condition and the recurrence
relation above. Note that in the recurrence relation bn+1 depends on all
values of bk for k = 0, · · · n.
Here is another example.
Example: Catalan numbers. How many unlabeled binary search trees
are there with exactly n nodes?
Answer: First recall that a binary search tree is made up of a root and a
right subtree and a left subtree each of which may be either empty or another
nonempty binary search tree. Here is a list of such trees for n = 1, 2, 3:
n= 1
n= 2
n= 3
Let the number of such trees be cn ; these are sometimes called the Catalan
numbers. Let us try to find a recurrence relation for cn+1 . If a tree has n + 1
nodes and assuming its left subtree has k nodes, then necessarily its right
subtree has n−k nodes. There are by definition ck ways to choose the k-node
left subtree and cn−k ways to choose the (n − k)-node right subtree. Thus
there are ck ×cn−k trees whose left subtree has exactly k nodes. Summing over
all possibilities of k = 0, 1, 2, · · · , n we account for all cn+1 trees. Therefore
10
we have shown the recurrence:
cn+1 = c0 cn + c1 cn−1 + c2 cn−2 + · · · + cn c0
We also know that c0 = 1: There is only one tree with no nodes and that is
the empty tree. We have all the information to compute these numbers from
our initial conditions and the recurrence relation. Again notice that in this
recurrence, we need all previous values of ck to compute cn+1 . Later when
we discuss generating functions, we will derive an explicit formula for these
numbers.
11