Download Dynamic Programming

Dynamic Programming
Econ 501
Fall 2009
Julieta Caunedo
1
Outline
Sequential Problem
Functional Equation
Principle of Optimality
Stochastic Version
References
2
Sequential Problem
M axfxt+1g1
t=0
s.t.
xt+1 2
( xt )
x0 2 X given
8t
1
X
t=0
t F (x ; x
t t+1 )
(SP)
Examples:
One sector model
M axfct;kt+1g1
t=0
s.t.
(kt+1 + ct; kt) 2 Y
Y
k0 2 R given
1
X
t U (c
(1)
t)
t=0
R2; or kt+1 + ct
f (k t )
Multiple sector
M axfct;kt+1g1
t=0
s.t.
(kt+1 + ct; kt) 2 Y
Y
k0 2 Rl given
R2l
1
X
t=0
t U (c
t)
(2)
3
Functional Equation
or recursive formulation: the general structure of the decision problem
recurs every period.
Bellman Equation
n
F (x; x0) +
W (x) = Sup x02 (x)
x0 2 X given, : X ! X
W ( x0 )
o
(FE)
4
Some De…nitions
The value function
V
Rl ! R
:
(3)
V (x0) = M axfxt+1g1
t=0
1
X
t F (x ; x
t t+1 )
t=0
A plan
fxt+1g1
t=0 in X
The set of feasible plans
( x0 ) =
n
fxt+1g1
t=0
: xt+1 2 (xt);
o
8t
5
Assumptions
(x) non empty
e2
8x0 2 X and x
(x0), limn!1
n
P
t=0
t F (x ; x
t t+1 )
exists.
6
Principle of Optimality
1. The value function de…ned in (5) satis…es the Bellman Equation, hence
If jV (x0)j < 1, then
V ( x0 )
F ( x0 ; x 0 ) + V ( x0 )
8x0 2 (x0)
and for any " > 0
V ( x0 )
F ( x 0 ; x 0 ) + V ( x0 ) + "
some x0 2 (x0)
n
o
If V (x0) = +1 , then there exist a sequence x0k 2 (x0) such that
lim F (x0; x0k ) + V (x0k ) = +1
k!1
If V (x0) =
1 , then 8x0 2 (x0) such that
F ( x0 ; x 0 ) + V ( x0 ) =
1
8x0 2 (x0)
n
o1
xt+1
is a solution to (1) then the sequence satis…es
t=0
2. If a sequence
the Bellman Equation
V (xt ) = F (xt ; xt+1) + V (xt+1)
e 2
3. Let W be a solution to (FE) and limn!1 nW (xn) = 0 8 x
(x0)8x0 2 X , then
W =V
e 2
4. If a feasible plan x
(x0) satis…es,
V (xt ) = F (xt ; xt+1) + V (xt+1)
and limn!1 nV (xn) = 0; then x is a solution to (1)
Sketch of the Proof:
in class...
8t
7
7.1
Stochastic Version
Sequential Problem
M axfxt+1g1 E 4
t=0
s.t.
xt+1 2
2
( xt ; z t )
x0 2 X z 0 2 Z
8t
1
X
t=0
3
t F (x ; z x
t t; t+1 )5
(SP)
given
where X is the set of possible values for the endogeneous state variable and Z is
that for the exogenous state space variable. (X; ) and (Z; ) are measurable
spaces and (S; )=(X xZ; x ) is the set of possible states of the system.
Shocks evolve according to a stationary transition function Q on (Z; ) and
the expectations is taken with respect to this meassure.
Information in this set up will be summarized by sequences z t = (z1; z2; ::::) 2
Z t where (Z t; t) is a product space.
Example:
One sector model
M axfct;kt+1g1 E
t=0
s.t.
(kt+1 + ct; kt) 2 Y (zt)
Y (zt )
k0 2 R z0 2 Z given
1
X
t U (c
t)
t=0
R2; or kt+1
+ ct
(4)
zt f ( kt )
7.2
Functional Equation
Bellman Equation
F (x; z; x0) +
W (s) = W (x; z ) = Sup x02 (x;z)
x0 2 X z0 2 Z given, : X xZ ! X
where
h
i
E W ( x0 ; z 0 ) =
Z
h
E W ( x0 ; z 0 )
i
(FE)
W (x0; z 0)Q(dz 0; z )
If there exist a function W satisfying the bellman equation, then we can de…ne
the associated policy correspondence
n o
G(s) = x0 2
( s ) : W (s ) =
F (x; z; x0) +
h
E W ( x0 ; z 0 )
i
7.3
Some De…nitions
The value function
V
:
Rl xZ ! R
2
V (x0; z0) = M axfxt+1g1 E 4
t=0
1
X
t=0
3
t F (x ; z x
t t; t+1 )5
A plan is a value 0 2 X and a sequence of measruble functions
Z t ! X , t = 1; 2; :::::
The set of feasible plans
t
0 2
(z t ) 2
from s0 2 S
( x0 ; z 0 )
( t 1 (z t 1 ); z t )
(5)
8zt 2 Z t, t = 1; 2; ::::
t
:
7.4
Assumptions
(x; z ) non empty-valued and the graph of
has a measurable selection,
9
h:S!X
F : graph( ) ! R; is
2 (s0),
is
x x
measurable.
such that h(s) 2 (s); all s 2 S
x x
measurable and for each s0 2 S and
F ( t 1(z t 1); zt; t(z t)) is t(z0;:) integrable
and
lim
n!1
1
X
t=0
t
Z
F ( t 1(z t 1); zt; t(z t)) t(z0;dz t) exist
where if B = A1xA2x::::: 2
t (z B )
0;
=
Z
A1
:::
Z
Z
At 1 At
t
Q(zt 1;dzt)Q(zt 2;dzt 1)::::::Q(z0;dz1)
7.5
Principle of Optimality
R
1. Let W be a solution to (5) and limt!1 z t W ( t 1(z t 1); zt) t(z0;dz t)
= 0 8 2 (s0)8s0 2 S , then
W =V
2. Let G de…ned as before, and suppose G is non-empty and admits a measurable selection, then any plan generated by G attains the supremum
of the sequential problem.
Sketch of the Proof:
see Stokey & Lucas
8
References
Stockey and Lucas (1989). Recursive Methods in Economic Dynamics,
Sections 2.1, 4.1-4.2 and 9.1
Sargent and Ljungqvist (2004). Recursive Macroeconomic Theory, Section
3.1.