Download Definition of the state in dynamic programming

Definition of the state in dynamic programming
Matthias Kredler, Universidad Carlos III de Madrid, 9 Mar 2016.
Example: Growth with productity shocks
Consider a standard growth model in which output is produced according
to yt = At ktα . At is a productivity shock that follows a first-order Markov
process with conditional density f (At+1 |At ), kt is capital at t. The investment technology is standard; the planner has to choose tomorrow’s capital
(the control ) given a feasible-set correspondence
kt+1 ∈ Γ(kt , At ) ≡ [0, At ktα + (1 − δ)kt ].
The period-t return to the planner is
F (kt , At ; kt+1 ) = ln At ktα + (1 − δ)kt − kt+1 .
The planner discounts the future at factor β ∈ (0, 1). Thus the Bellman
equation is given by
n
o
V (k, A) = max
F (k, A; k 0 ) + βE V (k 0 , A0 )|A .
k0 ∈Γ(k,A)
General formulation
Consider now a more general stochastic environment with infinite horizon.
There is a shock sequence zt ∈ RNz which follows a first-order Markov
process. In our example, At , zt (meaning: “A takes the role of z”) and
Nz = 1. Suppose that we are given a vector yt ∈ RNy which informs us about
the feasible set for a control ut ∈ RNu and the return in period t. Specifically,
we are given a feasibility correspondence, Γ, and a return function, F :
ut ∈ Γ(yt , zt ),
Γ : RNy +Nz ⇒ RNu ,
F (yt , zt , ut ),
F : RNy +Nz +Nu → R
In the example, kt , yt and kt+1 , ut . Finally, suppose that we are given
a law of motion for y, i.e. a function h that tells us which value y takes
tomorrow:
y
h : RNy +Nz +Nu +Nz → RN .
yt+1 = h(yt , zt , ut , zt+1 ),
1
In the example, the function h is of trivial form: kt+1 is both the control
at t and (part of) the state tomorrow, thus yt+1 = ut . Given this environment, we can always write down a valid functional equation for a value
function V (·) that reads:
Z
n
o
V (z, y) = max F (z, y; u) + β V z 0 , h(y, z, u, z 0 ) f (z 0 |z)dz 0
u∈Γ(z,y)
However, we will see this formulation may be wasteful. It may be that we can
condense the state of the economy into a vector x of lower dimensionality
than (z, y). This turns out to be of huge value for both analytical and
computational purposes.
Definition: The state of the economy is the smallest set of variables, a
vector x ∈ RNx , Nx ≤ Nz + Ny , that allows us to determine all of the the
following:
• the feasible set, i.e. Γ̃(x) = Γ(z, y) for some correspondence Γ̃ : RNx ⇒
RNu ,
• the return given a control u, i.e. F̃ (x, u) = F (z, y, u), for some function
F̃ : RNx +Nu → R,
• the law of motion for y given u and z 0 , i.e. y 0 = h̃(x, u, z 0 ) for some
function h̃ : RNx +Nu +Nz → RNx ,
• and the conditional expectation in the Bellman equation, i.e. f˜(z 0 |x) =
f (z 0 |z) for some function f˜.
The Bellman equation using the state x is then
Z
o
n
Ṽ (x) = max F̃ (x, u) + β Ṽ h̃(x, u, z 0 ) f (z 0 |x)dz 0 .
u∈G̃(x)
Back to the example: i.i.d. shocks and the cash-on-hand trick
We now go back to our example. It turns out that for a specific case it is
possible to return the dimensionality of the state space to one. This case is
the one where the shock At is i.i.d.Ȧs the new state, let us define cash-onhand as xt = At ktα + (1 − δ)kt . The other objects are
Γ̃(x) = [0, x],
F̃ (x, k 0 ) = ln(x − k 0 ),
h̃(x, k 0 , A0 ) = A0 (k 0 )α + (1 − δ)k 0 .
2
The Bellman equation is
Z
o
n
0
Ṽ (x) = max F̃ (x, k ) + β Ṽ h̃(x, k 0 , A0 ) f (A0 )dA0 .
k0 ∈G̃(x)
Since the conditional density of A0 equal the unconditional density by the
i.i.d. assumption, the expectation in the Bellman equation is correct. We
thus see that cash-on-hand gives us full information about the economic
environment at t and that the state can be condensed in this case.
3