Download Martingale problem approach to Markov processes

Martingale problem approach to Markov processes
Rajeeva L. Karandikar
Director
Chennai Mathematical Institute
[email protected], [email protected]
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Markov chains and Markov processes
Markov chains and Markov processes have played a very
important role in applications of probability theory to real
world problems.
The early development of probability theory focused on
experiments that could be repeated large number of times leading to the law of large numbers and central limit theorems
for partial sums of an i.i.d. sequence of random variables.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Markov chains and Markov processes
However, when applications to evolutionary phenomenon were
considered, it was clear that one has to consider situations
other than i.i.d- independent and identically distributed.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Markov chains and Markov processes
An early example of a situation where one has to go away
from i.i.d. nature is a Queueing system: where even if we
assume that arrivals in an hour are i.i.d. and number of
services completed in an hour is also i.i.d. - the resulting
queueing process is not partial sum of i.id. variables, as there
can be no service when the queue is empty!
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Markov chains and Markov processes
Branching processes and birth and death processes, used to
model spread of an infection or to model growth of population
are also examples that do not fit into partial sums of i.i.d.
random variables.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Markov chains and Markov processes
For a Markov chain, in many applications one is interested in
knowing if the Markov chain admits a stationary distribution,
and if so, is it unique.
The discrete time Markov Chain on a finite state space are
easiest to analyze: the transition probability matrix P and its
eigenvalues and eigenvectors yield most of the answers.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Markov chain in continuous time
For a Markov chain in continuous time, but finite state space
S, the quantity of interest is the transition probability function
pij (t) = P(Xt+s = j | Xs = i), i, j ∈ S.
pij (t) satisfies
pij (t + s) =
∑ pik (t)pkj (s).
k∈S
It can be shown that pij (t) is differentiable. Let
qij =
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
d
pij (t)t=0 .
dt
Director, Chennai Mathematical Institute
Forward and Backward equations
It can be shown that pij (t) satisfies the equations (shown on
next slide) known as Kolmgorov backward equation and
Kolmogorv forward equation Kolmgorov backward equation:
d
pij (t) = ∑ qik pkj (t)
dt
k∈S
Kolmogorv forward equation:
d
pij (t) = ∑ pik (t)qkj
dt
k∈S
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
The Q matrix
When the state space is countable, under mild regularity
conditions, the forward equation still is valid and yields the
transition probability function.
Thus, the matrix Q = ((qij )) can be used to recover properties
of pij (t) .
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Markov processes on a continuous state space
Let the state space S = R or S = Rd or S be a general
complete separable metric space.
For a Markov process X , the function P:
P(t, Xs , A) = P Xt+s ∈ A | {Xu : 0 ≤ u ≤ s, }
is the transition probability function and determines the
properties of the process.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Markov process and its semigroup
Here too, one has
Z
P(t + s, x, A) =
P(s, x, dy )P(t, y , A).
(1)
The Semi-group theory comes in useful for the study of
Markov process. The operators Tt is defined for a bounded
continuous f on S (f ∈ Cb (S)) by
Z
(Tt f )(x) =
f (y )P(t, x, dy ).
Using (1) one has: Tt ◦ Ts = Tt+s (the semi-group property).
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Generator of the semi-group
The generator L of the semi-group is defined as follows: Let
D0 (L) be the class of f ∈ Cb (S) such that t 7→ Tt f (x) is
continuous for all x. Let D(L) (domain of L) be the class of
f ∈ Cb (S) such that
Tt f (x) − f (x)
= g (x)
t
t↓0
lim
(2)
exists for all x ∈ S and g ∈ D0 (L).
For f ∈ D(L), Lf is defined to be g where g is given by (2).
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Stationary distribution
One of the questions of interest for a Markov process is :
Does it admit a stationary distribution π?
i.e. if X0 has distribution π, then for each t, distribution of
Xt is also π.
It can be shown that π is a stationary distribution if and only if
Z
(Lf )dπ = 0 ∀f ∈ D(L)
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
(3)
Director, Chennai Mathematical Institute
Evolution equation
Notation: for a function f and measure µ on S, let hf , µi
R
denote the integral fd µ.
For initial distribution µ0 (distribution of X0 ), let µt denote
the distribution of Xt . Then (µt ) satisfies the equation
hf , µt i = hf , µ0 i +
Z t
0
hLf , µu idu ∀f ∈ D(L), ∀t ≥ 0.
(4)
Indeed, (under mild conditions on the process (Xt )) the
equation (4) admits a unique solution and thus characterizes
µt - the law of Xt
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Evolution equation
Let (Xt ) be the Markov process with generator L and initial
distribution µ0 . For g ∈ Cb (S) and 0 ≤ t < ∞, let the measure
νt be defined by
h
i
R
νt (A) = E 1A (Xt ) exp( 0t g (Xs )ds) .
Then it can be shown that νt satisfies
hf , νt i = hf , µ0 i +
Z t
0
hLf , νu idu +
Z t
0
hgf , νu idu ∀f ∈ D(L)
(5)
and that the perturbed evolution equation (5) admits a unique
solution.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Filtering theory
Filtering theory:
Suppose a signal (Xt ) (assumed to be Markov) is transmitted
over a noisy channel and one observes
Z t
Yt =
0
h(Xs )ds + Wt
where Wt is a wiener process. The interest is in computing the
conditional distribution of Xt given {Yu : u ≤ t}:
πt (A)(Y ) = E (Xt ∈ A | Yu : u ≤ t).
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
FKK and Zakai equations of filtering
In early seventies, it was shown that the non linear filter
satisfies an equation called the Fujisaki-Kallianpur-Kunita
(FKK) equation- this is a non liner perturbation of the
evolution equation (4). In a special case Zakai introduced a
quantity σt (A)(Y ) such that σt satisfies a linear equation (a
perturbation of (4)) and when normalized to have measure
one, yields the conditional distribution πt . Hence σt has been
called the unnormalized conditional distribution. Both FKK
and Zakai equations are infinite dimensional stochastic
differential equations.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
FKK and Zakai equations of filtering
The question of uniqueness of solution to the FKK and Zakai
equation is connected with uniqueness question for the
perturbed evolution equation (5).
The uniqueness of solution to Zakai equation and the FKK
equation was shown in the case when the function h is
bounded.
Even in this special while the equations characterize the filter
πt , a major difficulty is that like the perturbed evolution
equation (5), the class of test functions is D(L).
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
FKK and Zakai equations of filtering
In most cases, the class D(L) is difficult to characterize. Even
in the simplest case when S = Rd and Xt is the Brownian
motion, it can be shown that Cb2 (Rd ) - the class of twice
continuously differentiable functions - is contained in D(L) and
for f ∈ Cb2 (Rd ),
1
Lf = ∆f .
2
Thus L is an extension of the Laplacian, but L is not equal to
the Laplacian. It is difficult to give a complete description of
D(L) and hence a criterion which involves ∀f ∈ D(L) is not
useful.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Martingale Problem
Question: Can we restrict the operator L to a subdomain
D(A) such that the restricted operator A (A = L on D(A))
characterizes the process: for example, Does
Z
(Lf )dπ = 0 ∀f ∈ D(A)
(6)
imply that π is a stationary distribution?
Does the equation
hf , µt i = hf , µ0 i +
Z t
0
hLf , µu idu +
Z t
0
hgf , µu idu ∀f ∈ D(A),
(7)
admit a unique solution?
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Martingale Problem
When (Xt ) is a Markov process and L is its generator, it can
be shown that
Mtf = f (Xt ) −
Z t
0
(Lf )(Xu )du
(8)
is a martingale for all f ∈ D(L). Indeed, If for a process (Xt ),
Mtf is a martingale for all f ∈ D(L), then (Xt ) is the Markov
process corresponding to L.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Martingale Problem
Question: Can we choose D(A) such that the restriction A of
L to D(A) satisfies the following: The requirement that
Mtf
= f (Xt ) −
Z t
0
(Af )(Xu )du
(9)
is a martingale for all f ∈ D(A) implies that (Xt ) is the
Markov process corresponding to L?
For Brownian motion, we can take D(A) to be Cb2 (Rd ) or even
Cb∞ (Rd ) and Af = 12 ∆f and the answer is Yes.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Martingale Problem
This leads to the Martingale problem:
Let A be an operator on Cb (S) with domain D(A) ⊆ Cb (S),
A is a function from D(A) into Cb (S).
Does there exist a process (Xt ) with r.c.l.l. paths such that
Mtf
= f (Xt ) −
Z t
0
(Af )(Xu )du is a martingale ∀f ∈ D(A) (10)
and does the requirement (10) determine the process (i.e. its
law or its distribution) uniquely? If answer is YES, the
martingale problem is said to be well posed.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Martingale Problem
The martingale problem for A is said to be well posed if for all
probability measures µ0 on S, there exists a process (Xt ) with
r.c.l.l. paths with distribution of X0 being µ0 and
Mtf
= f (Xt ) −
Z t
0
(Af )(Xu )du is a martingale ∀f ∈ D(A).
Further, if (Yt ) is another process with r.c.l.l. paths with
distribution of Y0 being µ0 and if
Ntf = f (Yt ) −
Z t
0
(Af )(Yu )du is a martingale ∀f ∈ D(A)
then the distribution of the process (Xt ) equals distribution of
the process (Yt ).
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Martingale Problem
Note that we have defined well posedness of martingale
problem for an operator A without any reference to a Markov
process or a semigroup. We have just started with a linear
operator whose domain and range are subsets of Cb (S).
Stroock-Varadhan introduced and studied Martingale problem
when S = Rd and A is a second order differential operator
with D(A)=Cb2 (Rd ) and constructed diffusion processes.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Diffusion processes
For f ∈ D(A)=Cb2 (Rd ), let
(Af )(x) =
d
1 d
∂
∂2
f
](x)
+
bi (x)[
f ](x) (11)
a
(x)[
ij
∑
∑
2 ij=1
∂ xi ∂ xj
∂ xi
i=1
where
aij ∈ Cb (Rd ), bi ∈ Cb (Rd ) 1 ≤ i, j ≤ d
(12)
and for all x ∈ Rd
((aij (x))) is a (strictly) positive definite symmetric matrix.
(13)
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Diffusion processes
Under conditions (11), (12) and (13), Stroock-Varadhan
proved that the martingale problem for A is well posed and
that the resulting solution is a Markov process and the
generator of the Markov process is an extension of the
differential operator A.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Martingale problem
The same technique has been used to show that similar result
is true in general. Under a mild regularity condition on the A,
well posedness of the martingale problem in the class of
processes with r.c.l.l. paths leads to its solution being a
Markov process with the generator of the Markov process
being an extension of A.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Martingale problem
Subsequently Eicheveria (a student of Varadhan) showed : Let
S be locally compact and let martingale problem for A be well
posed and suppose that D(A) is an algebra. Then
Z
(Lf )dπ = 0 ∀f ∈ D(A)
(14)
implies that π is a stationary distribution. Further
hf , µt i = hf , µ0 i +
Z t
0
hLf , µu idu +
Z t
0
hgf , µu idu ∀f ∈ D(A),
(15)
admits a unique solution.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Martingale problem
The requirement that S be locally compact was removed by
Karandikar and Bhatt : it was shown that
(i) if D(A) is an algebra
(ii) there exists a solution to the martingale problem in the
class of r.c.l.l. processes
(iii) if uniqueness holds in the class of measurable solutions
then the results on stationary distribution and evolution
equations are valid.
They further extended the result to cover the case when Af
may be unbounded and has discontinuities of certain kind.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Martingale problem
B. V. Rao, Karandikar and Bhatt showed that if the
martingale problem is well posed in the class of measurable
processes, then the results on stationary distributions and
evolution equations are true.
These extensions were crucial for application to Filtering
theory. Kallianpur, Karandikar and Bhatt used these results to
prove that Fujisaki-Kallianpur-Kunita equation and Zakai
equation (these are infinite dimensional stochastic differential
equations) admit unique solution and thus characterize the
filter.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
Filtering theory
Indeed, using these techniques, Kallianpur, Karandikar and
Bhatt have proven the uniqueness of solution to FKK and
Zakai equation, about 25 years after these equations were
derived. These techniques are also useful is proving robustness
of the filter- an important property.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
References
1. A. G. Bhatt, G. Kallianpur, and R. L. Karandikar.
Uniqueness and robustness of solution of measure-valued
equations of nonlinear filtering. Ann. Probab.,
23(4):1895 − 1938, 1995.
2. A. G. Bhatt and R. L. Karandikar. Invariant measures and
evolution equations for markov processes charachterized via
martingale problems. Ann. Probab., 21(4):2246 − 2268, 1993
3. A. G. Bhatt and R. L. Karandikar. Evolution equations for
markov processes: application to the white-noise theory of
filtering. Appl. Math. Optim., 31(3):327 − 348, 1995.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
References
4. A. G. Bhatt and R. L. Karandikar. Characterization of the
optimal filter: the non-markov case. Stochastics Stochastics
Rep., 66:177 − 204, 1999.
5. A. G. Bhatt, R. L. Karandikar, and B. V. Rao. On
characterisation of markov processes via martingale problems.
Proc. Indian Acad. Sci. Math. Sci., 116(1):83 − 96, 2006.
6. P. Echeverra. A criterion for invariant measures of markov
processes.
Z. Wahrsch. Verw. Gebiete, 61(1):1 − 16, 1982. 7. S. N.
Ethier and T. G. Kurtz. Markov Processes: Characterization
and Convergence. Wiley, New York, first edition, 1986.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute
References
8. G. Kallianpur and R. L. Karandikar. Measure-valued
equations for the optimum filter in finitely additive nonlinear
filtering theory. Z. Wahrsch. Verw. Gebiete, 66(1):1 − 17,
1984.
9. G. Kallianpur and R. L. Karandikar. White noise calculus
and nonlinear filtering theory. Ann. Probab.,
13(4):1033 − 1107, 1985.
10. G. Kallianpur and R. L. Karandikar. White noise theory of
Prediction, Filtering and Smoothing, volume 3 of Stochastics
Monographs. Gordon and Breach, New York, first edition,
1988.
Rajeeva L. Karandikar
Martingale problem approach to Markov processes
Director, Chennai Mathematical Institute