Download Markov Models - Computer Science

Markov Models
CS6800 Advanced Theory of Computation
Fall 2012
Vinay B Gavirangaswamy
Markov Property
Processes future values are conditionally
dependent on the present state of the
system.
Strong Markov Property
Similar as Markov Property, where values are
conditionally dependent on the stopping time
(Markov time) instead of present state.
Introduction
Markov Process
Process that follows Markov Property.
e.g. First order Markov Process for probability space (, F, P)
P(Xt+1=Sj | Xt=Si, Xt-1=Sk ,...) = P(Xt+1=Sj | qt=Si) , for t = 1, 2, 3, …
where,  = {S1, S2, S3, …, Sn }, State at time t is Xt=Si
Markov Model
Model’s a stochastic system which assumes Markov property. It is formally
represented by triplet M = (K, , A), where
K is a finite set of states
 vector of initial probabilities for each of the states.
A is a matrix representing transition probabilities.
Markov model is an non deterministic finite state machine.
Introduction (Contd.)
Stochastic FSM
0.42
0.45
SUNNY(0.30)
0.38
CLOUDY(0.40)
0.10
0.20
0.20
0.20
RAINY(0.30)
The transition matrix:
 0.42 0.38 0.20 


A   0.10 0.45 0.45 
 0.20 0.20 0.60 


0.45
0.60
• Stochastic matrix:
Rows sum up to 1
• Double stochastic matrix:
Rows and columns sum up to 1
Simple Markov Model of Weather




If we want to know probability of the sequence
SUNNY SUNNY SUNNY SUNNY SUNNY
Take initial probability of SUNNY day i.e. on a
any given day probability that it will be SUNNY is
0.30
And for use to get another SUNNY day after a
SUNNY day is 0.42
So, by using Markov Chain we can say probability
of getting 5 consecutive SUNNY day is
0.3  0.42  0.09335088
4
Prediction Based on State
Transition Probability
System is
autonomous
System is
controlled
System state is
fully observable
System state is
partially observable
Markov chain
Hidden Markov
model
Markov decision
process
Partially observable
Markov decision
process
Types of Markov Model
Markov Chain
 Simplest Markov model
 System state at time t+1 depends on state at t.
Hidden Markov Model
 Similar to Markov chain but system states are
unobservable.
Markov decision process
 Models a system where outcomes are partially
random and partially under the control of a
decision maker.
Types of Markov Model
(Contd.)
Markov decision process
It is formally represented by 4-tuple M = (K, ,
A, R), where
K is a finite set of states
 vector of initial probabilities for each of the
states.
A is a matrix representing transition probabilities.
R is reward given for transition from state St to
Sp with transition probability given in matrix A.
Types of Markov Model
(Contd.)
Partially observable Markov decision process
Similar to Markov decision process where state of
system is partially observed. It is formally
represented by 4-tuple M = (K, , O , A, ), where
K is a finite set of states.
 is a set of actions.
O is a set of observations.
A is a matrix representing transition probabilities.
 is a set of conditional observation probabilities.
Types of Markov Model
(Contd.)

System that transitions from one state to another, between a finite
number of possible states.

Next state depends only on current state and not its history.

Follows Markov property.

Formally for a stochastic process { Xt }
P{ Xt+1 = j | X0 = k0, . . . , Xt-1 = kt-1, Xt = i }
= P{ Xt+1 = j | Xt = i }
for every

i, j, k0, . . . , kt-1 and for every t.
Stationary Markov Chains
Pr{ Xt+1 = j | Xt = i } = Pr{ X1 = j | X0 = i } for all t
Markov Chain

If the state space S = { 0, 1, . . . , m–1} then
we have
j pij = 1  i and pij  0  i, j
(we must go
some where)
 p00
A   p10

 p20
(each transition has
probability  0)
p01
p11
p21
p02 
p12 

p22 
where pij = Pr{ X1 = j | X0 = i }
Transition Matrix
11
Each directed edge AB is associated with the positive
transition probability from A to B.
0.95
0.2
0.5
0.3
0.8
B
0
C
0.05
0.2
0.5
0
0.3
0
0.2
0
0.8
0
0
1
0
D
0
A
0.2
0.05
A
0.95
B
C
D
1
Representation of a Markov Chain as a
Digraph
12


States of Markov chains are classified by the digraph
representation (omitting the actual probability values)
A, C and D are recurrent states: they are in strongly
connected components which are sinks in the graph.
B is not recurrent – it is a transient state
Alternative definitions:
A state s is recurrent if it can be
reached from any state reachable from
s; otherwise it is transient.
Properties of Markov Chain states
13
A and B are transient states, C and D
are recurrent states.
Once the process moves from B to D,
it will never come back.
Another example of Recurrent and
Transient States
14
A Markov Chain is irreducible if the corresponding graph is strongly
connected (and thus all its states are recurrent).
E
1
Irreducible Markov Chains
15


A state s has a period k if k is the GCD of the
lengths of all the cycles that pass via s. (in the
shown graph the period of A is 2).
A Markov Chain is periodic if all the states in it
have a period k >1. It is aperiodic otherwise.
E
Periodic States
16
A Markov chain is ergodic if :
1. the corresponding graph is
strongly connected.
2. It is not peridoic
Ergodic Markov Chains are important since they guarantee the
corresponding Markovian process converges to a unique
distribution, in which all states have strictly positive
probability.
Ergodic Markov Chains
17
A Markov chain is reversible if
 P X  j | X  i    P X  i | X  j 
i
n 1
n
j
n 1
n
Reversible Markov chain












Physics
Chemistry
Testing
Information sciences
Queuing theory
Internet applications
Statistics
Economics and finance
Social science
Mathematical biology
Games
Music
Markov Model Applications


"Wkipedia-Markov model," [Online]. Available:
http://en.wikipedia.org/wiki/Markov_chain.
[Accessed 28 10 2012].
P. Xinhui Zhang, "DTMarkovchains," [Online].
Available:
http://www.wright.edu/~xinhui.zhang/.
[Accessed 28 10 2012].
References