Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Assignment 3 • Chapter 3: Problems 7, 11, 14 • Chapter 4: Problems 5, 6, 14 • Due date: Monday, March 15, 2004 Example Inventory System: Inventory at a store is reviewed daily. If inventory drops below 3 units, an order is placed with the supplier which is delivered the next day. The order size should bring inventory position to 6 units. Daily demand D is i.i.d. with distribution P(D = 0) =1/3 P(D = 1) =1/3 P(D = 2) =1/3. Let Xn describe inventory level on the nth day. Is the process {Xn} a Markov chain? Assume we start with 6 units. Markov Chains {Xn: n =0, 1, 2, ...} is a discrete time stochastic process Markov Chains {Xn: n =0, 1, 2, ...} is a discrete time stochastic process If Xn = i the process is said to be in state i at time n Markov Chains {Xn: n =0, 1, 2, ...} is a discrete time stochastic process If Xn = i the process is said to be in state i at time n {i: i=0, 1, 2, ...} is the state space Markov Chains {Xn: n =0, 1, 2, ...} is a discrete time stochastic process If Xn = i the process is said to be in state i at time n {i: i=0, 1, 2, ...} is the state space If P(Xn+1 =j|Xn =i, Xn-1 =in-1, ..., X0 =i0}=P(Xn+1 =j|Xn =i} = Pij, the process is said to be a Discrete Time Markov Chain (DTMC). Markov Chains {Xn: n =0, 1, 2, ...} is a discrete time stochastic process If Xn = i the process is said to be in state i at time n {i: i=0, 1, 2, ...} is the state space If P(Xn+1 =j|Xn =i, Xn-1 =in-1, ..., X0 =i0}=P(Xn+1 =j|Xn =i} = Pij, the process is said to be a Discrete Time Markov Chain (DTMC). Pij is the transition probability from state i to state j Pij 0, P00 P 10 . P . Pi 0 . . i, j 0 P01 P11 . . Pi1 . . P02 P12 . . Pi 2 . . ... ... . . ... . . P: transition matrix j 0 Pij 1, i 0,1,... Example 1: Probability it will rain tomorrow depends only on whether it rains today or not: P(rain tomorrow|rain today) = a P(rain tomorrow|no rain today) = b Example 1: Probability it will rain tomorrow depends only on whether it rains today or not: P(rain tomorrow|rain today) = a P(rain tomorrow|no rain today) = b State 0 = rain State 1 = no rain Example 1: Probability it will rain tomorrow depends only on whether it rains today or not: P(rain tomorrow|rain today) = a P(rain tomorrow|no rain today) = b State 0 = rain State 1 = no rain a 1 a P b 1 b Example 4: A gambler wins $1 with probability p, loses $1 with probability 1-p. She starts with $N and quits if she reaches either $M or $0. Xn is the amount of money the gambler has after playing n rounds. Example 4: A gambler wins $1 with probability p, loses $1 with probability 1-p. She starts with $N and quits if she reaches either $M or $0. Xn is the amount of money the gambler has after playing n rounds. P(Xn=i+1|Xn-1 =i, Xn-2 =in-2, ..., X0 =N}=P(Xn =i+1|Xn-1 =i}=p (i≠0, M) Example 4: A gambler wins $1 with probability p, loses $1 with probability 1-p. She starts with $N and quits if she reaches either $M or $0. Xn is the amount of money the gambler has after playing n rounds. P(Xn=i+1|Xn-1 =i, Xn-2 =in-2, ..., X0 =N}=P(Xn =i+1|Xn-1 =i}=p (i≠0, M) P(Xn=i-1| Xn-1 =i, Xn-2 = in-2, ..., X0 =N} = P(Xn =i-1|Xn-1 =i}=1– p (i≠0, M) Example 4: A gambler wins $1 with probability p, loses $1 with probability 1-p. She starts with $N and quits if she reaches either $M or $0. Xn is the amount of money the gambler has after playing n rounds. P(Xn=i+1|Xn-1 =i, Xn-2 =in-2, ..., X0 =N}=P(Xn =i+1|Xn-1 =i}=p (i≠0, M) P(Xn=i-1| Xn-1 =i, Xn-2 = in-2, ..., X0 =N} = P(Xn =i-1|Xn-1 =i}=1– p (i≠0, M) Pi, i+1=P(Xn=i+1|Xn-1 =i}; Pi, i-1=P(Xn=i-1|Xn-1 =i} Pi, i+1= p; Pi, i-1=1-p for i≠0, M P0,0= 1; PM, M=1 for i≠0, M (0 and M are called absorbing states) Pi, j= 0, otherwise random walk: A Markov chain whose state space is 0, 1, 2, ..., and Pi,i+1= p = 1 - Pi,i-1 for i=0, 1, 2, ..., and 0 < p < 1 is said to be a random walk. Chapman-Kolmogorv Equations Pijn P{ X n m j | X m i}, n 0, i, j 0 Chapman-Kolmogorv Equations Pijn P{ X n m j | X m i}, Pij1 Pij n 0, i, j 0 Chapman-Kolmogorv Equations Pijn P{ X n m j | X m i}, n 0, i, j 0 Pij1 Pij Pijn m k 0 Pikn Pkjm for all n, m 0, and i, j 0 (Chapman - Kolmogrov equations) Pijn m P{ X n m j | X 0 i}, Pijn m P{ X n m j | X 0 i}, = k 0 P{ X n m j , X n k | X 0 i} Pijn m P{ X n m j | X 0 i}, = k 0 P{ X n m j , X n k | X 0 i} k 0 P{ X n m j | X n k , X 0 i}P{ X n k | X 0 i} Pijn m P{ X n m j | X 0 i}, = k 0 P{ X n m j , X n k | X 0 i} k 0 P{ X n m j | X n k , X 0 i}P{ X n k | X 0 i} k 0 P{ X n m j | X n k}P{ X n k | X 0 i} Pijn m P{ X n m j | X 0 i}, = k 0 P{ X n m j , X n k | X 0 i} k 0 P{ X n m j | X n k , X 0 i}P{ X n k | X 0 i} k 0 P{ X n m j | X n k}P{ X n k | X 0 i} k 0 P P k 0 Pikn Pkjm m n kj ik P ( n ) : the matrix of n transition probabilities Pijn P ( n ) : the matrix of n transition probabilities Pijn P( nm) P( n) × P( m) P ( n ) : the matrix of n transition probabilities Pijn P(nm) P(n) × P(m) (Note: if A [ aij ] and B [bij ], then A × B [ k 1 aik bkj ]) M Example 1: Probability it will rain tomorrow depends only on whether it rains today or not: P(rain tomorrow|rain today) = a P(rain tomorrow|no rain today) = b What is the probability that it will rain four days from today given that it is raining today? Let a = 0.7 and b = 0.4. State 0 = rain State 1 = no rain What is P004 ? What is P004 ? 0.7 0.3 P 0.4 0.6 What is P004 ? 0.7 0.3 P 0.4 0.6 0.7 0.3 0.7 0.3 0.61 0.39 (2) P × 0.4 0.6 0.4 0.6 0.52 0.48 What is P004 ? 0.7 0.3 P 0.4 0.6 0.7 0.3 0.7 0.3 0.61 (2) P × 0.4 0.6 0.4 0.6 0.52 0.61 0.39 0.61 (4) (2) (2) P P ×P × 0.52 0.48 0.52 0.39 0.48 0.39 0.5749 0.4251 0.48 0.5668 0.4332 What is P004 ? 0.7 0.3 P 0.4 0.6 0.7 0.3 0.7 0.3 0.61 (2) P × 0.4 0.6 0.4 0.6 0.52 0.61 0.39 0.61 (4) (2) (2) P P ×P × 0.52 0.48 0.52 P004 0.5749 0.39 0.48 0.39 0.5749 0.4251 0.48 0.5668 0.4332 Unconditional probabilities How do we calculate P( X n j )? Unconditional probabilities How do we calculate P ( X n j )? Let a i P( X 0 i ) Unconditional probabilities How do we calculate P( X n j )? Let a i P( X 0 i ) P( X n j ) i 1 P ( X n j | X 0 i ) P ( X 0 i ) Unconditional probabilities How do we calculate P( X n j )? Let a i P( X 0 i ) P( X n j ) i 1 P( X n j | X 0 i ) P( X 0 i ) i 1 Pijna i Classification of States State j is accessible from state i if Pijn 0 for some n 0. Two states that are accessible to each other are said to communicate (i j ). Any state communicates with itself since Pii0 1. Communicating states If state i communicates with state j , then state j communicates with state i. Communicating states If state i communicates with state j , then state j communicates with state i. If state i communicates with state j , and state j communicates with state k , then state i communicates with state k . Proof If i communicates with j and j communicates with k , then there exist some m and n for which Pijn 0 and Pjkm 0. Pikn m r 0 Pirn Prkm Pijn Pjkm 0. Classification of States (continued) Two states that communicate are said to belong to the same class. Classification of States (continued) Two states that communicate are said to belong to the same class. Two classes are either identical or disjoint (have no communicating states). Classification of States (continued) Two states that communicate are said to belong to the same class. Two classes are either identical or disjoint (have no communicating states). A Markov chain is said to be irreducible if it has only one class (all states communicate with each other). 1/ 2 1/ 2 0 P 1/ 2 1/ 2 1/ 4 0 1/ 3 2 / 3 1/ 2 1/ 2 0 P 1/ 2 1/ 2 1/ 4 0 1/ 3 2 / 3 The Markov chain with transition probability matrix P is irreducible. 0 1/ 2 1/ 2 0 1/ 2 1/ 2 0 0 P 1/ 4 1/ 4 1/ 4 1/ 4 0 0 1 0 0 1/ 2 1/ 2 0 1/ 2 1/ 2 0 0 P 1/ 4 1/ 4 1/ 4 1/ 4 0 0 1 0 The classes of this Markov chain are {0, 1}, {2}, and {3}. Recurrent and transient states • fi: probability that starting in state i, the process will eventually re-enter state i. Recurrent and transient states • fi: probability that starting in state i, the process will eventually re-enter state i. • State i is recurrent if fi = 1. Recurrent and transient states • fi: probability that starting in state i, the process will eventually re-enter state i. • State i is recurrent if fi = 1. • State i is transient if fi < 1. Recurrent and transient states • fi: probability that starting in state i, the process will eventually re-enter state i. • State i is recurrent if fi = 1. • State i is transient if fi < 1. • Probability the process will be in state i for exactly n periods is fi n-1(1- fi), n ≥ 1. State i is recurrent if P and transient if n n 1 ii n P n1 ii Proof 1, In 0 if X n i if X n i Proof 1, In 0 if X n i if X n i I X 0 i : number of periods the process is in state i. n0 n given that it starts in i Proof 1, In 0 if X n i if X n i I X 0 i : number of periods the process is in state i. n0 n given that it starts in i E n 0 I n X 0 i n 0 E[ I n X 0 i ] n 0 P{ X n i X 0 i} n 0 Piin • Not all states can be transient. •If state i is recurrent, and state i communicates with state j, then state j is recurrent. Proof Since i j , there exists k and m for which Pijk >0 and Pjim >0. Pjjm n k Pjim Piin Pijk , for any n. n1 P m nk jj n 1 P P P P P m n k ji ii ij m k ji ij n P n1 ii . • Not all states can be transient. • If state i is recurrent, and state i communicates with state j, then state j is recurrent recurrence is a class property. • Not all states can be transient. • If state i is recurrent, and state i communicates with state j, then state j is recurrent recurrence is a class property. • Not all states can be transient. • If state i is transient, and state i communicates with state j, then state j is transient transience is also a class property. • If state i is recurrent, and state i communicates with state j, then state j is recurrent recurrence is a class property. • Not all states can be transient. • If state i is transient, and state i communicates with state j, then state j is transient transience is also a class property. • All states in an irreducible Markov chain are recurrent. 0 1 P 0 0 0 1/ 2 1/ 2 0 0 0 1 0 0 1 0 0 0 1 P 0 0 0 1/ 2 1/ 2 0 0 0 1 0 0 1 0 0 All states communicate. Therefore all states are recurrent. 0 0 1/ 2 1/ 2 0 1/ 2 1/ 2 0 0 0 P 0 0 1/ 2 1/ 2 0 0 1/ 2 1/ 2 0 0 1/ 4 1/ 4 0 0 1/ 2 0 0 1/ 2 1/ 2 0 1/ 2 1/ 2 0 0 0 P 0 0 1/ 2 1/ 2 0 0 1/ 2 1/ 2 0 0 1/ 4 1/ 4 0 0 1/ 2 There are three classes {0, 1}, {2, 3} and {4}. The first two are recurrent and the third is transient