Download Markov Chains and their Generalizations

28/04/2008
Scuola di Dottorato in Scienza ed Alta
Tecnologia – Dottorato in Informatica
Universita’ di Torino
Outline
Introduction to Stochastic Processes
Markov Chains
Stochastic Models:
Markov Chains and
their Generalizations
Renewal Theory
Markov Renewal Processes
References
E. Cinlar, Introduction to Stochastic Processes, Prentice Hall, 1975;
V.G. Kulkarni, Modeling and Analysis of Stochastic Systems,
Chapman & Hall, 1995
Gianfranco Balbo e Andras Horvath
Aprile – Maggio, 2008
Aprile - Maggio, 2008
Continuous Time Stichastic Processes
Exponential Distribution (1)
Exponential Distribution
Poisson Process
Continuous Time Markov Chains
Aprile - Maggio, 2008
3
„
A non-negative random variable X is said to have a negative exponential
distribution with parameter λ if
„
The probability density function (pdf) of a negative-exponential random
variable is
„
The Laplace transform of the exponential random variable is
Aprile - Maggio, 2008
4
Exponential Distribution (3)
Exponential Distribution (2)
„
2
The most important property of the negative exponential distribution is the
memoryless property
Minimum of exponentials
Given two random variables X and Y with negative exponential pdf
X
the new random variable Z = min(X,Y) has also an exp. pdf

‘
e-
x
since
0
a
x
Aprile - Maggio, 2008
Documento interno all'Università degli
Studi di Torino
5
Aprile - Maggio, 2008
6
1
28/04/2008
Exponential Distribution (4)
Exponential Distribution (5)
Strong Memoryless Property
Probability of First Failure
Given two random variables X and Y with negative exponential pdf, it is possible to show
Aprile - Maggio, 2008
7
Given two random variables X and Y with negative exponential pdf, it is possible to show
Aprile - Maggio, 2008
8
Exponential Distribution (7)
Exponential Distribution (6)
Sum of n independent identically distributed (i.i.d.) negative exponentials
Random Sum of n i.i.d. negative exponentials
Given a sequence of independent random variables {Xn, n¸ 1 } all characterized
by a common negative exponential pdfs with parameter λ, let N
be a geometric random variable with parameter p:
Given n random variables X1, X2, …, Xn with negative exponential pdfs with
parameter λ, let
By using the properties of Laplace transform, it is possible to show
Assume that N is independent of {Xn, n¸1 } and define the following random
sum of random variables
From which follows
By using the properties of Laplace transforms and exploiting the independence
of before, it is possible to show
and
from which we can conclude that Z has a negative-exponential distribution
with parameter λp
Aprile - Maggio, 2008
9
Poisson Process (1)
Aprile - Maggio, 2008
10
Poisson Process (2)
A Poisson process is often used to count events occurring one at the time during a
certain interval. Events of this type are assumed to be generated by an infinite set of
sources acting very slowly, independently of each other and with a “complete” random
behaviour.
Here we define the Poisson process in terms of the probability distribution of the
intervals among successive events.
Recalling the distribution of a sum of k exponential i.i.d. random variables derived
before, it is easy to show that
When t is a nonnegative value, then
In fact,
Let {Xn, n¸ 1 } be a sequence of random variables representing the inter-event times
Hence
Sn is the time of the occurrence of the n-th event.
Define
From which
N(t) is the number of events that have taken place during (0,t] and {N(t), t¸ 0 } is called
a counting process
When {Xn, n¸ 1 } is a sequence of i.i.d exponential variables with parameter λ,
the counting process {N(t), t¸0 } is called a Poisson Process with parameter λ
Aprile - Maggio, 2008
Documento interno all'Università degli
Studi di Torino
11
Aprile - Maggio, 2008
12
2
28/04/2008
Properties of Poisson Processes
Superposition and Splitting of Poisson Processes
For a fixed s ¸ 0
Merging two (or more) Poisson Processes to generate a new process is called
“superposition”
The operation of generating two (or more) counting processes out of a single
Poisson Process is called “splitting”
is a Poisson Process with parameter λ, independent of {N(u), 0 · u · s }.
A Poisson Process {N(t), t¸ 0 } has stationary and independent increments, i.e.
Superposition
Let {Ni(t), t ¸ 0 } be independent Poisson Processes of parameters {λi, 1 · i · r }.
Define
A stochastic process {N(t), t ¸ 0 } is a Poisson Process with parameter λ iff
The process {N(t), t¸ 0 } is called the superposition of {Ni(t), t ¸ 0 } ( 1 · i · r )
{N(t), t ¸ 0 } is a Poisson Process with parameter λ = λ1 + λ2 + Λ + λr.
A counting process {N(t), t ¸ 0 } is a Poisson Process with parameter λ iff
N1(t)
N2(t)
N(t)
Aprile - Maggio, 2008
13
Superposition and Splitting of Poisson Processes
Aprile - Maggio, 2008
14
Other types of Poisson Processes
Splitting
Non-Homogeneous Poisson Processes
Consider a Poisson Process {N(t), t ¸ 0 } that counts certain events that can be classified
into r categories.
It is possible to constructs r counting processes {Ni(t), t ¸ 0 }, i=1,2,…, r where Ni(t) is
the number of events of type i that occurred up to time t. Again, we can recognize that
A non-homogeneous Poisson process can be thought of as a process that counts events
that occur in a non-uniform fashion so that the probability that an event takes place in
an interval (t, t+h ] is given by λ(t) h + o(h)
Let {N(t), t¸ 0 }be a Poisson Process of parameter λ and let {Ni(t), t¸ 0 } (i=1,2,…,r ) be
generated by a Bernoulli Splitting Merchanism with parameters p1, p2, …, pr such that
Compound Poisson Process
Then each {Ni(t), ¸ 0} is a Poisson Process of parameter λ pi
A compound Poisson Process can be constructed out of a “standard” Poisson Process
N(t)
o
N1(t)
N2(t)
*
o
o
o
o
*
o
* *
o
* *
*
o
o
o
o
*
o
*
A compound Poisson process can be thought of as a process that counts events that are
allowed to occur in “batches” of arbitrary sizes (even negative!}
{N(t), t¸ 0 } of parameter λ, combined with a sequence of i.i.d random variables
(independent of N(t) ) {Zn, n¸1 } such that
*
Aprile - Maggio, 2008
Documento interno all'Università degli
Studi di Torino
15
Aprile - Maggio, 2008
16
3