Download Handout #5 (Random Signals)

Probability
• Sample Space (S)
– Collection of all possible outcomes of a random
experiment
• Sample Point
– Each outcome of the experiment (or)
element in the sample space
• Events are Collection of sample points
• Ex: Rolling a die (six sample points), Odd number thrown
in a die (three sample point – a subset), tossing a coin (two
sample points: head,tail)
Prof. Sankar
Review of Random Process
1
Probability
• Null Event (No Sample Point)
• Union (of A and B)
– Event which contains all points in A and B
• Intersection (of A and B)
– Event that contains points common to A and B
• Law of Large Numbers
– Probability of event A
Lim N A
P( A) 
N  N
N – number of times the random experiment is repeated
NA- number of times event A occurred
Prof. Sankar
Review of Random Process
2
Probability
• Properties
i ) 0  P( A )  1
ii ) P(S)  1 ; P( )  0
iii ) A, B  S, for mutually exclusive events ie., A  B    P( AB)  0
P( A  B)  P( A )  P( B)  P( A  B)  P( A .or.B)
For non - mutually exclusive events,
P( A  B)  P( A )  P( B)  P( AB)
where P( AB)  P( A.and .B)  P( A  B) is the joint probabilit y
Prof. Sankar
Review of Random Process
3
Probability
• Conditional Probability
– Probability of B conditioned by the fact that A
has occurred
P( AB)
P( B | A) 
 P( A)
P( B) P( A | B)

P( A)
 Bayes ' theorem
– The two events are statistically independent if
P( AB)  P( A) P( B)
Prof. Sankar
Review of Random Process
4
Probability
• Bernoulli’s Trials
– Same experiment repeated n times to find the
probability of a particular event occurring
exactly k times
Pn (k )   kn  p k q n  k
 
n!
 n  C 
 k  n k k!(n  k )!
Prof. Sankar
Review of Random Process
5
Random Signals
•
Associated with certain amount of uncertainty
and unpredictability. Higher the uncertainty
about a signal, higher the information content.
– For example, temperature or rainfall in a city
– thermal noise
•
•
Information is quantified statistically
(in terms of average (mean), variance, etc.)
Generation
– Toss a coin 6 times and count the number of heads
– x(n) is the signal whose value is the number of heads
on the nth trial
Prof. Sankar
Review of Random Process
6
Random Signals
• Mean
1
x
N
N
x
i 1
i
• Median: Middle or most central item in an
ordered set of numbers
• Mode = Max{xi}
• Variance
• Standard
1 N
2
x 
(
x

x
)
 i
N  1 i1
Deviation  x  var iance
2
measure of spread or deviation from the mean
Prof. Sankar
Review of Random Process
7
Random Variables
• Probability is a numerical measure of the outcome
of the random experiment
• Random variable is a numerical description of the
outcome of a random experiment, i.e., arbitrarily
assigned real numbers to events or sample points
– Can be discrete or continuous
– For example: head is assigned +1
tail is assigned –1 or 0
Prof. Sankar
Review of Random Process
8
Random Variables
•
Cumulative Distribution Function (CDF)
–

FX ( x ) P( X  x )

Properties: FX (x)  0; FX ()  0; FX ()  1
FX (x1 )  FX (x 2 ), if x1  x 2
•
Probability Density Function (PDF)
x
dF ( x )
p (x)  x
or F ( x )  P( X  x )   p ( y)dy
x
x
x
dx


–
Prof. Sankar
Properties:
p
x
( x )dx  1; p x ( x )  0

Review of Random Process
9
Important Distributions
• Binary distribution (Bernoulli distribution)
– Random variable has a binary distribution
– Partitions the sample space into two distinct
subsets A and B
– All elements in A are mapped into one number
say +1 and B to another number say 0.
P[X  1]  p P[X  0 ]  q  1  p
Mean(mx )  p, Variance ( x2 )  pq
pdf :
Prof. Sankar
p X ( x)  P[ X   ] ( x   )
Review of Random Process
10
Important Distributions
• Binomial Distribution
– Perform binary experiment n times with
outcome X1,X2,…Xn, if X   X i , then X has
i
binomial distribution
 n  k nk
CDF  P[X  k ]   p q
k
n
pdf : p( x )   P[X  k ]( x  k )
k 0
m x  np
Prof. Sankar
 2x  npq
Review of Random Process
11
Important Distributions
• Uniform Distribution
– Random variable is equally likely
– Equally Weighted pdf
 1

a xb
p X ( x)   b  a
 0 , elsewhere
ba
mX 
,
2
Prof. Sankar
 X2
p X (x)
2

b  a

12
Review of Random Process
1
ba
a
b
12
Important Distributions
• Poisson Distribution
– Random Variable X is Poisson distributed
with parameter m with
k
m
pk  P[ X  k ]  e  m
k! k

m
m
pdf p X ( x)  e   ( x  k )
k  0 k!
Mean  m
Variance  x2  m
– Approximation to binomial with p << 1,
and k << 1, then
k


np
n
k
n

k

np
 p q  e
k
 
Prof. Sankar
np  1
k!
Review of Random Process
13
Important Distributions
• Gaussian Distribution
1
p X ( x) 
2  x

x  mx 2

e
2 x2
x
Mean : m X , Variance :  x2
x
1

   mx 2
du
 p (u)du  2   e
• Normalized Gaussian pdf - N(0,1)
FX ( x) 
2 x2
x

x 
– Zero mean, Unit Variance
x
2
1
z
2
 z ( x) 
e
dz

2 
2
1
x
p X ( x) 
e 2
2
Prof. Sankar
Review of Random Process
14
Important Distributions
• Normalized Gaussian pdf
(x )  1  (x )
Alternativ e notations
 ( x )  erf ( x ) (error function )
Q( x )  1   ( x )  erfc ( x ) (complement ary error function )
For large values of x (ie ., x  3)
Q( x ) 
Prof. Sankar
1
2x 2
e
x
2
2
Review of Random Process
15
Joint and Conditional PDFs
• For two random variables X and Y
–
Two random variables X and Y
FX Y (x,y)  P(X  x,Y  y)
Joint pdf
 2 FX Y (x,y)
p X Y (x,y) 
xy
P(X 1  X  X 2,Y1  Y  Y2 ) 
X 2 Y2
 p
XY
(x,y) dy dx
X1 Y1
Volume under the pdf
If X and Y are independen t
Prof. Sankar
p X Y (x,y)  p X (x)p Y (y)
Review of Random Process
16
Joint and Conditional PDFs
• Marginal pdfs

p X (x) 
p
XY
(x,y)dy
XY
(x,y)dx


p Y (y) 
p

• Conditional pdfs
p X|Y (x | y) 
Prof. Sankar
p X Y (x,y)
p Y (y)
p Y|X (y | x) 
Review of Random Process
p X Y (x,y)
p X (x)
17
Expectation and Moments

First moment of X (mean) : m x  E( x ) 

(1st order statistic)
x 2 p X ( x ) dx ( 2nd order statistic )


n th moment of X : E( x n ) 

X ( x ) dx


Second moment of X : E( x 2 ) 
xp
x n p X ( x ) dx

• Centralized Moment
– Second centralized moment is variance
 2X  E( X  m X ) 2  E( X 2 )  m 2X
n th Centralize d Moment : E( X  m X ) n
Prof. Sankar
Review of Random Process
18
Expectations and Moments
• (i,j) joint moment between random variables X and Y
X2
Y2
 
E( X Y )  x i
i
j
X1
y j p X Y (x,y) dy dx
Y1
First joint moment (i  j  1)
E( XY )  R X Y is Correlatio n
Determines the nature of relationsh ip between the
two random variables
Prof. Sankar
Review of Random Process
19
Expectations and Moments
• (i,j) joint central moment

  x     y    f
E X   X  Y   Y  
i
j

Y2
i
j
X

Y
XY
(x,y) dy dx
Y1
First joint moment (i  j  1)
EX   X Y   Y   C X Y
Covariance
CovX, Y   C X Y  EX   X Y   Y 
 EX Y    X  Y
If X and Y are statistica lly independen t
EX Y   EX EY   CovX, Y   0
Then X and Y are said to be uncorrelat ed,
converse not necessaril y true
Prof. Sankar
Review of Random Process
20
Expectations and Moments
• Auto-covariance


C X X  Cov( X , X )  E x   X    X2
Correlatio n Coeff. PX Y 
( Normalized
2
CX Y
 X Y
covariance )
• Characteristic Function (moment generator)


 X (t ) E e jxt   e jxt f X ( x)dx


 
1
 f X ( x) 
2

 jxt

(
t
)
e
dx
X


F
 X (t ) 
f X ( x) Fourier tr ansform pair
Prof. Sankar
Review of Random Process
21
Random Process
• If a random variable X is a function of another
variable, say time t, x(t) is called random process
• Collection of all possible waveforms is called the
ensemble
• Individual waveform is called a sample function
• Outcome of a random experiment is a sample
function for random process instead of a single
value in the case of random variable
Prof. Sankar
Review of Random Process
22
Random Process
• Random Process X(.,.) is a function of time
variable t and sample point variable s
• Each sample point (s) identifies a function of time
X(.,s) referred as “sample function”
• Each time point (t) identifies a function of sample
points X(t,.), i.e., a random variable
• Random or Stochastic Processes can be
– continuous or discrete time process
– continuous or discrete amplitude process
Prof. Sankar
Review of Random Process
23
Random Process
• Ensemble statistic : Ensemble average at a
particular time

X  E ( x)   x f x  x dx

– Temporal average for a sample function
T
1 2
Lim
X   T  
x(t )dt

T T
2
• Random Process Classifications
– Stationary Process : Statistical characteristics of the
sample function do not change with time (time-invariant)
Prof. Sankar
Review of Random Process
24
Random Process
• Second Order joint pdf
E[ x(t1 ) x(t2 )]  Rx (t1 , t2 )  Rx (t2  t1 )  Rx ( )
– Autocorrelation is a function of only time difference
• Wide Sense (or Weak) Stationary
E[ x(t )]   x  constant
Rx (t1 , t2 )  Rx ( )
  t2  t1
– Independent of time up to second order only
• Ergodic Process
– Ensemble average = time average
Prof. Sankar
Review of Random Process
25
Random Process
• Mean E[ X (t )]
 m X (t )

– Mean of the random process at time t is the mean of the
random variable X(t)
• Autocorrelation
E[ X (t1 ) X (t 2 )] 
 RXX (t1 , t 2 )
• Auto-covariance
E X (t1 )  mX (t1 )  X (t2 )  mX (t2 )  
C XX (t1 , t2 )
C X (t1 , t2 )  RX (t1 , t2 )  mX (t1 )mX (t2 )
Prof. Sankar
Review of Random Process
26
Random Process
• Cross Correlation and covariance
RXY (t1 , t 2 ) 
 E X (t1 )Y (t 2 )
C XY (t1 , t 2 ) 
 E  X (t1 )  m X (t1 ) Y (t 2 )  mY (t 2 ) 
For discrete time process,
m X ( n) 
 E[ X n ]
RX (n1 , n2 ) or RX (n, m)  E[ X n X m ]  E[ X n1 X n2 ]
• Power Density Spectrum

S X ( f )  F [ RX ( )] 
 j 2 f
R
(

)
e
d
 X

Prof. Sankar
Review of Random Process
27
Random Process
• Total Average Power
T


1 2 2
Lim
PX  T   E   x (t )dt 
 T T 2



 RX (0) 
S

Prof. Sankar
X
( f )df or
 S
X
( )d

Review of Random Process
28