Download Probability and Statistics (part 2)

Probability and Statistics (part 2)
April 6, 2016
Last Time
Big Ideas:
I
Random variable X is distributed according to some
distribution f (x) for x ∈ Ω
Examples:
I
f(x) is Gaussian; Ω = (−∞, ∞)
Last Time
Big Ideas:
I
Random variable X is distributed according to some
distribution f (x) for x ∈ Ω
Examples:
I
f(x) is exponential; Ω = [0, ∞)
Last Time
Big Ideas:
I
Random variable X is distributed according to some
distribution f (x) for x ∈ Ω
Examples:
I
f(x) is Poisson; Ω = {0, 1, 2, . . .}
Last Time
Big Ideas:
I
Random variable X is distributed according to some
distribution f (x) for x ∈ Ω
I
E(X) is the expected value of the random variable X:
Z
E(X) =
x f (x) dx
Ω
Last Time
Big Ideas:
I
Random variable X is distributed according to some
distribution f (x) for x ∈ Ω
I
E(X) is the expected value of the random variable X:
Z
E(X) =
x f (x) dx
Ω
I
var(X) is the variance of the random variable X:
Z
var(X) = (x − E(X))2 f (x) dx
Ω
Empirical Data
I
Observe X to obtain the sample of data points
{x1 , x2 , . . . , xn }
Empirical Data
I
Observe X to obtain the sample of data points
{x1 , x2 , . . . , xn }
I
The sample average is
n
µn =
1X
xk
n
k=1
Empirical Data
I
Observe X to obtain the sample of data points
{x1 , x2 , . . . , xn }
I
The sample average is
n
µn =
1X
xk
n
k=1
I
The sample variance is
n
σn2
1X
=
(xk − µn )2
n
k=1
Convergence of empirical mean to expected value
The Central Limit Theorem
The distribution of the sample average, µn , converges as n → ∞
to a normal distribution
f (x) = √
1
2πσ 2
e−
(x−µ)2
2σ 2
with mean µ = E(X) and variance σ 2 =
,
var(X)
n .
How do we generate random numbers
I
For small numbers, flip a fair coin or roll a dice
I
Algorithmically generate
Hardward generators
I
I
Electronic roulette wheel
http://www.rand.org/pubs/monograph_reports/MR1418.html
I
I
I
Track random noise (thermal noise in a resistor, clock “drift”
between 2 clocks)
Takes relatively long time
Better than algorithmic when true randomness is needed
Psuedorandom number generators
I
Algorithmic approach to generate “random” numbers
Psuedorandom number generators
I
Algorithmic approach to generate “random” numbers
I
Middle Square Method
To generate a sequence of k-digit “random” numbers take a number with
k digits, square it (and add leading zeros to get 2k digits), then extract
the middle k digits.
Example:
2, 9162 = 8, 503, 056 → 5, 030
5, 0302 = 25, 300, 900 → 3, 009
3, 0092 = 9, 054, 081 → 0540
5402 = 291, 600 → 2, 916
Psuedorandom number generators
I
Algorithmic approach to generate “random” numbers
I
Middle Square Method
To generate a sequence of k-digit “random” numbers take a number with
k digits, square it (and add leading zeros to get 2k digits), then extract
the middle k digits.
Example:
2, 9162 = 8, 503, 056 → 5, 030
5, 0302 = 25, 300, 900 → 3, 009
3, 0092 = 9, 054, 081 → 0540
5402 = 291, 600 → 2, 916
“Anyone who considers arithmetical methods of producing
random digits is, of course, in a state of sin.” - John von
Neumann (1949)
Psuedorandom number generators
I
Algorithmic approach to generate “random” numbers
I
Middle Square Method
Psuedorandom number generators
I
Algorithmic approach to generate “random” numbers
I
Middle Square Method
I
Linear Congruential Generators
Generate a sequence of pseudorandom numbers by the process
xn+1 = (axn + c)
mod (m),
specifying modulus m, multiplier a, increment c, and seed x0 .
Examples:
I
I
ANSI-C and glibc (gcc): m = 232 , a = 1, 103, 515, 245,
c = 12, 345
Apple Carbonlib: m = 231 − 1, a = 75 , c = 0 (called MINSTD)
Numbers generated exhibit some correlation and may have short periods.
Psuedorandom number generators
I
I
I
Algorithmic approach to generate “random” numbers
Middle Square Method
Linear Congruential Generators
Psuedorandom number generators
I
I
I
I
Algorithmic approach to generate “random” numbers
Middle Square Method
Linear Congruential Generators
Mersenne Twister (1997)
I
I
I
I
most commonly used (MATLAB, Python, C++11)
generates numbers with nearly uniform distribution
complicated to program
requires more memory
Psuedorandom number generators
I
I
I
I
Algorithmic approach to generate “random” numbers
Middle Square Method
Linear Congruential Generators
Mersenne Twister (1997)
I
I
I
I
I
most commonly used (MATLAB, Python, C++11)
generates numbers with nearly uniform distribution
complicated to program
requires more memory
Blum Blum Shub (1986)
xn+1 = x2n
mod m
where m = pq, p and q prime, p mod 4 = q mod 4 = 3
I
I
I
has advantage that it can be made “secure”
by using only the last few digits of xn as the “random”
number, it’s difficult to predict the next “random” number
from a sequence of past numbers.
one needs to observe only 624 iterates of MT19937 to predict
all future iterates