Download distribution

Sampling random numbers
(from various distributions)
Probability
 Refers to randomness or uncertainty
 Sample space, S = set of all possible
outcomes of an experiment
 Event = collection (subset) of outcomes
contained in the sample space (ES)
2
Probability axioms
1. For any event A, P(A) >= 0.
2. P(S) = 1.
3. If A1, A2, …, An is a finite collection of
mutually exclusive events then
P(A1 U A2 U … U An) =  P(Ai)
If A1, A2, … is an infinite collection of
mutually exclusive events then
P(A1 U A2 U …) =  P(Ai)
3
Probability properties
1. For any event A, P(A) = 1 – P(A’) where A’
=S-A
S
4
Probability properties
Joint probability, P(AB), is the probability of
two events in conjunction. That is, it is the
probability of both events together.
2. If A and B are mutually exclusive, then
P(AB) = 0.
5
Probability properties
3. For any two events A and B, P(AB) =
P(A) + P(B) – P(AB).
6
Probability properties
4. Definition of conditional probability:

For any two events A and B with P(B)>0, the
conditional probability of A given that B has
occurred (i.e., I’m 100% sure of B) is:
 P(A|B) = P(AB) / P(B)
(prob of A given B)
 P(AB) = P(A|B) * P(B)
(multiplication rule)
7
The Law of Total Probability
 Let A1, …, An be mutually exclusive and
exhaustive events. Then for any other
event B,
 P(B) = P(B|A1)*P(A1) + … + P(B|An)*P(An)
 P(B) = ∑ P(B|Ai)*P(Ai)
event B
 recall P(A|B) = P(AB) / P(B)
A1
A2
P(B|A1)
S
P(B|A3)
A3
P(B|A2)
P(B|A4)
A4
8
Independence
 Two events A and B are independent if P(A|B) =
P(A) and are dependent otherwise.
 A and B are independent iff P(AB) = P(A) * P(B).
 Examples:
 The event of getting a 6 the first time a die is rolled and the event of getting
a 6 the second time are independent.
 By contrast, the event of getting a 6 the first time a die is rolled and the
event that the sum of the numbers seen on the first and second trials is 8
are dependent.
 If two cards are drawn with replacement from a deck of cards, the event of
drawing a red card on the first trial and that of drawing a red card on the
second trial are independent.
 By contrast, if two cards are drawn without replacement from a deck of
9 a
cards, the event of drawing a red card on the first trial and that of drawing
red card on the second trial are dependent.
Bayes’ Theorem
 Let A1, A2, …, An be a collection of n
mutually exclusive and exhaustive events
with P(Ai)>0 for i=1 … n. Then for any
event B for which P(B)>0,
P(Ak|B) = P(AkB) / P(B)
= P(B|Ak)*P(Ak) / ∑ P(B|Ai)*P(Ai)
= P(B|Ak)*P(Ak) / P(B)
10
Bayes’ Theorem
 Suppose there is a school with 60% boys and 40% girls as
its students. The female students wear pants or skirts in
equal numbers; the boys all wear pants. An observer sees
a (random) student from a distance, and what the observer
can see is that this student is wearing pants. What is the
probability this student is a girl? The correct answer can
be computed using Bayes' theorem.
11
from http://en.wikipedia.org/wiki/Bayes%27_theorem
Bayes’ Theorem
 Suppose there is a school with 60% boys and 40% girls as
its students. The female students wear pants or skirts in
equal numbers; the boys all wear pants. An observer sees
a (random) student from a distance, and what the observer
can see is that this student is wearing pants. What is the
probability this student is a girl? The correct answer can
be computed using Bayes' theorem.
 Given P(A) = 0.4 = P(girl), P(B) = 0.8 = 0.5 * 0.4 + 1.0 * 0.6 =
P(pants), and P(B|A) = 0.5 = P(pants given girl).
 Therefore, P(A|B) = 0.5 * 0.4 / 0.8 = 0.25.
from http://en.wikipedia.org/wiki/Bayes%27_theorem
12
Bayes’ Theorem

Richard Price and the Existence of a Deity: Richard Price discovered Bayes' essay and
its now-famous theorem in Bayes' papers after Bayes' death. He believed that Bayes'
Theorem helped prove the existence of God ("the Deity") and wrote the following in his
introduction to the Essay.
 “The purpose I mean is, to shew what reason we have for believing that there are in
the constitution of things fixt laws according to which things happen, and that,
therefore, the frame of the world must be the effect of the wisdom and power of an
intelligent cause; and thus to confirm the argument taken from final causes for the
existence of the Deity. It will be easy to see that the converse problem solved in this
essay is more directly applicable to this purpose; for it shews us, with distinctness
and precision, in every case of any particular order or recurrency of events, what
reason there is to think that such recurrency or order is derived from stable causes
or regulations in nature, and not from any irregularities of chance.” --Philosophical
Transactions of the Royal Society of London, 1763.
13
from http://en.wikipedia.org/wiki/Bayes%27_theorem
DISTRIBUTIONS
14
Distributions
 “In probability theory and statistics, a probability
distribution identifies either the probability of each
value of a random variable (when the variable is
discrete), or the probability of the value falling
within a particular interval (when the variable is
continuous). The probability distribution describes
the range of possible values that a random
variable can attain and the probability that the
value of the random variable is within any
(measurable) subset of that range.”
15
 from http://en.wikipedia.org/wiki/Probability_distributions
Uniform and normal distribution
examples
 Throw a fair die
 uniform
 Flip a coin
 uniform
 Measurement errors
 normal
 Physical characteristics of biological specimens
 normal
 Distribution in testing and intelligence
 normal
16
Uniform distribution
(from wolfram.com)
 A uniform distribution is a distribution that
has constant probability.
17
Uniform distribution
 Ex. Roll one die.
18
Poisson distribution
 In probability theory and statistics, the
Poisson distribution is a discrete
probability distribution that expresses the
probability of a number of events occurring
in a fixed period of time if these events
occur with a known average rate, and are
independent of the time since the last
event.
19
Poisson distribution
The probability that there are exactly k occurrences (k being a nonnegative integer, k = 0, 1, 2, ...) is
where
e is the base of the natural logarithm (e = 2.71828...),
k is the number of occurrences of an event - the probability of which
is given by the function, k! is the factorial of k,
λ is a positive real number, equal to the expected number of
occurrences that occur during the given interval.
For instance, if the events occur on average every 4 minutes, and you
are interested in the number of events occurring in a 10 minute
interval, you would use as model a Poisson distribution with λ = 10/420
= 2.5.
Poisson distribution
21
Poisson distribution examples
Examples of events that can be modelled as Poisson
distributions include:
 The number of cars that pass through a certain point on a road
(sufficiently distant from traffic lights) during a given period of time.
 The number of spelling mistakes one makes while typing a single
page.
 The number of phone calls at a call center per minute.
 The number of times a web server is accessed per minute.
 The number of roadkill (animals killed) found per unit length of road.
 The number of mutations in a given stretch of DNA after a certain
amount of radiation.
 The number of unstable nuclei that decayed within a given period of
time in a piece of radioactive substance. The radioactivity of the
substance will weaken with time, so the total time interval used in
the model should be significantly less than the mean lifetime of the
22
substance.
Poisson distribution examples
Examples of events that can be modelled as Poisson
distributions include:
 The number of pine trees per unit area of mixed forest.
 The number of stars in a given volume of space.
 The number of soldiers killed by horse-kicks each year in each
corps in the Prussian cavalry. This example was made famous by a
book of Ladislaus Josephovich Bortkiewicz (1868–1931).
 The distribution of visual receptor cells in the retina of the human
eye.
 The number of V2 rocket attacks per area in England, according to
the fictionalized account in Thomas Pynchon's Gravity's Rainbow.
 The number of light bulbs that burn out in a certain amount of time.
 The number of viruses that can infect a cell in cell culture.
 The number of hematopoietic stem cells in a sample of
unfractionated bone marrow cells.
23
Sampling from a Poisson
distribution (Knuth)
init:
L = e−λ
k=0
p=1
Donald Ervin Knuth (born January 10, 1938) is a
computer scientist and Professor Emeritus at
Stanford University. He is the author of the
seminal multi-volume work The Art of Computer
Programming. Knuth has been called the
"father" of the analysis of algorithms.
do:
k=k+1
Generate uniform random number u in [0.0-1.0].
p=p*u
while p ≥ L
return k - 1
24
Sampling from a Poisson
distribution (Knuth)
init:
public static int poisson ( int lambda ) {
L = e−λ
Random r = new Random();
k=0
final double L = Math.pow( Math.E,
p=1
-lambda );
do:
int k = 0;
k=k+1
double p = 1;
Generate uniform random number do {
u in [0.0-1.0].
++k;
p=p*u
//generate a unform random number, u
while p ≥ L
double u = r.nextDouble();
return k - 1
p *= u;
} while (p >= L);
return k - 1;
}
25
Gaussian or normal distribution
(or bell-shaped curve)
26
27
28
29
Given a uniformly distributed
random number generator,
how can we generate
normally distributed random
numbers?
Method types
1. Rejection

(ex. basic Box-Muller)
2. Transform

(ex. polar Box-Muller)
31
Box-Muller Transform
(from wikipedia)
 A Box-Muller transform (by George Edward
Pelham Box and Mervin Edgar Muller 1958) is a
method of generating pairs of independent
standard normally distributed (zero expectation,
unit variance) random numbers, given a source of
uniformly distributed random numbers.
 It is commonly expressed in two forms.
1. The basic form maps uniformly distributed Cartesian
coordinates falling inside the unit circle to normally
distributed Cartesian coordinates.
2. The polar form maps uniformly distributed polar
coordinates to normally distributed Cartesian coordinates.
uniform
pairs to
normal
pairs
 Alternatively, one could use the inverse transform
sampling method to generate normally-distributed
random numbers instead; the Box-Muller
transform was developed to be more
computationally efficient. The more efficient
Ziggurat algorithm can also be used.
32
Box-Muller Transform
 Cartesian form
1. Given x and y independently uniformly
distributed in [−1,1], set s = x2 + y2.
2. If s > 1, throw them away and try another pair
(x, y), until a pair with s in (0,1] is found.
3. Then, for these filtered points, compute (pairs
of results):
33
Box-Muller Transform
 Polar form
 Suppose U1 and U2 are independent random variables that are
uniformly distributed in (0,1].
 Let
 Then Z0 and Z1 are independent random variables with a normal
distribution of standard deviation 1. The derivation is based on
the fact that, in a two-dimensional cartesian system where X and
Y coordinates are described by two independent and normally
distributed random variables, the random variables for R2 and Θ
(shown above) in the corresponding polar coordinates are also
independent and can be expressed as:
34
Box-Muller Transform
 Contrasting the two forms
 The Cartesian method (as opposed to the
polar method) is a type of rejection sampling,
and throws away some generated random
numbers, but it is typically faster than the
polar method because it is simpler to
compute, provided that the random number
generator is relatively fast, and is more
numerically robust.
35
Box-Muller Transform
 Contrasting the two forms
 The Cartesian method avoids the use of
trigonometric functions, which are expensive
in many computing environments.
36
Box-Muller Transform
 Contrasting the two forms
 The Cartesian method throws away 1 − π/4 ≈
21.46% of the total input uniformly distributed
random number pairs generated, i.e. throws
away 4/π − 1 ≈ 0.2732 uniformly distributed
random number pairs per Gaussian random
number pair generated, requiring 4/π ≈
1.2732 input random numbers per output
random number.
37
Converting from mean=0,
stdev=1.0 to other normal
distributions
1. Generate random numbers (with mean=0 and
stdev=1.0) as before.
2. Multiply result by desired stdev.
3. Add desired mean.
38
Simple, discrete method.
1. Roll N (uniformly distributed) dice.
2. Sum them up.
39
Exercise using Excel