Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Lecture 2 Probability and what it has to do with data analysis Abstraction Random variable, x it has no set value, until you ‘realize’ it its properties are described by a probability, P One way to think about it x p(x) pot of an infinite number of x’s Drawing one x from the pot “realizes” x Describing P If x can take on only discrete values, say (1, 2, 3, 4, or 5) then a table would work: 40% probability that x=4 x 1 2 3 4 5 P 10% 30% 40% 15% 5% Probabilities should sum to 100% Sometimes you see probabilities written as fractions, instead of percentages x 1 2 3 4 5 P 0.10 0.40 0.40 0.15 0.05 Probability should sum to 1 0.15 probability that x=4 And sometimes you see probabilities plotted as a histogram 0.5 0.15 probability that x=4 P(x) 0.0 1 2 3 4 5 x If x can take on any value, then use a smooth function (or “distribution”) p(x) instead of a table probability that x is between p(x) x1 and x2 is proportional to this area x1 x2 mathematically P(x1<x<x2) = x1 p(x) dx x2 x p(x) x Probability that x is between - and + is 100%, so total area = 1 Mathematically - + p(x) dx = 1 One Reason Why all this is relevant … Any measurement of data that contains noise is treated as a random variable, d and … The distribution p(d) embodies both the ‘true value’ of the datum being measured and the measurement noise and … All quantities derived from a random variable are themselves random variables, so … The algebra of random variables allows you to understand how … … measurement noise affects inferences made from the data Basic Description of Distributions want two basic numbers 1) something that describes what x’s commonly occur 2) something that describes the variability of the x’s 1) something that describes what x’s e commonly occur that is, where the distribution is centered Mode x at which distribution has peak most-likely value of x p(x) peak xmode x The most popular car in the US is the Honda CR-V Honda CR-V But the next car you see on the highway will probably not be a Honda CR-V Where’s a CV-R? But modes can be deceptive … 100 realizations of x Sure, the 1-2 range has the most counts, but most of the measurements are bigger than 2! N 3 18 11 8 11 14 8 7 11 9 p(x) peak x 0-1 1-2 2-3 3-4 4-5 5-6 6-7 7-8 8-9 9-10 0 xmode x 10 Median 50% chance x is smaller than xmedian 50% chance x is bigger than xmedian p(x) No special reason the median needs to coincide with the peak 50% 50% xmedian x Expected value or ‘mean’ P(x) value you would get if you took the mean of lots of realizations of x Let’s examine a discrete distribution, for simplicity ... 4 3 2 1 0 1 2 3 x Hypothetical table of 140 realizations of x x 1 2 3 Total N 20 80 40 140 [ 20 1 + 80 2 + 40 3 ] / 140 mean = = (20/140) 1 + (80/140) 2 + (40/140) 3 = p(1) 1 + p(2) 2 + p(3) 3 = Σi p(xi) xi by analogy for a smooth distribution Expected (or mean) value of x E(x) = - + x p(x) dx 2) something that describes the variability of the x’s that is, the width of the distribution p(x) Here’s a perfectly sensible way to define the width of a distribution… 50% 25% W50 25% x … it’s not used much, though Width of a distribution Here’s another way… p(x) Parabola [x-E(x)]2 E(x) x … multiply and integrate p(x) Idea is that if distribution is narrow, then most of the probability lines up with the low spot of the parabola [x-E(x)]2 p(x) E(x) But if it is wide, then some of the probability lines up with the high parts of the parabola Compute this total area … E(x) Variance = x s2 = - + [x-E(x)]2 p(x) dx x variance = s p(x) A measure of width … s E(x) x we don’t immediately know its relationship to area, though … the Gaussian or normal distribution s2 is variance x is expected value p(x) = 1 (2p)s exp{ - (x-x)2 / 2s2 ) Memorize me ! p(x) x=1 s=1 Examples of x Normal p(x) Distributions x=3 s = 0.5 x Properties of the normal distribution Expectation = p(x) Median = Mode = x 95% x x-2s x x+2s 95% of probability within 2s of the expected value Again, Why all this is relevant … Inference depends on data … You use measurement, d, to deduce the values of some underlying parameter of interest, m. e.g. use measurements of travel time, d, to deduce the seismic velocity, m, of the earth model parameter, m, depends on measurement, d so m is a function of d, m(d) so … If data, d, is a random variable then so is model parameter, m All inferences made from uncertain data are themselves uncertain Model parameters are described by a distribution, p(m) Functions of a random variable any function of a random variable is itself a random variable Special case of a linear relationship and a normal distribution Normal p(d) with mean d and variance s2d Linear relationship m = a d + b Normal p(m) with mean ad+b and variance a2s2d multivariate distributions Example Liberty island is inhabited by both pigeons and seagulls 40% of the birds are pigeons and 60% of the birds are gulls 50% of pigeons are white and 50% are tan 100% of gulls are white Two variables species s takes two values pigeon p and gull g Of 100 birds, 20 are white pigeons color c takes two values white w and tan t 20 are tan pigeons 60 are white gulls 0 are tan gulls What is the probability that a bird has species s and color c ? a random bird, that is p 20% 20% g 60% 0% s w t c Note: sum of all boxes is 100% This is called the Joint Probability and is written P(s,c) Two continuous variables say x1 and x2 have a joint probability distribution and written p(x1, x2) with p(x1, x2) dx1 dx2 = 1 You would contour a joint probability distribution and it would look something like x2 x1 What is the probability that a bird has color c ? Of 100 birds, start with P(s,c) p 20% 20% s g and sum columns 60% 0% w t c To get P(c) 80% 20% 20 are white pigeons 20 are tan pigeons 60 are white gulls 0 are tan gulls What is the probability that a bird has species s ? start with P(s,c) p 20% 20% s Of 100 birds, g 60% 0% 20 are white pigeons 20 are tan pigeons 60 are white gulls 0 are tan gulls w t c 40% and sum rows 60% To get P(s) x2 These operations make sense with distributions, too x2 x2 x1 p(x1) x1 x1 p(x1) = p(x1,x2) dx2 distribution of x1 (irrespective of x2) p(x2) = p(x1,x2) dx1 distribution of x2 (irrespective of x1) p(x2) Given that a bird is species s what is the probability that it has color c ? Of 100 birds, 20 are white pigeons p 50% 50% g 100% 0% s 20 are tan pigeons 60 are white gulls 0 are tan gulls w t c Note, all rows sum to 100 This is called the Conditional Probability of c given s and is written P(c|s) similarly … Given that a bird is color c what is the probability that it has species s ? Of 100 birds, 20 are white pigeons 20 are tan pigeons p 25% 100% g 75% 0% s 60 are white gulls 0 are tan gulls So 25% of white birds are pigeons w t c Note, all columns sum to 100 This is called the Conditional Probability of s given c and is written P(s|c) Beware! P(c|s) p 50% p 50% s P(s|c) 25% 100% 75% 0% s g 100% 0% w t c g w t c Lot of errors occur from confusing the two: Probability that, if you have pancreatic cancer, that you will die from it 90% Probability that, if you die, you will have died of pancreatic cancer Actor Patrick Swayse pancreatic cancer victim 1.4% note P(s,c) = P(s|c) P(c) p 20 p 20 25 100 =s s g 60 w 0 c t 25% of 80 is 20 g 75 80 20 0 w w c t t c and P(s,c) = P(c|s) P(s) 50% of 40 is 20 p 20 p 20 50 50 =s s g 60 w 0 c t p g 100 40 s 0 g w c t 60 Note that since P(s,c) = P(s|c) P(c) = P(c|s) P(s) then P(s) = c P(s,c) = c P(s|c) P(c) and P(c) = s P(s,c) = c P(c|s) P(s) Continuous versions: p(s) = p(s,c) dc = p(s|c) p(c) dc and p(c) = p(s,c) ds = p(c|s) p(s) ds Also, since P(s,c) = P(s|c) P(c) = P(c|s) P(s) then P(s|c) = P(c|s) P(s) / P(c) and P(c|s) = P(s|c) P(c) / P(s) … which is called Bayes Theorem In this example bird color is the observable, the “data”, d bird species is the “model parameter”, m P(c|s) “color given species” or P(d|m) is “making a prediction based on the model” Given a pigeon, what the probability that it’s grey? P(s|c), “species given color” or P(m|d) is “making an inference from the data” Given a grey bird, what the probability that it’s a pigeon? Bayes Theorem with data d and model m P(m|d) = P(d|m) P(m) / P(d) = P(d|m) P(m) / i P(d|mi) P(mi) Bayesian Inference: Interpret P(m) as our knowledge of m before measuring d. Then P(m|d) is our updated state of knowledge after measuring d. Example of Bayesian Inference Scenaio: A body of a man is brought to the morgue. The coroner wants to know, “did the man die of pancreatic cancer?”. Thus there is one model parameter, m, takes one of two values, Y (he died of pancreatic cancer) and N (he didn’t). Before examining the body, the best estimate of P(m) that can be made is P(Y)=0.014 and P(N)=0.986, the rate of death by pancreatic cancer in the general population. Now the coroner performs a test for pancreatic cancer, giving one data, d, and its positive, + (as contrasted to negative, -). But the test is not perfect. It has a non-zero rate of both false-positives and false-negatives, as quantified by the conditional distribution: false negatives (didn’t have cancer but tested +) Y 0.995 0.01 N 0.005 0.99 + - false positives (did have cancer but tested -) P(d|m) P(Y|+) = P(+|Y) P(Y) / [P(+|Y) P(Y)+P(+|N) P(N)] = 0.9950.014 / [0.9950.014+0.0050.986] A 74% chance that person died of pancreatic = 0.74 or 74% cancer is not all that conclusive! Why Bayes Theorem is important It provides a framework for relating making a prediction from the model, P(d|m) to making an inference from the data, P(m|d) Bayes Theorem also implies that the joint distribution of data and model parameters p(d, m) is the fundamental quantity If you know p(d, m), you know everything there is to know … Expectation Variance And Covariance Of a multivariate distribution x2 The expectation is computed by first reducing the distribution to one dimension x2 x1 p(x1) x1 take the expectation of p(x1) to get x1 x2 x2 x1 x1 p(x2) take the expectation of p(x2) to get x2 x2 The varaince is also computed by first reducing the distribution to one dimension x2 x1 p(x1) x1 s1 x1 take the variance of p(x1) to get s12 x2 x2 x1 s2 p(x2) take the variance of p(x2) to get s22 Note that in this distribution if x1 is bigger than x1, then x2 is bigger than x2 and if x1 is smaller than x1, then x2 is smaller than x2 x2 This is a positive correlation x2 x1 x1 Expected value Conversely, in this distribution if x1 is bigger than x1, then x2 is smaller than x2 and if x1 is smaller than x1, then x2 is smaller than x2 x2 This is a negative correlation x2 x1 x1 Expected value This correlation can be quantified by multiplying the distribution by a four-quadrant function x2 x2 x1 - + + - x1 And then integrating. The function (x1-x1)(x2-x2) works fine C = (x1-x1) (x2-x2) p(x1,x2) dx1dx2 Called the “covariance” Note that the matrix C with elements Cij = (xi-xi) (xj-xj) p(x1,x2) dx1dx2 has diagonal elements of sxi2 the variance of xi and off-diagonal elements of cov(xi,xj) the covariance of xi and xj s12 C= cov(x1,x2) cov(x1,x3) cov(x1,x2) s22 cov(x2,x2) cov(x1,x3) cov(x2,x2) s32 The “vector of means” of multivatiate distribution x and the “Covariance matrix” of multivariate distribution Cx summarized a lot – but not everything – about a multivariate distribution Functions of a set of random variables, x A vector of of N random variables in a vector, x Special Case linear function y=Mx the expectation of y is y=Mx Memorize! the covariance of y is So Cy = M Cx MT Memorize! Note that these rules work regardless of the distribution of x if y is linearly related to x, y=Mx then y=Mx (rule for means) Cy = M Cx MT (rule for propagating error) Memorize!