Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Scientific Methods 1 ‘Scientific evaluation, experimental design & statistical methods’ COMP80131 Lecture 4: Statistical Methods-Probability Barry & Goran www.cs.man.ac.uk/~barry/mydocs/myCOMP80131 30 Nov 2012 COMP80131-SEEDSM12-4 1 Probability There are two useful definitions of probability: 1. Bayesian probability: A person’s belief in the truth of a statement S, quantified on a scale from 0 (definitely not true) to 1 (definitely true). 2. Experimental (or frequentist) probability: Limit of M / N as N tends to infinity, where M = number of times that a statement S is found to be true if it is tested N times . 30 Nov 2012 COMP80131-SEEDSM12-4 2 Different language • By either defn, probability P(S) is a number in range 0 to 1. • Multiply by 100 to express as a percentage. • Or express as odds: e.g. ‘4 to 1 against’ means 1/5 = 0.2 = 20%. • What do odds of ‘4 to 1 on’ mean? • What does ‘50-50’ mean ? 30 Nov 2012 COMP80131-SEEDSM12-4 3 Calculating probability • The 2 defns of probability are related but subtly different. • By examining a coin, we could give ourselves good reason for believing that tossing it just once will give an even chance of getting heads, i.e. that the Bayesian defn of P(S) = 0.5 where S = ‘get heads’. • If coin is then tossed N = 100 times we would expect about M = 50 occurrences of heads meaning that M/N 0.5. • Increasing N to 1000 & then to 1000000 would be expected to produce closer & closer approximations to P(S) = 0.5. • If this does not happen, our ‘a-priori’ belief may be wrong. • The coin may be ‘weighted’. 30 Nov 2012 COMP80131-SEEDSM12-4 4 Random process • • • • Tossing a coin is a random process. It generates a ‘random variable’ Heads or Tails. Random because the outcome cannot be predicted exactly. If 1= heads & 0 = tails we have a random binary number. • • • • Throwing a dice generates a random integer in range 1-6. Roulette wheel generates random integers in range 0-36. Exams produces random numbers in range 0-100 These are random processes producing discrete variables. • Some random processes produce continuous variables. e.g. measuring people’s heights. 30 Nov 2012 COMP80131-SEEDSM12-4 5 Simulating random process • MATLAB has functions that generate pseudo-random numbers. • ‘rand’ produces a pseudo-random number ‘uniformly distributed’ in the range 0 to 1. • May be considered ‘continuous’ since floating pt is very accurate. • Calling ‘rand’ repeatedly produces numbers evenly distributed across the range 0 to 1. • ‘Pseudo-random’ because if we know the algorithm used, we can predict the numbers. • So we pretend we do not know the algorithm. • ‘rand’ may be considered to simulate some random process that generates truly random numbers, uniformly distributed. 30 Nov 2012 COMP80131-SEEDSM12-4 6 Simulating coin tossing in MATLAB for n=1:20 R = rand; if R > 0.5, Heads(n)=1 else Heads(n) = 0; end; end; % of n loop Heads 10110001110101011101 - 12 heads & 8 tails • I changed 20 to 10,000 & got 5066 heads: P(Heads) 0.5066 • I ran it again & got 4918 heads : P(Heads) 0.4918 30 Nov 2012 COMP80131-SEEDSM12-4 7 Using an unfair coin for n=1:20 R = rand; if R > 0.4, Heads(n)=1 else Heads(n) = 0; end; end; % of n loop Heads 00101001110101010101 - 10 heads & 10 tails • I changed 20 to 10,000 & got 6012 heads: P(Heads) 0.6012 • When I ran it again, I got 5979 heads : P(Heads) 0.5979 30 Nov 2012 COMP80131-SEEDSM12-4 8 Estimating probability experimentally • Cannot measure probability with 100% accuracy. • All measurements are estimates • They may be slightly or totally wrong. • According to experimental defn, we must perform an expt an infinite number of times to measure a probability. • This is clearly impossible. • In practice, we have to perform the experiment a finite number of times • (Cannot spend all our lives tossing coins) • Accept resulting measurement as estimate of true probability. 30 Nov 2012 COMP80131-SEEDSM12-4 9 Bayesian Definition • According to Bayesian defn of probability, a person’s belief in the truth of a statement may be affected by one or more assumption (hypotheses). • “I assume it is a fair coin” • Different people may have different beliefs. • Can only estimate probability using information we have available, • Can modify this estimate later if we get new information. 30 Nov 2012 COMP80131-SEEDSM12-4 10 Conditional probability • P(S S1) = probability of ‘statement S’ being true given that we know that another statement, S1, is definitely true. • If S ‘get heads’ we may at first believe that P(S) = 0.5. • But what if someone tells us that the statement S1: ‘coin is weighted with heavier metal on one side’, is true? • Change our measurement of probability to P(S S1). • P(S) is then referred to as the ‘prior’ probability • P(S S1) is the ‘conditional’ or ‘posterior’ probability. 30 Nov 2012 COMP80131-SEEDSM12-4 11 Bayes’s Theorem • Expresses probability of some fact ‘A’ being true when we know that some other fact ‘B’ is true: P( A B) P( B A) P( A) P( B) • E.g. let A = ‘coin is fair’ & B = ‘get 12 heads out of 20’ • P(A) is ‘prior’ as it does not take into account any information about B. • Similarly P(B) is ‘prior’. • P(A|B) and P(B|A) are ‘conditional’ or ‘posterior’ prob. • Can write P(B) = P(B | A)P(A) + P(B | not A)P(not A) 30 Nov 2012 COMP80131-SEEDSM12-4 12 What is prob of getting 12 heads out of 20? clear all; % WITH FAIR COIN H=zeros(21,1); for rep=1:1000 for n=1:20 R = rand; % Unif random number between 0 & 1 if R > 0.5, Heads(n)=0; else Heads(n)=1; end; end; % of n loop Count = sum(Heads); H(1+Count) = H(1+Count)+1; end; % of rep loop figure(1); stem(0:20,H); 30 Nov 2012 COMP80131-SEEDSM12-4 13 Histogram for 1000 trials FAIR COIN Frequency out of 1000 trials 200 180 160 140 120 100 80 60 40 20 0 0 30 Nov 2012 2 4 6 8 10 12 14 16 18 Number of Heads obtainable with 20 coin-tosses COMP80131-SEEDSM12-4 20 14 Estimate of prob distribution based on 1000 trials Estimate of probability distribution FAIR COIN 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 0 30 Nov 2012 2 4 6 8 10 12 14 16 18 20 Number of Heads obtainable with 20 coin-tosses COMP80131-SEEDSM12-4 15 Probability distribution (discrete) • When there is a finite number of possible outcomes, • For each possible outcome, gives its probability of occurring. Probability 0.2 0.1 Outcome 1 30 Nov 2012 2 3 4 COMP80131-SEEDSM12-4 16 Probability density function(pdf) • For an infinite number of possible outcomes, • Real numbers, continuous over range -∞ to • Prob outcome lies betw P & Q: pdf Q P pdf ( x)dx • Area under the curve. • NB prob (P) = prob(Q) = 0 P 30 Nov 2012 Q Possible outcome x COMP80131-SEEDSM12-4 17 Continuous random processes • Characterised by probability density functions (pdf) pdf(x) Uniform pdf: Prob of the random variable x lying between a and b is: 1 b x ab pdf(x) pdf ( x)dx b a a 1 Gaussian (Normal) pdf with mean m & std dev . 1 pdf ( x) e 2 1 x m 2 2 b Pr ob pdf ( x)dx a m- 30 Nov 2012 m m+ 68% ab x COMP80131-SEEDSM12-4 95.5% for m 2 99.7% for m 3 18 Continuous distributions • More about these later. • Back to discrete now. 30 Nov 2012 COMP80131-SEEDSM12-4 19 Probability estimate (fair coin) • Estimated discrete probabilities: for 0:9 heads 0 0 0 0 0.008 0.011 0.024 0.087 0.119 0.160 for 10:20 heads 0.194 0.157 0.115 0.076 0.03 0.012 0.003 0.003 0.001 0 0 • Estimated probability of getting 12 heads out of 20 with a fair coin is 0.115. 30 Nov 2012 COMP80131-SEEDSM12-4 20 What is prob of getting 12 heads out of 20? clear all; %WITH 60-40 WEIGHTED COIN H=zeros(21,1); for rep=1:1000 for n=1:20 R = rand; % Unif random number betw 0 & 1 if R > 0.4, Heads(n)=1; else Heads(n)=0; end; end; % of n loop Count = sum(Heads); H(1+Count) = H(1+Count)+1; end; % of rep loop figure(1); stem(0:20,H); 30 Nov 2012 COMP80131-SEEDSM12-4 21 HISTOGRAM for ‘60-40’ weighted coin 200 Frequency out of 1000 trials 180 160 140 120 100 80 60 40 20 0 0 30 Nov 2012 2 4 6 8 10 12 14 16 18 Number of Heads obtainable with 20 coin-tosses COMP80131-SEEDSM12-4 20 22 Estimate of prob distribution based on 1000 trials Prob distribution estimate for ‘60-40’ weighted coin 0.2 0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 0 30 Nov 2012 2 4 6 8 10 12 14 16 18 Number of Heads obtainable with 20 coin-tosses COMP80131-SEEDSM12-4 20 23 Cumulative prob distrib (discrete CDF) • Prob of ordered output being less than or equal to a given outcome x. CDF 1 0.1 1 30 Nov 2012 2 3 4 Outcome x COMP80131-SEEDSM12-4 24 Estimate Cumulative Prob Distrib • Prob of getting between 0 and n Heads • Easily derived from a Histogram or Prob Distribution. CDF(1)= H(1)/1000; for n=2:21, CDF(n)=CDF(n-1) + H(n)/1000; end; figure(3); stem(0:20,CDF); 30 Nov 2012 COMP80131-SEEDSM12-4 25 Estimate of cumulative prob dist based on 1000 trials Estimate of Cumulative Prob Dist 1 FAIR COIN 0.9 0.8 0.7 Usually an S shaped function 0.6 0.5 0.4 0.3 0.2 0.1 0 0 30 Nov 2012 2 4 6 8 10 12 14 16 18 20 Number of Heads obtainable with 20 coin-tosses COMP80131-SEEDSM12-4 26 4 coin-tosses: how many possible outcomes? 0000 0001 0010 0011 0100 0101 0110 0111 1111 1001 1010 1011 1100 1101 1110 1111 30 Nov 2012 How many with 0 heads? 1 How many with 1 heads? 4 = 4C1 How many with 2 heads? 6 = 4C2 = 43/ (2!) How many with 3 heads? 4 = 4C3 How many with 4 heads? 1 Combinations: nCr = no of ways of choosing r from n = n(n-1) …(n-r+1) / (r!) COMP80131-SEEDSM12-4 27 Binomial Prob Distribution • Distributions have up to now been estimated. • For random processes with just 2 outputs, we can derive a true distribution: • If p=prob(Heads), prob of getting Heads exactly r times in n independent coin-tosses is: r (1-p)(n-r) C p n r • For a fair coin. p=0.5, this becomes nCr /2n • For a fair dice, prob of throwing 3 sixes in five throws is: [54/(3 2 1)] (1/6)3 (5/6)2 30 Nov 2012 COMP80131-SEEDSM12-4 28 Implementing formula (fair coin) • • • • • • • • 30 Nov 2012 p = 0.5; % for fair coin tossing n=20; for r=0:n nCr = prod(n:-1:(n-r+1))/prod(1:r); P(1+r) = nCr * (p^r) * (1-p)^(n-r); end; figure(4); stem(0:20,P); axis([0 20 0 0.2]); grid on; COMP80131-SEEDSM12-4 29 True probability of getting that no of heads True prob distribution (n=20) 0.2 0.18 Fair coin 0.16 0.14 0.12 0.1 0.08 0.06 0.04 0.02 0 0 30 Nov 2012 2 4 6 8 10 12 14 16 18 20 No of heads obtainable with n coin-tosses COMP80131-SEEDSM12-4 30 True probability from formula • For 0-9 heads: 0 0 0.0002 0.0011 0.0046 0.015 0.037 0.074 0.12 0.16 • For 10-20 heads: 0.176 0.16 0.12 0.074 0.037 0.015 0.0046 0.0011 0.0002 00 • True prob of getting 12 heads with a fair coin is 0.12. – True prob of getting ≥12 heads with fair coin is 0.252. • Changing p to 0.4, we find that true probability of getting 12 heads out or 20 with a ‘60-40’ weighted coin is: 0.18 – True prob of getting ≥12 heads with weightd coin is 0.61. 30 Nov 2012 COMP80131-SEEDSM12-4 31 Back to Bayes’s Theorem • There are 2 coins a fair one & a ‘60-40’ weighted one. • Choose a coin at random & toss it 20 times. • What is the probability of having a weighted coin when I get 12 heads out of 20? • A ‘coin is weighted 60-40’ & B ‘get 12 heads from 20’ • We know that P(B Fair coin) is 0.12 & P(B A) is 0.18. • P(B) = P(B | not A) P(not A) + P(B | A) P(A) = 0.12 0.5 + 0.18 0.5 = 0.15 • P(A B) = P(B A) P(A) / P(B) = 0.18 0.5 /0.15 = 0.6 • If B ‘get ≥12 heads from 20’, P(A/B)=0.60.5/(0.6 0.5+0.252 0.5) = .3/.42 = 0.71 30 Nov 2012 COMP80131-SEEDSM12-4 32 Further illustration of Bayes Theorem • At a college there are: 10 students from France 5 girls & 5 boys 15 from UK 5 girls & 10 boys 20 from Canada 5 girls & 15 boys 30 Nov 2012 COMP80131-SEEDSM12-4 33 Calculation • If we choose a student at random, the a-priori probability that this student is French is P(French) = 10/45 = 2/9 0.22 • If we notice that this student is a boy, how does this change the probability that the student is French? • Use Bayes’ Theorem as follows: P( French Boy ) P( Boy French) P( French) P( Boy ) = 0.5 (10/45) / (30/45) = 1/6 0.167 • The fact that we notice that the chosen student is a boy gives us additional information that changes the probability that the student chosen at random will be French. 30 Nov 2012 COMP80131-SEEDSM12-4 34 Check the calculation • We can check the previous result by common sense, • Notice that out of 30 boys, 5 are from France. • Therefore, P(FB) = 5/30 = 1/6. 30 Nov 2012 COMP80131-SEEDSM12-4 35 Usefulness of Bayes Theorem • It allows us to take additional information into account when calculating probabilities. • Without the additional information, we have a ‘prior’ probability • With it we have a ‘conditional’ or ‘posterior’ probability. 30 Nov 2012 COMP80131-SEEDSM12-4 36 Bayes Theorem in medicine • A patent goes to a doctor with a bad cough & a fever. The doctor needs to decide whether he has ‘swine flu’. • Let statement S = ‘has bad cough and fever’ and statement F = ‘has swine flu’. • The doctor consults his medical books and finds that about 40% of patients with swine-flu have these same symptoms. • Assuming that, currently, about 1% of the population is suffering from swine-flu and that currently about 5% have bad cough and fever (due to many possible causes including swine-flu), we can apply Bayes theorem to estimate the probability of this particular patient having swine-flu. 30 Nov 2012 COMP80131-SEEDSM12-4 37 Another problem to solve • A doctor in another country knows form his text-books that for 40% of patients with swine-flu, • The statement S, ‘has bad cough and fever’ is true. • He sees many patients and comes to believe that the probability that a patient with ‘bad cough and fever’ actually has swine-flu is about 0.1 or 10%. • If there were reason to believe that, currently, about 1% of the population have a bad cough and fever, what percentage of the population is likely to be suffering from swine-flu? 30 Nov 2012 COMP80131-SEEDSM12-4 38 Concept of a ‘null-hypothesis’ • A null-hypothesis is an assumption that is made and then tested by a set of experiments designed to reveal that it is likely to be false, if it is false. • Testing is done by considering how probable the results are, assuming the null hypothesis is true. • If the results appear very improbable the researcher may conclude that the null-hypothesis is likely to be false. • This is usually the outcome the researcher hopes for when he or she is trying to prove that a new technique is likely to have some value. 30 Nov 2012 COMP80131-SEEDSM12-4 39 An example • Assume we wish to find out if a proposed technique designed to benefit users of a system is likely to have any value. • Divide the users into two groups and offer the proposed technique to one group and something different to the other group. • The null-hypothesis would be that the proposed technique offers no measurable advantage over the other techniques. 30 Nov 2012 COMP80131-SEEDSM12-4 40 The testing • Carried out by looking for differences between the sets of results obtained for each of the two groups. • Careful experimental design to eliminate differences not caused by the techniques being compared. • Must take a large number of users in each group & randomize the way the users are assigned to groups. • Once other differences have been eliminated as far as possible, remaining difference will hopefully be indicative of effectiveness of techniques being investigated. • Vital question is whether they are likely to be due to the advantages of the new technique, or the inevitable random variations that arise from the other factors. • Are the differences statistically significant? • Can employ statistical significance test to find out. 30 Nov 2012 COMP80131-SEEDSM12-4 41 Failure of the experiment • If results are not found to look improbable under the nullhypothesis, i.e. if differences between the two groups are not statistically significant, then no conclusion can be made. • Null-hypothesis could be true, or it could still be false. • It would be a mistake to conclude that the ‘null-hypothesis’ has been proved likely to be true in this circumstance. • It is quite possible that the results of the experiment give insufficient evidence to make any conclusions at all. 30 Nov 2012 COMP80131-SEEDSM12-4 42 P-Value • Probability of obtaining a test result at least as extreme as the one that was actually observed, assuming that the null hypothesis is true. • Reject null-hypothesis if p-value is less than some value α (significance level) which is often 0.05 or 0.01. • When null-hypothesis is rejected, result is statistically significant. 30 Nov 2012 COMP80131-SEEDSM12-4 43 Checking whether a coin is fair • Suppose we obtain heads 14 times out of 20 flips. • The p-value for this test result would be the probability of a fair coin landing on heads at least 14 times out of 20 flips. • This is: (20C14 + 20C15+20C16+20C17+20C18+20C19+20C20) / 220 = 0.058 • This is probability that a fair coin would give a result as extreme or more extreme than 14 heads out of 20. 30 Nov 2012 COMP80131-SEEDSM12-4 44 Significance test • Reject null-hypothesis if p-value α . • If α= 0.05, rejection of null-hypothesis is at the 5% (significance) level. • Probability of wrongly rejecting null-hypothesis (Type 1 error) will be equal to α. • This is considered ‘sufficiently low’. • In this ‘coin testing’ case, p-value > 0.05, • Therefore observation is consistent with null-hypothesis and we cannot reject it. • Cannot conclude that coin is likely to be unfair. • But we have NOT proved that coin is likely to be fair. • 14 heads out of 20 flips can be ascribed to chance alone. • It falls within the range of what could happen 95% of the time with a fair coin. 30 Nov 2012 COMP80131-SEEDSM12-4 45