Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Hypothesis Testing - University of Illinois Urbana
Transcript
Hypothesis Testing or How to Decide to Decide Edpsy 580 Carolyn J. Anderson Department of Educational Psychology University of Illinois at Urbana-Champaign Hypothesis Testing or How to Decide to Decide Slide 1 of 54 Outline ■ Basic Logic 1. Definitions 2. Six Steps ■ Means – 1 population 1. When σ 2 is known 2. When σ 2 is not known ■ Types of errors and correct decisions. ■ When the alternative is true. 1. Power 2. Ways of increasing power 3. Choosing sample size Basic Logic 6 Steps Means 1 population Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 2 of 54 Hypothesis Testing– Definitions Basic Logic ● Hypothesis Testing– Definitions ● Scientific Hypotheses ■ A hypothesis is a statement that may or may not be true. ■ There are two kinds of hypotheses. ◆ Scientific ◆ Statistical ■ Scientific Hypotheses is a statement about what should be observed based on some theory about a particular behavior(s).// The scientific theory provides ● Statistical Hypotheses ● Two Types of Statistical Hypotheses ● Testing “THE” Hypothesis ● Testing Hypotheses & Assumptions ● Hypotheses versus Assumptions 6 Steps Means 1 population Errors and Correction Decisions Power Things that Effect Power ◆ ◆ Guidance about what to observe and what should happen. Explanations about why it occurs. Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 3 of 54 Scientific Hypotheses ■ Phenomenon: Students in academic high school programs tend to have higher achievement test scores than those in general and/or vocational/techinical programs. ■ Scientific hypotheses (theories): Basic Logic ● Hypothesis Testing– Definitions ● Scientific Hypotheses ● Statistical Hypotheses ● Two Types of Statistical Hypotheses ● Testing “THE” Hypothesis ● Testing Hypotheses & Assumptions ● Hypotheses versus Assumptions ◆ Certain students are in academic programs because they are good in academic subjects, plan to go to college, etc. ◆ Students in academic programs are taught more academic subjects than those in general and/or vocational-technical programs. 6 Steps Means 1 population Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 4 of 54 Statistical Hypotheses ■ Statistical Hypotheses are statements about one or more population distribution(s) and usually about one or more population parameter. ■ Examples: Basic Logic ● Hypothesis Testing– Definitions ● Scientific Hypotheses ● Statistical Hypotheses ● Two Types of Statistical Hypotheses ● Testing “THE” Hypothesis ● Testing Hypotheses & Assumptions ● Hypotheses versus Assumptions ◆ H: Reading scores of high school seniors attending academic programs are N (55, 80). ◆ H: Reading scores of high school seniors attending academic programs are N (55, 80) and scores for students attending general programs are N (50, 100). 6 Steps Means 1 population Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 5 of 54 Two Types of Statistical Hypotheses Basic Logic ● Hypothesis Testing– Definitions ■ ● Scientific Hypotheses ● Statistical Hypotheses ● Two Types of Statistical Simple hypothesis completely specifies a population distribution −→ the sampling distribution of any statistic is also completely known (once you know the sample size). Hypotheses e.g., The ones given on previous page. ● Testing “THE” Hypothesis ● Testing Hypotheses & Assumptions ● Hypotheses versus Assumptions 6 Steps Means 1 population ■ Composite hypotheses — the population distribution is not completely specified. e.g., Errors and Correction Decisions Power ◆ H: population is normal and µ = 55 Things that Effect Power (“exact” parameter value) Choosing Sample Size ◆ Hypothesis Testing or How to Decide to Decide H: µ ≥ 50 (“range” of parameter values). Slide 6 of 54 Testing “THE” Hypothesis Basic Logic ● Hypothesis Testing– Definitions ■ “THE” hypothesis is the null hypothesis, “Ho ” ■ Testing the null hypothesis is making a choice between two hypotheses or models. ■ These two hypotheses are: ● Scientific Hypotheses ● Statistical Hypotheses ● Two Types of Statistical Hypotheses ● Testing “THE” Hypothesis ● Testing Hypotheses & Assumptions ● Hypotheses versus Assumptions ◆ 6 Steps Means 1 population Null hypothesis, HO : ■ Assumed to be true. ■ It determines the sampling distribution of the test statistic −→ Ho must be an exact hypothesis. Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size ◆ Hypothesis Testing or How to Decide to Decide The Alternative hypothesis, Ha , is true if the null is false −→ Ha is a composite hypothesis. Slide 7 of 54 Testing Hypotheses & Assumptions ■ The alternative hypothesis is usually what corresponds to the expected result (based on your scientific hypothesis). ■ E.g., Ho : µ = 50 and Ha : µ > 50. Basic Logic ● Hypothesis Testing– Definitions ● Scientific Hypotheses ● Statistical Hypotheses ● Two Types of Statistical Hypotheses ● Testing “THE” Hypothesis ● Testing Hypotheses & Assumptions ● Hypotheses versus Assumptions 6 Steps Means 1 population ■ ◆ Ho only specifies a particular value for the mean. ◆ Ha gives a range of possible values. Need the sampling distribution for a test statistic −→ must make assumptions. Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 8 of 54 Hypotheses versus Assumptions ■ The difference between assumptions and hypotheses is that assumptions are exactly the same under Ho and Ha but the hypotheses differ. ■ For example, Basic Logic ● Hypothesis Testing– Definitions ● Scientific Hypotheses ● Statistical Hypotheses ● Two Types of Statistical Hypotheses ● Testing “THE” Hypothesis ● Testing Hypotheses & Assumptions ● Hypotheses versus Assumptions 6 Steps Assumptions Means 1 population Errors and Correction Decisions Power Things that Effect Power Hypothesis Null Ȳ is Normal Independence σ2 Ho : µ = 55 Alternative Ȳ is Normal Independence σ2 Ha : µ 6= 55 Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 9 of 54 6 Steps in a Statistical Hypothesis Test Basic Logic 6 Steps ● 6 Steps in a Statistical Hypothesis Test For illustration, use test on mean of a single variable (i.e., HSB who attend academic programs and reading scores). 1. State the null and alternative hypotheses. ● Step 4 Ho : µ = 50 vs ● Step 4 ● Step 4 ● Step 4 & Step 5 ● Rules for Non-directional Alternatives ● Rejection Region ● Rules for Non-directional Ha : µ 6= 50 (or Ha : µ > 50) 2. Make assumptions so that the sampling distribution of the sample statistic is completely specified. Ha ● Critical Mean Values ● Last Rule for Non-directional Ha ● Directional Alternatives ● Directional Alternatives – continued ● 1 vs 2 Tail Tests ■ Observations are independent. ■ Come from a population with σ 2 = 100. ■ Sampling distribution of Ȳ is normal (note: n = 308). ● 1 vs 2 Tail Tests ● Summary of the Six Steps Means 1 population Errors and Correction Decisions Power 3. Specify the degree of “risk” or α level. This equals the probability of rejecting the null hypothesis when the null hypothesis is true. α = .05 Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 10 of 54 Step 4 4. Basic Logic Compute the probability of obtaining the sample statistic that differs from the hypothesized valued (from Ho ). This probability is called the p–value. 6 Steps ● 6 Steps in a Statistical Hypothesis Test ● Step 4 ● Step 4 ● Step 4 ● Step 4 & Step 5 ● Rules for Non-directional To find the p–value, you need to know what is the sampling distribution of a test statistic.. . . ■ Test statistic: For our example, 55.89 − 50 ȳ − 50 √ = √ z= = 10.34 σ/ n 10/ 308 Alternatives ● Rejection Region ● Rules for Non-directional Ha ● Critical Mean Values ● Last Rule for Non-directional Ha ● Directional Alternatives ● Directional Alternatives – continued ● 1 vs 2 Tail Tests ■ Sampling distribution of z is N (0, 1) because ◆ Assumed that Ȳ is normal. ◆ A linear transformation of a normal R.V. is a normal variable. ◆ E(z) = 0 (if null is true) and var(z) = 1. ● 1 vs 2 Tail Tests ● Summary of the Six Steps Means 1 population Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 11 of 54 Step 4 The Sampling Distribution of Ȳ assuming Ho : Basic Logic 6 Steps ● 6 Steps in a Statistical Hypothesis Test ● Step 4 ● Step 4 ● Step 4 ● Step 4 & Step 5 ● Rules for Non-directional Alternatives ● Rejection Region ● Rules for Non-directional Ha ● Critical Mean Values ● Last Rule for Non-directional Ha ● Directional Alternatives ● Directional Alternatives – continued ● 1 vs 2 Tail Tests ● 1 vs 2 Tail Tests ● Summary of the Six Steps Means 1 population Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 12 of 54 Step 4 Basic Logic The Sampling Distribution of Ȳ and test statistic 6 Steps ● 6 Steps in a Statistical Hypothesis Test ● Step 4 ● Step 4 ● Step 4 ● Step 4 & Step 5 ● Rules for Non-directional Alternatives ● Rejection Region ● Rules for Non-directional Ha ● Critical Mean Values ● Last Rule for Non-directional Ha ● Directional Alternatives ● Directional Alternatives – continued ● 1 vs 2 Tail Tests ● 1 vs 2 Tail Tests ● Summary of the Six Steps Means 1 population Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 13 of 54 Step 4 & Step 5 Basic Logic 6 Steps ● 6 Steps in a Statistical Hypothesis Test ● Step 4 ● Step 4 ● Step 4 ● Step 4 & Step 5 ● Rules for Non-directional Alternatives ● Rejection Region ● Rules for Non-directional 4. (continued) The probability of observing a standard normal random variable with a |z-score| of 10.34 or larger is p = .5 × 10−24 . 5. Make a decision. Reject Ho or Retain Ho . There are equivalent ways of doing this, but they all depend on Ha ● Critical Mean Values ● Last Rule for Non-directional Ha ● Directional Alternatives ● Directional Alternatives – continued ◆ Your alternative hypothesis ◆ The following probability Ȳ − µ ≤ zα/2 Prob −zα/2 ≤ σȲ ● 1 vs 2 Tail Tests ● 1 vs 2 Tail Tests ● Summary of the Six Steps Means 1 population Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 14 of 54 Rules for Non-directional Alternatives “Two-Tailed” Tests −→ Ha : µ 6= 50. Basic Logic 6 Steps ● 6 Steps in a Statistical Hypothesis Test ■ Using p–values and α, the rule is If p If p ● Step 4 ● Step 4 ● Step 4 ● Step 4 & Step 5 ● Rules for Non-directional Alternatives > α/2 then retain Ho < α/2 then reject Ho e.g., p = .5 × 10−24 << .05/2 = .025 ● Rejection Region ● Rules for Non-directional Ha ● Critical Mean Values ● Last Rule for Non-directional Ha ■ Compare the test statistic to a critical value ● Directional Alternatives ● Directional Alternatives – continued If ● 1 vs 2 Tail Tests ● 1 vs 2 Tail Tests If ● Summary of the Six Steps −zα/2 ≤ z ≤ zα/2 , z < −zα/2 or z > zα/2 , then retain Ho then reject Ho Means 1 population Errors and Correction Decisions Power e.g., z = 10.34 > z.05/2 = 1.96, so reject Ho . Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 15 of 54 Rejection Region Basic Logic 6 Steps ● 6 Steps in a Statistical Hypothesis Test ● Step 4 ● Step 4 ● Step 4 ● Step 4 & Step 5 ● Rules for Non-directional Alternatives ● Rejection Region ● Rules for Non-directional Ha ● Critical Mean Values ● Last Rule for Non-directional Ha ● Directional Alternatives ● Directional Alternatives – continued ● 1 vs 2 Tail Tests ● 1 vs 2 Tail Tests ● Summary of the Six Steps Means 1 population Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 16 of 54 Rules for Non-directional Ha ■ Basic Logic 6 Steps ● 6 Steps in a Statistical Hypothesis Test ● Step 4 Compare Ȳ to the “critical” mean value. Compute Ȳlower = µ − zα/2 σȲ and Ȳupper = µ + zα/w σȲ . The rule is ● Step 4 If ● Step 4 ● Step 4 & Step 5 ● Rules for Non-directional Alternatives ● Rejection Region ● Rules for Non-directional Ha ● Critical Mean Values ● Last Rule for Non-directional Ha ● Directional Alternatives ● Directional Alternatives – continued ● 1 vs 2 Tail Tests ● 1 vs 2 Tail Tests ● Summary of the Six Steps Ȳlower ≤ Ȳ ≤ Ȳupper , then retain Ho Ȳ < Ȳlower or Ȳ > Ȳupper , then reject Ho √ e.g., Ȳlower = 50 − 1.96(10/ √ 308) = 48.88 and Ȳupper = 50 + 1.96(10/ 308) = 51.12. If The obtained sample mean Ȳ = 55.89 is not in the interval (48.88 to 51.12), reject Ho . Means 1 population Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 17 of 54 Critical Mean Values Basic Logic 6 Steps ● 6 Steps in a Statistical Hypothesis Test ● Step 4 ● Step 4 ● Step 4 ● Step 4 & Step 5 ● Rules for Non-directional Alternatives ● Rejection Region ● Rules for Non-directional Ha ● Critical Mean Values ● Last Rule for Non-directional Ha ● Directional Alternatives ● Directional Alternatives – continued ● 1 vs 2 Tail Tests ● 1 vs 2 Tail Tests ● Summary of the Six Steps Means 1 population Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 18 of 54 Last Rule for Non-directional Ha ■ Basic Logic Ȳ fall inside the (1 − α) × 100% confidence interval? The (1 − α) × 100% confidence interval is 6 Steps ● 6 Steps in a Statistical Hypothesis Test Ȳ ± zα/2 σȲ ● Step 4 ● Step 4 ● Step 4 ● Step 4 & Step 5 ● Rules for Non-directional Rule: If the hypothesized value is in interval, retain Ho , and if the hypothesized value is outside the interval, reject Ho . Alternatives ● Rejection Region ● Rules for Non-directional Ha ● Critical Mean Values ● Last Rule for Non-directional Ha ● Directional Alternatives ● Directional Alternatives – continued e.g., 95% confidence interval √ for µreading is 55.89 ± 1.96(10/ 308) −→ (54.77, 57.01) The interval does not contain 50 → reject Ho . ● 1 vs 2 Tail Tests ● 1 vs 2 Tail Tests ● Summary of the Six Steps Means 1 population Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 19 of 54 Directional Alternatives e.g., Ha : µreading > µ0 or Ha : µreading < µo Basic Logic 6 Steps ● 6 Steps in a Statistical Hypothesis Test ■ Using p-values: ● Step 4 If p> α, then retain Ho if p< α, then reject Ho ● Step 4 ● Step 4 ● Step 4 & Step 5 ● Rules for Non-directional Alternatives ● Rejection Region ● Rules for Non-directional Ha ● Critical Mean Values ● Last Rule for Non-directional ■ Compare test statistic to critical value of the test statistic. Either Ha If Ha : µ < µo then if z < −z/alpha , reject Ho ● Directional Alternatives ● Directional Alternatives – continued ● 1 vs 2 Tail Tests ● 1 vs 2 Tail Tests ● Summary of the Six Steps or If Ha : µ > µo , then if z > z/alpha , reject Ho Means 1 population Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 20 of 54 Directional Alternatives – continued Basic Logic ■ 6 Steps ● 6 Steps in a Statistical Hypothesis Test Compare obtained sample statistic (e.g., Ȳ ) to the critical value of the statistic. ● Step 4 ● Step 4 ● Step 4 ◆ If Ha : µ < µo , then compute Ȳcrit = µo − zα σȲ . Reject Ho if Ȳ < Ȳcrit . ◆ If Ha : µ > µo , then compute Ȳcrit = µo + zα σȲ . Reject Ho if Ȳ > Ȳcrit . ● Step 4 & Step 5 ● Rules for Non-directional Alternatives ● Rejection Region ● Rules for Non-directional Ha ● Critical Mean Values ● Last Rule for Non-directional Ha ● Directional Alternatives ● Directional Alternatives – continued ● 1 vs 2 Tail Tests ■ A one-sided confidence interval. . . ● 1 vs 2 Tail Tests ● Summary of the Six Steps Means 1 population Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 21 of 54 1 vs 2 Tail Tests ■ The “rejection regions” for 1 and 2 tailed tests with the same α level are different. ■ The rejection region for 1 tail test is larger. ■ You must specify Ha before you look at your data; otherwise, your probability values are not valid. ■ It’s “cheating” to go “data snooping” as way to decide on Ha . ■ “Statistical significance” means that the difference between what you observed/obtained and what you expected assuming HO is “statistically large” and it was unlikely to have happened. The result is more likely under Ha . ■ It is not correct to state that HO (or HA ) is “true” or “false”. Always possible that we’ve made a mistake. Basic Logic 6 Steps ● 6 Steps in a Statistical Hypothesis Test ● Step 4 ● Step 4 ● Step 4 ● Step 4 & Step 5 ● Rules for Non-directional Alternatives ● Rejection Region ● Rules for Non-directional Ha ● Critical Mean Values ● Last Rule for Non-directional Ha ● Directional Alternatives ● Directional Alternatives – continued ● 1 vs 2 Tail Tests ● 1 vs 2 Tail Tests ● Summary of the Six Steps Means 1 population Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 22 of 54 1 vs 2 Tail Tests ■ The decision does not tell you that Basic Logic 6 Steps ● 6 Steps in a Statistical Hypothesis Test ● Step 4 ● Step 4 ◆ The difference is important. ◆ The result is meaningful. ◆ The scientific theory (explanation) guiding your research is correct. ● Step 4 ● Step 4 & Step 5 ● Rules for Non-directional Alternatives ● Rejection Region ● Rules for Non-directional Ha ● Critical Mean Values ● Last Rule for Non-directional Ha ● Directional Alternatives ● Directional Alternatives – continued ■ Step 6: interpretation. ● 1 vs 2 Tail Tests ● 1 vs 2 Tail Tests ● Summary of the Six Steps Means 1 population Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 23 of 54 Summary of the Six Steps Basic Logic 6 Steps ● 6 Steps in a Statistical Hypothesis Test ● Step 4 ● Step 4 1. State the null and alternative hypotheses. 2. Make assumptions. ● Step 4 ● Step 4 & Step 5 ● Rules for Non-directional Alternatives 3. Specify the α–level. ● Rejection Region ● Rules for Non-directional Ha ● Critical Mean Values ● Last Rule for Non-directional Ha ● Directional Alternatives ● Directional Alternatives – continued 4. Compute test statistic, p-value, CI, or critical value of sample statistic. ● 1 vs 2 Tail Tests ● 1 vs 2 Tail Tests ● Summary of the Six Steps 5. Make a decision. Means 1 population Errors and Correction Decisions 6. Interpret the result. Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 24 of 54 When σ 2 is Unknown The process is the same, just two details change Basic Logic 6 Steps ■ Must estimate σ 2 . Means 1 population ● When σ 2 is Unknown ● Different Sampling ◆ We’ll use the unbiased one: Distribution n ● Student t-Distribution ● Standard Normal vs t-Distribution ● Standard Normal vs t-Distribution 1 X 2 (yi − ȳ)2 s = n − 1 i=1 p and use s/ (n) as an estimate of σȲ . Errors and Correction Decisions Power ◆ Things that Effect Power The test statistic is now, Choosing Sample Size Ȳ − µo Ȳ − µo √ = t= s/ n sȲ ■ Need a different sampling distribution for our test statistic. . . . Hypothesis Testing or How to Decide to Decide Slide 25 of 54 Different Sampling Distribution ■ Ȳ is the only random variable in Basic Logic z= 6 Steps Means 1 population ● When σ 2 is Unknown ● Different Sampling Ȳ − µ σȲ ■ Both Ȳ and sȲ are random variables in ■ Ȳ − µo Ȳ − µo √ = sȲ s/ n Therefore, expect that our t statistic to be more variable than z. The sampling distribution for t must have heavier tails than the normal distribution (“leptokurtic”). ■ Student t-Distribution Distribution ● Student t-Distribution ● Standard Normal vs t-Distribution ● Standard Normal vs t-Distribution Errors and Correction Decisions t= Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 26 of 54 Student t-Distribution ■ Symmetric around the mean µ = 0, “central t-distribution” ■ Bell shaped. Heavier tails than normal distribution. Depends on the “degrees of freedom”. Basic Logic 6 Steps Means 1 population ● When σ 2 is Unknown ● Different Sampling ■ Distribution ● Student t-Distribution ● Standard Normal vs t-Distribution ● Standard Normal vs t-Distribution ■ ◆ Notation: Greek “nu”, ν = degrees of freedom. ◆ ν is a parameter of the central t–distribution. Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size ■ ν =n−1 Hypothesis Testing or How to Decide to Decide Slide 27 of 54 Standard Normal vs t-Distribution ■ Basic Logic ν 6 Steps ■ Means 1 population ● When σ 2 is Unknown ● Different Sampling The standard normal is the “limiting distribution” of the t-distribution: As ν → ∞, t → N (0, 1) ■ For very large n, the t-distribution is very close to N (0, 1). HSB reading Ho : µ = 50 vs Ha : µ > 50, Distribution 55.89 − 50 5.89 t= p = 11.098 = .53 87.14/308 ● Student t-Distribution ● Standard Normal vs t-Distribution ● Standard Normal vs t-Distribution p-value = .32 × 10−23 . Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size ■ Note z.025 = 1.968 compared to t.025,ν=307 = 1.969 Hypothesis Testing or How to Decide to Decide Slide 28 of 54 Standard Normal vs t-Distribution Basic Logic 6 Steps Means 1 population ● When σ 2 is Unknown ● Different Sampling Distribution ● Student t-Distribution ● Standard Normal vs t-Distribution ● Standard Normal vs t-Distribution Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 29 of 54 Errors and Correction Decisions Basic Logic 6 Steps Means 1 population Errors and Correction Decisions ● Errors and Correction Decisions ● Errors and Correction Decisions ● Errors and Correction HO Decision Decisions ● Graphically, what’s going on Power HA True State of the World HO HA Correct Type II Error (1 − α) β Type I Error Correct α power (1 − β) 1.00 1.00 Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 30 of 54 Errors and Correction Decisions Basic Logic ■ Probabilities of events when the null is true: 6 Steps Correct P (retainHO |HO is true) = Type 1 error: P (rejectHO |HO is true) = Means 1 population Errors and Correction Decisions ● Errors and Correction Decisions ● Errors and Correction Decisions ● Errors and Correction Decisions ● Graphically, what’s going on ■ 1−α α Probabilities when the alternative is true: Power Type 2 error: P (retainHo |HA is true) = β Power: P (rejectHo |HA is true) = 1 − β Things that Effect Power Choosing Sample Size ■ These are conditional probabilities. Hypothesis Testing or How to Decide to Decide Slide 31 of 54 Errors and Correction Decisions ■ Generally, making a type I error is considered to be worse than making a type II error; therefore, we set the type I error rate α so that it is small. ■ When we preform a test of Ho , we assume that it is true. ■ Now we’ll consider what happens when Ha is really true when we’ve assumed or acted as if Ho is true. ■ Suppose that the true µ = 55.00 and σ 2 = 100 and that we test Ho : µ = 50 vs HA : µ using z. > 50 Basic Logic 6 Steps Means 1 population Errors and Correction Decisions ● Errors and Correction Decisions ● Errors and Correction Decisions ● Errors and Correction Decisions ● Graphically, what’s going on Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 32 of 54 Graphically, what’s going on Basic Logic 6 Steps Means 1 population Errors and Correction Decisions ● Errors and Correction Decisions ● Errors and Correction Decisions ● Errors and Correction Decisions ● Graphically, what’s going on Power Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 33 of 54 Power ■ Basic Logic 6 Steps What’s the probability of rejecting Ho : µ = 50 given that the true mean is µ = 55? Means 1 population P (reject Ho |µ = 55) = POWER Errors and Correction Decisions Power ● Power ● Graphically: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing ■ This is the shaded area under the curve for distribution on the next page. . . Power ● Power of t-test ● Power of t-test ● Power of t-test & Non-central t-distribution ● Power of t-test: ● Power of t-test: µ = 49 µ = 48 ● Two-Tailed Test ● Power of t-test: SAS/Power Analysis ● SAS/Power Analysis: Results Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 34 of 54 Graphically: Computing Power Basic Logic 6 Steps Means 1 population Errors and Correction Decisions Power ● Power ● Graphically: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power ● Power of t-test ● Power of t-test ● Power of t-test & Non-central t-distribution ● Power of t-test: ● Power of t-test: µ = 49 µ = 48 ● Two-Tailed Test ● Power of t-test: SAS/Power Analysis ● SAS/Power Analysis: Results Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 35 of 54 The Algebra: Computing Power ■ Basic Logic 6 Steps We rejected Ho : µ = 50 vs Ha : µ > 50 for the reading scores of students that go to academic programs. The test statistic was Means 1 population z= Errors and Correction Decisions Ȳ − µo 55.89 − 50.00 √ = √ = 10.34 σ/ n 10/ 308 Power ● Power ● Graphically: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power and z.95 = 1.645 (i.e., α = .05). ■ Assume actual mean is µ = 55. ■ We would have rejected Ho for any test statistic ≥ 1.645 ● Power of t-test ● Power of t-test ● Power of t-test & Non-central t-distribution ● Power of t-test: ● Power of t-test: µ = 49 µ = 48 ● Two-Tailed Test ● Power of t-test: SAS/Power Analysis ● SAS/Power Analysis: Results Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 36 of 54 The Algebra: Computing Power Basic Logic ■ 6 Steps Would have rejected Ho for any Ȳ ≥ Ȳcrit Means 1 population zcrit Errors and Correction Decisions Ȳcrit − µo √ = 1.645 ≥ σ/ n Power ● Power ● Graphically: Computing Power ● The Algebra: Computing ■ After a little algebra, Ȳcrit = zcrit σȲ + µo Power ● The Algebra: Computing Power ● The Algebra: Computing Power ● Power of t-test ● Power of t-test ● Power of t-test & Non-central ■ So for our example, t-distribution µ = 49 ● Power of t-test: µ = 48 ● Power of t-test: ● Two-Tailed Test ● Power of t-test: SAS/Power Analysis Ȳcrit = 1.645( p 100/308) + 50 = 50.937 ● SAS/Power Analysis: Results Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 37 of 54 The Algebra: Computing Power ■ If µ = 55, this corresponds to a z-score of Ȳcrit − µa 50.937 − 55 √ za = = p = −7.125 σ/ n 100/308 Basic Logic 6 Steps Means 1 population Errors and Correction Decisions Power ● Power ● Graphically: Computing Power ● The Algebra: Computing ■ Probability of Type II error Power ● The Algebra: Computing Power ● The Algebra: Computing β Power ● Power of t-test ● Power of t-test ● Power of t-test & Non-central = P (z ≤ −7.125|µ = 55) = .1 × 10−11 t-distribution ● Power of t-test: ● Power of t-test: µ = 49 µ = 48 ● Two-Tailed Test ● Power of t-test: SAS/Power Analysis = P (retain Ho |µ = 55) ■ Probability of correct decision, Power ● SAS/Power Analysis: Results Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide P (reject Ho |µ = 55) = 1 − β = .99999 Slide 38 of 54 Power of t-test ■ Same procedure; however, use a “non-central” t-distribution instead of normal. ■ Example: HSB reading scores of students who attend general high school program: ● Power ● Graphically: Computing Power ● The Algebra: Computing ■ Hypotheses: Ho : µ = 50 vs Ha : µ < 50 Power ● The Algebra: Computing Power ● The Algebra: Computing ■ α = .05 Power ■ Summary Statistics: Basic Logic 6 Steps Means 1 population Errors and Correction Decisions Power ● Power of t-test ● Power of t-test ● Power of t-test & Non-central t-distribution ● Power of t-test: ● Power of t-test: µ = 49 µ = 48 ● Two-Tailed Test ● Power of t-test: SAS/Power Analysis ● SAS/Power Analysis: Results n sȲ = 145, Ȳ = 49.06, p = 80.09/145 = .74 s2 = 80.09, Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 39 of 54 Power of t-test ■ Test statistic: t = (49.06 − 50)/.74 = −1.27 ■ Using t-distribution with ν = 144: tcrit = t144,.05 = −1.6555 or p-value = .10. ■ Decision: retain Ho . ■ Interpretation: The data do not provide evidence that the mean reading scores for students that attend general high school programs differs from the overall mean of 50. ■ What about power of the test? Basic Logic 6 Steps Means 1 population Errors and Correction Decisions Power ● Power ● Graphically: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power ● Power of t-test ● Power of t-test ● Power of t-test & Non-central t-distribution ● Power of t-test: ● Power of t-test: µ = 49 µ = 48 ● Two-Tailed Test ● Power of t-test: SAS/Power Analysis ● SAS/Power Analysis: Results Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 40 of 54 Power of t-test & Non-central t-distribution Basic Logic ■ Need to use a “Non-central” t-distribution ■ The non-centrality parameter equals 6 Steps Means 1 population Errors and Correction Decisions µa − µo √ δ= s/ n Power ● Power ● Graphically: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power ■ Example. . . ● Power of t-test ● Power of t-test ● Power of t-test & Non-central t-distribution ● Power of t-test: ● Power of t-test: µ = 49 µ = 48 ● Two-Tailed Test ● Power of t-test: SAS/Power Analysis ● SAS/Power Analysis: Results Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 41 of 54 Power of t-test: µ = 49 Basic Logic 6 Steps Means 1 population Errors and Correction Decisions Power ● Power ● Graphically: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power ● Power of t-test ● Power of t-test ● Power of t-test & Non-central t-distribution ● Power of t-test: ● Power of t-test: µ = 49 µ = 48 ● Two-Tailed Test ● Power of t-test: SAS/Power Analysis ● SAS/Power Analysis: Results Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 42 of 54 Power of t-test: µ = 48 Basic Logic 6 Steps Means 1 population Errors and Correction Decisions Power ● Power ● Graphically: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power ● Power of t-test ● Power of t-test ● Power of t-test & Non-central t-distribution ● Power of t-test: ● Power of t-test: µ = 49 µ = 48 ● Two-Tailed Test ● Power of t-test: SAS/Power Analysis ● SAS/Power Analysis: Results Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 43 of 54 Two-Tailed Test Basic Logic 6 Steps Means 1 population Errors and Correction Decisions Power ● Power ● Graphically: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power ● Power of t-test ● Power of t-test ● Power of t-test & Non-central t-distribution ● Power of t-test: ● Power of t-test: µ = 49 µ = 48 ● Two-Tailed Test ● Power of t-test: SAS/Power Analysis ● SAS/Power Analysis: Results Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 44 of 54 Power of t-test: SAS/Power Analysis ■ Solutions → Analysis → Analyst → Statistics → Sample Size → Select the test for which you want to computer power. ■ For one sample t-test enter requested information. Note that “standard deviation” refers to s. ■ Output: what you put in and the computed power. ■ Demonstration. Basic Logic 6 Steps Means 1 population Errors and Correction Decisions Power ● Power ● Graphically: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power ● Power of t-test ● Power of t-test ● Power of t-test & Non-central t-distribution ● Power of t-test: ● Power of t-test: µ = 49 µ = 48 ● Two-Tailed Test ● Power of t-test: SAS/Power Analysis ● SAS/Power Analysis: Results Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 45 of 54 SAS/Power Analysis: Results For a one-sample t-test, 1-sided alternative with Null Mean = 50, Standard Deviation = 8.949, and Alpha = 0.05 Basic Logic 6 Steps Means 1 population Errors and Correction Decisions Power ● Power ● Graphically: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power ● The Algebra: Computing Power ● Power of t-test ● Power of t-test ● Power of t-test & Non-central Null Mean 49.75 49.5 49 48 47 N 145 145 145 145 145 Power .095 .164 .379 .849 > .99 t-distribution ● Power of t-test: ● Power of t-test: µ = 49 µ = 48 ● Two-Tailed Test ● Power of t-test: SAS/Power Analysis ● SAS/Power Analysis: Results Things that Effect Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 46 of 54 Things that Effect Power Basic Logic 6 Steps Alpha Level: as α decreases, power decreases and β increases Means 1 population Errors and Correction Decisions Power Things that Effect Power ● Things that Effect Power ● Things that Effect Power— µa ● Things that Effect Power— n ● Things that Effect Power— error ● Things that Effect Power— error ● How to Increase Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 47 of 54 Things that Effect Power— µa Basic Logic 6 Steps Alternative Mean: The greater the distance between µo and µa , the greater the power. Means 1 population Errors and Correction Decisions Power Things that Effect Power ● Things that Effect Power ● Things that Effect Power— µa ● Things that Effect Power— n ● Things that Effect Power— error ● Things that Effect Power— error ● How to Increase Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 48 of 54 Things that Effect Power— n Basic Logic The variance of the sampling distribution: Sample size n 6 Steps Means 1 population Errors and Correction Decisions Power Things that Effect Power ● Things that Effect Power ● Things that Effect Power— µa ● Things that Effect Power— n ● Things that Effect Power— error ● Things that Effect Power— error ● How to Increase Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 49 of 54 Things that Effect Power— error The variance of sampling distribution: Measurement error Basic Logic 6 Steps ■ Means 1 population If observed scores include (random) measurement error, which is uncorrelated with the actual score: Errors and Correction Decisions X = True Score + e Power Things that Effect Power ■ ● Things that Effect Power ● Things that Effect Power— The variance of the observed score X equals σ 2 = var(X) = var(True Score) + var(e) µa ● Things that Effect Power— n ● Things that Effect Power— error ● Things that Effect Power— ■ The variance of the sample distribution of the mean equals error ● How to Increase Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide 2 var(True Score) + var(e) σ 2 = σȲ = n n Slide 50 of 54 Things that Effect Power— error Basic Logic The variance of the sampling distribution: Error 6 Steps Means 1 population Errors and Correction Decisions Power Things that Effect Power ● Things that Effect Power ● Things that Effect Power— µa ● Things that Effect Power— n ● Things that Effect Power— error ● Things that Effect Power— error ● How to Increase Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 51 of 54 How to Increase Power Basic Logic ■ Increase α (make it easier to reject Ho ), but this should not be done. Why? ■ True parameter deviates more from the null value. ■ Decreases the variance of the sampling distribution 6 Steps Means 1 population Errors and Correction Decisions Power Things that Effect Power ● Things that Effect Power ● Things that Effect Power— µa ● Things that Effect Power— n ● Things that Effect Power— error ● Things that Effect Power— ◆ Increase sample size ◆ Better measurement error ● How to Increase Power Choosing Sample Size Hypothesis Testing or How to Decide to Decide Slide 52 of 54 Choosing Sample Size Basic Logic Use concept of power to help determine sample size. You need 6 Steps Means 1 population ■ Need an estimate of σ 2 . ■ Need an estimate size of effect. ■ Desired level of precision. Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size ● Choosing Sample Size ● Sample Size and Power ■ ◆ Power ◆ How many units from the actual mean. Demonstration of SAS/Analyst Hypothesis Testing or How to Decide to Decide Slide 53 of 54 Sample Size and Power From SAS/Analyst Basic Logic 6 Steps Means 1 population Errors and Correction Decisions Power Things that Effect Power Choosing Sample Size ● Choosing Sample Size ● Sample Size and Power Hypothesis Testing or How to Decide to Decide Slide 54 of 54
