Download Introduction: Research Methods in Political Science May 12

Tests of Significance
June 11, 2008
Ivan Katchanovski, Ph.D.
POL 242Y-Y
Tests of Statistical Significance
• Tests of Statistical Significance: Formal and
exact way to test hypotheses
– Derived with help of advanced mathematics
– Is a relationship between independent and
dependent variables statistically significant?
• Widely used in the social sciences
– Often misused
• Focus on application in research methods
2
Null Hypothesis
Research Hypothesis (H1)
Null Hypothesis (H0)
• A statement about
relationship between
independent and
dependent variables that
we want to prove or
disprove.
• A statement of "no
difference” between
independent and
dependent variables
– Example: people with
college education have
higher incomes than people
with high school education
– Example: people with
college education and
people with high school
education have the same
incomes
3
Statistical Significance
• The null hypothesis: Dependent and independent
variables are statistically unrelated
• If a relationship between an independent variable and
a dependent variable is statistically nonsignificant
– Null hypothesis is true
– Research hypothesis is rejected
• If a relationship between an independent variable and
a dependent variable is statistically significant
– Null hypothesis is false
– Research hypothesis is supported
4
Criteria of Statistical Significance
• Statistical significance:
• SPSS p(obtained)<p.=.001, or p=.01, or p=.05
• Conventional levels of statistical significance:
• Less than .001: Probability that a tested relationship
occurred by chance is less than .001, or 1 in 1000, or .1%
• Less than .01: Probability that a tested relationship
occurred by chance is less than .01, or 1 in 100, or 1%
• Less than .05: Probability that a tested relationship
occurred by chance is less than .05, or 1 in 20, or 5%
• Less than .10 (can be used if N is small)
5
Chi Square Test of Significance (  )
2
• Can be used with variables at any level of
measurement
– Most appropriate for nominal and ordinal variables
– Used in cross-tabulation analysis
•
Pearson’s Chi square distribution
–
–
•
Karl Pearson
Eugenics
Limitations
–
Problematic if expected frequencies in cells are small (5 or
less)
6
Chi Square Distribution
7
Steps of Hypothesis Testing
using Chi Square
• Step 0. Research hypothesis
– Example: political party support in the US differs by gender
• Step 1. Assumptions: independent random sampling;
variables are at nominal level of measurement
• Step 2. Null Hypothesis: The dependent and the
independent variables are not related
– Example: political party support is not related to gender
• Step 3. Selecting sampling distribution: SPSS does this
automatically
– Example: Chi-Square
8
Steps of Hypothesis Testing
using Chi Square (Cont.)
• Step 4. Computing the test statistic using Chi-square
formulas or SPSS command (Crosstabs)
• Step 5. Making a decision whether to reject or
accept the null hypothesis.
– If test statistic falls in the critical region:
– SPSS p(obtained)<p=.05
• Reject the null hypothesis and accept research hypothesis
– Statistically insignificant if test statistic (Chi-Square) does
not fall in the critical region:
– SPSS p(obtained)>p=.05
• Accept the null hypothesis and reject research hypothesis
9
Example: Political Party Support by
Gender
• Bivariate (two variables) table of frequency distribution
• The dependent variable (political party support) is in
rows
• The independent variable (gender) is
Political party
in columns
Male, % Female, %
Republican
50
37
Democrat
Total, %
50
100
63
100
N
503
551
Source: 1996 Lipset/Meltz Survey
10
Example: Chi Square Test
• SPSS Chi square test:
– Pearson Chi Square value= 16.219
– P = 0.000
• Pearson Chi Square value (16.219) falls in the critical region
of Chi Square distribution (Determined manually)
• SPSS automatic determination of statistical significance
– SPSS p(obtained)=0.000<p=.05: Statistically significant
– Select the lowest level of statistical significance
• SPSS p(obtained)=0.000<p=.001
• Reject the null hypothesis
• Accept the research hypothesis:
– Political party support in the US differs by gender.
– The difference is statistically significant at the .001 level
11
Limitations of Tests of Statistical
Significance
• Type I error (alpha) - rejecting a true null hypothesis
• Type II (beta) - failing to reject a false null
hypothesis
• Equating statistical significance with real-life
significance
– Computers made statistical tests easy and fast
– Almost any relationship can become statistically significant
in surveys with very large number of respondents
– Statistical significance does not always means real-life
significance
12