Download Significance

S519: Evaluation of
Information Systems
Social Statistics
Inferential Statistics
Chapter 8: Significantly
significant
Last week
This week




What is significance and why it is important
Type I and Type II errors
How inferential statistics works
How to select the proper statistical test for
your research
The concept of significance

Significance

Any difference between the attitudes of the two
groups is due to some systematic influence and
not due to chance.
Significance

Example

Hypothesis: There is a significant difference in
attitude toward maternal employment between
adolescents whose mothers work and
adolescents whose mothers do not work, as
measured by a test of emotional state.

There are many other reasons to affect this
hypothesis, for example?
Significant level

Significant level is the risk associated with not
being 100% confident that the null is true
(there is no difference between data or
variables)

If the significant finding occurred at the 0.05 level
(p<0.05), this means that there is 1 chance in 20
(5%) that there is no difference in data (null is
true): any differences found were not due to the
hypothesized reason, but to some other unknown
reasons, or by chance.
Significant level

p-value




The probability for the null hypothesis to be true
The probability for no difference in data or
variables
p>0.05 (non significant): more than 5% chance
(5% to 99%) that the null is true (no difference in
data) – accept null
p<0.05 (statistically significant): less than 5%
chance that the null is true (no difference in data)
– reject null
Null and research hypothesis

Research hypothesis


Null hypothesis


There is a difference in the academic achievement of
children who participated in a preschool program and
children who did not participate.
The two groups are equal to each other on some
measure of achievement.
As a good researcher, your job to show that any difference that
exists between these two groups is due only to the effects of the
preschool experience.
Statistical significance


A result is called statistically significant  it is
unlikely to cause by chance
A statistically significant difference  there is
statistical evidence that there is a difference

It does not mean that the difference is necessarily
large, important, or significant in common
understanding
Null hypothesis




Either true or false
But null cannot be tested directly (as it is
applied to the population)
The researchers do not know the real true
nature of the null hypothesis, and it is hard to
know and to test
That is why we need inferential statistics
Table 8.1 (s-p179)
Accept the null hypothesis Reject the null hypothesis
The null
hypothesis is
really true
, you accepted a null when
it is true
[there is really no difference
between the groups]
Type I error: reject a null
hypothesis when it is true
(represented by the Greek
letter alpha, α)
[there is really no difference
between the groups]
The null
hypothesis is
really false
Type II error: accepted a
false null hypothesis
(represented by Greek letter
beta, β)
[there really are differences
between the two groups]
, you rejected the null
hypothesis which is false.
[there really are differences
between the two groups]
Type I error

Defines the risk that you are willing to take in
any test of the null hypothesis



Conventional: 0.01 ~ 0.05
Example: if the level of significance is 0.05  there is
a 5% chance you will make the Type I error: to reject it
when the null is true.
It is not proper to say “on 100 tests of the null
hypothesis, I will make an error on only 5”

As it is normally associated with one test
Type I error

p<.05 or p<.01 (reject the null)


The risk to make the Type I error (reject the real true
null) is less than 5% or 1% chance
p>.05 or p=n.s. (nonsignificant) (accept the null)

The probability to make the Type I error (reject the
real true null) exceeds .05
Citation counts
Degree
Closeness
Betweenness
0.4223
0.5614
0.7282
p=0.0060
p=0.0100
p=0.0003
Type II error


The risk of accepting the real false null
hypothesis
It is sensitive to the number of subjects in a
sample


The size of sample increases  Type II error
decreases
If the sample is more closer to the population, the
likelihood that you will accept a false null
hypothesis decreases
Significance

Statistical significance means




Example: Group 1 with training to read using
computer, Group 2 with training to read using
classroom teaching
A reading test: Group 1=75.6, Group 2=75.7
When using t test: result is statistically significant
at the .01 level
How to interpret: computers do better than
classroom teaching
Inferential statistics


Descriptive statistics: describe the characters
of a sample
Inferential statistics: infer something about
the population based on the sample’s
character
How inference works

Mother-work group and mother-not-work
group




Select representative samples of two groups
Conduct a test for each member in these two
groups, calculate the mean scores
Select a proper statistical test
Draw a conclusion (to a population):


If statistically significant: the difference is due to moms
If not significant: the difference is not due to moms
How to select which test


Flow chart (s-p186)
http://rimarcik.com/en/navigator/
A template for significant test







1. a statement of the null hypothesis
2. setting a level of risk associated with the null hypothesis (level of
significance or Type I error, p)
3. select a proper statistical test (see Fig 8.1)
4. set up the sample and experiment, and compute the test statistic
value
5. determine the value needed for rejection of the null hypothesis
using proper tables – critical value (see appendix)
6. compare the computed value and the obtained value
7. if computed value > critical value: reject the null;
if computed value < critical value: accept the null
Exercise

Are the following statements true, and why:



A Type I error of 0.05 means that in 5 tests out of
100 tests of the research hypothesis, I will reject a
true null hypothesis
It is possible to set the Type I error to 0
The smaller the Type I error rate, the better the
results