Download A summary on tests of significance.

The issue
A theory (a “box theory”) is formulated:
the distribution(s) of the
random variable(s) is (are)
supposed to be known
Refute!
Yes
Improve!
A doubt...:
Was the exp.
well­designed?
KO
Be
humble!
OK
Test of significance
No
Predictions
Experiment
Observations
Comparison
via...
Proving and disproving a theory
… Every genuine test of a theory is an attempt to falsify it, or to refute it. Testability is falsifiability; [...] Confirming evidence should not count except when it is the result of a genuine test of the theory; and this means that it can be presented as a serious but unsuccessful attempt to falsify the theory. [...] One can sum up all this by saying that the criterion of the scientific status of a theory is its falsifiability, or refutability, or testability.
Karl Popper, “Science: Conjectures and Refutations”, 1953
Tests of significance: an example
A measurement, given by the sample mean x, is compared with the expected value, .
If there is a difference between experiment and theory, i.e. between x and ...
“... is the observed difference
due to chance?”
Key idea:
if the observed value x is too many standard errors from its expected value , then it's hard to explain the result by chance.
0) Box theory and Experiment
The box theory is the theory by means of which predictions (expected values) of random variables are made.
The experiment is a set of measurements that produce values of the random variables.
1) Null and Alternative Hypothesis
Null hypothesis H0: any observed difference is due to chance.
Alternative hypothesis HA (H1): the observed difference is real.
2) Test statistic
A statistic that measures the difference, suitably normalized, between the observed value, determined through the experiment, and the expected value, determined through the box theory.
A test statistic is necessary in most cases. In the case of sequences of Bernoulli trials, true P­values (see next slide) are evaluated by directly using the binomial distribution.
Ex. gr.: z, the difference, normalized on the standard error, between the observed value x and the expected value .
3) P­value
Observed significance level = P­value =
chance of getting a value of the test statistic
as extreme as or more extreme than the observed one.
For a supporter of H0, the P­value is the chance that things could go that way or worse!
For an opponent of H0, the P­value is the chance that things could go that way or better!
!
The P–value is not (ex.gr.) the probability of the null hypothesis being right!

3) P­value
Convention related to the evidence against H0 and the box theory (levels of significance):
P < 5 % ⇒ significant evidence against H0
●
P < 1 % ⇒ highly significant evidence against H0
●