Download Home - Moodle

Home
Beginning Research | Action Research | Case Study | Interviews |
Observation Techniques | Education Research in the Postmodern
Evaluation Research in Education | Narrative| Presentations |
Qualitative Research | Quantitative Methods | Questionnaires |
Writing up Research
Quantitative Methods in
Education Research
Prepared by Professor John Berry
© J Berry, Centre for Teaching Mathematics, University of
Plymouth, 2005
(links updated August 2006)
CONTENTS


A. INTRODUCTION
B.
QUANTITATIVE AND QUALITATIVE RESEARCH
C.
INGREDIENTS OF QUANTITATIVE RESEARCH
D. STATISTICS
E.
STATISTICAL CONCEPTS & QUANTITATIVE
PROCEDURES
F. TASKS
G. FURTHER READING

H.





WEBSITES
A.
INTRODUCTION
This component is unable to do more than help you to begin
thinking about quantitative methods in educational research. Its
aim is to give you an insight into the issues should you choose
quantitative methods as part of your research methodology.
We will briefly address the following questions:


What are quantitative methods?
What are the ingredients of quantitative methods?
How do you go about research design?
Education research has moved away largely from the numbers
approach in recent years, and the emphasis has been on qualitative
methods. However, the use of numbers can be a very useful tool,
either as part of a larger project that employs many different
methods or as a basis for a complete piece of work. With the use of
sophisticated software packages such as SPSS it is relatively easy to
deal with the computation side of things and it is possible to come
up with numerous tables and charts almost instantly once your data
is installed. However, it is very important that the underlying
principles of statistical analysis are understood if sense is to be
made of the results spewed out by such a package in terms of your
research.
This component consists of two sections; we begin with an overview
of quantitative methods and finish with a brief introduction to some
of the basic statistical concepts to be looking for when you read
research papers that use quantitative methods of research.
Back to CONTENTS list
B.
QUANTITATIVE AND QUALITATIVE RESEARCH
In simple terms we can think of two approaches to investigations in
educational research: qualitative and quantitative. In the former we
use words to describe the outcomes and in the latter we use
numbers.
Quantitative research methods were originally developed in the
natural sciences to study natural phenomena. However examples of
quantitative methods now well accepted in the social sciences and
education include:

surveys;



laboratory experiments;
formal methods such as econometrics:
numerical methods such as mathematical modelling.
Qualitative research methods were developed in the social sciences
to enable researchers to study social and cultural phenomenon.
Examples of qualitative methods include:



action research aims to contribute both to the practical
concerns of people in an immediate problematic situation and
to the goals of social science by joint collaboration within a
mutually acceptable ethical framework;
case study research - a case study is an empirical enquiry
that investigates a contemporary phenomenon within its reallife context;
ethnography- the ethnographer immerses her/himself in the
life of people s/he studies and seeks to place the phenomena
studied in its social and cultural context.
Other components of this module cover various qualitative research
methods.
Structure of Research Papers
When setting out on educational research you will be (have been)
encouraged by your supervisor to read appropriate publications and
this is a good way of identifying the methods of research that seem
most used in your research area. A typical structure for a research
paper is summarised in the table below:
literature survey
other people’s work
methodology
qualitative or quantitative
results
your work
discussion/conclusions
your discussion and reference to others
Table 1 Structure of typical research paper
Activity 1
Scan read the following three papers:
Mark Cosgrove: A study in science-in-the-making as students
generate an analogy for electricity. International Journal of Science
Education. 17 No 3, pp 295-310, 1995
Susan Picker: Using Discrete Mathematics to give Remedial
Students a Second Chance. DIMACS, 36, pp 35-41, 1997
John Berry and Pasi Sahlberg: Investigating Pupils' Ideas of
Learning. Learning and Instruction. 6 No 1, pp 19-36, 1996
Identify the research method being used in each paper.
Answer the question 'Why use numbers in education
research?' with reference to these examples.
Back to CONTENTS list
C.
INGREDIENTS OF QUANTITATIVE RESEARCH
As part of your research you will be looking at certain
characteristics (variables) and endeavouring to show something
interesting about how they are distributed within a certain
population. The nature of your research will determine the variables
in which you are interested. A variable needs to be measured for
the purpose of quantitative analysis.
We may collect data concerning many variables, perhaps through a
questionnaire, or choose to measure just two or several variables
by observation or testing. The variables we are interested in may be
dependent or independent. There will be other features present in
the problem that may be constant or confounding.
Using the data that you have collected then you can:


Describe variables in terms of distribution: frequency, central
tendency and measures and form of dispersion. Descriptive
statistics include averages, frequencies, cumulative
distributions, percentages, variance and standard deviations,
associations and correlations. Variables can be displayed
graphically by tables, bar or pie charts for instance.
This may be all the statistics you need and you can make
deductions from your descriptions. In fact univariate (one
variable) analysis can only be descriptive.
But descriptive statistics can be used to describe a significant
relationship between two variables (bivariate data) or more
variables (multivariate).
Infer significant generalisable relationships between
variables. The tests employed are designed to find out
whether or not your data is due to chance or because
something interesting is going on.
See the section on Variablesin The Research Methods
Knowledge Base.
Often it is not possible to undertake a true experiment as part of
your research and a common research approach in educational
research is called quasi-experimental design represented in the
following diagram:
Experimental
Group
O1
Control Group
O3
X
O2
O4
In this figure O1 and O3 represent initial testing of the two groups;
X represents some intervention or experimentation strategy with
one of the groups and O2 and O4 represent final testing of the two
groups. We would use the test results to investigate whether the
experimental teaching approach has led to an improvement in the
feature being tested.
Back to CONTENTS list
D.
STATISTICS
Perhaps the best way to begin to appreciate the kind of statistics
that you might employ in your own research is to have a look at
what others have done.
Read the paper by John Berry and Pasi Sahlberg: Investigating
Pupils' Ideas of Learning. Learning and Instruction. 6 No 1, pp 1936, 1996 and attempt to identify the statistics that are used.

The section ‘Findings’ on pages 28 – 32 contains some of the
important measures that we use in quantitative research
methods. The mean and standard deviation tell us about the
average and spread of the data. The bar graphs on page 31
allow a visual comparison of the means between the two
groups of samples.

The symbol ‘p’ represents the probability of a significant
difference between the two groups. This is probably the most
difficult concept to grasp because in some senses it is counter
intuitive. The probability of an event happening lies between 0
and 1. A large probability (i.e. p close to 1) implies a high
likelihood of the event happening. For example if you are told
that there is a 95% chance of winning a game (p = 0.95) then
put your money on winning! On the other hand if there is a
5% chance of you winning (p = 0.05) it’s probably best not to
bet on yourself!

So if p is small (close to 0) the event is less likely to happen
than if p is large (close to 1). Let’s see what this means in the
context of educational research. As an example look at page
31 Berry and Sahlberg. If we compare the means of the
scores of the UK and Finland pupils for the statement "I learn
better by doing work by myself than by watching the
teacher", then there is a probability of less than 0.001 (i.e.
0.1%) that the difference in the means will occur by chance.
In ordinary language the probability of it happening by chance
is so small that we say it is a significant result. Because it is
unlikely to occur, the reason that it does is significant.

In our analysis we look for small probabilities usually less than
5% or p = 0.05. Then we say that the result is significant at
the 5% level of significance.

There is also evidence from Figure 3 that there is some
difference between the UK and Finland pupils. For Q2 the UK
rating is positive and the Finland rating is negative.

Very often a comparison of the means as in Figure 3 is a clue.
For example look at Q7 in Figure 3. For the UK pupils the
rating is positive whereas for the Finland pupils it is negative.
The ‘p – value’ for this feature is less than 0.001 (0.1%) and
so we conclude that there is a significant difference between
the two groups of pupils for the statement "I like most
teachers in my school".

One of the first steps in the design of a piece of quantitative
research is setting up what is called a hypothesis. For
example, we might propose the hypothesis that there is no
difference between the UK pupils and Finnish pupils views of
their teachers (item 7 in Berry and Sahlberg). This is called
the null hypothesis. We then need to use the pupil responses
to try to disprove this hypothesis.

To be able to compare quantities we need to define a statistic
whose distribution is known. In the paper by Berry and
Sahlberg the t-statistic is used as a measure of the difference
between the means of the two groups of pupils. Having
calculated the value of the t-statistic for the feature under
investigation we then look up in tables the probability of the
feature occurring. (At this stage don’t worry about how it is
calculated!) You can see on page 31 the t-statistic and its
associated p-value.
At the heart of quantitative research methods is some very sound
statistical theory. If you are planning to carry out a research
investigation using quantitative research methods you do not need
a thorough grounding in this theory but you will need an
understanding of the statistical methods. We use statistical software
packages to do the arithmetical calculations so the important skill is
not doing the mathematics but is interpreting the results.
In what follows we have gathered together some of the essential
statistical ideas needed for quantitative research. It is a summary
with some examples to provide a flavour of the ideas.
Back to CONTENTS list
E.
STATISTICAL CONCEPTS & QUANTITATIVE
PROCEDURES
NB What follows is merely an introductory overview of some of the
relevant concepts and procedures. To find out more go first to a
general textbook such as Denscombe (1998), Chapter 10, and then,
for a much fuller account, try Peers (1996).
1.
Variables

numerical measurements: person’s age or weight; size of a
family

non-numerical measurements: position on a scale indicating a
level of agreement e.g. Likert rating scale

continuous data: measurement that can, in principle, take any
value within a certain range e.g. time, age and weight

categorical data: (or discrete data): measurements that can
take only known discrete values e.g. the number of rooms in
a house, the number of children in a family
o nominal data: numerical values are assigned to
categories as codes e.g. in coding a questionnaire for
computer analysis, the response ‘male’ might be coded
as ‘1’; and ‘female’ as ‘2’. No mathematical analysis is
usually possible and no ordering is implied.
o ordinal data: numerical values are assigned in
accordance with a qualitative scale e.g. in coding a
questionnaire for computer analysis, the responses ‘very
good’, ‘good’, ‘poor’ and ‘very poor’ are coded ‘4’, ‘3’, ‘2’
and ‘1’ respectively.
See also the section on Variables in The Research Methods
Knowledge Base.
2.
Basic Measures

mean: is a measure of the central location or average of a set
of numbers, e.g. the mean of 2 7 2 1 8 2 6 9 10 5 1 4 is 4.75

standard deviation: is the square root of the variance!!

variance: is a measure of dispersion (or spread) of a set of
data calculated in the following way:
3.

median: is the centre or middle number of a data set, e.g. the
median of 2 7 2 1 8 2 6 9 10 5 1 4 is 4.5

quartiles: divide a distribution of values into four equal parts.
The three corresponding values of the variable are denoted by
Q1, Q2 (equal to the median) and Q3

range: is a measure of dispersion equal to the difference
between the largest and smallest value.
Frequency Distribution
Example A
A new hybrid apple is developed with the aim of producing larger
apples than a particular previous hybrid. In a sample of 1000
apples, the distribution of weights was as follows:
Weight (g)
0-50
50100
100150
150200
200250
250300
300350
350-400
frequency
20
42
106
227
205
241
106
53
1. Apples can only be sold to a particular supermarket with a
weight greater than 150g. What proportion of the new hybrid
apples would be rejected by this supermarket.
2. How many grams above this weight of 150 g is the mean
weight of apples?
3. What is this difference in weights in units of the standard
deviation of apple weights?
Suppose that we graph the data using columns to show the amount
in each group. We get a frequency distribution.
From the data there are 168 apples whose weight is less than 150 g
and 832 apples whose weight is greater than 150 g. There are 1000
apples altogether.
We can deduce that the proportion
sold to the supermarket.
= 16.8 % cannot be
The mean weight of the apples is 223.35 g and the standard
deviation is 78.9 g.
The difference in weight between 150 g and the mean is 73.35 g
and this is
4.
of a standard deviation.
Measures of Location and Dispersion
A distribution is symmetrical if the difference between the mean and
the median is zero. An appropriate pictorial representation of the
data, (histogram, stem and leaf diagram etc.) would produce a
mirror image about the centre:
A distribution is positively skewed (or skewed to the right) if the
mean - median is greater than zero. Such data when represented
by a histogram would have a right tail that is longer than the left
tail:
A distribution is negatively skewed (or skewed to the left) if the
mean - median is less than zero. Such data when represented by a
histogram would have a left tail that is longer than the right tail:
If data are skewed then the best measure of location is the median
and the best measure of dispersion is the interquartile range.
If data are symmetrical then the best measure of location is the
mean and the best measure of dispersion is the standard deviation
or variance.
5.
Probability
This is an important concept in statistics and is an important part of
our story.
It is defined in the following way: if an experiment has n equally
likely outcomes and q of them are the event E, then the probability
of the event E, P(E), occurring is
P(E) =
Some simple examples:
the probability of getting a head from the toss of a fair coin is
the probability of getting a six from the throw of a fair die is
the probability of getting the ace of spades from cutting of a pack of
cards is
The smaller the probability the more unlikely the event is to
happen. This is an important concept in quantitative methods in
education research as we shall see.
There is an important link between probability and the frequency
distribution. Consider again the hybrid apple example above. We
saw that the proportion of apples weighing less than 150 grams was
0.168. If we pick up one of the apples ‘at random’ then it could
weigh less than 150 g and be rejected or it could weigh more than
150 g and be accepted by the supermarket.
The probability of picking such an apple is 0.168.
Exercise
To illustrate this idea further complete the following:



the probability of picking an apple in the weight range 200250 g is
the probability of picking an apple in the weight range 350400 g is
the probability of picking an apple with a weight greater than
300 g is
The important idea here is that the probability is associated the
amount of data under the distribution graph.
6.
Testing an hypothesis
There are two basic concepts to grasp before starting out on testing
an hypothesis.

Firstly, the tests are designed to disprove hypotheses. We
never set out to prove anything; our aim is to show that an
idea is untenable as it leads to an unsatisfactorily small
probability.

Secondly, the hypothesis that we are trying to disprove is
always chosen to be the one in which there is no change. For
example there is no difference between the two population
means.
This is referred to as the null hypothesis and is labelled H0. The
conclusions of a hypothesis test lead either to acceptance of the null
hypothesis or its rejection in favour of the alternative hypothesis H1.
Hypothesis testing: a hypothesis test or significance test is a rule
that decides on the acceptance or rejection of the null hypothesis
based on the results of a random sample of the population under
consideration.
step 1: Formulate the practical problem in terms of hypotheses. The
null hypothesis needs to be very simple and represents the status
quo, i.e. there is no difference between the processes being tested.
step 2. Calculate a statistic that is a function purely of the data. All
good statistics should have two properties: (i) they should tend to
behave differently when H0 is true from when H1 is true; and (ii) its
probability distribution should be calculable under the assumption
that H0 is true.
step 3: Choose a critical region. We must be able to decide on the
kind of values of the test statistic, which will most strongly point to
H1 being true rather than H0. The value of the test statistic in this
critical region will lead us to reject H0 in favour of H1; otherwise we
are not able to reject H0 in favour of H1. We should never conclude
by accepting H0.
step 4: Decide the size of the critical region. i.e. a 1% probability of
H0 being rejected etc.
For more on this see the section on Hypotheses in The Research
Methods Knowledge Base.
Example B
In an educational research programme two groups of students are
taught a topic in different ways. An experimental group uses a
spreadsheet to explore the topic and a control group uses a more
traditional pen and paper activity. Each group contains 20 students.
At the end of the topic the teacher tests the two groups on their
understanding of the topic and obtains the following data:
Experimental
5
11
25
33
35
40
45
46
52
55
56
56
57
59
69
74
75
89
92
97
Control
33
39
44
45
45
46
47
48
49
49
53
54
54
55
58
60
61
63
65
69
Extra data:
mean
standard deviation
Experimental
53.6
25.0
Control
51.8
9.1
Experimental Group
Control Group
How would you interpret these findings?
Some analysis
The researcher might be tempted to conclude that the experiment
has had little or no effect on the performance of the experimental
group as judged by the means. However the large difference in
standard deviations might suggest that the experimental group is
much more variable in performance than the control group.
The researcher might also be tempted to deduce that the
experiment has turned some of the pupils off the task. Look at the
three low scores!
Suppose that we investigate the difference in the means: Let
H0: there is no difference between the means of the two groups: m
1 = m 2
H1: the score of the experimental group is greater than the score of
the control group:
m
1
>m
2
We use a two-sample t-test to get
The p value is 0.39 (39%) so we deduce that there is not enough
evidence to reject the null hypothesis. The researcher could not
deduce that there was an improvement in student performance.
7.
Statistical tests
t tests
In hypothesis testing, the t test is used to test for differences
between means when small samples are involved. (n £ 30 say). For
larger samples use the z test.
The t test can test
i) if a sample has been drawn from a Normal population with known
mean and variance. (Single sample)
ii) if two unknown population means are identical given two
independent random samples. (Two unpaired samples)
iii) if two paired random samples come from the same Normal
population. (Two paired samples (paired differences))
Any hypothesis test can be one tailed or two tailed depending on
the alternative hypothesis, H1.
Consider the null hypothesis, H0: m =3
A one tailed test is one where H1 would be of the form m > 3.
A two-tailed test is one where H1 would be of the form m ¹ 3.
Click here for more information on t-tests.
Single sample test
Let X1, X2, ¼ , Xn be a random sample with mean and variance s2.
To test if this sample comes from a Normal population with known
mean m and unknown variance s2, the test statistic
is used to test the null hypothesis H0: the population mean equals
m. If the test statistic lies in the critical region whose critical values
are found from the distribution of Tn, a, H0 is rejected in favour of
the alternative hypothesis H1. n are the degrees of freedom and for
a single sample test n = n-1, and a is the significance level of the
test.
Two unpaired samples
Let X1, X2, ¼ , Xm be a random sample with mean and variance sx2
drawn from a Normal population with unknown mean mx and
unknown variance sx2.
Let Y1, Y2, ¼ , Yn be a random sample with mean and variance sy2
drawn from a Normal population with unknown mean my and
unknown variance sy2.
To test the null hypothesis that the two unknown population means
are the same we use the test statistic
where
standard deviation.
, the estimate of the common population
The test statistic T is distributed Tn, where n =(m-1)+(n-1) for two
unpaired samples. If the test statistic lies in the critical region
whose critical values are found from the distribution of Tn, a, H0 is
rejected in favour of the alternative hypothesis H1.
Example C
It is claimed that the concentration period of students doing a
particular task is normally distributed with mean 44mins. A sample
of 21 students were taken, and their concentration period
measured. The mean time of the sample was found to be 42mins
and the sample variance was calculated to be 36min. Is there any
evidence at the 5% level of significance against the claim that the
population mean is 44min?
Solution
Here m = 44,
= 42, s = Ö 36 = 6 and n = 21.
This is a two-tailed test since we are looking for any difference.
H0: m = 44
H1: m ¹ 44
Since p = 0.1423 = 14.23% there is insufficient evidence to reject
the null hypothesis. We therefore conclude that the population
mean concentration period is 44 minutes.
Example D
A researcher investigating the effects of pollution on two rivers
takes an independent random sample of fish of a certain species
from each river, measures their mass in ounces and obtains the
following results.
River 1
20
10
17
7
10
18
River 2
14
6
10
8
9
7
7
6
8
Test at the 5% level of significance if there is any evidence of a
difference in the mean weight between the two rivers.
Solution
Assume that each sample is taken from a normal population.
Here m = 6,
n = 9,
= 13.667, sx2 = 23.556
= 8.333, sy2 = 5.556.
Let m 1 be the population mean of river 1, and m
population mean of river 2.
H0: m
1
=m
H1: m
1
¹m
2
be the
2
2
Since p = 0.043 = 4.3% < 5% we deduce that there is sufficient
evidence to reject the null hypothesis at the 5% level of
significance. We therefore conclude that the mean weight of fish in
River 1 is not equal to the mean weight of fish in River 2.
Back to CONTENTS list
F.
TASKS
(NB: only for those University of Plymouth students
undertaking the ‘Research in Education’ module as part of
the preparation for the submission of a MA dissertation
proposal)
Tasks, once completed, should be sent to
[email protected], making clear:



which component it is from;
which task it is (A, B or C);
the name of your dissertation supervisor.
It will then be passed on to the component leader (and
copied to your supervisor). The component leader will get
back to you with comments and advice which we hope will
be educative and which will help you in preparing your
dissertation proposal once you are ready. (Remember that
these tasks are formative and that it is the proposal which
forms the summative assessment for the MERS501 (resined)
module.) This email address is checked daily so please use it
for all correspondence about RESINED other than that
directed to particular individuals for specific reasons.
TASK A (NATURE OF EDUCATION RESEARCH)

Does research have to be quantitative to be scientific?
See Cohen et al (2000), Chapter 1, and the section on
Positivism and Post-Positivism in Trochim (2000),
examine the terms used in this question and discuss
with reference to examples.
TASK B (DATA COLLECTION)

A research group suspects that girls and boys adopt
different styles of working when using ICT in
mathematics lessons. Outline a quantitative research
method to investigate the effect of gender on the use of
calculators on children’s understanding of
mathematics. What are your variables? Formulate an
experimental hypothesis for the research.What features
might cause problems in your research?
TASK C (DATA ANALYSIS)

Twelve students fail an examination. After a period of
revision and tutorial support they resit by taking a new
examination. The marks for the original examination
and the resit are shown in the table below.
Student number

1
2
3
4
5
6
7
8
9
10
11
12
Examination score
30
31
20
17
25
32
35
29
30
27
32
30
Resit score
42
38
30
21
40
45
31
32
38
50
34
40
What would you do with the data now?
Back to CONTENTS list
G.
FURTHER READING
Blaxter, I., Hughes C. and Tight M. (1996) How to Research.
Buckingham, Open University Press.
Bryman, A. and Cramer D. (1999) Quantitative Data Analysis
with SPSS 8 Release for Windows: a guide for social
scientists. London, Routledge.
Cohen, L ; Manion, L & Morrison, K (2000) Research Methods
in Education (5th edition). London, RoutledgeFalmer.
Denscombe, M. (1998) The Good Research Guide.
Buckingham, Open University Press.
Greenfield, Tony (ed) (1996) Research Methods – Guidance
for Postgraduates. London, Arnold.
Kanji, Gopal (1993) 100 Statistical Tests. London, SAGE
Publications.
Peers, Ian (1996) Statistical Analysis for Education &
Psychology Researchers. London, Falmer.
Plewis, Ian (1997) Statistics in Education. London, Arnold.
Robson, C. (1990) Experiment, Design and Statistics in
Psychology. Middlesex, Penguin Books.
Rose, D. and Sullivan, O. (1993) Introducing Data Analysis
for Social Scientists. Buckingham, Open University Press.
MORE ADVANCED READING
http://www.cf.ac.uk/socsi/capacity/References.html
H.
WEBSITES
Trochim, William M. The Research Methods Knowledge Base,
2nd Edition. Internet WWW page, at URL:
http://www.socialresearchmethods.net/kb/ (version
current as of
). (This is the excellent site referred [and
linked] to several times in the sections presented above.)
SURFSTAT australia
http://www.anu.edu.au/nceph/surfstat/surfstathome/surfstat.html
Electronic Statistics Textbook
http://www.statsoft.com/textbook/stathome.html
Back to CONTENTS list
Home
Beginning Research | Action Research | Case Study | Interviews | Observation
Techniques | Education Research in the Postmodern
Evaluation Research in Education | Narrative| Presentations | Qualitative Research |
Quantitative Methods | Questionnaires | Writing up Research
© J Berry, Centre for Teaching Mathematics, University of
Plymouth, 2005