Download Aspect 1 - IDt - Mälardalens högskola

Generalisability
in Science Education Research
FontD Seminar 2004 in Mälardalens Högskola
Hans Niedderer
Mälardalens Högskola
Institutionen för Matematik och Fysik
Part 1
Theoretical aspects about generalisability
Aspects of generalisability

Aspect 1: Statistics and classical quantitative design

Aspect 2: Generalisability as essential feature of theory

Aspect 3: Generalisability and paradigm (Kuhn)
Aspect 1:
Statistics and classical quantitative design

Petri 1996: ONE Student

Hake 1998: 5000 students
Aspect 1:
Generalisability guaranteed by design?


Weaknesses of case studies (Wirth & Leutner 2004)

… No discovery of a universal, generalizable

… No discovery of a cause-effect relationship

… Not appropriate for testing hypotheses
truth
I could think of cases: with/without computer,
with/without electronium
General empirical approach (Wirth & Leutner 2004)
Question
Problem
Deduction of
hypothesis
Scientific
observation
Theory
Generalization
Induction
Modification
Internal
Validity
General empirical approach (Wirth & Leutner 2004)
Question
Problem
Deduction of
hypothesis
Scientific
observation
Theory
Internal
Validity
Generalization
Induction
Modification
=> "External validity"
External validity

Extent to which conclusions drawn from a scientific
observation can be generalized to other persons,
situations or points of time.

Control

environmental conditions,

real life setting,

representative sample,

replication (in different contexts),

theory use
Possibilities in science education
Where we have strong hypotheses from previous
qualitative research ...


Investigate the effect of using an electronium model of the
atom compared to a course using a propability model of the
atom

Doing similar courses in optics with and without using
different kinds of computer software
Example: In Roger's dissertation project at KAU, we
try to develop a classical design, using different
treatments with different use of ICT in optics
Aspect 1:
Statistics and classical quantitative design

Generalisability ...
seen as a problem of design and statistical evidence.
Aspect 2:
Generalisability as essential feature of theory
Reif, F., Larkin, J. H. (1991)
Cognition in Scientific and Everyday Domains:
Comparison and Learning Implications,
JRST Vol. 28, NO. 9
Dom a i n goa l s
Ev e r y da y dom a i n
Sc i e nt i f i c dom a i n
Mai n goal s
Cent r al goal
Leadi ng a good l i f e
Subgoal
Adequat e pr edi ct i on and
expl anat i on
Requi r ement s
Adequat e gener al i t y,
par si mony, pr eci si on,
consi st ency
Maxi mal gener al i t y,
par si mony, pr eci si on,
consi st ency
Few i nf er ences
Many i nf er ences
Wor k i ng Goal s
Under st andi ng
Opt i mal pr edi ct i on
and expl anat i on
Aspect 2:
Generalisability as essential feature of theory
Reif, F., Larkin, J. H. (1991)
Cognition in Scientific and Everyday Domains:
Comparison and Learning Implications,
JRST Vol. 28, NO. 9
Leadi ng a good l i f e
Opt i mal pr edi ct i on
and expl anat i on
Adequat
e
pr
edi
ct
i
on
and
Dom a i n goa l s
Ev e r y da y dom a i n
expl
anat
i
on
Mai n goal s
Cent r al goal
Leadi ng a good l i f e
Sc i e nt i f i c dom a i n
Incl. Science education !
Opt i mal pr edi ct i on
and expl anat i on
Adequat
e
gener
al
i
t
y,
Maxi
mal
gener
al
i
t
y,
Subgoal
Adequat e pr edi ct i on and
expl anat i on
par
si
mony,
pr
eci
si
on,
par
si
mony,
pr
eci
si
on,
Requi r ement s
Adequat e gener al i t y,
Maxi mal gener al i t y,
si mony, pr eci si on,
par si mony, pr eci si on,
consi st ency par
consi
st ency
consi st ency
consi st ency
Wor k i ng Goal s
Little
generalisability in EDL thinking ("cluster concepts")
Under
st andi ng
i nf er ences
Many
i nf er ences
"Generalisability
asFew
definition
of science" (Reif, Larkin,
Schecker,
Example of working for more generalisability

In Margareta's work about ownership we are working
hard on a better theoretical definition of the concept
of ownership, with maximal

Generality

Parsimony

Precision

Consistency

Furthermore we try to define in such a way that it has
predictive power.

No statistics will be needed in this task.
Aspect 2:
Generalisability as essential feature of theory

Generalisability ...
seen as amount and quality of use of theory.
Theory with general concepts

Try to come to general definitions of concepts: use
the same definition for every case, not one definition
in one case and a different definition in an other case
(as we would all do it in everyday life context!)

Similar: work on general claims or hypotheses

Try to build up confidence by telling frequencies - how
often you were able to apply this concept - in
qualitative work - and how often you were unsure
about it. A negative statement - category does clearly
NOT fit - is a positive statement in this sense!
Aspect 3: Generalisability as acceptance in the
scientific community
T. S. Kuhn distinguishes three phases of development
of a scientific theory:

the pre-paradigmatic phase: Many different questions,
many "theories", little generalisability

the paradigmatic phase, high generalisability of
paradigmatic research

the revolutionary phase: generalisability of the new
paradigm takes time to build up.
Aspect 3: Generalisability as acceptance in the
scientific community


In the paradigmatic phase of research, there are
agreed

questions

concepts

repeated and agreed results

meanings
In SER we have at least one such field:
alternative conceptions of learners


Many Swedes contributed to it (B. Andersson, ..., F. Marton)
This gives a body of agreed knowledge which gives
the highest amount of generalisability
(Generalisability by cummulation in the scientific
community)
Number of investigations in Pfund&Duit
[email protected]
1991
1994
1998
Mechanics
281
421
687
Elect ricit y
146
218
379
Heat
68
111
159
Opt ics
60
162
Part icles
69
190
Energy
69
151
Topic
Ast ronomy
Quant um Physics
1998
36
11
R. Duit:
Bibliography
"Students'
Alternative
Frameworks
and Science
Education"
Aug. 2002
35
57
> 4000 entries
alltogether
Part 2
Cases
Cases

Case 1: 6000 students study by Hake (1998)

Case 2: Doctoral study of Bethge (1988)

Case 3: Doctoral study of Petri (1996)
Case 1: Empirical Results Hake 1998

Interactive-engagement vs traditional methods
A six-thousand-student survey of mechanics test data
for introductory physics courses
R. Hake, Am.J.Phys. 1998 (1)
Traditional methods

"Traditional" (T) courses as those reported by
instructors to make little or no use of IE methods,
relying primarily on

passive-student lectures

recipe labs, and

algorithmic-problem exams
“Interactive Engagement” (IE)
Methods as those designed at least in part to promote

conceptual understanding through interactive
engagement of students

in "heads-on" (always) and

"hands-on" (usually) activities

which yield immediate feedback through discussion
with peers and/or instructors
Interactive-engagement vs traditional methods
G  %post  %pre
gain vs pretest - universities
4832 students
G in %
%post  %pre
g
100  %pre
g<1
Generalisability in case 1
Aspect 1: Statistics and design

6000 students is very impressive

Control of variables?
"interactive teaching as reported by the teachers"
and
"better understanding as measured by the FCI"?
More qualitative tasks related to alternative conceptions
about force?
FCI measures only one aspect of competence, namely
qualitative, multiple choice tasks related to many different
alternative conceptions (not only Newton's force)
Generalisability in case 1


Aspect 2:
Generalisability as essential feature of theory

General theoretical claim

Overgeneralised? Other factors not taken into account?
Aspect 3:
Generalisability as acceptance in the scientific
community
Case 2: Students' alternative conceptions in
atomic physics (Bethge 1988)
Case 2: Students' alternative conceptions in
atomic physics (Bethge 1988)
Methods

1) Audio recordings of current physics lessons were our main
data source.

2) A pair-relation questionnaire with associative elements. In this
type of questionnaire students were asked to make statements
using two given concepts, for example:

wave
-
energy level

wave function
-
trajectory

trajectory
-
energy level

position
-
wave function

electron
-
wave

trajectory
-
probability

3) A questionnaire with seven "thinking type" tasks

4) Interviews with nine pairs of students
Conceptions related to orbits (trajectories) in
quantum physics after teaching
(O1) Classical orbits (about 50%)
(O2) Only special orbits allowed
about 25%
(O3) Smeared orbits
The concept of "trajectory" is combined with notions of
"probability" and "wave function" from wave mechanics in
several ways to form a new "intermediate" conception:
the orbits are "smeared", not exactly determined,
"fuzzy"
the probability for a special orbit is given
the probability of parts of the orbit is given
(O4) Trajectories do not exist in quantum physics
(about 25%)
Generalisability in case 2
Aspect 1: Statistics and design


Criteria of Wirth & Leutner

external validity, generalisability to other persons,
situations or points of time, representative sample:
This research was with data from many students
and classes,

real life setting: with data from real teaching
The results seem thus be generalisable from a
statistical view with respect to 17 to 19 age students,
after teaching in atomic physics with more than
Bohr's model
Generalisability in case 2

Aspect 2:
Generalisability as essential feature of theory



General theoretical claim: students also in quantum atomic
physics show a limited number of alternative conceptions.
Some of these are ...
Aspect 3:
Generalisability as acceptance in the scientific community

Replication in different contexts (Aspect 1) and
cummulation (Aspect 3)

… was done to some extent later by dissertations (Lichtfeldt
1992, Mashhadi 1996, Deylitz 1999) and other research
(Harrison et al. 1999, Müller et al. 2002)
So the results are - today ! - partially generalisable
Case 3: Learning pathways in atomic physics
(Petri 1996)


Building on theoretical ideas in the community

Driver 1989, Scott et al. 1991: conceptual pathways

Bremen workshop 1991: need to describe learning pathways
"(1) Need to document learning pathways
for different content areas in physics"
"(2) Need to construct ways of describing cognitive systems
that are useful to researchers in physics education"

Niedderer (1991 to 1996): Prior example electric circuits

Psillos 1999: Conceptual evolution
Giving ONE example in great detail
(analysing huge amount of data from ONE student)
Carls Learning Pathway “Modell of the Atom"
Petri 1996; Petri&Niedderer 1998
Carl's first view
(" Planetary model")
Carl's second view
(" Probability-orbits",
"smeared orbits")
orbit /shell
Carl's third view
("Quantum model")
Carl's fourth view
("Orbital model")
Example of working for more generalisability

In developping a new dissertation project at MdH,
Peter Gustavsson and I are planning to have a new
doctoral student working on a learning pathway for a
whole class in distant education.
An advanced model of the learning process
strength
Example: conceptions of an atom
Planetary view
Smeared orbits view
Quantum particle view
Quantum cloud view
time
Generalisability in case 3

Aspect 1: Statistics and design


Aspect 2:
Generalisability as essential feature of theory


1 student: No generalisability to other students
Showing new, general model of a "learning pathway" as a
theoretical idea (GENERALISABILITY-claim!)
in ONE example in great detail, thus making it work!
Holzkamp: An experiment is a realisation of theory.
Aspect 3:
Generalisability as acceptance in the scientific
community

The theoretical ideas were later used - and cited - by other
researchers
(Psillos 1999 "conceptual evolution"; Taber (2000) "manifold
conceptions in cognitive structure")
Part 3
Conclusions
Final conclusion

Try to come to theoretical definitions of concepts and
claims, thus building up potential generalisability

Generalisability is a decision of the community of
researchers, developping a paradigm (Kuhn), coming
to paradigmatic research

This decision is based on empirical research in
combination with normative decisions about relevant
questions and theoretical approaches

To formulate it the other way round:
I do not believe in generalisability from one study.
Final conclusion (ctd.)


In this view, generalisability means

similar questions are asked

with similar theory

with similar results

by (many) other researchers
This is why literature search and writing a
"state of the art" in a doctoral dissertation and
relating this to own results is so important!
Aspect 3: Generalisability as acceptance in the
scientific community

In one field we are in a revolutionary phase:
the basic understanding of teaching and learning
Old paradigm about teaching and learning
T ransmissive Instruction
T eacher
Information
Student
Paradigm shift about teaching and learning
T ransmissive Instruction
T eacher
Student
Information
Co nst r uc t ivist Inst r uc t io n
e
a
r
n i
n
g
L
St ud e nt
Mind
Te a c he r
E
n
v
i r o n m
e
t
n
Paradigm shift about teaching and learning
T ransmissive Instruction
T eacher
Student
Information
Co nst r uc t ivist Inst r uc t io n
e
a
r
n i
n
g
L
St ud e nt
Mind
Te a c he r
E
n
v
i r o n m
e
t
n
Acceptance in the
community of
practioners:
still high
Acceptance in the
scientific
community: high,
but consequences
are still under
development
Paradigm shift about teaching and learning
T ransmissive Instruction
T eacher
Student
Information
Co nst r uc t ivist Inst r uc t io n
e
a
r
n i
Example of a
generalisable statement,
not yet fully accepted:
n
g
L
St ud e nt
Mind
Te a c he r
E
n
v
i r o n m
Acceptance in the
community of
practioners:
still high
e
t
n
A learner's
constructions are
different from the
teaching input - as a
rule, not as an
accident
Case 4: Mixed methods approach
(Deylitz 1999, Budde 2003)
Pre- and post test results of special items
Results
Learning environment
Cognitive system of student
Problem with quantum interpretation
Uli:
Well, actually I completely put aside the
concept of trajectory in the area of atomic physics.
The function you have is nothing but the
probability of presence of an electron. ... you can't
say it moves on an orbit. To explain - well, the
motion cannot really be explained anymore ... It's
however, not an orbit any more. .. The electron
must move somehow - very strange, it is now here
and then there .. That gets crazy ..
Diagram of probability
distribution shown during
interview
Damned, it could theoretically move in between. Just that it moves in a
strange zigzag, but that would mean again something like an orbit. And
that's crazy again. Well, somehow I can't get that clear.
Elke: If they didn’t move, a probability distribution would make no sense
at all . The electron would stay in one position - that’s all ... As it shows up at
different places it must move! Otherwise, it would be present always at only
one point.
Summary of findings

25% of students use conceptions near to modern
physics

25% use some typical intermediate conceptions such
as "smeared orbits"

50% stick to classical orbits ("Bohr model")
... even after teaching
Part 2
Jürgen Petri (1996)
Research on learning processes
Research questions

How can we describe a learning process from a
cognitive point of view?
Related theory

International workshop on physics learning in 1991

Learning processes studies

Cognitive modelling of physics learning

Conceptual or learning pathways (Driver, Scott;
Niedderer, Petri; Psillos et al.)

Conceptual change (Hewson, Duit, ...)
Data

1 student, 80 lessons on QAP, grade 13 gymnasium

6 interviews at different points in time

Audio tapes from group work of students, partially
transcribed

Written artefacts (reports etc.) from 1 student
Methods: Scheme for interpretive analysis
of qualitative and quantitative data
- Results of previous research

xxx
- Discussion about
the improvement of
science teaching
- Own experiences,
ideas and expectations
Hypotheses
about ....
1
e.g. cognitive elements,
prior conceptions,
intermediate conceptions
2
- Analysis of
content
Data
- transcripts
of teaching
- transcripts
of interviews
- tests or
questionaires
with statistical
analysis
???
1
Process of looking to the data and trying
to find evidence or counter evidence
2
a. Getting new ideas about new
hypotheses by going through the data
b. Reformulating hypotheses
Carls LearningPathway “Modell of the Atom"
Petri&Niedderer 1998
Carl' s firs t conception of the atom
Carl' s s econd conception of the atom
("Planetary model")
("Probability-orbits model",
"smeared orbits")
orbit /shell
Carl' s third conception of the atom
("Quantum model")
Carl' s fourth conception of the atom
("Orbital model")
Carls initial conception about an atom
Planetary model
orbits
C: The electrons,
negatively charged tiny
balls, move around the
nucleus in definite orbits
or circles, even several
electrons in one orbit or
shell.
Carls second conception about an atom
Smeared orbits
C: If I take the nucleus and
think of a field around, a
wave-like probability-field,
then I think of drawing a psifunction as a wave that
spreads out equally in all
directions. And everywhere,
where the probability is
higher, there is an orbit. I
don't know, I can't get rid of
these orbits, though I don't
know, where I got them.
IC smeared orbit in an atom ("cognitive attractor")
The conception "smeared orbits model of the atom"
Propositi ona l The o rbit s are combined wit h a quan tum idea of
representation probabili ty, wave or unce rtainty , thus becoming a
mixture between orbit s and orbit als.
Image
representation
Recons tructed by researcher (Baye r 1985)

Single facets 
Electron clouds are pic tures of orbit s. (C3P)
The wave func tion describes the trajectory of an
electron. (C3P)
Found by different authors, with different teaching approaches:
Bayer 1985, Bethge 1988, Petri 1996
Final state “model of an atom“ (Petri&Niedderer 1998)
Final state of Carl‘s cognitive system “atom"
Layer
Strength
Status
high
low
Probability
model
middle
middle
Electron cloud
model
middle
high
Planetary
model
orbit s
An advanced model of the learning process
strength
Example: conceptions of an atom
Planetary conception
Smeared orbits conception
Quantum particle conception
Quantum cloud conception
time
Part 3
Stefan Deylitz (1999)
Research about evaluation
Erg

Testaufgaben

Einzelne Items
Research questions

Evaluation of a new approach to quantum atomic
physics (QAP)
Related theory/literature: different types of evaluation

Comparative E.

Formative E.

Summative E.
==> How far were our own main goals achieved?
Data

26 students, 3 courses, grade 12 gymnasium

New test constructed according to goals

Pre and post test

Post interviews with all students
Selected questions of the questionnaire
1a Draw a picture of an atom and label it!
1b Describe your model with a few sentences.
1e Can you determine the size of an atom in your model
of an atom?
3b Given are three drawings. Describe commonalties
and differences between these three atomic models
and use the notions "electron orbit", "probability
density", and "charge cloud".
Methods
Performance levels 0, 1, 2, 3
Because of open-ended questions , we had to take
answers to different questions related to the same
knowledge domain. So, item combinations were defined to
determine performance levels.
Level
3
2
1
0
Description
Students' answers were along the
expectations from our teaching approach.
Students' answers were partially along the
expectations from our teaching approach.
Students' answers showed some major
deficiencies.
Students' answers were weak in relation to
our expectations.
Evaluation: To what extent have the main goals
been achieved by the students?
The first three goals
Total performance level 0
0
1
2
Model of an atom
3
50% -
A classical
model is
dominating
Psi-function
No connection between
psi-function and
model of an atom
The concept of state has
no meaning with relation
to quantum physics
Equation is
hidden behind
Notion of state
Schrödinger equation
Total performance level 3
A quantum
mechanical model
is dominating
50% -
The psi-function is the
main component of the
model of an atom
50% -
The concept of state
has a new meaning
with relation to
quantum physics
50% -
Deylitzformulate
1999 the
Students
50% -
psi-function is the
what extent have the The
main
goals
main component
of the
model of an atom
been achieved by the students?
No connection between
Evaluation:
To
psi-function and
model of an atom
The concept of state has
no meaning with relation
to quantum physics
Total performance level 0
Equation is
hidden behind
A classical
STELLA
models
model is
dominating
Observed phenomena
are not related
to a
No connection
between
quantumpsi-function
model of atom
and
model of an atom
No understanding
The concept
of state has
of
shielding
effects
no meaning with relation
to be
seen
to quantum
physics
Equation is
hidden behind
Notion of state
50% -
The second three goals
0
The concept of state
has a new meaning
with relation to
quantum physics
Total performance level 3
1
2
3
Schrödinger equation
Model of an atom 50% - Students formulate the
50% process
of application
A
quantum
without help model
mechanical
is dominating
Relating measurements to theory
Psi-function
50% - Observed phenomena
50% - are explained by
The psi-function is the
relating
them to a of the
main component
quantum
of atom
model ofmodel
an atom
Higher order atoms
A qualitative
Notion of state
50% - The concept of state
50% - understanding of
has a newused
meaning
shielding;
in own
with
relation
to
STELLA models
quantum physics
Schrödinger equation
50% -
Deylitz 1999
Students formulate the
process of application
Part 4
Marion Budde (2003)
Research about learning effects of
content specific teaching inputs
Research questions

What effects has a certain teaching input on the
learning process of an individual student?
Theoretical perspective:
Co nst r uc t ivist Inst r uc t io n
e
a
r
n i
n
g
L
St ud e nt
Mind
Te a c he r
E
n
v
i r o n m
e
t
n
==>
Resonance
Detail of teaching approach ("teaching input")
The probability model
The electronium model
Data

2 students: Audio tapes from teaching (five lessons of
45 minutes per week, for approximately18 weeks)

1 student attended private lessons (approximately 50
lessons), which were also audio-recorded.

Pre and post test for understanding of atomic models.
Methods


Criteria for statements showing a learning effect:

Statements made spontaneously

Statement with different content

Contradicting statement

Consequent statement in a complex discussion

Use of own formulation or in a new context
Development of different types of resonance

No resonance

Congruent/non-congruent resonance
Research
during teaching
Overviewabout
of theresonances
observed resonances
Student “Klaus”
Student “Thomas”
Liquid
Spontaneous congruent
intended resonance
Slightly delayed congruent
intended resonance
Continuum /
Continuous
density
Not intended non-resonance
Not intended non-resonance
Particle conception is too
strong: even liquids are seen
as consisting of particles
Distance conception of density
No motion
Slightly delayed
congruent intended
resonance
Spontaneous congruent
intended resonance
Trial of summary 1

Expect strong mechanistic thinking

Particle

Orbits, trajectories

Planetary model

Stability from equal forces: F-coulomb = F-centrifugal
Trial of summary 2

Expect intermediate conceptions
(self constructed in students heads,
not intended by the teacher)
a wave orbit
a smeared orbit
a sample of neighbour orbits
with high p robability
Trial of summary 3
The "electronium" interpretation of psi in atoms seems

... to be intuitive

... to develop a meaning of a distribution

... to help to replace the particle conception ("cognitive tool")
by a similar powerful conception of a compressible gas or
liquid
Cognitive attractor "smeared orbits"
Bayer (1986) describes it as a spherically smeared shell with a
certain thickness, coming from a kind of wave orbits.
Bethge (1988) describes it as a new and broader understanding of
orbits, taking into the notion of orbits some elements of smeared
or most probable orbits or probabilities along the orbit.
Kuehnen (1994) describes conceptions which are no more purely
mechanistic seeing the electron with some probability in a
special space region, where they move along orbits. But they
still are particles.
These different investigations show that the conception of
”smeared orbits” probably is what could be called a cognitive
attractor in this content area, which describes a cognitive
construction using a conception of orbits to make sense of a new
teaching input with the idea of probability.