Download HawkinsFilipovicAILABeijing2011

Identifying Criterial Features for
Reference Level Descriptions
John A. Hawkins
(1,2) & Luna Filipović (1)
(1) University of Cambridge, (2) University of
California Davis
The CEFR levels have been defined hitherto
in functional terms, i.e. in terms of the uses
to which language can be put and the
various functions that learners can perform
as they gradually master a second language.
See e.g. the "can-do" statements of the Council of
Europe (2001) document, pp.244-257.
Learners at B1 “can express opinions on
abstract/cultural matters in a limited way or offer
advice within a known area”.
Learners at B2 “can follow or give a talk on a
familiar topic”.
Learners at C1 “can contribute effectively to
meetings and seminars within own area of work or
keep up a casual conversation with a good deal of
fluency”.
By focusing on functions the CEFR attempts
to define these levels in a way that is
independent of the different grammatical
and lexical details of the languages of
Europe.
But learners who perform each of these
functions may be using a wide variety of
grammatical constructions and words in
order to do so.
And the ability to do each of the tasks does
not tell us with precision which grammatical
and lexical properties of English (and of
other target languages) the learner actually
knows and uses at each level.
We need this precision and added
specificity, because the grammatical and
lexical details of each language are key
properties that examiners look for when
they assign candidates to a proficiency level
and score,
because teachers need to teach these
properties
and because learners need to learn them.
So in this talk we ask:
1) how much of the grammar and lexicon of
English do learners actually know and/or
produce at each of these CEFR levels?
2) and what patterns and principles are there in
these developing second language acquisition
stages of English?
The way we have chosen to proceed is to
search for “criterial features”.
These are properties of English
(constructions, words, rule types, errors and
their frequencies, etc) that are distinctive
and characteristic of L2 proficiency at the
different levels.
See Hawkins & Filipović (2011), Hawkins & Buttery (2009,
2010).
Criterial features can be found in all areas of
English: syntax, morphology, phonology, the
lexicon, semantics, and discourse. They
enable us to distinguish higher proficiency
levels from lower levels in an efficient way.
An analogy for Criterial Features
The defining characteristics for recognising faces in
a police identikit. You don’t need to see all the
features of a person’s face in order to distinguish
that person from others, just the important defining
characteristics that capture essential qualities
Criterial features are similar: they capture essential
distinguishing properties of the CEFR proficiency
levels.
Another example:
Languages change over time, and when historians of
English examine Old English, Middle English and
Early Modern English, they focus on important
differences between these stages, not on what
stayed the same.
So too in second language learning. We need to
discover what is characteristic of each level as
learners progress.
More generally, this research programme is centered
on the notion of criteriality and on criterial
differences between levels.
I.e. we are trying to capture key changes from level
to level in what has been learned and in what can be
produced, based NOT on what we hope learners have
learned and on what they have been taught, but
based on an empirical investigation of what they
actually do, in the CLC.
Electronic corpora of learner English make it
possible for us to discover criterial features.
The Cambridge Learner Corpus (CLC) gives
us empirical evidence for developmental
stages in the learning of new constructions,
words and word meanings.
It also gives us quantitative data on learner
errors in syntax, morpho-syntax and lexical
choice.
This corpus has been tagged for parts of
speech and parsed using a sophisticated
automatic parser (Briscoe et al. 2006) permitting
numerous grammatical and lexical searches
to be conducted. Between a third and one
half has been error-coded using codes
devised by researchers at CUP.
Sample Error Codes in the CLC
RN Replace noun Have a good travel (journey)
RV Replace verb I existed last weekend in London
(spent)
MD Missing determiner I spoke to President (the)
I have car (a)
AGV Verb agreement error The three birds is singing
(are)
IV Incorrect Verb Inflection I spended last week in London
(spent)
FJ Wrong Adjective Form
The situation got worst (worse)
UQ Unnecessary Quantifier A little bit quite common (quite
common)
DY Derivation of Adverb
It happened fastly (fast)
Briscoe’s RASP (Robust Accurate Statistical
Parser)
•
•
•
identifies parts of speech (PoS) probabilistically
tagging
generates a parse forest representation containing
all possible subanalyses with associated
probabilities
weighted Grammatical Relations yielded by the nbest parses of the input.
There are different types of criterial features
(Hawkins & Buttery 2010, Hawkins & Filipović 2011). Here
we refer to just two:
Positive Linguistic Features
These refer to positive, i.e. correct, linguistic
properties of English that have been acquired at a
certain L2 level and that generally persist at all
higher levels. A property P (e.g. a new construction
type) acquired at B2 may differentiate that level and
higher levels from [A1, A2, B1] and will be criterial
for the former. Or P may be acquired at C2 and
differentiate this level from all lower levels.
Negative linguistic features
These are incorrect properties of English, or errors,
that occur at a certain level or levels and with a
characteristic frequency. Both the presence versus
absence of the errors, and especially their
frequency (the "error bandwidth"), can be criterial
for the level(s).
Examples of Positive Criterial Features
The A levels (A1 and A2)
Simple intransitive clauses (NP-V) and the
slightly more complex transitive (NP-V-NP)
sentence types are present from the
beginning:
He went. (NP-V) A1
He loved her. (NP-V-NP)
A1
Modal auxiliary verbs like may, might, can
and must appear first at A1 or A2, but only
in some of their senses.
Can is first attested in the PERMISSION sense at A1
and in the POSSIBILITY sense at A2:
And if you want, you can bring pencils or pens. (PERMISSION)
A1
In this magazine you can see all the new C.D.[s] and all the
dates of the concerts. (POSSIBILITY) A2
Noun Phrase sequences of Pronoun plus
Infinitive are found at A2:
something to eat
nothing to do
A2
A2
as are postnominal modifiers with participial
–ed:
beautiful paintings [painted by famous Iranian painters]
A2
Lexical verbs appearing at the A levels are
typically among the most basic and frequent
verbs of English; they appear first in their
most basic and frequent senses.
Verbs attested at A1 include:
catch, eat, give, put, take and walk
New lexical verbs appearing at A2 include:
break, cut, hit, push, stand, and fall
again typically in their most basic and literal senses.
For break this includes its primary physical sense:
I broke a beautiful glass. A2
for cut it includes the following example in its
primary sense:
First I cut the cake with my mother. A2
The B Levels (B1 and B2)
The new features at B1 involve more complex
syntax, e.g. an “Object Control” structure such as:
I ordered him [to gather my men to the hall]
B1
him is both the object of ordered and the logical
subject of gather here.
This is a criterial construction for B1 and higher
levels which distinguishes them from the A levels.
Structures like the following with finite or
non-finite subordinate clauses and
movement of the WH-word (how, where, etc)
to the front of its clause are also first
attested at B1:
I don’t know [how I could have done it]
I did not know [where to look for it]
B1
B1
And postnominal modifiers in participial –ing
become productive at B1:
I received your mail [asking for the sales report]
B1
Structures with a finite subordinate clause
positioned to the right of predicates like is
true and seems with a subject it are also
criterial for B1 and higher levels:
It’s true [that I don’t need a ring to make me remember you]
B1
i.e. so-called “Extraposition” structures
A large number of new lexical verbs appear
for the first time at B1 including:
divide, fit, grab, spill, stick and tear
And the meanings of the verbs that appeared first
at A1 and A2 begin to expand from their basic
senses.
break appears for the first time in the extended
sense of INTERRUPT at B1:
At last I managed to break the routine of the city … B1
Constructions that are criterial for B2 and
higher levels include “secondary predications”
go and paint the houses yellow and blue
B2
with yellow and blue predicated of the direct object
houses
“Extraposition” structures with a non-finite
subordinate clause positioned to the right of
its predicate are B2
It would be helpful [to work in your group as well]
B2
And so-called “Pseudocleft” structures with
an initial what functioning as subject of its
verb:
What fascinated me was [that I was able to lie on the sea surface]
B2
“Subject-to-Subject Raising” constructions
appear first at B2 with most of the higher
verbs and adjectives that trigger this rule,
for example prove:
The car has proved [to be one of the most important
inventions of our century]
B2
Similar examples are found at B2 with other raising verbs and
adjectives (The car happened to be …, The car appeared to be
…, The car turned out to be …, The car is likely to be …, etc)
New lexical verbs at B2 include
acquire, capture, drag, rush, spread,swallow
and new meanings and uses are attested for the
verbs that appeared earlier.
For break, first attested at A2, these include new
collocations such as break a promise or break the
law:
For cut, also an A2 verb, they include new meanings
at B2 such as REDUCE in cut the cost
The C Levels (C1 and C2)
“Subject-to-Object Raising” constructions with the
verb believe appear first at C1 and are criterial for C
levels:
I believe her [to be this country’s best representative]
C1
Passivized Subject-to-Object Raising
constructions such as the following with
assumed are also criterial for C1:
the low cost of membership and entry was assumed to be an
advantage. C1
Sequences of two prenominal –s genitives are
found at C1:
in the bride’s family’s house
C1
Structurally:
in [[[the bride’s] family’s] house]
New lexical verbs appearing first at C1
include
accumulate, boast, quote, reassure, shape and
stain
along with new meaning possibilities for the verbs
already introduced. E.g. break appears first in the
idiomatic sense of break the bank at C1.
New features appearing at C2 include less
common Subject-to-Object Raising
constructions with higher predicates such as
presume, declare and remember:
He presumed work [to be the way to live]
C2
New lexical verbs at C2 include
stagger, sway, limp, saunter, raid,squander
New meanings for break at C2 include original
figurative senses such as the attested break the
wall that surrounds him.
Negative Linguistic Features
One major distinguishing feature of the C levels can
be seen in the low frequencies for “negative
features” or errors.
There appear to be significant improvements in ALL
of the syntactic and morpho-syntactic error types at
the C levels.
At the B levels, by contrast, improvements are
relatively modest, and for many error types the
scores actually get worse, especially at B2, before
they get better again at C1.
The error codes defined and assigned by
our CUP colleagues involve morphosyntactic errors of inflection, derivation and
grammatical form, and syntactic errors of
omission, positioning and co-occurrence. It
appears that learners at the C levels are
increasingly mastering these rules of
English, whereas B-level learners are not.
We must now ask: WHY we see these
patterns in the data and why we see the
criterial features changing the way they do
at the different levels? In particular, WHAT
is it about the features of the higher
proficiency levels that makes them late
acquired rather than early?
It cannot simply be that learners are
imitating the words and constructions they
are explicitly taught in their textbooks.
First, because there are many different textbooks
and teaching methods around the world.
But secondly because learners learn more than they
are explicitly taught, from their reading materials,
papers, magazines, movies, TV, conversations, and
so on.
I.e. second language learning shares many
similarities with first language learning, but not all
obviously.
For example, more frequently occurring
words and constructions are learned before
less frequent ones,
and simpler words, constructions and
meanings are learned before more complex
ones,
in both first and second language
acquisition.
E.g. learning English nouns and verbs with high
frequencies of use is easier than learning those
with lower frequencies, because they are
encountered more frequently (greater exposure);
frequent lexical items are overrepresented at first
in L2 English, moving gradually to L1 English
norms (see Hawkins & Buttery 2009, Hawkins & Filipović 2011)
More frequent construction types
(“subcategorization” frames or “verb cooccurrences”) are acquired earlier than more
infrequent ones, in general (see C. Williams 2007,
Hawkins & Filipović 2011).
The constructions of English that are
learned earliest are those that occur most
frequently in the input, as reflected in e.g.
the British National Corpus. This could be
established by comparing the CLC with the
British National Corpus (BNC), see Williams
(2007).
In fact, the new constructions that are
criterial for A2, B1 and B2, in Williams’ data,
appear to be learned in direct proportion to
their frequency in the input, as reflected in
the BNC. The more exposure, the earlier the
acquisition and the easier the learning.
This is shown in Tables 1 and 2.
Table 1 Frequencies for Verb Co-occurrence
Frames in English Corpora (including BNC)
Average Token Frequencies in the BNC for
the new Verb Co-occurrence Frames
appearing at the learner levels
A2
1,041,634
B1
38,174
B2/C1/C2
27,615
Table 2 Frequency Ranking
Average Frequency Ranking in the BNC for
the new Verb Co-occurrence Frames
appearing at the learner levels
A2
8.2
B1
38.6
B2/C1/C2
55.6
These kinds of data enable us to set up the
following principle of second language
learning (for which there are also well-attested
parallels in first language learning, see e.g.
Tomasello 2003, Diessel 2004, MacWhinney 2005):
Maximize Frequently Occurring Properties (MaF)
Properties of the L2 are learned in proportion to
their frequency of occurrence (as measured, for
example, in the BNC): more frequent exposure of a
property to the learner facilitates its learning and
reduces learning effort.
I.e. more frequent properties will result in earlier L2
acquisition, more of the relevant properties learned, and
fewer errors, in general. Infrequency makes learning more
effortful, with precise predictions depending on other factors.
Another principle of second language learning
(shared with first language learning) involves the
relative simplicity or complexity of structures and
meanings. The criterial grammatical features of
earlier levels are, in general, simpler than those of
later levels.
Also, in phonology simpler consonants and
consonantal distinctions are acquired earlier
than more complex ones (see e.g. Eckman
1984).
Simpler and more basic meanings for verbs
are acquired earlier than more complex and
derived extensions in meaning, figurative
uses, etc.
The verb break in its basic physical sense at A2
break in the sense of INTERRUPT (break the routine) B1
break an agreement, promise, etc. B2
break the bank (idiomatic) C1
break the wall that surrounds him (original figurative) C2
Maximize Structurally and Semantically Simple
Properties (MaS)
Properties of the L2 are learned in proportion to
their structural and semantic simplicity: simplicity
means there are fewer properties to be learned and
less learning effort is required.
I.e. simpler properties will result in earlier L2 acquisition,
more of the relevant properties learned, and fewer errors.
Complexity makes learning more effortful, in general, since
there are more properties to be learned, with precise
predictions depending on other factors.
In second language learning we also see
“transfer” effects from the first language,
either positive (when the transfer results in a
correct L2 property) or negative (when it
results in an error).
This is one thing that differentiates second from
first language acquisition.
E.g. speakers of languages with definite and
indefinite articles find it easier to acquire
the article system of English than do
speakers of languages without articles (see
Hawkins & Buttery 2009, 2010)
Errors involving missing definite and
indefinite articles in the L2 English of the
CLC are consistently low when the L1s also
have articles.
Recall: MD
I spoke to President (the)
I have car (a)
Table 3 (next slide) shows missing
determiner error rates for “the” and “a” at all
proficiency levels for French, German and
Spanish as first languages. All three
languages have an article system. (Data from
Hawkins & Buttery 2009)
The figures indicate the percentage of errors with respect to
the total number of correct uses. For instance a percentage of
10.0% would indicate that a determiner was omitted 1 in every
10 times that it should have appeared.
We see generally low error rates for these languages, without
significant deviation between levels.
Table 3 Missing Determiner Error Rates for L1s with
Articles
French
German
Spanish
French
German
Spanish
A2
4.76
0.00
3.37
Missing “the”
B1
B2
4.67
5.01
2.56
4.11
3.62
4.76
C1
3.11
3.11
3.22
C2
2.13
1.60
2.21
A2
6.60
0.89
4.52
Missing “a”
B1
4.79
2.90
4.28
C1
4.76
3.62
5.16
C2
3.41
2.02
3.58
B2
6.56
3.83
7.91
Table 4 (next slide) shows missing
determiner error rates for “the” and “a” at all
levels for Turkish, Japanese, Korean and
Russian as first languages. These
languages do not have an article system.
There is a general linear improvement, i.e. a decline, in error
rates across the levels with increasing proficiency (shown
from left to right).
Table 4 Missing Determiner Error Rates for L1s
without Articles
Turkish
Japanese
Korean
Russian
Missing “the”
A2
B1
B2
22.06
20.75
21.32
27.66
25.91
18.72
22.58
23.83
18.13
14.63
22.73
18.45
C1
14.44
13.80
17.48
14.62
C2
7.56
9.32
10.38
9.57
Table 4 continued
Turkish
Japanese
Korean
Russian
Missing “a”
A2
B1
24.29
27.63
35.09
34.80
35.29
42.33
21.71
30.17
B2
32.48
24.26
30.65
26.37
C1
23.89
27.41
32.56
20.82
C2
11.86
15.56
22.23
12.69
One of the learning principles proposed in Hawkins &
Filipović (2011) to account for these data is
Maximize Positive Transfer:
Maximize Positive Transfer (MaPT)
Properties of the L1 which are also present in the
L2 are learned more easily and with less learning
effort, and are readily transferred, on account of
pre-existing knowledge in L1.
Shared L1/L2 properties should result, in general, in earlier L2
acquisition, in more of the relevant properties being learned,
and in fewer errors, unless these shared properties involve e.g.
high complexity and are impacted by other factors. Dissimilar
L1/L2 properties will be harder to learn by virtue of the
additional learning that is required, again in general.
More generally, in ongoing work we provide a
multi-factor model of learning comprising a
set of interacting principles such as those
illustrated.
This model is a type of “complex adaptive
system” (see Gell-Mann 1992) and is called
“CASP”, short for “complex adaptive system
principles of SLA”.
These principles interact, sometimes reinforcing
each other (e.g. early acquired frequent items are
also often simple), sometimes competing to produce
variable outputs and alternative interlanguages.
Some of the principles are more general, others more
specific. One of the more general principles is
Minimize Learning Effort.
Minimize Learning Effort (MiL)
◦ Learners of a second language (L2) prefer to
minimize learning effort when they learn the
grammatical and lexical properties of the L2.
Learning effort is minimized when shared properties of the L1
can be transferred directly into the L2 (MaPT), when properties
of the L2 are frequently occurring in the L2 input (MaF), when
structural and semantic properties of the L2 are simple rather
than complex (MaS), and when there are fewer linguistic items
to be learned in a given grammatical or lexical domain.
Once criterial features and transfer effects
have been identified at the different
proficiency levels they can be put to use for
learning, teaching and assessment
purposes, as illustrated in other talks in this
session.
NB! Our criterial features are taken only from
candidates who scored passing grades at each level.
Hence learners who are studying for the relevant
level can now be told explicitly what their successful
peers have mastered, and teachers can incorporate
these features in their teaching and materials,
thereby optimizing the learners’ chances of success.
Theoretical interest of this work
The criterial features we are extracting from the
corpus are also of interest to theoreticians studying
language acquisition. They provide a new set of
empirical patterns that can be used to inform
predictive and multi-factor theories of learning,
such as CASP.
References
Briscoe, E., J. Carroll and R. Watson (2006) ‘The second release of the
RASP system’. In Proceedings of the COLING/ACL 2006 Interactive
Presentation Sessions, Sydney, Australia.
Council of Europe (2001) Common European Framework of
Reference for Languages: Learning, teaching, assessment. CUP,
Cambridge.
Diessel, H. (2004) The Acquisition of Complex Sentences. CUP,
Cambridge.
Eckman, F.R. (1984) ‘Universals, typologies, and interlanguage’.
In:W.E. Rutherford, ed., Language Universals and Second Language
Acquisition, John Benjamins, Amsterdam, 79-105.
Gell-Mann, M. (1992) ‘Complexity and complex adaptive systems’. In
J.A. Hawkins & M. Gell-Mann, eds., The Evolution of Human
Languages, Addison-Wesley, Redwood City, CA.
Hawkins, J.A. (2004) Efficiency and Complexity in Grammars. OUP,
Oxford.
Hawkins, J.A. & P. Buttery (2009) ‘Using learner language from
corpora to profile levels of proficiency: Insights from the English
Profile Programme’. In L. Taylor & C.J. Weir, eds., Language
Testing Matters, Proceedings of the 3rd ALTE Conference 2008,
CUP, Cambridge, 158-175.
Hawkins, J.A. & P. Buttery (2010) ‘Criterial features in learner corpora:
Theory and illustrations’, English Profile Journal 1.
Hawkins, J.A. & L. Filipović (2011) Criterial Features in L2 English:
Specifying the Reference Levels of the Common European
Framework. CUP, Cambridge.
MacWhinney, B. (2005) ‘A unified model of language acquisition’. In
J.F. Kroll & A.M.B. de Groot, eds., Handbook of Bilingualism:
Psycholinguistic Approaches, OUP, Oxford.
Milanovic, M. (2009) ‘Cambridge ESOL and the CEFR’, University of
Cambridge ESOL Examinations Research Notes 37: 2-5.
Tomasello, M. (2003) Constructing a Language: A Usage-based Theory
of Language Acquisition, Harvard University Press, Cambridge,
Mass.
Williams, C.A.M. (2007) ‘A preliminary study into verbal
subcategorisation frame usage in the CLC’, MS, RCEAL, University of
Cambridge.
Acknowledgments
The findings reported here would not have been possible without
the assistance of many collaborators. Special thanks to:
Co-author Paula Buttery (see Hawkins & Buttery 2009, 2010);
Annette Capel of CUP for help with the wordlist searches;
Ted Briscoe of the Cambridge Computer Lab and his colleagues for use of
the RASP parser;
Mike Milanovic & Nick Saville of Cambridge ESOL for theoretical and practical
guidance and financial support (see below);
Mike McCarthy of CUP and Penn State U for advice and input;
Roger Hawkey of Cambridge ESOL for advice and English Profile Programme
co-ordination;
Lu Gram of the Computer Lab for help with error calculations and other
searches;
Caroline Williams of Cambridge University for verb subcategorisation data;
and to
CUP’s computational linguists who prepared "The <#S>
Compleat|Complete</#S> Learner Corpus Document" 2006, from which the
error codes and examples sentences in slide 18 are taken.
Financial Support
The work reported here was made possible by
generous financial support from Cambridge
Assessment and from Cambridge University Press,
within the context of the Cambridge English Profile
Programme. This support is gratefully
acknowledged here.