Evaluation of improvement in grammatical
and listening comprehension skills
of first year language students
taking English as a “minor”
after following two
experimental modules
during their first semester in the RomaTre
language program.
(A talk given for a group of language teachers, IAL,
Rome, 25.5.2006)
Patrick Boylan – University of Rome III
–
All incoming students at RomaTre University are given a
Foreign Language Placement Exam (run by the Facoltà,
not by the language department) to determine their entry level for
the language program. Students who choose to take an English
minor, in order to major in some other language, take three “English
minor” exams during the three years of the degree course. Since
these students are generally certified by the Placement Test at
between CEF* Level A1 and A2 upon arriving, language instruction is
geared to that level for the first year course, and raised by half a
CEF level in each of the following two years. (Half a level is all
that can be taught since the official yearly language courses do not
last a year, but rather only one semester.)
*CEF
= Common European Framework of reference for language competence.
In 2002 and 2003 I called in an outside
language school to run their computerized “TOEFL”-style
English Test (called the “L-Test®”)
on the students of my first-year English modules. This test is
traditional in nature, focusing on grammar and discrete-point
reading+listening comprehension.
I chose this test for 3
reasons:
1. I wanted a test similar to the Facoltà
Placement Exam in order to verify that exam's findings.
2. I
was curious to see if my students made improvement according to
“traditional criteria” (i.e., according to whether their
grammar and discrete-point knowledge improved), since my experimental
modules do not focus on such things and I do not
evaluate them in the final exam;
3. I wanted an exam that did
not cost anything, since I had no research funds at the time. I knew
the L-Test®
creator and I knew that he wanted to try his new computerized version
out on a large number of students before using it at his language
school. Manus manum lavat.
My
hypothesis was that, since all my students are regularly classified
by the entry test as false beginners (A1 to A2, some B1s), they must
have latent grammatical and discrete-point knowledge that a holistic
style of teaching, of the kind I do in my experimental modules, would
most likely revitalize.
In other words, according to my
hypothesis, the students' grammatical and discrete-point knowledge of
English ought to improve noticeably in only one semester, not because
specific explanations or exercises were given, but only by being
“dynamized” in carrying out the required research
tasks.
This proved to be the case – and resoundingly
so.
As for the answer to the third question – whether
the L-Test®
was valid (for the areas it was designed to measure) and reliable
(over a large number of students) – it also was affirmative,
and resoundingly so.
As for the answer to the first
question (whether the Facoltà Placement Test was accurate),
the issue will be discussed in another report.
DATA
for academic years
2002-03,
and
2003-04,
(partial data for 2001-2002)
Academic year 2002-03
Population: 201 first year “English minor” students students (M=37 F=164) taking the first semester modules “Seeing and saying things in English” (October to November, 3 credits) and “Accommodation in intercultural conversations in English” (December to January, 2 credits).
(NOTE: The class was exceptionally
large and required an amphitheatre to permit the 201 students to form
the work groups required to carry out the various research projects.
The number of students assigned per class was thankfully reduced in
the following years.)
Total class contact hours: 40
NOTE: In addition to my teaching,
students were also supposed to take “practical language
classes” taught by lettori at level A2/B1. Because
of conflicting time schedules, however, less than one fourth attended
the lettori lessons regularly. Another reason for non
attendance was the highly demotivating University policy of
certifying only the attainment of level B1 (for speaking/writing) for
English minors upon their graduation. This means that even if
students brought their level up to B2 or C1, this fact would not be
certified on their graduation certificate. Since many English minor
students were already at level A2 upon entering the University, they
had no incentive to take higher level lettori courses
voluntarily. They (wrongly, in my opinion) chose to abandon the
lettori courses until their third year, which is the only
course needed to bring their level up to the minimum required (B1 –
speaking/writing).
(First exam: 29 October 2002, Second
Exam: 28 February 2003)
Average results:
CEF* Level in English |
|
2002-03 |
2003-04 |
||
Entry |
1st semester |
2nd semester |
1st semester |
2nd semester |
|
Level C2 (6.0) |
|
|
|
|
|
(5.5) |
|
|
|
|
|
Level C1 (5.0) |
|
|
|
|
|
(4.5) |
|
|
|
|
|
Level B2 (4.0) |
|
|
|
|
|
(3.5) |
|
|
|
|
|
Level B1 (3.0) |
|
2.78 |
|
|
|
(2.5) |
|
|
|
|
|
Level A2 (2.0) |
1.76 |
|
|
|
|
(1.5) |
|
|
|
|
|
Level A1 (1.0) |
|
|
|
|
|
In addition, during the Academic
year 2002-03 I asked students from my 2001-02 first-year
(first semester) class to retake the single L-Test®
they had taken a year before, at the end of their semester with me.
(I had not yet arranged for an L-Test®
at the beginning of the semester, as I would do in the following
years.)
I wanted to see if their English had continued
to improve after their semester in the experimental modules and if
progress was as noticeable.
Fifteen students (M=0, F=15)
volunteered to take the test. (They were compensated with the
bestowal of a certificate by the outside Language School, useful in
applying for a job. One of the students' certificates, with her
marks, can be seen here.).
At
the time of the exam, the students were completing their second
semester of their second year.
“One year after” Exam: 30 April 2003 (Exam results from 27 Feb. 2002 also appear for comparison)
Average results:
CEF* Level in English |
|
2001-02 |
2002-03 |
||
Entry |
1st semester |
2nd semester |
1st semester |
2nd semester |
|
Level C2 (6.0) |
|
|
|
|
|
(5.5) |
|
|
|
|
|
Level C1 (5.0) |
|
|
|
|
|
(4.5) |
|
|
|
|
|
Level B2 (4.0) |
|
|
|
|
|
(3.5) |
|
|
|
|
3.50 |
Level B1 (3.0) |
|
3.03 |
|
|
|
(2.5) |
|
|
|
|
|
Level A2 (2.0) |
|
|
|
|
|
(1.5) |
1.5 ? |
|
|
|
|
Level A1 (1.0) |
|
|
|
|
|
Academic year 2003-04
Population: 76 first year “English minor” students students (M=11 F=65) taking the first semester modules “Seeing and saying things in English” (October to November, 3 credits) and “Accommodation in intercultural conversations in English” (December to January, 2 credits).
Total class contact hours: 40
NOTE: In addition to my teaching,
students were also supposed to take “practical language
classes” taught by lettori at level A2/B1. Three
quarters did not. See the note in Academic year 2002-03 for
the reasons.
(First exam: 30 October 2003, Second
Exam: 26 February 2004)
Average results:
CEF* Level in English |
|
2003-04 |
2004-05 |
||
Entry |
1st semester |
2nd semester |
1st semester |
2nd semester |
|
Level C2 (6.0) |
|
|
|
|
|
(5.5) |
|
|
|
|
|
Level C1 (5.0) |
|
|
|
|
|
(4.5) |
|
|
|
|
|
Level B2 (4.0) |
|
|
|
|
|
(3.5) |
|
|
|
|
|
Level B1 (3.0) |
|
3.10 |
|
|
|
(2.5) |
|
|
|
|
|
Level A2 (2.0) |
1.92 |
|
|
|
|
(1.5) |
|
|
|
|
|
Level A1 (1.0) |
|
|
|
|
|
Update to the talk (June, 2007)
I
said at the beginning that my exams do not focus on grammar or
discrete-point skills. They attempt to measure holistic knowledge, in
particular (1.) the capacity to grasp and verify what a speaker of
English means with a given tone, expression or gesture and to
appropriate that expressive mode authentically for personal use; (2.)
the capacity to see beyond the common myths and understand what
“English” really is and what it means to learn it
(students will have to be lifelong learners in our globalized world).
Just
before the end of the 2006-07 first-year course, |