Thesis Format

Integrated Article

Does Lexical Frequency affect rater judgement of essays? An experimental design using quantitative and qualitative data

Mohammad Muneer Ul-Huda, The University of Western OntarioFollow

Degree

Master of Education

Program

Education

Supervisor

Faez, Farahnaz

2nd Supervisor

Boers, Frank

Abstract

Many correlational studies show a positive relation between written assessments of language and use of more diverse vocabulary (Lexical Diversity) and more infrequent words (Lexical Frequency). However, there have been no experimental studies that have isolated the effects of Lexical Frequency from Lexical Diversity. In the present study, 14 raters judged two versions of the same essay that differed only in Lexical Frequency. A Paired T-test showed no difference in mean scores between essays (t(13) = .396, p = .70) when the Lexical Frequency of 23.5% of Content Words were changed in a 347 word essay. Comments explaining scores given to essays showed that features other than vocabulary had a far greater influence on rater judgement. It is possible that the Lexical Frequency manipulations were not great enough to affect rater judgement, whether subliminal or conscious. Implications of these results for standardized language proficiency tests and future research in vocabulary are discussed.

Summary for Lay Audience

Studies have shown a positive relation between essay scores and the use of a greater variety of words (referred to as Lexical Diversity) and use of words that are less common (referred to as Lexical Frequency). These two variables, Lexical Diversity and Lexical Frequency, are often measured together. Because of this, we don’t know how Lexical Frequency alone (use of less common words) affects a rater’s perception of a writing. We conducted an experimental design where we produced two essays that were identical, except with regard to their Lexical Frequency: one essay used more common words (High Frequency Essay) and the other used less common words (Low Frequency Essay). Statistical analysis showed that raters (n=14) gave both essays the same score on average. After looking at rater comments explaining why they scored essays the way they did, we see that features of the essay other than vocabulary affected rater judgement far more. This could be one reason why we saw no difference in essay scores. Another reason could be that the change in Lexical Frequency that we created was simply not big enough to affect rater judgement. Implications of these results are discussed.

Recommended Citation

Ul-Huda, Mohammad Muneer, "Does Lexical Frequency affect rater judgement of essays? An experimental design using quantitative and qualitative data" (2019). Electronic Thesis and Dissertation Repository. 6407.
https://ir.lib.uwo.ca/etd/6407