What is the Sawtooth Effect?

Documents

Maintaining standards during normal times and when aqualifications are reformed

Ref: Ofqual/20/6710/2PDF, 463KB, 26 pages

Details

This research, and the overview, bring together the ideas that we use to explain what we are trying to achieve when we attempt to maintain standards: attainment-referencing, comparable outcomes, the sawtooth effect.

Maintaining exam standards during normal times is not simple, and is even harder during periods of qualification reform when curriculum, syllabus and assessment arrangements change. The short overview explains how the ideas of attainment-referencing, comparable outcomes and the sawtooth effect relate to each other. It provides a comprehensive overview of the maintenance of standards.

During normal times, we tend to view the maintenance of standards in terms of meaning: how grades need to be interpreted. In short, what do they mean? We strive to ensure that equivalent grades, across successive versions of the same subject exam, can be interpreted in the same way; that is, in terms of attainment. We say that exam standards have been maintained when equivalent grade boundary marks across adjacent exams correspond to equivalent levels of attainment. We call this principle attainment-referencing.

Conversely, across periods of qualification reform, we tend to view the maintenance of standards in terms of consequence. In short, how can grades be used? We strive to ensure that equivalent grades, across successive versions of the same subject exam – pre-reform versus post-reform – can be used in the same way, and used fairly.

We switch our focus from meaning to consequence – from the last exam pre-reform to the first exam post-reform – because we expect the quality of candidates’ exam performances to drop significantly across this period of transition. It is not the fault of the first cohort post-reform that their performances (and their attainments) are lower. This is a consequence of their teachers not yet being up to speed with teaching the new content elements, or with preparing for the new assessment structure or formats.

So, we compensate for this effect – the sawtooth effect, which represents a sudden dip in performance followed by a gradual rise back up again over time – by applying the comparable outcomes principle. We do this to be fair to candidates.

This introductory overview and our research explains these ideas in more depth. They explain the logic of what we are trying to achieve when we attempt to maintain exam standards, and the methods that we use to maintain them.

This sawtooth research is about the interpretation of assessment results, and the steps that assessment agencies can take – either before or after awarding those results – to make sure they are interpreted appropriately; or, at least, to minimise the likelihood of misinterpretation.

Finally, it is worth mentioning that coronavirus (COVID-19) times are even less normal than periods of qualification reform. However, very similar considerations arise, especially the need to be fair to candidates, and similar methods can be employed as well. However, because the effects of coronavirus (COVID-19) will be far more severe for some learners than for others, applying the comparable outcomes principle can only represent a partial solution to the problem of learning loss attributable to coronavirus (COVID-19).