The assessment of qualifications in 2021 and lessons learned since last year

Speaking at the Schools and Academies Show @Ofqual’s Chief Regulator spoke about the assessment of qualifications in 2021 and lessons learned since last year: 

Good morning. It is a great pleasure to be here with you for this year’s second online Schools and Academies Show. My name is Simon Lebus and I am the Interim Chief Regulator at Ofqual.

I have found it to be a daunting task to have so many of my interactions with various different interlocutors mediated exclusively through a computer screen rather than face-to-face. I can only hope I am able to rise to the challenge as effectively and with as much panache as so many of you have managed to do.

It is a problem I have also faced at Ofqual. Having started there on 4 January when we were in lockdown I have never actually managed to meet any of my colleagues there face to face. That was also, of course, the day on which the Prime Minister announced that the summer’s exams would not proceed normally and my time at Ofqual has, therefore, been very different to how I imagined it when I agreed last year to take the role on. In particular, I have found myself presiding over a major departure in how we manage Year 11 and Year 13 assessment in the form, for general qualifications at least, of a wholesale migration to teacher assessment, a change that in normal times would have been regarded as revolutionary, and which, like all revolutionary changes in education, would have been preceded by many years of debate and discussion and extensive piloting.

But the pandemic has changed our lives, in many ways almost unrecognisably over the past year, and we are becoming adept at expecting the unexpected.

In education, we have had to deal with schools being closed to most students, which has had multiple impacts, one being a greatly accelerated move to online learning using new technologies and pedagogies – something which would have taken years in normal times. Less positively is the exacerbation of stress and anxiety and mental health problems being experienced by some students, the loss of opportunities for learning, social interaction and development, and the significant workload increases that teachers have had to face – having to carry out all sorts of extra tasks and duties and, in some respects, having to reinvent your own roles.

This, of course, is going to be the second year where a combination of public health concerns and worries about fairness have led to exams not being made available.

I’m going to talk to you today about how we will regulate the awarding of qualifications in 2021, clarify the arrangements, and respond to some of the feedback that we have had so far.

But first of all, let’s look back on last year and the impact of COVID-19 on general qualifications, learners and teachers.

Lessons learned

One of the lessons we took from 2020 was the absolute need to have arrangements in place that we could be confident would command public support. Last year concerns about the impact of the moderation algorithm, about the appeals arrangements, and a general sense of lack of student agency led to a feeling the system was not fair. This resulted in a wholesale loss of public support.

In developing the approach for regulating the awarding of qualifications in 2021, we have made sure to listen to as many different voices as possible, and in particular, those at the ‘sharp end’. We carried out a wide-ranging public consultation jointly with the Department for Education at the beginning of this year. We received more than 100,000 responses to this – around 50,000 from students, 25,000 from parents and 10,000 from teachers – in addition to responses from school groups, teacher representatives and other professionally-interested groups. We had a team that read through all of them and the consultation responses have provided valuable input for the principles that we have embedded within this year’s approach. This process helped to surface the issues and, I hope, has engendered a sense of involvement with this year’s arrangements and how teacher-assessed grading will work in practice. We have built a system that is based on the professional judgement of teachers, and that is something I feel we should all view positively.

How it will work this year

The scaffolding that is being put in place to support you in making your assessment judgements as part of that system is designed to help support judgement being applied in a consistent way across and between schools.

I know you will all take seriously the heavy responsibility this places on teachers who find themselves not merely preparing their students for the next step of their life journey but who have also the job of allocating the exam grades that provide a passport to it. We know that teachers feel the weight of this professional responsibility and are not always comfortable with some of the moral dilemmas and conflicts it potentially creates.

That is one reason why we have worked closely with the exam boards to develop a range of materials for teachers that will provide a framework to draw on in developing and deploying their assessment strategies.

Arising out of this, there have been queries about why we have asked exam boards to make available questions from past papers instead of new materials.

The benefit of these assessment materials drawn from past papers is that teachers will also have access to mark schemes and exemplar material from previous years that will allow them to select appropriate content to use with their students. This can also help when making decisions about the grades to award to students in the absence of standardised exams.

Our aim was to enable exam boards to provide as broad a range of questions as possible so that teachers, who will know what has been taught to their students in this disrupted year, can select the questions that are most suitable.

Using the assessment materials isn’t compulsory – it forms part of the range of evidence teachers can use to determine grades. Equally, there is no requirement to use new materials to determine grades. The Joint Council for Qualifications has published guidance and support materials for administering summer 2021 assessments that will help you to clarify what we are expecting.

Grade inflation

As I mentioned earlier, I started this job on 4 January. One of the unexpected consequences and, for me, benefits of that, and also the astonishing number of meetings that can be squeezed into a single day of Zoom time, has been that I have been able to engage with a much broader range of stakeholders than would ever have been possible in normal times. That has made me very aware of the clear commitment that we all share to making sure that students this year do not suffer further disadvantage by being deprived of their right to the GCSE and A level grades that will support the next stage of their progression into learning, training or employment.

This is a collective effort involving multiple actors, many of them playing roles or operating in ways that would have been unfamiliar in pre-COVID times. We are all, as a consequence, having to cope with higher levels of uncertainty than would normally be tolerable.

One manifestation of this is worry about grade inflation. I would not wish to be drawn into anticipation of this, but would emphasise that some of the factors driving inflation last year will not apply, in particular the sense that teachers might have had, when recommending grades, that their recommendations would be subject to moderation by an algorithm and some of the challenges associated with a prescriptive approach to rank ordering.

We would also be expecting schools and colleges, as part of the quality assurance process, to identify any outcomes that look atypical – in either inflationary or deflationary terms – against previous years’ outcomes, and to provide some sort of explanation as to what might have caused that.

Overall, however, I believe that teacher judgement, supported with the scaffolding and guidance about assessment standards that have been provided by exam boards, is a trustworthy and sound basis on which to operate. I am also sure that teachers understand the responsibility not to award students grades that would mislead them into making unwise choices about their next steps.

The main driver contributing to inflation is likely therefore to be the effect of operation of ‘benefit of the doubt’. Say you have a class of thirty year 11 GCSE candidates, and five of them have produced work, on more than one occasion and under fairly controlled circumstances, which leads you to believe they are capable of getting a grade 9 on the day of the exam. In reality, we know that all five probably won’t quite manage it on the day as they may have a bad day, some problems at home or the wrong questions come up. Inevitably, it is impossible to be sure which of the five will, and which won’t.

So, acting with complete professional integrity, using the knowledge you have of normal grading standards, the range of evidence you have of their performance, and following exam board guidance, you would likely submit a grade 9 for all 5 of them. That small act of professional judgement, made in perfectly good conscience, and with good evidence, available for scrutiny if requested, will inevitably have an impact when repeated across the system, but that will lead only to some small upward pressure on outcomes, not the ‘Weimar-style inflation’ ‘or ‘prizes for all’ that some commentators have unhelpfully suggested. That seems to me an entirely legitimate consequence of deploying teacher judgement for this purpose, something that I hope will be recognised and respected in the public discourse.

I have also heard concerns expressed about undue parental and student pressure being placed on teachers to try to influence their judgement. We will be providing for the reporting of all such activity, as it is essential that teachers’ already difficult task is not made more difficult by having to deal with these additional and unacceptable pressures.

We also know there is a need for official ‘Ofqual branded’ support and information – for you to refer students and parents to, which will engender a greater understanding of what is, and what isn’t, in your power. To this end, we have just published an updated version of the Guide for Heads of Centres and we are also about to publish an updated version of our Student Guide.

Quality assurance

However, the most important tool we have to protect teachers is a robust quality assurance system that allows meaningful external checking and scrutiny to take place. These measures not only give confidence to students, parents, and the wider public, but will also give you confidence as teachers, that when you are making dispassionate and even-handed evidence-based judgements, other teachers and schools are doing likewise.

A great deal of work has gone into designing these quality assurance processes at the heart of which lies the centre policy, which summarises the centre’s approach to assessment and their approach to quality assuring the grades they award to students.

The process starts with all schools and colleges receiving a call to discuss the requirements of the centre policy on assessment and quality assurance to ensure that they are clear about what it needs to include.

Then in Stage One of the process this Centre Policy is submitted, the deadline, as set out in the JCQ guidance being in fact today.

In Stage Two, exam boards will contact all schools and colleges to let them know either that their policy has been accepted, or that they will receive a virtual visit. But please don’t wait to continue with your approach. The main aim of these visits will be to make sure that schools and colleges have appropriate arrangements in place for internal standardisation, but the focus could be on any aspect of the centre’s policy.

Stage Three then is the sampling of centre judgements. Schools and colleges must submit all grades by 18 June and exam boards will review all grades as part of their quality assurance.

You will be asked to provide samples of student work and the evidence used to determine the grades for the students selected. You will need to provide work samples from at least five students in at least one subject at A level and two subjects at GCSE, one of which is likely to be either English language or maths.

Exam boards will decide on the subjects and the students based on their analysis of the data received when TAG results are submitted on 18 June and they will let you know which have been selected, in the week beginning 21 June.

Schools and colleges will need to submit this evidence promptly – within two days of the request being made – so it’s important that evidence and records are in good order ahead of that date. This part of the process is designed to be non-labour intensive and administratively straightforward for exams officers to manage.

Exam board senior examiners and experienced examiners will then select evidence from schools and colleges, making sure that all centre types and regions are included.

They will compare results against grades from previous years when exams took place so that’s 2017, 2018 and 2019, and identify centres where the proportion of grades in 2021 appears significantly higher or lower. We are aware that results for individual subjects, especially those with small cohorts, can vary more from one year to the next. So, the comparison for a centre will be made at qualification level – for all GCSE subjects combined and all A level subjects combined.

This doesn’t mean, I should stress, that you must award grades to closely match those in previous years, or that the information from previous years should be used to artificially suppress results. That’s definitely not the case. There can be good reasons for results to vary from one year to the next, and the requirement is that you record and explain the reasons for any substantial variances, in line with your policy. The quality assurance checks will focus on whether the range of evidence supports the grading judgments for those students.

For the majority of centres, where there is no reason for further scrutiny, this will be all that they need to do.

However, if exam boards do have questions about how grading decisions have been made, they will then contact the school or college. Where there are serious concerns, they could ask for more work to be provided. And, in some cases, if the evidence does not support the grades submitted, centres will be asked to reconsider their grades. Where the exam board disagrees with the centre’s grade, they can ultimately withhold results.

I know concerns have been expressed about the administrative burden associated with these arrangements and am very conscious of the amount of work we are asking teachers to do at a time when you are already under considerable pressure but I am also acutely aware that robust quality assurance processes are central to reassuring students and parents that results are fair, and hope that they will also reassure you that others are acting with the same integrity and even-handedness in awarding grades.

This, I hope, will allow us to avoid a recurrence of the febrile atmosphere experienced last summer. News headlines such as ‘The Story of the Big U-Turn of the Summer’; ‘Exam Grades in Chaos, Teachers in Despair’; ‘Students’ fury as grades marked down’; and ‘Cruel A level results inequality exposed’, made for difficult reading and adversely affected the morale of everyone involved in awarding qualifications – we are, after all, only human.

That being said, I recognise the headlines are never going to be glowing about a process that in many ways is simply the best we can do under the circumstances. But that’s not what is important. What is important is being able to restore public confidence in the grades that our young people receive. What is important is being able to look ourselves in the eye and feel confident that we have done everything we can to make those grades reliable, fair and a true representation of achievement and ability. The quality assurance process that we have set out will help us to do just that. It tells you that the school or college down the road is playing by the same rules. And it tells the world at large that the system has confidence in the decisions that you, as teachers, have made.

Appeals

And now onto appeals – although last year these attracted much publicity the actual numbers were much lower than a normal year. When I checked our records, I noted that there were around 300,000 appeals in 2019 resulting in around 70,000 grade changes. This compared to under 10,000 in 2020, though I recognise this is not a like-for-like comparison as the grounds for appeal were much more restricted.

Our consultation on how appeals will work in 2021 is live on our website now and is open until 5 May. We are seeking views on proposed guidance relating to: a learner’s right to a review and appeal; grounds of appeal; procedural and administrative errors; academic judgement; and correcting results.

I would not want to make estimates for what we should expect this year, where the system is again slightly different, but I do think it is worth emphasising that many of the normal incentives that encourage high levels of appeals will not apply. In particular, because students will be receiving holistic grades based on teacher judgement, there will not be the usual pattern of students entering speculative applications for re-marks on the basis that they are one or two marks off a grade boundary.

But please take a look at our consultation – as I said, you have until 5 May to respond.

Looking ahead now to the summer and results days – these will be the culmination of all the extremely hard work that you have all done over the year and the outcome of the trust that has been placed in the system, and particularly our teachers, to do its best for our young people.

A levels and GCSE results will be coming out in the same week, so you may already be considering what that will look like in practice. Students in all likelihood will be receiving their results using Apps rather than coming into the schools and colleges, and they will be sharing their outcomes with friends on social media. But although results days will look different on the surface, it will be, as always, an anxious time for students. They will continue to need your support, and no doubt many of you are thinking about how to scale up the pastoral care that is offered during that week, and how you can make sure that an appropriate level of advice and guidance is on hand to enable students to make the right decisions and move on with their education or start their careers. Where they go next is a collective societal responsibility, and it’s for universities, colleges and employers to understand that, and to work together to support them as they take their next steps.

2022

I have focused mainly so far on 2021 and we are in the thick of that at the moment, but we are also turning our thoughts to the future. The 2022 Year 13 cohort is evidently going to be unusual in that it will be coming to A levels never having before sat public exams and both Years 11 and 13 will have suffered high levels of learning disruption. Things certainly won’t be ‘business as usual’ for some time to come.

Discussions about arrangements for 2022 are already underway. Longer term, however, there is clearly going to be scope to reflect on what we have learned during this time and what implications it might have for assessment. I am thinking especially of the large scale of adoption of technology and online learning and its integration into pedagogy and whether that will ultimately have a washback into assessment. I am also hopeful that a successful experience this year will allow us to reflect in a more substantial way on the role of teacher judgement in assessment and the contribution it can make.

Closing remarks

Turning back to this year I am struck by the coincidence that the date for submission of teacher-assessed grades is 18 June, two days after ‘Thank a Teacher Day’ on 16 June. That will be an opportunity to celebrate and acknowledge the work you and your colleagues have done in helping to lead our young people through an incredibly tough year.

In anticipation, I join in offering you my sincere thanks for that.

Simon Lebus, Chief Regulator, Ofqual