Pioneering assessment pilot leads the way in ethical AI

First-of-its-kind research into the use of artificial intelligence (AI) in formative assessment has shown clear benefits for both learners and educators. 

The pilot project, supported by NCFE’s Assessment Innovation Fund, saw Bolton College use its FirstPass platform to investigate the use of trained AI to assist learners with answering open-ended questions, as well as the impact on teacher workloads.

Results showed that nine in 10 learners (92.3%) found the feedback they received from FirstPass helped them to compose better answers to open-ended questions, and 82% of teachers believe it’s an effective service for supporting formative assessment. 

Aftab Hussian, ILT Manager at Bolton College, said:

“We wanted to assess the efficacy of training the AI models to support the delivery of open-ended questions, and we wanted to assess if the AI model that underpinned the FirstPass platform could support the students with real-time feedback as they compose freeform text responses to open-ended questions which have been posed by their teachers. 

“One of the things we did on the platform was to demonstrate that an AI education formative assessment service could use a crowdsourcing model, and we also demonstrated that multiple teachers and students could support the training of the AI models that underpin formative assessment practices.

“Without the Assessment Innovation Fund, and without the support of NCFE, we wouldn’t have been given the opportunity to demonstrate this groundbreaking research. The fund has allowed us to demonstrate that AI services, like FirstPass, can be developed, trained, used and sustained to benefit students and teachers.” 

The FirstPass pilot involved six colleges, with teachers and subject specialists across the UK working in a coordinated way to train machine learning models which underpinned the delivery and assessment of open-ended questions to students. 

These machine learning models supported the delivery of real-time feedback to students as they composed their free-form text responses to open-ended questions. This reduced the delay in receiving feedback, which can be over a week depending on teacher workload. 

For teachers, while most value the use of open-ended questions for conducting formative assessments, marking and providing feedback to these responses is time consuming. When assessing large numbers of students, human factors, such as bias, fatigue and inconsistency, can add to the complexities of conducting these assessment activities. 

FirstPass allowed for the evaluation of these questions in a more detailed and objective manner, as well as in a far shorter time. 

The awarding organisation and educational charity NCFE is dedicated to promoting and advancing learning and breaking the boundaries of what’s possible within education. Its pioneering £1m Assessment Innovation Fund, launched in 2021, provides evidence-based, alternative assessment solutions, where the impact can be tested in real-life within a supportive environment.  

More than 3,000 learners and over 200 educators have already participated in 12 funded pilots to date as part of the fund, exploring the use of VR and AR in assessment, digital badging, the effects of immersive and interactive story-based assessments, and online platforms.

Gray Mytton, Assessment Innovation Manager at NCFE, said:

“The FirstPass platform is a fantastic example of innovation in the FE sector and the potential of artificial intelligence in education when it’s used ethically and responsibly.

“Bolton College has developed this solution to support their learners and teachers on their learning journeys, and we’re proud to be part of the project that opened the platform to other colleges for the first time. The feedback from teachers and learners about their experience has been incredibly positive and we’re excited about the further work we have planned to make FirstPass even better.”

Bolton College is set to continue its work with NCFE through a second phase of the project where it will look to further develop and test FirstPass – supported again by the Assessment Innovation Fund. This research will focus on formative assessment in two specific courses; the Level 2 Diploma in Early Year Practitioner and the Technical Award in Child Development and Care at Levels 1 and 2. 

Additionally, Bolton College will be funded by the Education Endowment Foundation (EEF) to undertake a small-scale pilot with 8 – 12 further education colleges. The pilot will assess how FirstPass can support formative assessment practices to aid learners within GCSE English Language resits. If the project is successful, the EEF will increase its funding to support a larger scale project during 2024-25.

For more information about the initial pilot and to read the final report, visit here.