Cirrus Assessment x ICAS: ChatGPT and Exam Integrity

In a digitally saturated education landscape, navigating the labyrinth of e-assessment and AI seems daunting. The Cirrus Assessment x Institute of Chartered Accountants of Scotland webinar “E-Assessment in 2023: ChatGPT and Exam Integrity” held on 21 June aimed to demystify this topic, with a particular focus on e-assessment tools to protect the integrity and the implications of AI in exams – the good, the bad, and the ugly. This dynamic panel discussion generated much discourse among educators globally and featured insightful presentations by three seasoned industry experts.

The opening act of the webinar was delivered by Martijn van der Jagt, Cirrus Assessment’s Account Manager. With a keen understanding of online assessments, Martijn presented an engaging session about the “Tools in the E-Assessment Toolbelt”. He offered a look into the various tools that e-assessment platforms, such as Cirrus, (should) have under their belt when tackling AI-assisted cheating, such as with ChatGPT. 

Following Martijn, Vegard Sivertsen, CEO of Cirrus Assessment, took the virtual stage. Known for his pragmatic approach to technology, Vegard discussed how educators could cautiously embrace AI in his presentation, “AI in Exams: Opportunity or Threat?”. He carefully examined what needs to transpire for AI to be successfully incorporated into exams, discussing potential challenges, opportunities, and future directions.

The presentation was concluded by Robert Burns, the Delivery Lead Assessment at the Institute of Chartered Accountants of Scotland. With a wealth of experience in managing assessments, Robert’s enlightening talk, “The Future of E-Assessment with AI,” offered a glimpse into what lies ahead. Despite addressing the concerns surrounding AI in education, Robert’s forward-thinking approach was clear: AI will revolutionise education, and we should anticipate this change.

In the coming sections, we will break down each presentation, unpacking the ideas and perspectives these thought leaders share. Stay with us as we dive into a future where AI is a prominent actor in e-assessments, turning education on its head – in the best possible way.

Martijn van der Jagt: Detect and Prevent – Tools to Watch

In his insightful presentation, Martijn delved into the complexities of preventing academic dishonesty through various “tools” in your exam platform toolbelt that not only detect AI cheating, but stop it in its tracks. He shed light on the transformative potential of employing multiple or random test forms, advocating for unique exam structures for each candidate to deter collaborative cheating. Additionally, he proposed time-based questions as an effective measure to curtail the usage of external AI tools like ChatGPT. The premise is simple but effective: if questions are timed, candidates are left with limited scope to resort to dishonesty.

Martijn also introduced the audience to a myriad of advanced cheating detection/prevention features. He emphasised the importance of data analysis in invigilation logs to monitor suspicious activities. Turning to secure exam delivery, he discussed various measures to preempt using AI-generated content during online exams, such as the built-in Safe Exam Browser and remote proctoring techniques.

These include AI Proctoring and Live Proctoring, which employ automated AI algorithms or live human proctors to ensure exam integrity. He presented the potential of harnessing tools such as fullscreen mode and clipboard disabling to prevent candidates from navigating away from the test or copying and pasting content. These, alongside the introduction of new features like disabling new tabs, single-screen use, and cache clearing, add another layer of security.

Martijn also mentioned Proctorio’s planned improvements, designed to whitelist specific programs, websites, and applications, thereby eliminating any unapproved resources. He concluded by highlighting the emerging domain of plagiarism and AI detection, citing the successful integration of Turnitin’s AI detection features into Cirrus’s platform, which consistently flags plagiarised, copy-pasted, or AI-generated content.

Martijn is always more than happy to give a demonstration of the Cirrus platform and all of these features in Cirrus; contact him at [email protected] for more information!

Vegard Sivertsen: Fostering a Culture of Integrity

Martijn handed the mic over to the next speaker, Vegard Sivertsen, CEO of Cirrus Assessment, who shifted the lens to focus on using AI within examination settings. Sivertsen proposed an approach that didn’t solely aim at restricting AI tools but instead focused on its potential benefits. He raised the intriguing possibility of incoporating AI tools like ChatGPT into exams, provided that candidates accurately quote any copied text. This approach facilitates a harmonious merging of AI into exam procedures, encouraging students to leverage advanced tools responsibly to enhance their learning and assessment experience.

Vegard also discussed a diverse strategy for exam design. He recommended blending exams with both timed questions that require readily available knowledge, and open-ended questions where the use of AI could be allowed. This fosters an environment where students are compelled to apply higher-order thinking skills, including analysis, synthesis, and evaluation, rather than merely relying on factual recall. Additionally, Sivertsen stressed the importance of clear communication and setting expectations to help foster a culture of academic integrity. He reminded the audience that academic integrity isn’t just about curtailing cheating, but also involves promoting ethical and honest behaviour in all aspects of the educational process. This perspective underscores establishing a value-based, technologically adept educational framework.

“It’s important that you communicate clearly with your candidates, what the expectations are, what the rules are, etc, and what is expected of them. Not only during the exam, but as a workforce later in life. And that you foster a culture of academic integrity, that you actually focus on making sure that candidates have integrity during exam delivery instead of trying to fight the use of AI. Try to find ways to work with it.” –Vegard Sivertsen

Robert Burns: AI – Gathering Storm or The Parting of Clouds

In his session, Robert Burns, the Delivery Lead Assessment at the Institute of Chartered Accountants of Scotland, navigated the audience through the potential and pitfalls of AI in exams. His presentation, aptly titled “AI – Gathering Storm or The Parting of Clouds,” dived into how AI can enhance the validity and reliability of exams, provide swifter and superior feedback, and dramatically increase the efficiency of exam creation and maintenance. However, he also cautioned about the risk of introducing new biases into assessment content, reduced transparency in assessment design, and potential ethical and legal issues surrounding content.

Robert outlined how AI can be employed to improve the validity and reliability of exams. Through automated grading, adaptive testing, and personalised feedback and guidance, AI offers an innovative approach to evaluation, enhancing its accuracy and relevance. He envisioned a future where AI aids in designing and delivering a more inclusive and interactive learning and assessment style, with tailored feedback and support for candidates – introducing game-changing exam environment, and a more personalised candidate experience.

However, he didn’t shy away from addressing potential fears and challenges. He acknowledged concerns about AI replacing human tutors, eliminating jobs, and making it difficult to detect cheating. He provided a balanced view, emphasising that these apprehensions must be carefully managed as we move forward with our AI journey.

“Change is always quite frightening.  It wasn’t that long ago that everyone was throwing their hands in the air in horror at the notion of digital delivery of exams, and some thought it would dilute the qualification. Others thought it would, perhaps, reduce the integrity of the assessments.

But then Covid came along and it forced us all to a place that we were possibly quite uncomfortable with. And if you look at back now, at the way that the assessment world has evolved in the last couple of years, how it’s embraced digital delivery, the transformation has been quite incredible. And I do think that it is the potential that the AI could bring the industry towards just as big a transformation, if not bigger” – Robert Burns

AI has the incredible potential to transform education.  Although we must protect exam integrity with features that detect AI-assisted dishonesty, or prevent it altogether, we also must evolve with the rapidly changing landscape.  Finding ways to embrace and interweave AI into the fabric of education means that we can be agile with AI, and continue to innovate and reform assessment for the future.

Summary

“The Cirrus Assessment x Institute of Chartered Accounts of Scotland webinar explored AI’s role in e-assessments and maintaining exam integrity. Martijn van der Jagt delved into prevention tools against AI-assisted cheating. Vegard Sivertsen discussed embracing AI responsibly in exams, fostering a culture of academic integrity. Robert Burns navigated through potential and pitfalls of AI, envisaging a future with personalised AI-enhanced assessments, while acknowledging potential fears and challenges. The goal is to protect exam integrity, while embracing and innovating with AI.”