Guests Discuss The Design and Implementation of Online Exams on The Score Podcast
Episodes 009 and 010 of the academic integrity podcast The Score features Kylie Day and Sarah Thorneycroft, leaders in the field of design and implementation of online examinations. Both guests work at the University of New England (UNE), in Australia. Kylie Day is the manager of exams and e-assessments, and Sarah Thorneycroft is the director of digital education.
Host Kathryn Baron (@TchersPet) speaks with both about their process and experiences in developing exams and assessments at UNE in Australia.
This 2-episode series is combined into one below. High points of the conversation follow. Listen to their full episodes on Apple, Spotify, and The Score Website.
Note: Removal of filler words and minor edits have been made for clarity.
PART 1 – Episode 009
Kathryn Baron (06:19): And so with the flexibility that you have in testing at UNE, what’s the process? How does it work?
Kylie Day (07:06): … We put our effort towards the student’s feelings and attitudes and decisions before the exam ever starts. So, in the same way as a community safety program or a community health program, you would do population-wide communications to talk about the risks involved, expected behavior, alternatives to risky behavior.
In the same way that the highway patrol police are not expected to catch every single person who might speed, they have a presence and that serves a purpose to make it risky, to dissuade people from speeding.
But that’s not the only thing that one would do if you wanted to reduce, say the road toll or the incidents of people breaking the road rules. You would also expect to have a community safety program and narrative happening along with that. And when we catch people who might be cheating it’s not a good outcome for them, it’s not a good outcome for us as an institution.
Sarah Thorneycroft (08:53): Reframing from catching cheating to understanding cheating is important, I think. And I know we’ll talk about risk taking a bit later on as well. But just to build on what Kylie said, I think the notion of cheating as risk taking behavior, it’s important to understand the decision making and the cost benefit analysis that students do that end up with cheating being the least risky option.
Kylie Day (09:36): The example of how flexibility can prevent cheating. A person in an insecure employment situation…can choose between paying the rent, not paying the rent and getting their assignment done, failing, or cheating. None of those options are very pleasant. Or they can get an extension on that assignment.
Kylie Day (10:05): … We see flexibility and easy flexibility as a key factor in letting students manage their own pressures in ways that allows them to succeed and not have to cheat to do that.
Sarah Thorneycroft (10:17): That changes the cost benefit analysis.
Kylie Day (10:20): So, we work with an online exam proctoring service where our exams live in our learning management system, but we have highly skilled and trained supervisors who can… They have a view of the students’ screen. They can use software to lock down that student’s computer in ways that we ask them to, and they can also watch the student.
Kylie Day (12:01): And that’s the first thing that our faculty said when we started having conversations about flexibility, flexibility is an F word, if I can be cheeky. Students will cheat, and so that’s when we talk about design. The assessment needs to be designed in the mode or in the context of the mode that it’s held. It should not be that we are just doing paper exams on a web page, it’s a whole second order change.
Kylie Day (12:31): So, the design features might include using a question bank. So you would have just enough. I can get a different question to you. It’s still the same topic, same degree of difficulty. But if I say, “Hey, what did you put on question one?” That kind of collaboration will be disrupted because we get different questions.
Sarah Thorneycroft (15:12): This is where it’s really useful to help people make comparisons between the paper examination paradigm in which somebody is watching them, and often in more embodied ways of walking up and down and patrolling the physical room that people are located in. But we’ve also discovered, because online the proctor and student relationship is one to one, whereas in an exam hall it’s one to many. Yes, that proctor is watching because that’s the cultural condition for examinations that we’ve agreed on regardless of where they’re held.
Sarah Thorneycroft (15:49): But the proctor can actually also provide support in situ, which can be both technical support or general encouragement. And we’ve had a lot of comments come through student evaluation that actually talk about how helpful and supportive the proctor was. So that’s one of the key reasons that we focus on human invigilation, not AI-only invigilation, because of that personalized element and the ability to also provide benefits, not just stress and monitoring.
Kathryn Baron (22:57): Do you have online practice exams to help students as well?
Kylie Day (23:05): We do, and that’s one of our favorite things. We call it a ‘try it out exam’. And you have to book it, it’s supervised. You have to follow the rules, but it’s got questions like, Hey, did you know this is where you can see the countdown clock on your screen. Or a question that suggests that you change the batteries in your wireless mouse or keyboard before your exam and do all your windows updates. It’s instructional around, how do I have a good time in my online exam? It has a thing to draw us a graph, which you can do, showing the correlation between the amount of caffeine that you consume compared to the amount of assignments you have due.
So it’s intentionally lighthearted, but it allows a student to work out what buttons do I have to push? How does this thing work out? What does it feel like? What does it look like? What do I need to do in my own space to conform to exam conditions? And will my computer actually sustain the technical requirements and the bandwidth that I need?
Kylie Day (30:31): What rings in my head a lot is the phrase demonstration beats explanation. So just starting with people who wanted to come and play really and making sure that went really well. Those people then become champions. You can publicize details and say, “You know what? We can talk all we like, but we tried it and this is what happened.” And having evidence to show people.
Kathryn Baron (32:44): What are the concrete steps that these other universities can take?
Kylie Day (32:48): One of the pieces of advice I give to people at other universities is that they should not consider it to be an IT project, nor should it be seen as an admin logistics project. That those pieces are really important, but the structure of the team I think is one of the reasons for our success in doing it.
Sarah Thorneycroft (33:10): Yeah, I think I tend to frustrate my sector colleagues who hope that there might be a nice recipe of concrete steps and you just follow the steps and then it works, and it’s all good. And they come and talk to us and we are like, “Oh actually it’s a cultural change piece.”
PART 2 – Episode 010
Kathryn Baron (01:46) …Now, let’s talk about how it works. The tests are run through an exam center team. My first question is then how does that work exactly and is it just at UNE or is this the process that’s used across the country at universities and colleges?
Kylie Day (03:58): … We do have a central team and that’s been a feature at Australian universities for a long time. But what we’ve seen at other universities in Australia lately is that it’s being distributed back out to academic areas. And I think I would say that’s a loss because I think it requires professional expertise to run what is probably the largest event a university will hold, high stress, high stakes, high numbers of people, really, really quite important.
And to pull that expertise in terms of how do I wrangle 10,000 people without making them cry, to be a little bit cynical, but that’s a skill. How do I communicate with people to achieve compliance with lots of different rules? How do I get people to actually do what they need to do so that everything coincides nicely for everyone and everyone has a good experience and how do I manage academic integrity issues well? I think distributing that out to academics who already have plenty to do, might not be their area of expertise, but to outsource that to them as well. I think you lose something there.
Kylie Day (07:43): Covid helped us because we were at about 25% online exams before covid, in the before times. And then we had a very rapid shift to 100% of all exams had to be held online with a 24-hour window in the online proctoring. So that really helped tear the bandaid off. And I think it helped people just take that step that they might not have been keen on doing. What we, my team, put a lot of effort into was to make it really safe for them and massive amounts of support for students and for staff, so that nothing was too hard and that nothing went badly. And that’s why we put effort into being on call till 1:00 AM so that there were no stories from students about how they were just left at midnight with no one to help them. And I think that really helped. And when we did have people who wanted to be a bit innovative, we went out of our way to support that.
And so those then became the stories, the good examples that we could say, hey, your colleague tried this and here are the metrics where we can see that student success increased. Students are happier. Students have more agency over all the demands on themselves. So they’re much more settled and more engaged. And just supporting that in a really safe way with a lot of support. The whole flexibility piece did take a lot of time for people to get their heads around. And I think that exams exist as a cultural archetype, that they’re hard, they’re tricky, they’re secret, they’re tough. You have to turn up or else, all this stuff that people have embedded in their brains about exams. Helping people realize that the way exams have been managed in the past is not necessarily the way exams should be managed and really calling into question every assumption that people have consciously or unconsciously about assessment and exams and flexibility and students. So it really has been a long change piece.
Sarah Thorneycroft (10:45): Access too is key for students that don’t have to engage in geographical travel to get to locations. That can sometimes be a real barrier for our demographic. So being able to access online in your own home makes a real difference for a lot of students.
Kylie Day (11:02): We had a student early on who actually rang crying tears of happiness and no one rings, right, to say what a wonderful exam they’ve just had, right? It’s an occupational hazard in our line of work that you only ever hear from people who have a bad time, but this student rang early on in the project when she really realized the flexibility that she could have.
She rang, crying tears of happiness to thank us to say that she had a spinal injury, which meant she was in chronic pain. Traveling was really hard and would make her really unwell with pain. And that she asked for a comfortable chair, but our idea of comfortable chair was not the same as her idea. And we couldn’t provide what she needed in the exam venue. When she realized that she could do it, she had three exams in two days, and she physically was not going to be able to do that at an exam center. Which meant that she wasn’t going to finish her degree, which meant that she wasn’t going to be able to get that job that she had lined up, this dream job.
Once she realized that she could actually choose the timing of her three exams and sit one on a weekend, sitting in her lounge chair, which was much better for her and lay down if she needed to, she realized that she could access those exams. She could finish her degree. She was going to get that dream job that she’d lined up and that moved her to tears and probably moved us to tears a bit too when she rang to tell us that. So exams are an institutional barrier. Traditional exams are an institutional barrier to accessibility.
Kylie Day (13:13): Certainly easier to get those metrics in an online assessment mode rather than paper. From my perspective, we do a survey after every exam period to say, how was it? Which bits were good, which bits were bad? Why did you like it? Why didn’t you like it? What impact did it have? And we also get various other pieces of feedback. And what we know is that students really appreciate being able to choose a time that suits them. They don’t like having to sit in an exam hall with 300 other people, sniffling and tapping and wobbling their desks. They don’t like having to travel, but I think Sarah can speak on the kind of metrics that you could get that would influence design.
Sarah Thorneycroft (13:58): So in terms of designing our approach, getting metrics around when students choose to have their exams is really useful, because you can actually see the uptake of flexibility and understand when you make this available to students, how are they making use of it? And thus, to what extent you want to make sure you’re designing your assessments to maximize that capacity. And some of the other metrics, I know that some of the ones that we use a lot are around things like the test taker experience. So this isn’t necessarily about the design of assessment. A lot of the most effective actions you can take for assessment design are the things that don’t look like assessment design. Metrics around the test taker experience in terms of satisfaction, technical issues, academic integrity issues, the incidents of actual confirmed breaches and that kind of thing.
When you’re talking about an academic, or you probably use the term professor, who’s talking to a student who had a bad experience in the exam, that’s really easy to understand as oh actually online exams are bad, but understanding that out of 10,000 exams, 85 to 90% of students are having a really positive experience.
Kathryn Baron (16:22): So how expensive is your system?
Kylie Day (16:25): It’s on a par with external exams per exam sitting. I think it’s actually these days, it’s probably a lot cheaper because we had 400 exam venues worldwide and renting a room at the Australian embassy in Paris might cost 600 Euro for a day, but the bigger church hall in Australia, they charge you 50 cents electricity a day. So it’s hard to really come up with a round figure, but it’s comparable with off campus exams, indirect costs.
Sarah Thorneycroft (18:26): The intangible costs are an important part of the conversation. In terms of dollars for instance, it’s reasonably more expensive than our learning management system, just as an example. But the key thing is because human individualization, human proctoring is a key part of our strategy. It’s not a platform cost, it’s people, it’s people cost. So I think it’s important to contextualize that way so that it’s not a really expensive piece of technology. It’s actually a part of a whole ecosystem and it’s paying for the human experience.
University of New England (Australia)Listen to the entire episode 010 of The Score with guests Kylie Day and Sarah Thorneycroft here
Responses