The Score Podcast Episode 005, Guests Mark Biggin and Karen Symms Gallagher Talk about Detecting Collusion and Awareness
This episode features Dr. Mark Biggin and Dr. Karen Symms Gallagher (@USCRossierDean). Biggin is an affiliate of the Biological Systems and Engineering Division at Lawrence Berkeley National Lab. And he created a tool – Q-SID that can detect collusion which he has offered to faculty for free. He also teaches at UC Berkeley.
Gallagher is a professor and a dean of 20 years at the Rossier School of Education at University of Southern California. She is also a Veronica and David Hagen Chair in Women’s Leadership at Rossier School. Her recent opinion published in the LA Times drew much needed attention to the issue of academic integrity in higher education.
High points of the conversation follow. Listen to the full episode Apple, Spotify, and The Score Website. Note: Removal of filler words and minor edits have been made for clarity.
Kathryn Baron (02:30): Dr. Biggin, in an article in Forbes by Derek Newton, you described yourself as being naive about the amount of cheating in online tests. What happened that revealed this dark side to you?
Dr. Mark Biggin (02:41): Oh, oh, direct experience from a class I was teaching. During the lockdown, suddenly we were giving exams that were unproctored online as opposed to proctored in person. I assumed that if we just told the students to follow the honor code, they would do that. I didn’t imagine many students would cheat, but the readers, for the first time I was doing this, pointed out they found two students who obviously had copied their answers. They were very similar.
From there, being me, somewhat an analysis person, I started doing a statistical analysis and I found that some of the students had very unusually similar question scores. They got the same scores for many questions, and so the greater scores. When we looked at the written answers of those students, we found that many of them had cheated. And the students that we challenged, most of them confessed. Through some iterative process, we kept finding more and more students who cheated. We eventually found that in that particular class, it was the worst case we had actually. 19% of the students in the end, we found had cheated. I was floored. I kept saying, “Oh, I found say five or six groups, 15/17 students and, oh well, that must be most of them.” Then one of the students who cheated said, “Oh, no, no, no, I bet there’s more than that.” That student was right, and it just kept going. That was my entrée.
Kathryn Baron (04:14): Dr. Biggin, that statistical method you developed is called Q-SID, which stands for Questions Score Identity Detection. It detects collusion amongst students during a test. How were students sharing information?
Dr. Mark Biggin (04:29): It’s important to say these were online exams which were unproctored. They were actually open books so students were allowed to look at lecture notes, but that wasn’t sufficient for some of the students. What they do is they first go through and answer all the questions they can answer and then they collude by just literally sending an email with, or in some way, a text, whatever, literally the entire exam that they’ve written.
They copy those answers from their colleagues – that they didn’t know the answer to. It’s just wholesale copying. When you look at some of the copied answers, such as a chemical structure, they copy it minutely. It’s not that they … They’re really just blindly copying. This is not an intellectual collaboration. This is blind copying.
Dr. Mark Biggin: (06:48) I should say, I do think it’s important to say that Q-SID should not be used on its own to accuse a student of cheating. You use it to solve the needle in the haystack problem. It points you to students who are very likely to have cheated, but you then have to go and look in detail at their written answers and determine, are those written answers clearly copied as opposed to just two students who happened to have got very similar scores?
Dr. Karen Symms Gallagher (10:40): In the last couple of years, we’ve had over a thousand cases that have been referred to SJACS, [ the student run process for academic integrity cases at USC] not exclusively for cheating, but between the fall of 2019 and the fall of 2020, there was 115% increase in cheating, reported cheating by faculty. Most of it was what we call contract cheating. It was collusion, looking up answers during a test. Again, a lot of unproctored tests. It was, like many universities, we rapidly went into online education through Zoom and we saw this increase of reported cheating.
Kathryn Baron: (11:50): Dr. Gallagher, you write, “What surprised me most as an educator playing this cat and mouse game for decades is that cheating is now scaled and outsourced internationally and powered by venture capitalists, Wall Street investors and billion-dollar companies.” Would you tell us what you found about the scale and sophistication of these businesses?
Dr. Karen Symms Gallagher (12:10): We know the saying, there’s an app for that. Well, when it comes to cheating, there are hundreds of apps for that. That is because contract cheating, which really is outsourcing answers or essays for an exchange of money, is very lucrative. In the last 10 years alone, there has been a sizable investment by venture capitalists in apps that clearly are cheating apps. I mean, they say they’re for homework help, but they’re inexpensive.
They’re like signing up for Netflix so it’s possible for almost any student to use these apps. Now, to be clear, we’ve had cheating like this. We’ve actually had what you call contract cheating, but it was usually something that students who had the resources, either the money or the ability to find [ones] people have used. But with these apps… And as an aside, I’m not naming any of them. I don’t think that’s the important part of this. It’s just that it is very lucrative for these for-profit ed-tech companies.
They advertise on social media so that students are inundated, whether on Facebook or Twitter they get advertising for this. It’s in a kind of advertising that appeals to students about how overworked they are, how awful COVID is, let us help you. We can not only help you with your answers to your math problems in the STEM fields, but also we can write that essay for you with little as a couple of days’ notice. In exchange for money, we can have someone write that five-page paper for you, all the way up to we can have them write your research paper. Again, it’s the number of commercial cheating apps out there and their ability to advertise in places that they’ll find students.
Kathryn Baron (17:28): Because there’s a lot of harm brought by cheating, not just for the students who don’t learn and may even suffer some emotional consequences from resorting to cheating, that’s for the reluctant cheaters, but there’s also harm to universities and society. Dr. Biggin, this is one of your major concerns, especially because some important courses are more prone to student cheating.
Dr. Mark Biggin (17:49): We get a median of 4% of students cheating across all the exams we’ve looked at, but it varies greatly by class. Some classes seem to have a more persistent problem, probably because they’re considered by the students to be high value because they’re important. One that I teach is required for medical school and that’s one we’ve had the biggest problem with. That speaks to – I think it’s telling you to some extent why students are doing this.
Dr. Mark Biggin (18:52): I think academics are to some extent a little naive at ignoring that [economic] incentive. It’s an enormous effect. As to the harm done, we’ve already discussed [that]. I think If students know that other students are cheating, although only a minority cheat, the rest of the student body are aware this is happening, particularly during the pandemic. But if you have a lot of online courses or those courses where people can cheat and do cheat, the other students know. If the administration and the faculty aren’t making what are perceived to be sufficiently effective attempts to mitigate and stop that cheating, it creates a pall over the environment. Sort of the sense of trust and comfort with the system is corroded somewhat.
Dr. Mark Biggin (20:30): Well, because in most classes the students are judged relatively to the other students, for every student who goes up a grade, another student who didn’t cheat goes down a grade. If 10% of students are cheating, that’s 20% of the grades are inaccurate. 10% got a grade too high and 10% a grade too low.
Dr. Karen Symms Gallagher (21:47): Yes. I agree very much with Dr. Biggin, that we have to do something as administrators and as faculty. We cannot let students prosper from cheating. In the long run, if we erode the belief in the academic integrity of a college, a school, a department, we all suffer.
Dr. Karen Symms Gallagher (22:19): That is probably the most insidious part of cheating in general, but these contract cheating, these companies, these websites, these applications that are flagrantly selling cheating kinds of services. It is up to us. I think we, both administrators working with faculty and working with students, because the other students … I mean, it’s right. Other students don’t want the cheating to go on. They know it not only harms them on a grade, but in the end, it can harm the value of the degree that they get.
Dr. Mark Biggin (25:06): Well, the physicists are always telling us, if you haven’t measured it, you haven’t understood it. The first thing you have to do is measure the amount of cheating, the different forms of cheating that are occurring and know when and where it’s occurring.
Dr. Mark Biggin (26:55): We’ve been able to reduce cheating by about twofold, by informing students in advance of the method, and actually showing them the website. And the website has a specific page addressed to the students explaining that our goal is not to catch them, but to dissuade them and to tell the other students, “We’re doing this. Don’t feel threatened. We’re doing this to make sure you get the grade you deserve.”
I assumed cheating would plummet. It dropped about twofold. I still had 7% in my class this last summer collude, even with all that information and telling students. The website explained to them, “You’ll be caught if you cheat.” And they cheated. In fact, one of those students had been caught in the term before in the class, was failed, took the class again, cheated again, and was caught again.
Dr. Karen Symms Gallagher (32:08): I really want to pick up on this notion of measuring it. I found out about the 115% increase in SJACS at USC through a student publication. We do not publish what’s going on at USC in our handling of student disciplinary actions, nor do most universities. In fact, I went through several student newspapers to find that there’s been this increase, since the pandemic forced most classes to be online. Well, if you don’t know what’s going on, you’re both unaware, but also there’s not much we can do about it until we recognize it is an issue. Measuring it, that is seeing again reported issues. A lot of cheating does not get reported because faculty members say, “I’m the bad person if this happens. Students will give me bad reviews on my end of the semester [reviews].” It destroys the teacher-student relationship.
Responses