Detecting AI "plagiarism" and other wild tales

 

popular Scooby Doo meme where the professor (Fred) unmasks the evil doer (cheatrers) to find out that the cheaters where the professor's own insecurities
If only it weren't for those darned kids!

Admittedly I haven't been blogging a lot these days. I keep meaning to come back and actually get into the habit of writing more frequently, but as one of my Twitter acquaintances once observed, you make a note of it to come back to, but then lose motivation (loosely paraphrasing Matt Crosslin - I think).  In any case, à propos of TurnItIn's announcement this past week that they will now have an AI writing detector and AI writing resource center for educators as part of their offerings (wooooo! /s), I thought I'd spend a few minutes jotting down some thoughts. Warning: I am a  bit of a Dr. Crankypants on this one...

If you haven't been paying attention, the early research is out on these kinds of detector schemes. People have been playing around with ChatGPT and AI author detectors and the results are in. These detectors just aren't good. Even GPTZero "The World's #1 AI Detector" (🙄) isn't all that accurate.  Change some words around, paraphrase, change a bit of the syntactic structure, and boom! Something that was flagged as likely to be AI-generated is no longer marked as AI-generated.  Furthermore, texts that aren't AI-generated, are falsely flagged as AI text for...reasons.  Who knows how the system works? 🤔🤷‍♂️.

All this really rubs me the wrong way, as an instructional designer, as an EdTech person, and as someone who works in higher ed.  For reasons that I won't go into too much detail, my work requires students to submit their capstone examinations through TII.  The capstone examination is a culminating analytical essay (with citations and appropriate argumentation) that is undertaken once students have completed their MA coursework. You must pass this exam to earn your MA. As my department's EdTech Guru Extraordinaire, I administer this exam twice a year, so I get to set everything up in the LMS. 

The capstone exam is a stressful time in the lives of students because even though many complete their courses with flying colors, they cannot graduate until the capstone essay is deemed acceptable (based on the assessment rubric that is shared with learners ahead of time).  Students have two chances to pass, or they never receive a degree. Even if you are confident in your knowledge (like I was back in 2010 when I took the exam), you still feel the jitters and nervousness.  My hands were shaking going into the computer lab. Back then you had four hours to answer four essay-style questions, so restroom breaks were also on a real "must go!!!" basis. Questions such as What if I don't pass?  What if I fail twice? Two (or more) years wasted swirl through your mind.  Luckily the exam today is a bit different than my days, so students have more time, open books, and more opportunities to write something broader with more authentic prompts.  Still, those questions about "what if I don't pass?" remain.  TII, the run-of-the-mill, non-AI-enhanced variety, still provides false positives. it frikkin' flags the name of the university because it...(wait for it) appears in other students' submitted papers in the institutional repository! 🤦‍♂️

Now, as the exam gatekeeper, I view these results before I anonymize the exams for grading. I have to make a call as to whether someone plagiarized, and I always come to the conclusion that  TII is full of 💩.  On the plus side, because the exam is longer than it was when I was a student, students have access to the report and they can modify/update their submission to get rid of any inadvertent plagiarism (i.e., they forgot to cite something), but some folks get stressed out at seeing a 20% plagiarism report and they've cited everything properly. I'd be stressed out too if I only had two chances at an exam that determined if I would graduate or not and I got these reports!  Now, add to it some black box "likelihood of AI writing this" where you don't know how the mechanism works, and you have no idea how to evaluate such an output. Even though my default stance is to trust the student, it still takes energy to reassure students when they see this kind of junk output...

It annoys me that such "tools" increase stress upon learners, and add to the workload of care that professionals have to add to their schedule to reassure students who are, rightfully,  freaked out.

Comments

Post a Comment

Popular posts from this blog

Latour: Third Source of Uncertainty - Objects have agency too!

Image
Continuing on my exploration of ANT, and asynchronous and indirect dialogue with Latour - this blog post will cover the third source of uncertainty, which according to Latour, is that Objects have agency too! As with the previous blog posts, I've pulled out quotes from the book that seemed interesting, or that I reacted to in some way, and I am responding to them here. no tie can be said to be durable and made of social stuff (p. 66) This quote seems to continue Latour's assertion that there is no such thing as "social" or "social stuff" and that "social", or the meaning of needs to be negotiated and better understood.  It also continues the thought that social can only be seen from the actions of its actors, the traces they leave behind, and that these bonds are not durable because they need continuous reinforcement. I guess Social is a perishable item. Left to its own devices, a power relationship that mobilizes nothing but social skil
Read more

MOOC participation - open door policy and analytics

The other day I was reading ZML Didaktik on the topic of MOOC participants. In MOOCs, one of the big questions is why are people lurking and not participating? If more than 500 people join a MOOC, why are only 10% contributing with any amount of regularity?  On the same blog, in a previous blog post, I had commented (it was an open stream of thought really) that perhaps there should be an open enrollment period, and then if the system sees that certain participants are bellow a threshold of activity, the system may give them the option to self-identify as a lurker, or un-register from the class. This line of thought went along a view that compared to the traditional classroom; in a traditional classroom students register for the course, they can attend classes for a week and then decide whether or not they want to stay with the course or not. If they do stay with the course they know that there is a certain amount of "lurking" that they can get away with, but they do have
Read more

You've been punk'd! However, that was an educational experience

Image
It's now the end of Teaching Goes Massive: New Skills Required (aka #massiveteaching) on coursera. Well, almost, we still have a couple of days left. I guess that the lesson here is that we were (the "learners") were punk'd† by Paul-Olivier Dehaye of the Univerisity of Zurich. After that last blog post (and subsequent pickup of the post by George Siemens and others) Inside Higher Education and the Chronicle wanted to chat with me about this MOOC experience. I only had time for one of the two before things went to press, so my 15 minutes of fame went to IHE.  Both IHE and the C hronicle have written about the topic, and have received some information from Coursera on the incident.  Others have also written about it (see here , here and here ) It's surprising to me that the University, and Coursera, waited until after this thing was a big issue in order to respond in the class, and clue people into what was happening. On Wednesday (day 3 of Week 3/3 of the
Read more