There has been a bit of a splash in the last few days, publicity regarding a study of Turnitin by Susan Schorn of the University of Texas.
iSchoolGuide, for instance, splashed an item by Sara Guaglione: University Of Texas At Austin Writing Coordinator Susan E. Schorn Finds Turnitin Software Misses 39 Percent Of Plagiarized Sources, and EducationDive posts a similar take on the story, this by Tara García Mathewson, Plagiarism detection software often ineffective.
There is not a lot new here, not for regular readers of this blog. Turnitin is ineffective.
Both articles are based on a post in InsideHigherEd by Carl Straumsheim, What Is Detected? worth reading, for its content and for the comments it has generated. Again, not a lot new, not for regular readers of this blog. Turnitin is ineffective (as are other so-called plagiarism detectors, it is not just Turnitin which is problematic).
Straumsheim goes further (than Guaglione and Mathewson), pointing to Turnitin’s propensity to assign false negatives – passing as “original” material which is clearly and totally unoriginal. This is not original material ghost-written by another but passed off as a students’ own work – no “plagiarism detection software” can discover that if it truly is original. It isn’t work translated from one language to another. Plagiarism detection software cannot detect that either. It isn’t work copy-pasted from invisible web resources or copied from print material, neither of which can be accessed by Turnitin’s spiders. “False negatives” are sequences copied from the open web, places which Turnitin has visited, but in which the software still fails to find matches. Turnitin gives these sequences a clean bill of health when in fact they are blatant copies of previously published material. There is a blatant example of this on the Turnitin website itself, a promotion which Turnitin uses to demonstrate how wonderful Turnitin is. It isn’t (see Carried away).
To my mind, the big problem with false negatives is the confusion this produces in many students’ minds, perhaps already confused because practices they use in one teacher’s classes and found acceptable prove unacceptable in a different teacher’s classes; I am also sure that, for many students, the aim of writing assignments becomes, not good thinking and writing, but “beat Turnitin” – whether they deliberately intend to cheat or, in some senses far worse, because they think that this is what good writing is.
P.L. Thomas provides an even more telling analysis of Schorn’s study, in Technology Fails Plagiarism, Citation Tests. In this article, Thomas makes points similar to those above. Thomas too believes that many instructors put the emphasis in the wrong place. They teach negatively; they teach plagiarism (or plagiarism avoidance) rather than teaching citation and good writing practice positively. Thomas makes some grand points, and has some very useful links to support his views.
Both Straumsheim and Thomas go on to consider the problems and issues of what Evgeny Morozov calls “technological solutionism,” in this instance being some instructors’ seemingly-blind faith in and acceptance of technology, using it to replace teacher judgement and responsibility. Straumsheim uses machine grading as a further example, and quotes from a NCTE paper : “machine grading ‘compels teachers to ignore what is most important in writing instruction in order to teach what is least important’.”
Thomas’s tilt at technological solutionism is, possibly, more questionable: his further example of faith in software is reliance on “citation software” – and he specifically names NoodleBib and RefMe.
[I have to be careful here: I have several times worked with Debbie Abilock, a co-founder of Noodletools, so there may be a degree of bias here. I hope not. I strive, as always in these posts, for distance and objectivity.]
Thomas says that, although many universities (JR: I would add schools as well) encourage the use of citation generators, he himself “… discourage(s) the practice because almost always the generated bibliographies are incorrect.”
That is a sweeping generalisation, and I think he would have hard time supporting his statement that “almost always the generated bibliographies are incorrect.”
Often, very often, if we are looking at references produced through auto-citation features, agreed. The software relies on external sources, and if the external sources have recorded the data wrongly then the auto-generated reference will likewise be wrong. But “almost always…” may be an assertion too far.
To be fair, Thomas goes on to say, “… if we think of citation software as a tool, and if students can be taught to review and edit generated bibliographies, the technology has promise.” That would mitigate the effects of auto-citation – though of course those who use these tools (not just students!) do have to know what is right, what to look for and what to check, in order to review and edit!
And those who have blind faith in the technology (as well as those who do not know what to check for or who are too lazy or are pressed for time) do not review and edit.
But Noodletools, RefMe and other citation generators do offer alternative modes of input. Instead of relying on an outside agency to provide the data which the software then sorts and spits out as a bibliographical reference, they offer modes by which the user inputs the data herself – or himself. The user fills in a form. Author? [input box for the author’s name]; Title? [input box for the title]. The “better” software offers more options, giving the option of adding more than one author, for instance; “better” software offers help-in-context, such as reminding the user to use Upper Case for Titles for an MLA reference but Sentence case for titles for an APA reference.
The “better” software asks the right questions, and its algorithms work to provide accurate output. I am not sure about RefMe; I haven’t done a thorough analysis and review, but my early explorations did not impress. I have real concerns about EasyBib, BibMe and many other school-level citation generators; they do fall into that “often, very often” category of unreliable software. Noodletools though? I believe this software is far more reliable than these other products, and that is why I have been careful to disclose connection and possible bias.
Any mistakes in the generated bibliography are down to the user, the person who input the data, and not an outside agency which might, or might not, have been fed correct data.
Yes, the student or other user still needs to know what s/he is doing, still needs to review and edit. But heck, Mr Thomas, I write my references the old way, individually and by hand (by keyboard) and I still need to review and edit. Don’t you?
Besides which, the professional tools, EndNote and RefWorks and Mendeley and many many more… don’t you still need to proof-read and edit?
It’s a small point and I have probably made too much of it here. It’s what writing is all about, the craft of writing, going back and honing and changing and refining ( … until the deadline presses and it is too late to do any more). That’s the problem and the danger of technology, and I fully agree with P.L. Thomas there – technology doesn’t just sometimes get it wrong, technology doesn’t just encourage blind faith; The greater problem is that technology so often steers us in the wrong direction, and technology so often gives us wrong answers. Technology offers quick and easy solutions but so often to the wrong questions and problems. Don’t stop thinking!