Speaking through the Dean in Hogfather, Terry Pratchett remarked that what is good for a hangover is drinking heavily the night before.

I get that feeling thinking about Cite This For Me (a Chegg product).  It works, every time, and I don’t even need the alcohol.

Join me on this voyage of serendipity (if you dare).

It starts with the May 2018 subject report on the IB MYP Personal Project.  On page 3 we read

Those candidates who did not include an evaluation of their sources (which could be done through a CARRDS or OPVL tool) limited their achievement; there was often insufficient identification and evidence of other research skills

I know that OVPL is an evaluation tool often used in history and similar disciplines; you consider the Origin, Purpose, Value and Limitation of sources.  I could not recall meeting CARRDS before, but the context suggests that it is similar to the CRAAP tool, the acronym standing for Currency, Relevance, Accuracy, Authority and Purpose.

A quick search Google on Google confirms this:

Hit #1 appears to be irrelevant but hits #2, #3 and #5 (the hit just below the selected images results) look to be good. These are PDFs, each of which uses the term “evaluating” and “web (sources or sites)” either in the page title or the short extract beneath the URL.  Sure enough, this is just what they are – checklists to aid evaluation of web pages and websites.  CARRDS is a mnemonic for remembering Credibility, Accuracy, Reliability, Relevance, Date, Scope and Purpose.

I am not impressed – at least, not by the sub-questions suggested on these checklists. They ask questions about the source material, but none of them encourage users to look elsewhere to ascertain or verify the source and information being evaluated; only one (the Clifton Public Schools) explicitly suggests looking further on the same website for more information.  This approach, as I discussed in Not just CRAAP-3, is limiting.  You will often, usually,  get better impression of the authority and credentials of authors and of the bias and purpose of websites and of the organisations behind them (and so on) by looking at what others say about them than you do on the page or site you are actually evaluating.

But it was a hit further down the Google results list which really caught my eye and attention:

the entry Bibliography & CARRDS – Chemistry bibliographies – Cite This For Me looked interesting. I was drawn to see if this was a demonstration of how the CARRDS tool could be used to evaluate bibliographies, or maybe it was a bibliography of CARRDS-type tools.  If neither of these turned out not to be the purpose of the page, then I still wanted to find out what it was.  So, to the Cite This For Me (CTFM) site I went, following the link that Google gave me. This is what I got:

It appears to be a 10-item list of sources used as the basis of a bibliography. The page is headed:

These are the sources and citations used to research Bibliography & CARRDS. This bibliography was generated on Cite This For Me on Wednesday, October 28, 2015

but none of the items listed has anything to do with CARRDS.  They are all connected with spectrometry and/or drugs.  The right-hand column suggests that the student (?) used 9 websites and one journal in compiling this list, though a quick glance at the bibliographic references as listed on this page suggests that a range of types of source, from encyclopedia entries and chemistry sites to articles or papers in at least three academic journals was more likely to have been used.

Several of the bibliographic entries look wrong, even on the briefest of glances. This got me wondering why Cite This For Me has posted this list.  It seems pretty poor advertising if the list was generated by a CTFM staffer – though that in itself would not surprise me given the experience I relate in Double-dipping.  Cite This For Me is not the best reference generator on the market, far from it.  But chances are, I thought, it’s a list compiled by a student.  More digging needed.

The URL suggests that this page is curated in a section of the site called “Topic Ideas” so I right-clicked on the “Topic Ideas” tab.  This leads to a page with the banner

See what topics our users are researching and their bibliographies

and an invitation to Search for essay title / Or view by subject.

One of Chegg’s purposes is to enable students at all levels of education to share their work – and it looks as if CTFM subscribers do.  I put this part of my investigation aside until later, making mental notes to check on the extent of the sharing and also to look into CTFM’s terms and conditions (more below).  I wanted first to think through the Chemistry bibliographies & CARRDS list.

I wondered if Bibliography & CARRDS was a classroom exercise: first compile a bibliography of 10 items which you might want to use in your project, then evaluate each of these items using the CARRDS protocol and rank the 10 items accordingly.   It might be that students were asked to use CTFM to compile their lists; it might be that it was this one student who chose to use CTFM.  (A search for CARRDS in the essay titles search box pulls up just the one item, so there is no quick way of telling.)

Did the student use the CTFM auto-reference generator or did s/he complete a form manually?  I suspect the auto-reference generator was just too tempting – and I also suspect that whatever it came up with was accepted with no further checking or editing.

Here for instance is the original source used for bibliography entry #1:

At the foot of the page we can clearly see the name of the author and the date of publication:

Did our student just put the page URL into the CTFM auto-reference generator and accept what came up, without adding the author?  I strongly suspect that s/he did, for this is what I got when I did this, use CTFM’s auto-reference generator:

The author is apparently not found.  Nor is the title of the site (CHEMGUIDE).  CTFM uses the domain as the main entry (and the student accepts this).

This is not the only entry in the bibliography which uses domain as the author, several more do.   This is what CTFM does.  It is not when most bibliographic style guides suggests, but CTFM seems to be coded to do the unconventional.

CTFM does find a date of publication in the metadata, and this might explain why the student entry (compiled in 2015) found 2000 while my check was on a version which had been modified in 2019.

Note too that lower case t and full title given in the Page Information. These are  replicated in the CTFM reference, though the page itself uses only the short title, THE MASS SPECTROMETER.  That’s what I would have used, if I had been building my reference manually.

But never mind the quality, just feel the speed.

I will detail just one other bibliographic entry here, because it would seem further to confirm that this student was using the auto-reference feature.   Feel free to look at the others yourself – or if you’re a librarian or teacher, maybe get your students to do it?

The entry headed FLIKKA K, E.A. is for a journal article.

The student found it in PubMed, a search engine which indexes biomedical literature.  Apparently he did not need the actual article – note the button on the right and the DOI link at the foot of the abstract, either of which would bring this searcher closer to the full-text, the PubMed entry was presumably enough for whatever purposes this student had in mind.

Our student almost certainly classed this page as a website and used the PubMed URL to generate the CTFM reference.  If the student had recognised this as a journal rather than a website, the CTFM auto-generated reference (based on the title and not the URL) would have been different (as I found):

The clincher – that the student used CTFM auto-reference and classed this as a website – is the author entry in the Your Bibliography line: Flikka, K.e. – uppercase K and lower-case e.

Here’s the intermediate stage in CTFM:

The last name of the author is recognised as Flikka K and the initials of “et al” provide that E.A.

As I tell students whenever I discuss auto-referencing tools (and I am sure you do too), you do need to check the output for accuracy and consistency – and to be able to do that, you do need to know what your chosen style guide looks like, you do need to know the ground-rules.  If you don’t know what is right, you won’t know when you are fed garbage.  GIGO, it still holds.

There are similar issues with most entries in the CARRDS bibliography, but there is no need to discuss them here. (If you’d like to check for yourself, I’d recommend your first look be the item which gives Oxford Journals as the author and 2015 as publication date.).

I still wonder if and how this student applied CARRDS to this bibliography (and whether the teacher put her/him right).

Suffice it to say, CTFM issues include, as above, authors missed, dates missed, use of domain as the author even when the authors’ names were apparent at the tops of pages, wrong dates generated, and more.  Note too the inconsistencies of upper-/ lower-case in article titles throughout, quite intriguing.

But that’s Cite This For Me for you.  But definitely not for me.

The big question: why is CTFM posting this and similar “bibliographies”?

It seems a curious way to promote a product – show how very poor it is.  What is going on?

The Bibliography and CARRDS bibliography is in a section of the CTFM site “Topic ideas.”

and, as we saw earlier, the link takes us to a page on which one can search more than 30 subjects or search for bibliographies by keyword.

Some of these bibliographies are very recent; this bibliography on Brexit was, for instance, compiled within the last few days:

It is soon clear too that not all CTFM users treat all information found online as being found on a website:  there are bibliographies which include entries for journals, books, chapters in books of readings and so on.  Thank goodness for that.

It is not immediately clear whether the users have used the auto-reference feature or have completed the entry manually (or a combination of both) and ascertaining this is not my purpose here.

In passing, a search for [extended essay] brings up 25 entries, the top few entries also dated within the last few days.   These are not necessarily IB DP extended essays, of course, but again it is intriguing.

It looks as if there are some 25373 pages of bibliographies available on the CTFM site.  If all pages include 25 bibliographies, that’s a lot of bibliographies, the collection must go back several years (maybe to CTFM’s earliest days?)

All of which raises many questions.

Why are these bibliographies posted and publicly available?  Do students (and others?) who compile these lists know they are available? Have they given explicit permission for them to be made public (or is this automatically given by virtue of use of the site)? Is there a right to delete one’s bibliography? Do students use them to find ready made bibliographies on their subjects? Do students even know they are there and available (and so too will be their own compilations)?

Also, what does Chegg get out of making these bibliographies publicly available (other than bad reports when someone like me digs into them)?  How is data mined, what use does Chegg make of personal data and the use made of their site?

The answers to many of these questions are apparent in the three pages linked at the foot of the CTFM site, the links to the US Privacy Policy, the International Privacy Policy and a Cookie Notice – and some of these pages lead to further policy pages.

It is also worth reading Nadine Bailey’s investigation of Chegg’s terms and conditions and policies. Her article Buying the Future of Research is eye-opening and goes much deeper than I do in this post.  Her post is well worth a look (and having looked, do act accordingly).

What I do pick out starts with the Cookie Notice page.  This includes an opt-out option which they say you can control by clicking on Cookie Settings. Alas,  nothing happened when I clicked on this (and I tried in two different browsers).

There is an interesting paragraph on Do Not Track signals:

What about Do Not Track?

Please note that your browser setting may allow you to automatically transmit a “Do Not Track” signal to web sites and online service you visit. There is no consensus among industry participants as to what “Do Not Track” means in this context. Like many web sites and online services, Chegg does not alter its practices when it receives a “Do Not Track” signal from a visitor’s browser.

Chegg ignores any Do Not Track instruction, it “does not alter its practices when it receives a “Do Not Track” signal from a visitor’s browser.”   There is honesty there.  Your web-browsing habits will be tracked (the better for the data mining algorithms to know all there is to know about you). But you do have to read and understand this, and all the other policy pages.

The terms of use page, which has more than 50 different sections, makes it clear that you sign over all your rights in anything you post or upload, in any use you make of any Chegg service – and that Chegg then has the right to share these data with other agencies and services.

User Content and Activities

When you submit, post, upload, embed, display, communicate, link to, email or otherwise distribute or publish any review, problem, suggestion, idea, solution, question, answer, class notes, course outline bibliographic and citation information comment, testimonial, feedback, message, image, video, text, profile data or other material (“User Content“) to Chegg, any Chegg employee or contractor, or a Chegg Website, you grant Chegg and our affiliates, licensees, distributors, agents, representatives and other entities or individuals authorized by Chegg, a non-exclusive, worldwide, perpetual, unlimited, irrevocable, royalty-free, fully sublicensable (through multiple tiers) and fully transferable right to exercise any and all copyright, trademark, publicity, and database rights you have in the content, in any media known now or in the future, and to make, use, reproduce, copy, display, publish, exhibit, distribute, modify, sell, offer for sale, create derivative works based upon and otherwise use the User Content.

Subscribers are not just giving up their content, they are also giving up their anonymity, their personal information may be used, without the need for further consent, for advertising purposes.  Again from the Terms of Use:

Note that we may create, facilitate or display social advertisements, whereby your name, profile and photo may be used to advertise products and services to your network based on your use of the Services and your interactions with Chegg. You agree that Chegg may use your name and profile picture in connection with social ads to advertise products and services to your network based on your use of the Services and your interactions with Chegg and third parties through the Services.

You further agree that Chegg is free to use any ideas or concepts contained in any User Content for any purposes whatsoever, including, without limitation, developing, manufacturing and marketing products and services; and creating informational articles, without any payment of any kind to you. You authorize Chegg to publish your User Content in a searchable format that may be accessed by users of the Services and the Internet. To the fullest extent permitted by law, you waive any moral rights you may have in any User Content you submit, even if such User Content is altered or changed in a manner not agreeable to you.

This is honest too – if you read all the small print, if you understand what you are signing up to.  Have the libraries and the schools which promote and subscribe to Cite This For Me (or any other Chegg product) read all this and accepted it?

It used to be said that if you’re not paying, then you are the product. That is no longer true (if ever it was):  even if you are paying, you may still be the product – and with Chegg, you surely are.  We may not know how Chegg and the other services are using your data, but be sure, your data, you, are being used.

That hangover – how’s your head? Mine is aching…

And the message that comes across quite definitely, for a clear head, stay clear of Chegg.