Not just CRAAP – 2

Print Friendly
Share Button

In part 1 of this three-part article, I discussed a study by Sam Wineburg and Sarah McGrew into different methods of search-and-find as employed by three distinct groups, professional fact-checkers, professional historians and first-year undergraduates.  The researchers found that the methods used and the thinking processes of the historians and the students were different to the strategies and the thinking processes of the fact-checkers – and that the methods used by these historians and the students could be among the reasons why many of them made incomplete analyses of the sites visited and made flawed conclusions.

The three groups were asked to complete six tasks in timed conditions. The findings and ensuing discussion are detailed in the paper Lateral Reading: Reading Less and Learning More When Evaluating Digital Information.

In this earlier post (Not just CRAAP – 1), I invited readers to try one of the tasks for themselves. If you haven’t already done this, it might be a good idea to try before reading on here.

The task asked participants to imagine they looking for information on bullying, and describe their thought processes as they considered two particular articles on two different websites.  The articles were Bullying at School: Never Acceptable on the site of the American College of Pediatricians (ACPeds – the College) and then  Stigma: At the Root of Ostracism and Bullying on the site of the American Academy of Pediatrics (AAP – the Academy).

Participants were allowed to look elsewhere on the sites and anywhere else online that they wished.  They had to decide which website was the more reliable and trustworthy.

What the researchers found was that historians and students tended to spend much time looking at the actual articles before they moved elsewhere; some stayed on the target sites, some left them to look elsewhere.  By contrast, the fact-checkers spent very little time on the target pages – sometimes just seconds; they all quickly looked elsewhere, often outside the publishing sites.  That is not necessarily (at least in my eyes) a concern.  What does concern is that the evaluations made by the two groups were very different.   Historians and students tended to choose the College site as the more reliable and trustworthy, the Academy site as the less reliable and trustworthy.  [inaccurately expressed : see the correction below.] The fact-checkers were the exact opposite.

In this and in the five other tasks, the fact-checkers were much quicker at making decisions and finding corroborating information in reliable sources to support their thinking.  Historians and students were slower, and many did not try to verify information found. Of those who did try to verify what they found elsewhere, many accepted the opinions of any source rather than looking further for corroboration from more-reliable sources.

One more pair of reminders, and then we get to the meat of this present blog-post:

As discussed earlier, the College is a small fringe organisation of pediatricians with a conservative agenda.

The Academy is the world’s largest organisation of pediatricians; it is long-established, much respected, and is the publisher of Pediatrics, the flagship research journal of the profession.

Biased research?

Soon after the Wineburg-McGrew study was published as a Working Paper of the Stanford University History Education Group, Den Trumbull MD, the author of the original College paper Bullying at School: Never Acceptable wrote and posted Commentary on a Stanford University Study: Criticizing University Students and Doctorate Historians.

It is highly critical of the Wineburg-McGrew study, accusing them of bias and false reasoning.  Inter alia, he made the points that

  • their paper, Lateral Reading, was not peer-reviewed;
  • the fact-checkers were prejudiced because they were “influenced by non-objective sources” which prejudiced their opinions before ever they read the actual article;
  • his paper – which he describes as the College’s position statement on bullying – is “defended by referenced research;”
  • that the College has been “maligned” by opponents leading to the fact-checkers prejudging the College statement on bullying, even when that statement is “irrefutable.”

Trumbull declares that the Wineburg-McGrew study is totally flawed, totally biased. The fact-checkers, he says, were “opinion-checkers,” not fact-checkers. Moreover, he declares,  “facts are not a matter of opinion or popularity.”

He goes on to say:

True fact-checking would involve scrutinizing the text and the references that support the text. That’s what the students and historians mostly did. The fact-checkers were more likely to have been influenced by the all-too-common ad hominem attacks found on the Internet, and perhaps persuaded by the views of their professional associations with “news and political organizations.”

It comes down to: do we best consider the authority and reliability and usefulness of information found by considering only that information, or do/should context and reputation and other factors come into consideration too.

In part 1 of this article, I declared that I was not impressed by either the College paper nor the Academy article;  I said that I would probably not use either if I was looking for information on bullying.  I noted some of the flaws and shortcomings in Trumbull’s article I had found, replicating as best iI could and n timed conditions the task as posed in Wineburg and McGrew’s study. Dr Trumbull’s counter-blast to the Wineburg-McGrew study made me take a second and closer look.

A more careful look at the College paper shows that this is NOT an article or academic paper. I missed this first time through, and I suspect the fact-checkers did as well, so quick were they to leave the site without looking at the content of the page.  It is an opinion piece, right from the start. The abstract reads:

No child should be harassed for his or her unique characteristics. Schools should encourage an environment of respectful self-expression for all students, and no group should be singled out for special treatment.  Parental involvement should be a school’s primary method of resolution with programs emphasizing general respectfulness serving to set the tone in the classrooms.

First time through, I missed the full significance of the repeated use of “should.”   Though the first two sentences seem sound enough,  I did pause a moment over the final sentence.  It took a second reading, after reading the full paper, to appreciate that “should.”    Trumbull’s paper lacks an argument.  That “should” is not there pointing to the conclusion of the paper. It is establishing the conclusion as a given.  There is no attempt to establish the “should-ness” of the position, no background context or review of the literature.  There is no mention of contrary positions.

Of course, the “should-ness” is fully in line with the College’s lead core value as listed on the About Us page:

The American College of Pediatricians

  1. Recognizes that there are absolutes and scientific truths that transcend relative social considerations of the day.

This is not an academic paper as I had first thought, it is a position paper. It says so, in the second-top navigation bar.  I missed that, doing the test.

I would probably have been aware if I had followed links and menus on the site to reach this page as in a “normal” piece of research;  the test in the study is “un-normal,” in that it takes us to the page directly; it is all too easy to miss the clue in the navigation bar.  On the other hand, if a search-engine search had brought me directly to this page, I would have have missed that clue too.

The College is not trying to hide anything – though I do wonder it is standard practice for a  position paper to be set out like an academic paper? Again, given the speed with which they left the page, I wonder if the fact-checkers also missed this clue.  Come to that, the historians and the students tended to spend much time on this page – but now I wonder how many of them spotted this?  (There is no mention of this in Wineburg and McGrew’s paper.)

Despite its “feel,” most notably the abstract and the list of references, this is not an academic paper.

Before reading the page itself, I spent much time thinking about the footnote signals and the references. I quickly saw that one reference was to a dictionary definition, two were to blog posts – or is this a single blog post? Though the authors are different, the title and the URL are the same for both.

Five of the ten references are to different papers in the journal Pediatrics, published by the highly-respected Academy. That looks good.  But the citations in the text gave me pause, there is so little real content there. They could come from any source.  I noted in particular footnoted item #3.  The second section in the paper is headed Forms of Bullying and provides five bullet-pointed forms of bullying; the next section, Target Characteristics of Bullies, presents eight bullet-pointed characteristics.  Most of these thirteen bullet-pointed statements strike me as common knowledge; if I was asked to come up with a list of forms or characteristics of bullying, I think I would come up with a similar list – and I know very little about bullying.   I thought it odd that, of the thirteen items listed in these two sections, just one is attributed, “Physical inabilities and disabilities (3)

Could Dr Trumbull find a source only for this one item but not for any of the others?  Or  could he not be bothered to find references for the other items?

I was similarly struck by the next three citations (#4, 5 and 6). These all seemed fairly basic notions about bullying, so basic that any introductory article might list them. Did we really need three separate citations from three separate academic papers to make these points?  Students sometimes use this ploy to suggest wide reading, don’t they?

Citations #9 and 10 are striking. Trumbull provides two quotations, different superscript numbers, different speakers:

These two citations leads to references which each name the same blog post, but name different authors:

There is just one blog post from which these quotations are taken, Expert says media dangerously ignore mental illness in coverage of gay teen suicides.

It was written by Liz Goodwin, not by Haas and not by Bayard. What Trumbull has done is to take Goodwin’s paraphrases of what Haas and Bayard said and present them as direct quotations,

and then attributes the same blog post to each of them in turn. Liz Goodwin, the actual author of these words, gets not a mention.

It’s a petty point, but if Trumbull is going to accuse Wineburg and McGrew of a lack of scholarly rigor, then he needs to be a tad more rigorous himself.

Perhaps the only use of source material which is both accurate and telling is citation/ reference #7.

The reference shows that these thoughts and the quotation come from a paper in Journal of Criminology:

Once again, it might seem petty to point out (1) that Trumbull fails to name the authors of this paper (which are shown very clearly here on the Journal of Criminology website, at the DOI as recorded):

or (2) that the quotation he uses appears on page 8 of the paper and not pages 7 and 8 as detailed in his reference.  And, because his is not an academic paper which might present other viewpoints (the better to refute them),  Trumbull finds no need to mention comments elsewhere on this finding, including the notions that

  1. schools which introduce anti-bullying programs early tend to have more success in preventing bullying than schools which start late, (What Makes Anti-bullying Programs Effective? reported in Psychology Today)
  2. schools which introduce anti-bullying programs often do so because they have problems – possibly caused by failure to start the programs early (see (1)), (a Huffington blog post, Are Anti-Bullying Programs Counterproductive? includes a number of criticisms of the study)
  3.  schools which introduce anti-bullying programs often do so because they have problems (see the Huffington Post article) – and so “are more likely to have experienced peer victimization, compared to those attending schools without bullying prevention” (the key finding of Seokjin Jeong and Byung Hyun Lee’s paper),
  4. there are many studies and meta-studies which conclude that anti-bullying programs in schools do decrease (though not eliminate) bullying (for instance, Systematic reviews of the effectiveness of developmental prevention programs in reducing delinquency, aggression, and bullying by David Farrington and others).

There is evidence both ways and somewhere in-between too.  But again, if you believe your position is “irrefutable,” you don’t recognise possibly contrary views, do you?  If they exist, they are wrong, aren’t they?

All together, and although supposedly “defended by referenced research,” in academic and argumentative terms Trumbull’s defences are weak.

As I said, I doubt that I would use this position statement in a presentation on bullying.  Writing this article though, I realise that I might use it in an academic paper, an example of alternative and possibly fringe views (and go on to suggest why the College position is suspect).

As noted, I doubt that I would use the Academy page in my paper on bullying either. It is light on substance. On the other hand, it is not meant to convey anything weighty. It is a session description for an upcoming two-hour symposium to be held during a general meeting of four pediatric organisations.  The header of the article reads:

Experts in bullying and children’s mental health gather at the Pediatric Academic Societies meeting to describe new research and what it means for children’s mental health.

If I was writing a paper on bullying, this article could be useful as a starting-point for further research. Six presentations are mentioned, so I might search for any of the titles of these presentations which interest me, I might use the names of the presenters as possible experts to follow up. I might also note that the American Academy of Pediatrics is one of the four organisations sponsoring the Annual Meeting. The American College of Pediatricians is not one of the four sponsors. I wonder if anyone from ACPeds attended the meeting or the symposium?

Another angle

One of Trumbull’s complaints about the Wineburg-McGrew study is that

the College has been “maligned” by opponents leading to the fact-checkers prejudging the College statement on bullying, even when that statement is “irrefutable.”

I am reminded that two of the frames of the ACRL Framework for Information Literacy for Higher Education are

Scholarship as Conversation
Authority Is Constructed and Contextual

I do wonder how conversation can take place when the claim is made that the College’s views are “irrefutable” and also that there are “absolutes and scientific truths that transcend relative social considerations of the day.” This is claiming absolute authority, no room to question or to argue.

That contravenes the other frame I mention here: authority is not absolute. It is earned and it is situational.  We educate our students to appreciate (we hope) that information is not all equal, even when two writers say the same thing. We look for reliability and we look for authority – and while authority is bestowed, in part, by who the writer is, it also depends on the views of others, whether through reputation, qualification or deeds.  You are not trustworthy because you say you are. You have to show you are trustworthy, you have to earn trust.  We do well to look closely at the messenger as well as at the message.   If the messenger is untrustworthy, can we trust the message?

Space, time and your patience, gentle reader, suggest I bring this post to a close. It could be, though, that each of the other four frames of information literacy may have some relevance in this investigation.  I must think further, maybe another post.

More immediately, I want to address evaluation checklists. One of Wineburg’s points is that the tools we give students may not always serve them well. They did not in this study, the tasks that he and McGrew set. They did not serve the historians too well either.

We’ll look more carefully at CRAAP and other evaluation tools in the next post.

[In part 1 of this 3 part article we looked at Wineburg and McGrew’s study which suggests a fresh look at the way we evaluate web pages and sites.]
[In part 3 we look at the checklist approach to evaluation, and suggest that we don’t need to get rid of our CRAAP tests, we need to enhance them.]

Correction 31 March 2018

My summary of my summary is inaccurate and misleading, I misrepresent the historians.   Here I write:

Historians and students tended to choose the College site as the more reliable and trustworthy, the Academy site as the less reliable and trustworthy.  The fact-checkers were the exact opposite.

I expressed this better in part 1 of this investigation, where I wrote:

It is not surprising then that two-thirds of these students considered the College site as being the more reliable. Only 20% of the students opted for the Academy (the remaining students thought the two sites were equally reliable). The historians did a little better: while only one opted for the College site while as the more reliable, another 40% thought the two sites were equally worthy. Only half the historians thought the Academy the more reliable.

My apologies.

One thought on “Not just CRAAP – 2

  1. Pingback: Not just CRAAP – 1 | Honesty, honestly…

Leave a Reply

Your email address will not be published. Required fields are marked *