William F. Garrity

As reported in the July 8, 2008, Inside Higher Ed: 

Cite Check

Citations figure prominently in academic promotion and peer review. Theoretically, scholarly references serve a dual purpose: They indicate an author’s familiarity with established literature and assign credit to previous work, while from the other direction many would argue they signal a paper’s relevance and standing within a discipline.

That’s, of course, in theory. The reality may surprise many academics who might not stop to think about the system they rely on for the production of knowledge, or who studiously ignore those little superscript numbers that indicate (again, in theory), Read the referenced paper to learn why the preceding assertion is correct.

What if it isn’t correct? What if the authors didn’t even read half of the papers they cited?

Like any self-enclosed, loosely policed network, citations are far from perfect. It’s well documented, for example, that researchers tend to cite papers that support their conclusions and downplay or ignore work that calls them into question. Scholars also have ambitions and reputations, so it’s not surprising to hear that they might weave in a few citations to articles written by colleagues they’re trying to impress — or fail to cite work by competitors. Maybe they overlook research written in other languages, or aren’t familiar with relevant work in a related but different field, or spelled an author’s name wrong, or listed the wrong journal.

All of these shortcomings are reviewed and discussed in an article published this year1 in the management science journal Interfaces along with the critical responses to it.2

As it turns out, scholars have already done some work quantifying problem citations, divided into two categories, “incorrect references” and “quotation errors.” The authors of the paper, J. Scott Armstrong of the University of Pennsylvania’s Wharton School and Malcolm Wright of the Ehrenberg-Bass Institute at the University of South Australia, Adelaide, write of the former type, “This problem has been extensively studied in the health literature ... 31 percent of the references in public health journals contained errors, and three percent of these were so severe that the referenced material could not be located.”

More serious than such botched references are articles that incorrectly quote a cited paper or, as the authors put it, “misreport findings.” For example, in the same study of health literature3, they write, “authors’ descriptions of previous studies in public health journals differed from the original copy in 30 percent of references; half of these descriptions were unrelated to the quoting authors’ contentions.”

It wasn’t until Wright noticed that a paper Armstrong co-authored in 1977 had been inaccurately cited that they realized the extent of the problem, Armstrong said in an interview. It was Wright who suggested investigating the problem further in a more systematic way. So they focused on that specific article, which outlines a precise method for estimating the extent to which non-responses to mail surveys bias the results. Since the article has been heavily cited and the method it describes can be identified, they were able to trace how well articles that reference it represent the original material.

Using a combination of Google Scholar searches and the ISI Citation Index, Armstrong and Wright found results that are disconcerting even when acknowledging that Google doesn’t crawl every academic paper: Among academic studies using mail surveys, only 6 percent mentioned the non-response problem at all, and of those, 2.1 percent (339 articles) cited the 1977 paper. The authors found 36 variations of the paper’s citation among those that referenced it, with an “overall error rate” of 7.7 percent.

By analyzing a sample of 50 papers (out of 1,184) that cite the 1977 article (including the 30 most frequently cited of the bunch), the authors also found significant inaccuracies. By their standards, those papers didn’t fare especially well either: “In short, although there were over 100 authors and more than 100 reviewers, all the papers failed to adhere to the [1977 paper’s] procedures for estimating nonresponse bias. Only 12 percent of the papers mentioned extrapolation, which is the key element of [the paper’s] method for correcting nonresponse bias.”

The paper concludes: “Given the understandability of the recommendations and the fact that no one contacted Armstrong or [his 1977 co-author] for clarification, one might question whether the citing authors read the ... paper. To present their studies in a more favorable light, some authors may have wanted to dispel concerns about nonresponse bias; thus, they cited [it] for support for their own procedures. Interestingly, one of our colleagues said that it is common knowledge that authors add references that they have not read in order to gain favor with reviewers. One wonders: If it is possible to write a paper without reading the references, why should the authors expect readers to read the references?”

If the problem is as widespread as Armstrong and Wright suggest — and Armstrong said he believes the findings generalize to other scientific fields — then a more systemic fix might be warranted. They provide several common-sense remedies intended to address what the peer review system currently, it appears, is unable to counteract ("My experience is most peer reviewers don’t seem to be competent to do the job,” Armstrong says). “When an author uses prior research that is relevant to a finding, that author should make an attempt to contact the original authors to ensure that the citation is properly used,” they write.

“As I point out in the paper, I’ve been doing this for years, and it doesn’t really require that much work,” Armstrong said. “Generally, I found it to be easy to do. I do it as an author; I hardly get anybody asking me — they just go ahead and quote me incorrectly.”

The paper also argues that researchers should have to verify to journal editors that they’ve tried to contact the relevant authors, and that they’ve read the papers they cited. Furthermore, they suggest, there could be a solution waiting on the Web — one that sounds a lot like a cross between the Wikipedia model and Amazon.com reviews: “Journals should open Web sites (free to nonsubscribers) that allow people to post key papers that have been overlooked, along with a brief explanation of how the findings relate to the published study.”

Already, Interfaces and a journal Armstrong co-founded, the International Journal of Forecasting, are planning to introduce those suggestions into their editing processes. Rob J. Hyndman, the forecasting journal’s editor-in-chief, said in an e-mail that within two weeks, the Web submission system will include a check box with this text: “Confirm that the list of references has been checked carefully for accuracy and that each of the references has been read by at least one of the authors.”

And the 2008 paper? For the record, Armstrong said he and Wright followed their own advice in publishing their research: “Oh yeah, we talked about that. We had to make sure that each one of us had read every one of the papers.”

1Armstrong, J. S., M. Wright. 2008. The Ombudsman: Verification of Citations: Fawlty Towers of Knowledge? Interfaces 38(2) 125-139.

2This paper is available to download, in PDF format, by clicking here.

3See previous paragraph.

— Andy Guess

The June 17, 2008, http://www.insidehighered.com/ reports on Cornell's Undergraduate Information Competency Initiative: 

When “Google” has become a synonym for “research,” how should faculty respond? And if the answer doesn’t lie in musty books and stacks of journals, are libraries still part of the answer?

The problem is near-universal for professors who discover, upon assigning research projects, that superficial searches on the Internet and facts gleaned from Wikipedia are the extent — or a significant portion — of far too many of their students’ investigations. It’s not necessarily an issue of laziness, perhaps, but one of exposure to a set of research practices and a mindset that encourages critical thinking about competing online sources. Just because students walk in the door as “digital natives,” the common observation goes, doesn’t mean they’re equipped to handle the heavy lifting of digital databases and proprietary search engines that comprise the bulk of modern, online research techniques.

Yet the gap between students’ research competence and what’s required of a modern college graduate can’t easily be solved without a framework that encompasses faculty members, librarians, technicians and those who study teaching methods. After all, faculty control their syllabuses, librarians are often confined to the reference desk and IT staff are there for when the network crashes.

So instead of expecting students to wander into the library themselves, some professors are bringing the stacks into the classroom. In an effort to nudge curriculums in the direction of incorporating research methodology into the fabric of courses themselves, two universities are experimenting with voluntary programs that encourage cooperation between faculty and research specialists to develop assignments that will serve as a hands-on and collaborative introduction to the relevant skills and practices.

Kathy Lee Berggren, a professor at Cornell University, teaches oral communication with a “heavy research component.” Still, she pointed out, “a lot of my students really [only] scratch the surface with the type of research they’re doing.”

“Research isn’t a Google search,” she said.

That sentiment was echoed by several others involved with the Cornell Undergraduate Information Competency Initiative, a program that kicked off on Monday with a week-long summer institute aimed at understanding how students perceive university research, how to guide their habits and how to merge existing course goals with instruction in research methods. Those practices, of course, can apply whether inside a brick-and-mortar research facility or logged on from home. The goal is to “really learn how to use a library whether they’re in it or not,” Berggren said.

...

EDUCAUSE's ELI (EDUCAUSE Learning Initiative) releases two new reports in its series aimed at higher education administrators.  ELI is something you should track, I think. 

"7 Things You Should Know About Multi-Touch Interfaces" 

Multi-touch interfaces are input devices that recognize two or more simultaneous touches, allowing one or more users to interact with computer applications through various gestures created by fingers on a surface. Some devices also recognize differences in pressure and temperature. Multi-touch technology introduces users to swipes, pinches, rotations, and other actions that allow for richer, more immediate interaction with digital content. Multi-touch devices and supporting applications offer diverse ways of visualizing information to improve understanding, and they facilitate new ways to foster collaborative creation, permitting several users to work simultaneously on a single screen. 

"7 Things You Should Know About Ning" 

Ning is an online service that allows users to create their own social networks and join and participate in other networks. No technical skill is required to set up a social network, and there are no limits to the number of networks a user can join. Users of Ning social networks have access to functionality similar to that of more well-known social networks, such as Facebook and MySpace. Various features allow users to read news or learn about related events, join groups, read and comment on blog entries, view photos and videos, and other activities as set up by the network creator. RSS feeds let users subscribe to updates from specific parts of the social network. 

I've posted about Ning previously. 

EDUCAUSE is the association for higher education ICT.  It's an outstanding organization to track, and (I think) a model for a higher ed advocacy group.  (Were the Association of Research Libraries to be such!) 

Every year, EDUCAUSE publishes a list of the top issues of attention by campus ICT leaders.  Here's this year's list. 

1. Security
2. Administrative/ERP/Information Systems
3. Funding IT
4. Infrastructure
5. Identity/Access Management
6. Disaster Recovery / Business Continuity
7. Governance, Organization, and Leadership
8. Change Management
9. E-Learning / Distributed Teaching and Learning
10. Staffing / HR Management / Training

Microsoft's ESP is a product that supposedly "... is a visual simulation platform to brings immersive games-based technology to training and learning, decision support, and research and development modeling for government and commercial organizations." 

Microsoft ESP enables your organization to create, deliver, and realize the enormous benefits of immersive simulations while gaining a strong return on investment that's not readily available from other simulations tools today.

Cost-efficient simulations—Help reduce travel costs and augment costly full-flight and fixed-base simulator training time with realistic immersive simulations that run anytime, anywhere on Windows PCs.

Effective, engaging training—Give your workforce the experience of an immersive world for dynamic, engrossing, and memorable training. Enhance their willingness to learn, improve their recall, and prepare them for real-world action. The "Instructor" capability provides for real-time monitoring of student performance. After-Action Review enables instructors to replay a student's performance, break it down to discrete components, and coach the student to improve skills.

Multi-user, Internet, and VoIP support—Teach, test, rehearse, and evaluate teams of up to 30 people at once over a network or high-speed broadband connection. Peer-based connectivity takes advantage of the platform's native VoIP support to enable connectivity to anywhere broadband access is available.

Accurate, cost-effective modeling—Easily and affordably adjust simulation variables to visualize outcomes, plans, and design specs in 3D for decision making and R&D modeling.