Hacking Scientific Citations

Some scholars are inflating their reference counts by sneaking them into metadata:

Citations of scientific work abide by a standardized referencing system: Each reference explicitly mentions at least the title, authors’ names, publication year, journal or conference name, and page numbers of the cited publication. These details are stored as metadata, not visible in the article’s text directly, but assigned to a digital object identifier, or DOI—a unique identifier for each scientific publication.

References in a scientific publication allow authors to justify methodological choices or present the results of past studies, highlighting the iterative and collaborative nature of science.

However, we found through a chance encounter that some unscrupulous actors have added extra references, invisible in the text but present in the articles’ metadata, when they submitted the articles to scientific databases. The result? Citation counts for certain researchers or journals have skyrocketed, even though these references were not cited by the authors in their articles.

[…]

In the journals published by Technoscience Academy, at least 9% of recorded references were “sneaked references.” These additional references were only in the metadata, distorting citation counts and giving certain authors an unfair advantage. Some legitimate references were also lost, meaning they were not present in the metadata.

In addition, when analyzing the sneaked references, we found that they highly benefited some researchers. For example, a single researcher who was associated with Technoscience Academy benefited from more than 3,000 additional illegitimate citations. Some journals from the same publisher benefited from a couple hundred additional sneaked citations.

Be careful what you’re measuring, because that’s what you’ll get. Make sure it’s what you actually want.

Tags: , ,

Posted on July 15, 2024 at 1:13 PM • 9 Comments

Comments

DoneForNow • July 15, 2024 5:55 PM

Blimey, in the days when I submitted scientific papers for review the only metadata on offer was the watermark on the posh printer paper they insisted we use!

Paul Sagi • July 17, 2024 9:39 AM

In a research paper, the more citations the better, if the papers cited are real.
It makes the paper seem to be of better quality and increases the chance the paper will be cited, which makes the researcher appear better.
I wonder how the invisible metadata was injected, was it white text on a white background?

David in Toronto • July 18, 2024 9:51 AM

So what are the consequences for doing this? It would seem to me that there should be some consequences. And by that I mean a process with formal investigation and reporting to attempt to identify the source of the manipulation, followed by something like a watchlist for extra scrutiny, and lastly if the manipulation can be accurately attributed something more.

Winter • July 19, 2024 7:27 AM

@David

So what are the consequences for doing this?

Depends on your employer and position.

Where I have worked, being caught at manipulating your citation score in underhanded ways is a reason for summary dismissal. Other countries and institutions might differ in their response.

Ulf Lorenz • July 19, 2024 1:45 PM

Just another variant of the problem that KPIs should not be used for decision making on those same people that generate these KPIs. Or what was the saying, something like “If you pay people by lines of code, you get lines of code”.

M • August 15, 2024 9:34 AM

The fault lies in trusting the journal. It’s rather like being quoted in a newspaper story, and trusting the writer to get the quotes correct.

You don’t trust them; you read the story.

What if they don’t? You complain to the paper, you publicize the complaint, and (if they don’t correct it) you sue them.

Why would you trust a journal? Why is there not someone who has a grant to grab articles and ensure they are properly formatted (names spelled correctly, citations in order, etc.)?

It’s not like we haven’t seen many, many other examples of cheating like this.

Paul Sagi:

The article certainly implies that Technoscience Academy was inserting the extra citations, or at the very least doesn’t check them.

The example scientist who benefitted so much was associated with them (possibly an editor?) So either the journal looked at the author and let the articles through unexamined, or actually added the citations to benefit the scientist.

Sidebar photo of Bruce Schneier by Joe MacInnis.