Why AI invents citations and how to catch it

A language model generates text by predicting likely next words, based on patterns learned from a very large body of text. It does not look anything up while it writes, unless it is connected to a search system.

A citation is a highly regular pattern. Author names, a short title, a journal abbreviation, a year, a volume and a number follow a predictable shape, and the model has seen millions of them. When asked for a source, it produces something with exactly that shape.

The model stores patterns, not a catalogue. It has no reliable record of which combination of authors, title and journal really appeared together, so it can blend real authors with a real journal and a title that was never published.

Numeric identifiers make this worse. A PMID or an NCT number is a string of digits with no meaning the model can reason about, so a generated identifier may point to nothing or to an unrelated record.

The output is fluent and confident either way. Nothing in the wording signals whether a reference was remembered correctly or assembled from fragments.

• The model predicts text, it does not query a database.
• Citations have a regular form that is easy to imitate.
• Identifiers are arbitrary numbers, so wrong ones look as valid as right ones.
• Confidence of tone says nothing about accuracy.

Not every bad citation is a complete fabrication. In clinical evidence work, several kinds of error appear.

• A reference that does not exist at all.
• A real paper attached to a claim it does not make.
• A real identifier that belongs to a different study.
• A registry record cited as if it reported results it never posted.
• A conference abstract presented as a full peer reviewed paper.
• A correct paper with wrong numbers attributed to it.

The second and third kinds are the most dangerous, because a quick check that the reference exists will pass.

Retrieval augmented generation, often shortened to RAG, connects a language model to a search step. The system first retrieves relevant passages from a defined collection of documents, then asks the model to write an answer from those passages.

This changes the source of the facts. Instead of relying on what the model absorbed during training, the answer can be grounded in documents you can open and read.

Retrieval reduces the problem, but it does not remove it by itself:

• The search can miss the most relevant study, so the answer rests on weaker evidence.
• The model can still add statements that are not in the retrieved passages.
• A citation can be attached to the wrong passage, even when both are real.
• When retrieval returns nothing useful, a model may fall back on memory and write an answer anyway.

That is why grounding needs a second, independent step: checking each citation after the answer is written. Our overview of artificial intelligence in clinical trials covers where else these methods are used across research.

Whether a reference comes from a colleague, a slide deck or an AI tool, the same checks apply.

1. Confirm the identifier resolves. Look up the PMID in PubMed or the NCT number on ClinicalTrials.gov and make sure it opens a real record.

2. Match the metadata. Compare title, authors, journal, year or sponsor with the citation. A real ID with a different title is a red flag.

3. Find the passage. Locate the sentence, table or outcome that supports the claim, in the abstract, the full text or the registry results.

4. Check that it says what is claimed. Compare population, intervention, comparator, outcome, time frame and effect direction with the sentence that cites it.

5. Check the evidence type. Note whether the source is a randomized trial, an observational study, a review or a registry entry without results, and weigh it accordingly. Our guide to levels of evidence explains the hierarchy.

6. Check for corrections. Look for retractions, errata or updated registry records that change the picture.

7. Record what you checked. Note the date and the record you used, so the check can be repeated.

Steps 1 and 2 catch invented references. Steps 3 and 4 catch real references used for the wrong claim, which is the error that most often survives a quick review.

Before you use an AI tool for evidence questions in professional work, ask how it behaves on each of these points. A vendor that cannot answer clearly is telling you something.

• Source collection. Which databases and registries does it search, and how often are they refreshed?
• Grounding. Is the answer written only from retrieved passages, or can the model add knowledge from memory?
• Citation granularity. Is every sentence cited, or only the answer as a whole?
• Identifier verification. Does the tool check that each cited ID exists in its index before showing it?
• Support check. Does it verify that the cited passage actually supports the sentence?
• Empty results. What happens when there is no evidence? Does it say so, or does it produce an answer anyway?
• Transparency. Does it show which sources were actually searched for this question?
• One click access. Can you open each cited record directly?
• Evidence type. Does it distinguish trials, reviews and registry entries?
• Audit trail. Can you save, share and revisit an answer with its sources for later review?
• Data handling. Does it require patient data, and where are your questions stored?

ClinicalEvidence is built around the idea that an unverified answer should not be shown at all.

• Answers are written only from passages retrieved from our index of registered trials and published studies, covering ClinicalTrials.gov, PubMed, PubMed Central, the EU Clinical Trials Register and the WHO ICTRP where licences allow.
• Every sentence carries an NCT number or PMID that links to the source record.
• Automatic citation verification checks that each ID exists in the index and that the cited passage supports the sentence.
• A sentence that fails verification is removed, and an answer without verified sources is not shown.
• When the index holds no evidence for a question, you get a plain statement that no verified answer exists, with the closest matching studies if there are any.
• The tool never answers from model memory.
• The Sources searched panel lists only the sources actually searched for your question.

Read more about cited AI answers and about how ClinicalEvidence works from question to verified answer.

• Language models invent citations because they generate plausible text, not because they look records up.
• Retrieval grounds answers in real documents but needs an independent verification step.
• A real identifier attached to the wrong claim is the error most likely to slip through.
• A trustworthy tool says plainly when there is no evidence.
• You remain responsible for reading the source before you rely on it.