Peer review leaves the central claims of most biomedical preprints intact
A claim-level comparison of 72,644 bioRxiv preprints with their peer-reviewed publications, labelled by a large language model.
Hao Yin, Woosung Ahn, Patrycja M. Forster & Ruslan Rust · Claude Sonnet 4.6
Loading figures…
The headline · Figure 1a–b
Most primary claims survive peer review
For every pair, the model extracted the single primary claim of the abstract and judged whether it was unchanged, minorly revised, or majorly changed between the preprint and the published version. The large majority changed little.
–
Primary claim unchanged
–
Minor revision (wording only)
–
Major change in content
–
More cautious vs more confident wording
a · Content change of the primary claim
89.9% of primary claims were unchanged or only minorly revised; just 10.2% changed substantially.
b · Hedging shift of the primary claim
When wording shifted, it moved toward caution twice as often as toward confidence (8.4% vs 4.2%).
Caution scales with revision · Figure 1c
The bigger the revision, the more the wording softens
Among majorly revised claims, the wording became more cautious in 38.5% and more confident in only 19.8% of assessable cases. Claims labelled unchanged in content carry no hedging shift by definition, so the comparison runs between the minor and major strata.
c · Hedging direction within each content-change stratum
Percent of claims with an assessable hedging label that became more cautious (blue) or more confident (orange).
2:1
Across the whole corpus, weakened claims outnumber strengthened claims roughly two to one — a ratio below parity in every field.
P<0.001
Two-sided sign test on the 9,150 pairs with any hedging shift.
By field · Figure 1d–e
Revision rate varies by field, but the direction of caution does not
Major revision of the primary claim ranged from 7.2% in bioinformatics to 17.5% in microbiology across the 17 fields with at least 1,500 pairs. Yet the shift toward caution held everywhere: the strengthened-to-weakened ratio stayed below one in every field.
d · Content-change composition by field (ordered by major-revision rate)
e · Strengthened-to-weakened ratio of primary claims by field (dashed line = parity)
Claim types · Figure 1f
The kind of claim almost never changes
Among the 71,055 pairs whose primary claim was present on both sides and carried a valid type, the claim type was preserved in 97.7%. Among the 1,605 that did change type, transitions ran mostly between adjacent categories — for example mechanistic to descriptive or associative — rather than collapsing to a null result.
f · Primary claim-type transitions among pairs whose type changed
Band width is proportional to the number of pairs. Descriptive claims show the largest net gain; mechanistic claims are the most often reclassified.
Drivers of revision · Figure 2
Larger revisions track longer review and higher-impact journals
Revision moves the claims of an abstract together and rises with both the length of peer review and the impact of the destination journal.
a · Secondary claim follows the primary
When the primary claim was majorly revised, the first secondary claim also changed in 90% of pairs, versus only 34% when the primary was unchanged.
b · Major-revision rate by primary claim type
Method claims are the most stable (5.4% major). Secondary claims (red) are revised more often than primary claims (blue) for every type.
Suppl. Fig. 5a · Major-revision rate by year of preprint posting
Major revision fell from 17.0% in 2019 to 5.7% in 2024. Recent preprints that need substantial revision may not yet be published, so this is a descriptive pattern.
c · Content change by review-time tertile
Major revision rose from 7.0% in the fastest tertile (median 110 days) to 14.1% in the slowest (median 416 days).
d · Any revision vs journal impact (2-yr mean citedness, log scale)
Revision rises with journal impact and is not linear: a grouped binomial fractional-polynomial model (powers −0.5 and 0) over 59,012 pairs in 736 journals puts the fitted revision probability at 53.8% at citedness 2 and 67.5% at citedness 10 (OR 1.78, robust 95% CI 1.39–2.29).
e · Retraction rate: bioRxiv-linked vs not linked
In an article-level cohort of 399,928 papers published 2021–2024 in 47 journals, retraction was less frequent among bioRxiv-linked articles (0.79 vs 5.20 per 10,000 article-years; hazard ratio 0.22, 95% CI 0.10–0.50, Cox model stratified by journal and publication year). Exploratory; 6 events in the linked group.
Validation · Suppl. Figure 2
The model agrees with the raters as well as the raters agree with each other
Four raters independently labelled a stratified subsample of 550 pairs, blind to the model. They agreed at a quadratic-weighted Cohen's κ of 0.76 (Krippendorff's α 0.77). The model reached κ 0.75 against the rater consensus and 0.69 against individual raters, and was within one ordinal level of the consensus on 98.7% of pairs.