Why So Many Biomedical Findings Do Not Replicate — and What Journals Now Require

Biomedical findings fail to replicate for structural reasons — small studies, flexible analyses and selective reporting — first formalized in a 2005 PLOS Medicine essay arguing most research claims are more likely false than true. The institutional response has been funder policy, notably NIH's rigor and transparency requirements, and reporting guidelines enforced by journals.

Why do findings fail to replicate?

The 2005 essay by John Ioannidis, published in PLOS Medicine, built a simple argument: the probability that a research claim is true depends on study power, bias, the ratio of true to null relationships in a field, and the number of teams chasing significance. A finding is less likely to be true when studies are smaller, effect sizes are smaller, more relationships are tested with less preselection, and designs, definitions, outcomes and analytical modes are more flexible. Simulations in the paper showed that for most study designs and settings, a research claim is more likely false than true, and that claimed findings may often be accurate measures of prevailing bias.

None of this requires misconduct. A well-intentioned lab running underpowered experiments, analyzing outcomes in several ways and publishing the one that clears 0.05 will manufacture false positives at a steady rate. Replication failure is mostly the arithmetic of those incentives, which is why the fixes have targeted design and reporting rather than individual behavior alone.

What did NIH change?

NIH now expects grant applications to address rigor and reproducibility directly. Reviewers evaluate the scientific premise, the strength of the experimental design, consideration of relevant biological variables such as sex, and authentication of key resources. The agency points applicants to principles developed at a workshop of editors representing over 30 basic and preclinical science journals, and to guidance on what reviewers look for when evaluating scientific merit:

1. Scientific premise: the strength of the evidence underlying the proposed research question.
2. Rigorous experimental design: power calculations, predefined endpoints, allocation and blinding where feasible.
3. Biological variables: sex, age and strain accounted for in design rather than defaulted.
4. Authentication of key resources: cell lines, antibodies and model systems verified as what they are claimed to be.

What do reporting guidelines require?

The other pillar is checklists at the point of publication. The EQUATOR Network maintains a library of more than 708 reporting guidelines mapping study designs to minimum reporting standards, so that reviewers and readers can see what was planned, run and analyzed:

These checklists do not make a study better; they make omissions visible, which is the precondition for peer review functioning at all. Per the EQUATOR Network, the library covers main study types from randomized trials to economic evaluations, with extensions for trial protocols and specific designs.

What can peer review actually catch?

Peer review is a filter run by unpaid experts before publication, and its detection limits are well documented: it screens for plausibility, fit and internal coherence, but reviewers typically see only what authors report, rarely see raw data, and cannot detect selective reporting inside a flexible analysis. That is why the structural fixes matter more than reviewer diligence — preregistration ties authors to a plan, reporting guidelines expose what a paper omits, and data deposition lets others recompute results.

How should a professional reader read a paper in 2026?

Read the methods before the abstract's claims. Check whether the primary endpoint was predefined or chosen after the fact, whether the sample size was justified, whether key resources were authenticated, and whether negative or neutral analyses appear in supplement rather than being absent. The gap between a paper and a practice-changing result is closed only by replication in the intended population, and a reader who treats single-study claims as provisional is applying the same standard the funders now write into policy.

How did the policy response come together?

The modern framework was assembled from two directions at once. From the funding side, NIH built rigor and transparency expectations into grant applications and review language — training reviewers to probe scientific premise, experimental design strength, biological variables and resource authentication while minimizing added administrative burden. From the publishing side, editors representing more than 30 basic and preclinical science journals developed shared principles for publishing preclinical research at an NIH-hosted workshop, committing to transparent reporting of methods, sample sizes, inclusion and exclusion criteria, and randomization and blinding where applicable.

The two tracks reinforce each other because a funder cannot require what journals will not print, and journals cannot demand what funders will not pay for. A researcher planning a study today therefore faces a chain of expectations — design rigor to win the grant, reporting completeness to pass review, data and method availability to survive scrutiny — where a generation ago the binding constraint was often a positive result and little else.

What practices actually move the needle?

Across the policy documents and guideline library, a consistent set of practices recurs, and each attacks a specific failure mechanism rather than diligence in the abstract:

1. Prospective power calculation and sample size justification — attacks small-study noise.
2. Preregistration or protocol publication with predefined endpoints — attacks flexible analysis.
3. Randomization and blinding where feasible — attacks systematic bias in allocation and assessment.
4. Reporting of all analyzed outcomes and exclusions — attacks selective publication.
5. Authentication of cell lines, antibodies and models — attacks silent resource invalidity.
6. Public deposition of data and code — enables recomputation by others.

None of these is novel science; all of them are cheap relative to the cost of a research literature in which readers must discount every unreplicated claim by an unknown factor. The professional habit that ties them together is simple to state and rare in practice: treat a single paper as one observation, weight methods over results, and let replication — not press coverage — move a finding from interesting to usable.

How do retractions and corrections fit in?

The system's self-correction machinery runs on a separate track from review. Journals publish corrections for errors that do not affect conclusions, expressions of concern while questions are investigated, and retractions that remove a paper from the cited literature. The reporting guideline ecosystem makes each step more legible: when a checklist specifies what should have been disclosed, an omission is a documented deviation rather than a judgment call. NIH's rigor pages link editorial principles directly to what reviewers and readers can verify, which narrows the space in which an unreliable result can masquerade as a merely unconventional one.

What does this mean for industry readers?

For companies that consume academic literature — target selection, biomarker strategy, competitive intelligence — the reproducibility literature is a discount rate. A single paper supporting a target hypothesis is an input to prioritization, not a conclusion from it, and the same checklist a journal applies can be applied in-house: was the model authenticated, was the endpoint predefined, is the effect size plausible against the study's power. Diligence teams that formalize this filter spend less time discovering in Phase 1 what a careful read of the methods section would have shown.