What, Exactly, Did We Review?

Making Computational Review Version-Specific, 1 of 4

Computational review has a versioning problem.

A reviewer can inspect a repository, rerun a workflow, compare its outputs, and approve the associated claims. Then the repository changes. A dependency is updated, a data product is replaced, or a figure is regenerated from a different intermediate file. The approval remains visible—but what exact combination of claim, code, data, environment, execution, and output did it cover?

A link to “the repository” does not answer that question. Neither does a green test badge. Both can be useful, but both can move.

We have been developing a small design proposal around this problem: the Minimum Credible Reproducibility Protocol, or MCRP. Its central idea is that computational review should act on an immutable, version-specific object. We call that object a Scientific Record Release.

The release binds together:

• the material claims under review;
• the evidence and outputs offered for each claim;
• the executions, workflows, configurations, environments, and source artifacts that produced them;
• the checks that were actually performed; and
• the human decisions that remain necessary after the machines finish.

Each accepted release has a content identity. A machine result and a human approval refer to that identity, not to a mutable branch or a generic project page. If a material dependency changes, the old record is preserved, a new candidate is created, and affected checks and decisions must be re-established.

That sounds simple, but it changes the meaning of a review result. “The code ran” becomes “this identified workflow ran against these identified inputs, under this recorded environment, and produced outputs that satisfied these declared checks.” “The analysis was approved” becomes “these reviewers accepted these claims for this release, within these stated boundaries and limitations.”

MCRP is not a claim that provenance graphs, research objects, signed records, executable papers, or independent reruns are new. They are not. The narrower proposal is to join established pieces into a prospective lifecycle with three rules:

1. acceptance conditions are declared before the external verification run;
2. verification and scientific judgment are recorded separately from author-controlled production; and
3. changes have explicit consequences for the claims and decisions they can reach.

The current technical supplement is deliberately modest. It defines the lifecycle, compares it with adjacent work, and exercises a synthetic reference implementation. It does not show that MCRP improves reviewer accuracy, reduces effort, works across scientific domains, or deserves adoption as a standard. Those are evaluation questions, not conclusions.

Read the MCRP technical supplement.

This is the first of four short posts. The next will examine the claim–evidence structure: how a reviewer moves from a scientific statement to the exact outputs, executions, and dependencies offered in its support—and where a structurally complete record can still conceal a scientific gap.

Series roadmap

1. What, Exactly, Did We Review? — immutable releases and version-bound decisions
2. From Claims to Evidence — traversable review records and their blind spots
3. Who Is Allowed to Say It Passed? — agents, external enforcement, and scientific authority
4. What Happens After Acceptance? — freshness, failure, cost, and the evaluation agenda

---

Written by Codex and junior on 2026-08-16