That feels very familiar from chemistry.

I am not a social scientist, so I will leave the empirical social-science side there. The chemistry version is easy to recognize. There is a large gap between a typical experimental section that gestures at what was done and something like Organic Syntheses, where procedures are written in much more detail and each reaction and characterization dataset is checked for reproducibility in the laboratory of a member of the Board of Editors.

That standard exists for a reason. Rick Danheiser wrote in C&EN that, from 1982-2005, about 12% of submitted Organic Syntheses articles were rejected because the results could not be reproduced. After more detailed author instructions and a procedure checklist were introduced in 2005-07, more than 95% of submissions checked with satisfactory reproducibility.

That is the part I keep coming back to. The problem is not always that an author is hiding something. Often they know the procedure so well that they no longer notice which details are load-bearing. Stirring rate, addition order, concentration, drying time, workup details, vendor grade, how dry the solvent really was, what “room temperature” meant that week in that lab. Anyone who has tried to repeat a reaction from a too-short experimental section knows this feeling.

This is where AI tools could be useful without pretending to be the chemist. Not “write my experimental section,” and definitely not “certify that this procedure works.” More like: read this procedure as an annoying first-year graduate student who has to run it tomorrow, and ask what is missing.

There is already adjacent work here. A Nature Communications paper converted prose synthesis procedures into structured action sequences for chemical synthesis. That is not the same thing as reproducing a reaction, but it points in the right direction: if a procedure cannot be converted into concrete actions, quantities, conditions, and decision points, it probably is not as complete as it looks.

I wrote recently that ground truth in science is still reality, just harder to access than a test suite. That remains true. AI cannot tell you whether a reaction will work in the lab without someone eventually doing the experiment. But it may be able to tell authors where their methods section stops being a recipe and starts being a memory aid for the person who already knows what happened.

That would be a useful tool. Slightly irritating, probably. But useful.

Classic study gave 146 economist teams the same dataset & got wildly different answers. New paper reruns it with agentic AI. Claude Code & Codex land near the human median but with far tighter dispersion & no extremes.

I’m torn between the reproducibility (the tight clustering) and what it might cost in AI-assisted scientific creativity. Barry Marshall looked at the same gastric biopsy data as everyone else and reached the conclusion the field had ruled out. That kind of outlier isn’t noise; it’s occasionally how science moves. If AI reliably clusters near the median human interpretation, it scales up the research we already know how to do. It won’t find the next H. pylori.

“There just isn’t anybody whose job it is to actively look for it.”

Not surprising. Many researchers are working in Excel rather than reproducible pipelines, and journals don’t have the bandwidth to audit supporting data. If anything, 3% seems low?

The obvious fix is an AI toolbench for pre-publication data validation — something that catches this before it enters the literature rather than years after. The verification loop is precisely what experimental science is missing.

Anthropic is the sole major AI laboratory publishing system prompts for consumer-facing chat interfaces, with archives extending back to Claude 3.

Worth noting that if you’re using the API you might be writing your own system prompt anyway, so this mostly matters for claude.ai users. The harder problem is underlying model behavior: when a researcher publishes results generated with Claude Opus 4.6, what would it take to reproduce that in two years? Code Ocean does something like this for computational environments — pin the entire runtime alongside the paper, executable on demand, and Nature has integrated it into peer review. Nobody is doing the equivalent for AI model versions in research workflows yet.

The hard part is building the product sense necessary to evaluate the learnings and guide the direction.

Similarly, an AI editor could confirm or deny whether a paper’s claims are likely to be true, but in a coming age of radical overabundance of valid research it’s the taste of the editor that matters — selecting which papers their audience would actually care about. Product sense works the same way: it isn’t verification, it’s curation.

The obvious question is: checking against what?

That could mean internal self-consistency — trying a calculation two ways, looking for contradictions in its own reasoning. Useful, but it doesn’t escape the model’s own knowledge boundaries. Or it could mean external retrieval — and for most deployments today, that means a web search. That’s better than nothing, but it’s a weak verification tool for scientific claims. The web will tell you that fish oil is associated with cardiovascular health. It won’t tell you whether the mechanism-of-action proposed in a 2019 paper has been confirmed, challenged, or quietly superseded by six subsequent studies. For that, you need something structured.

Which raises a more interesting question: what would Opus 4.7’s verification loop look like if it had access to a proper scientific knowledge graph — not search, but a graph of claims made across the literature, tagged with confidence, provenance, and the network of studies that support or contradict them? Or better still, causal datasets: not “paper A mentions compound X and outcome Y” but “experiment N demonstrated cause-effect at dose Z, replicated three times.”

I’ve written before about how the speed of the verification loop is what separates fields where AI has transformed research from fields where it hasn’t (yet). Math closes the loop via proof assistants; drug discovery historically couldn’t close it in under months. That’s changing — Exscientia’s closed design-make-test-learn cycles, Periodic Labs building automated materials discovery. But closing the experimental loop is a separate problem from connecting AI reasoning to the existing literature — and that side has barely started.

A model that actively seeks to verify its reasoning is only as good as what it can verify against. Right now we’re giving it the open web. The more interesting engineering problem is connecting it to the structured record of what science has actually established — and what it hasn’t. Wiley’s Scholar Gateway and Nexus Domains are attempts at this — Scholar Gateway for in-session retrieval via MCP, giving Claude and other AI systems access to peer-reviewed literature rather than the open web; Nexus Domains for curated content feeds delivered via API and MCP to enterprise R&D pipelines. These are first steps in building the right verification layer. The question Opus 4.7 makes newly urgent is whether the rest of the field catches up.

The structural reason mathematics and software development got there first is worth pausing on. Both have fast automated verification built in — proof assistants like Lean for math, test suites and type checkers for code. The loop closes in seconds. Drug discovery has never had that — the verification step is a wet lab experiment that takes weeks or months.

That gap is getting shorter. A dynamic flow system at NC State, published last year in Nature Chemical Engineering, generates ten times more experimental data than previous approaches by monitoring reactions in real time rather than waiting for steady state. Exscientia has been running closed design-make-test-learn cycles in its Oxford robotics facility since late 2024. Periodic Labs, which launched last October with a $300M round from founders of ChatGPT and GNoME, is building explicitly toward this for materials discovery.

The distinguishing factor between disciplines where AI has already transformed research and those where it hasn’t isn’t the AI. It’s the speed of the verification loop. Mathematics and software development had that built in. Experimental science is engineering its way to the same place.

The Quanta piece reads like a preview.

Skills are great for pure knowledge and teaching an LLM how to use an existing tool. But for giving an LLM actual access to services, the Model Context Protocol (MCP) is the far superior, more pragmatic architectural choice.

In practice, some publishers aren’t forcing the choice. Wiley’s Knowledge Nexus offers both — MCP if you want to point an LLM at it directly, API if you’d rather build your own integration. Whichever fits your stack is probably fine.

Social media claims are mostly atomic: a politician said something, a statistic is cited correctly or not, an event happened or didn’t. You can check those against a corpus. Scientific claims are relational — they assert relationships between entities distributed across thousands of papers, and the “truth” of the claim is a property of the network, not any individual document. Asking an LLM to fact-check “compound X inhibits pathway Y at therapeutic doses” requires knowing what the literature establishes about X’s mechanism, Y’s context-dependence, and whether the relevant concentrations have ever appeared in the same study. A retrieval system can find text that mentions both; it can’t tell you whether the relationship holds.

This is precisely what knowledge graphs were built for. Don Swanson demonstrated it in 1986: he found that fish oil and Raynaud’s syndrome research had never cited each other, yet traversing the relationships — fish oil inhibits platelet aggregation, platelet aggregation implicated in Raynaud’s — produced a testable hypothesis. No document stated it. The connection existed only in the graph. A clinical trial three years later confirmed it.

Thirty years on, Himmelstein et al. built Hetionet: 47,000 nodes, 2.25 million relationships, 29 biomedical databases integrated into a single graph. They used it to generate drug repurposing predictions across 209,000 compound-disease pairs. Most of those candidates couldn’t be found by searching the literature because no paper had connected them — that’s what made them candidates worth testing.

The reason I keep coming back to this is that “fact-checking” is actually the least interesting thing a knowledge graph enables. Verification looks backward: does this claim hold given what we know? Discovery looks forward: what does the structure of existing knowledge imply that nobody has tested yet? Swanson and Himmelstein were doing the second thing. An AI system built on structured biomedical knowledge could do both simultaneously — flagging claims that contradict established relationships while surfacing hypotheses that the graph supports but the literature hasn’t yet stated.

The infrastructure question is the hard one, and also the interesting one. Building a knowledge graph like Hetionet is, in a real sense, constructing a digital twin of the scientific record — a computable representation of what the literature actually establishes about how the world works. Ground truth in science is still reality, just harder to access than a test suite. A well-constructed knowledge graph is the closest thing we have to making it queryable. Agents can already find errors faster than humans can triage them — the bottleneck isn’t computation, it’s the structured representation of what science actually knows. That’s a much larger project than building a better Community Notes, and a much more valuable one.