What data extraction errors actually look like

It’s 11pm and you’re staring at a cell in your extraction grid, not sure whether to trust it. The paper is open in the other window. You know the number is plausible. But plausible is exactly what a wrong number looks like, and the errors that survive to print are never the obvious ones. So it’s worth knowing, concretely, what a data-extraction error actually is. The audit literature (Gøtzsche and colleagues’ review of standardized-mean-difference meta-analyses among them) has catalogued these shapes for years. They fall into a small number of recurring types.

The taxonomy

Transposition and digit slips. The trial reports 13.4; the grid says 14.3 or 1.34. Pure transcription noise, uncorrelated with meaning, which is both its weakness and its danger: it produces a number that is off by an order of magnitude or a swapped pair, sometimes obvious, often not.

Wrong arm or wrong timepoint. The trial has three arms and four visits. You want the 12-week high-dose value and you copy the 8-week value, or the middle arm. Nothing is mistyped. The number is real, it just answers a different question than the one your row is asking.

SE reported, SD extracted (and its CI cousin). The paper prints a mean with a standard error or a confidence interval; the grid records it as a standard deviation. These differ by a factor of √n, so the error scales with sample size:

suppose mean = 5.0, SE = 0.4, n = 50
SD = SE × √n = 0.4 × 7.07 = 2.83
recording 0.4 as the SD understates spread ~7×

Unit drift. Glucose in mmol/L copied into a column of mg/dL; weight in pounds among kilograms; a percentage read as a proportion.

suppose fasting glucose = 5.5 mmol/L
in mg/dL: 5.5 × 18.0 = 99
a 5.5 sitting among 90-110 values reads as a crash, not a unit

Change-versus-endpoint mixups. One trial reports mean change from baseline; the next reports final values; your row treats them as the same quantity. The means may be comparable, but the SDs are not: the SD of a change score depends on the pre-post correlation and is generally smaller than the SD of the endpoint. Pool them naively and you mis-weight both trials.

What dual extraction catches, and what it can’t

Two independent extractors, then reconciliation, is the field’s gold standard for a reason: it catches independent errors. A transposition, a misread, a slip to the wrong arm: one person makes it, the other doesn’t, the disagreement surfaces at reconciliation, and you resolve it against the source.

The structural blind spot is correlated error. When the mistake is driven by the paper’s own ambiguity, both extractors tend to make the same one. A table labelled only “mean ± 0.4” invites both readers to call it an SD. A protocol with a fuzzy primary timepoint sends both to the same wrong visit. A results section that never states its units lets both assume the common one. There is no disagreement to reconcile, so the error passes silently through the process built to stop it. Dual extraction reduces variance; it does not remove shared bias.

What makes an error auditable

The reason a transposition can survive peer review is not that it’s subtle. It’s that, once transcribed, the number is detached from where it came from. An extracted value is auditable only if it stays bound to three things: the source quote and location it came from, the transformation applied to it (unit conversion, SE→SD via Handbook §6.5.2.3, the paired route for crossover in chapter 23), and the assumptions that transformation required (which n, which CI level, which correlation).

With that binding, every error type above becomes a question someone can answer two years later: which arm, which visit, was that an SE, what unit, change or endpoint. Without it, a reviewer can’t distinguish a typo from a deliberate derivation, so every number is equally suspect. This is the real argument for extracting numbers in a way that carries their provenance forward automatically: not that a machine reads more carefully than a person, but that a value which can still name its own source is one you can actually check.