abstract_only recovery census: how many references_cache entries have full text sitting in PMC

2026-09-16. Follow-up to #10876 / #10878. Generated by scripts/audit_abstract_only_recovery.py (just abstract-only-recovery-census); regenerate with --phase all. The recoverability phase took just under 4 hours to compute without an NCBI_API_KEY (E-utilities without a key is rate-limited to 3 req/s; setting the key raises that to 10 req/s) — the 1000-char recoverability floor and the HTML-fallback attempt described in Methodology add requests on top of a plain efetch sweep.

Summary

Issue #10878 asked whether the JATS extractor bug fixed in #10876 — a whole-document word match on "restricted" that discarded complete PMC full texts, plus the fact that tables were never extracted at all — left other references_cache/PMID_*.md entries stranded as abstract_only, and asked for a census before deciding a refetch policy rather than a blanket rewrite.

It found a lot more than the bug's own blast radius. Of the 31,757 references_cache/PMID_*.md files cached abstract_only as of this census's snapshot (see the note on drift in Methodology):

	Count	% of all abstract_only
Have a PMC record at all	12,355	38.9%
Genuinely recoverable (real body, via the actual production extractor)	8,482	26.7%
— of those, previously attempted and stuck anyway	2,610	8.2%
— of those, never attempted for full text at all	5,872	18.5%
— of those, would also gain ≥1 table	5,809	18.3%
Not recoverable (PMC record exists but not Open Access, or genuinely empty)	3,873	12.2%
No PMC record at all	19,402	61.1%

8,482 papers, not the ~280 the issue's own sample extrapolated to. That sample (200 files, 2 PMC hits) was off by roughly two orders of magnitude on the PMC-linkage rate alone — see Why the original sample missed this. This report is a full census over every current abstract_only file, not another sample, and its PMC-linkage step is cross-validated against the production resolver (Entrez.elink) independently of the bulk API it mostly uses (see Methodology).

**Only about a third of the recoverable set is actually attributable to the

10876 bug specifically.** 2,610 files (8.2% of all abstract_only) were

already marked full_text_attempted: true and are stuck as abstract_only anyway — these are the plausible bug victims (or casualties of some other transient fetch failure) #10878 was written to ask about. The other 5,872 (18.5%) were never attempted for full text at all — most likely cached before full-text fetching existed in this pipeline, or through a code path that never tries PMC. Refetching either group recovers the same missing evidence, but only the first group is a #10876 regression; the second is a separate, larger backlog that happens to share a remedy.

Why the original sample missed this

10878's 200-file random sample found 2 PMC-linked papers (1%) and

extrapolated to ~280 recoverable across the corpus. This census finds a PMC record for 38.9% of all abstract_only files — independently confirmed two ways:

The bulk PMC ID Converter API (pmc.ncbi.nlm.nih.gov/tools/idconv), used for the full census because it resolves up to 200 IDs per request.
Bio.Entrez.elink(dbfrom="pubmed", db="pmc", linkname="pubmed_pmc") — the exact call the production PMCFullTextProvider._resolve_pmcid makes — run individually against an independent random sample of 50 abstract_only PMIDs. 15/50 (30%) resolved to a PMCID, and every one of the 50 agreed with the bulk API's verdict (50/50 match).

At a true rate anywhere near 30–39%, a random sample of 200 finding only 2 hits is not plausible sampling noise (the expected count would be 60–78). This report cannot say what went wrong in the original check — the issue does not show its method beyond "asked PMC about each one" — but the cross-validated number here is the one to act on.

Why "has a PMCID" overstates recoverability by 1.46x

A PMCID only means PMC indexes the record, not that its body is servable. PMCFullTextProvider.locate first calls Entrez.efetch(db="pmc", id=pmcid, rettype="xml", retmode="xml") — this census calls that identical endpoint with identical parameters, not a reimplementation. Most PMC records are not in the Open Access subset, and for those efetch returns front matter only: no <body> element, often with an explicit  comment and <meta-name>pmc-prop-open-access</meta-name><meta-value>no</meta-value>. 3,842 of the 12,355 PMC-linked abstract_only files (31.1%) are exactly this case — old, PMC-indexed, not Open Access. If the resulting XML body is missing or falls at or under its _MIN_PMC_FULLTEXT_CHARS floor (1000 characters), production falls back to fetching the PMC article's HTML page instead. This census's _fetch_pmc_html copies that URL template and selector logic from upstream verbatim, but the HTML fallback recovered nothing in this run — 0 of the 3,873 not-recoverable records rescued a usable body that way — and that result is not currently trustworthy as a finding about PMC's access policy. A PR review caught a bug: the PMCID this census gets from the bulk ID Converter API already carries a PMC prefix (e.g. PMC5593426), while PMCFullTextProvider's own Entrez.elink path returns a bare numeric id, so the shared URL template — copied correctly for the numeric-id case — doubled the prefix here into .../articles/PMCPMC5593426/, a URL that 404s. That bug is now fixed (pmcid.removeprefix("PMC") before formatting), but a corrected-URL fetch of the known-good PMC5593426 (Open Access) from this environment still extracted no body: the page returns HTTP 200, but its article text now sits inside <section class="main-article-body">, not the <div class= "article-body">/<div class="tsec"> markup _fetch_pmc_html (and apparently the version of PMCFullTextProvider this was copied from) looks for. Whether that is a live PMC template change or something specific to scripted access from this environment is not established here. Given that, this report draws no conclusion about whether the HTML fallback would ever recover a body — the 0-of-3,873 figure reflects what this run's fallback path actually returned, not a verified statement about PMC's non-OA gate. This is why the recoverability phase fetches and classifies every PMC-linked candidate with the real, patched XMLExtractor.extract() rather than stopping at "has a PMCID": 12,355 PMC-linked candidates against 8,482 truly recoverable is a 1.46x overstatement (38.9% vs. 26.7% of all abstract_only, or equivalently 31.4% of PMC-linked records not recoverable) — a figure the XML-path fetch alone already establishes and the fallback question does not affect.

The false-positive skew is not uniform across the corpus either: the very lowest PMIDs (oldest papers) are overwhelmingly not_open_access, while recent papers are overwhelmingly recoverable. A partial or PMID-ordered sample will over- or under-state the true rate depending on which end it starts from — worth flagging since it is exactly what produced a misleading early read while this census was running (the first ~270 candidates checked, in ascending PMID order, showed under 1% recoverable; the full 12,355 showed 68.7%).

Tables

5,809 of the 8,482 recoverable papers (68.5%) also carry at least one JATS <table-wrap> that _jats_tables_as_text would extract — 14,363 tables in total across the recoverable set. Every one of these 5,809 came back through the XML path: no record in this run recovered a body via the HTML fallback (see Why "has a PMCID" overstates recoverability for why that 0 is not yet a settled finding), and in any case _jats_tables_as_text only ever parses JATS <table-wrap> markup, so an HTML-recovered body would carry text but never a table regardless. This is on top of the already-cached full_text_xml corpus: of 7,433 such files, only 313 currently carry a ## Table section, leaving up to 7,120 that predate table extraction and may be missing tables their cached body never captured (an upper bound — some genuinely have none; confirming which requires refetching each, the same as the abstract_only case).

Methodology

Three phases, offline where possible:

Enumerate (--phase missing-tables component; offline): scan references_cache/PMID_*.md, read content_type from the frontmatter using dismech.frontmatter.split_frontmatter (a naive text.split("---", 2) truncates the frontmatter on some files — see A frontmatter-parsing footnote below). The full_text_xml/table figures in this report (7,433 / 313 / 7,120) are current as of this report's own commit, since that phase re-scans the corpus fresh on every run. The abstract_only count that seeded the idconv/recoverability candidate list below is not: it was captured at 31,757 on 2026-09-16 ~14:25 UTC, before this branch merged in main's subsequent curation activity; references_cache/ now holds 31,810 abstract_only files. Curation adds a few dozen abstract_only files a day in the ordinary course of the KB growing, so this is expected drift, not an error — but the summary table's recoverable/attempted/never-attempted figures below are pinned to the 31,757 snapshot the idconv and recoverability phases actually ran against, not to today's live count.
idconv: bulk-resolve every abstract_only PMID against the PMC ID Converter API in batches of 180 (its cap is 200), ~180 requests total. Resumable — writes incrementally and skips PMIDs already resolved.
recoverability: for every PMC-linked candidate, efetch the actual article XML (identical call to PMCFullTextProvider._fetch_pmc_xml_bytes) and classify it with the real dismech-patched extractor — import dismech.patch_reference_validator (which applies the monkeypatch) then call the installed linkml_reference_validator.etl.extract.xml. XMLExtractor().extract() directly (with the same content_type= "application/xml" keyword production passes), plus dismech.patch_reference_validator._jats_tables_as_text() for the table count. This XML path is the live production code, unmodified, so the 8,482-recoverable headline is what a real refetch would produce today. If the extracted body is missing or at/under the production _MIN_PMC_FULLTEXT_CHARS floor (1000 characters), production additionally falls back to fetching the PMC article's HTML page (PMCFullTextProvider._fetch_pmc_html); this census's copy of that step had a bug during this run (see Why "has a PMCID" overstates recoverability) that is now fixed but not yet re-verified as recovering anything, so the fallback's contribution to this report is 0 by observation, not by a confirmed match to production behavior. Also resumable. Took just under 4 hours for the full 12,355-candidate set at the anonymous E-utilities rate limit (3 req/s).

Verified against ground truth throughout: the known BRIDA paper (PMID:28530713 / PMC5593426, the case that motivated #10876) is correctly classified has_body=True, num_tables=1; a known non-OA record (PMC3346298) is correctly classified not-recoverable; the idconv/elink cross-check above agreed on every one of 50 independently sampled PMIDs.

scripts/audit_abstract_only_recovery.py implements all three phases; just abstract-only-recovery-census --phase all reruns them (default --out-dir tmp/abstract-only-census, gitignored — this report's numbers are the frozen record, not the live CSVs). --phase summarize reports on results already on disk without hitting the network again, and now also breaks the recoverable count down by which path produced it (XML vs. HTML fallback) and by attempted-vs-never-attempted, so both axes in this report are direct tool output rather than hand-computed.

A frontmatter-parsing footnote

Using the delimiter-aware split_frontmatter instead of a naive text.split("---", 2) changes the enumerated count: 3 files — PMID_1928091.md, PMID_1899320.md, and PMID_2229018.md — have a --- sequence inside their body text ahead of the real frontmatter close, so the naive split truncates their frontmatter and a content_type read against that truncated text finds nothing at all (rather than merely finding it in the wrong place, which is what byte-windowed reads used in an earlier version of this script did to two other files). All three are correctly abstract_only and are included in the 31,757 above.

What this means for policy

10878 explicitly declined to propose a blanket refetch, since rewriting

thousands of references_cache/ files in one PR is exactly the merge-invisible-to-both-sides failure mode #8623 and #10061 already document for duplicate keys and retired enums — every open curation PR touching one of those 8,482 files would go conflict-red on merge, not error, just silently wrong.

That argues for batching, not for abandoning the recovery. Two batching axes this census makes available for free:

By attempted-vs-never-attempted. The 2,610 full_text_attempted: true files are the closest thing to a #10876 regression list and the most defensible first tranche.
By whether a table is gained. The 5,809 files that would also gain a table are higher-value re-curation candidates than the 2,673 that would gain body text only.

This report does not pick a batch size or cadence — that is a maintainer call informed by how much churn the repo can absorb per week, not something a census can decide. The full candidate list is reproducible on demand via just abstract-only-recovery-census --phase all (or --phase idconv / --phase recoverability alone, both resumable) rather than committed here as a static file, so whoever picks a batch size can regenerate a fresh, current list rather than working off a snapshot that drifts as curation continues elsewhere in the KB.