GOAL: Reference-Title Backfill & Evidence Re-Verification

Proposed qualityevidencereferencescampaign NIH: Enhancing Scientific Rigor, Transparency and Replicability
Home / Projects / GOAL: Reference-Title Backfill & Evidence Re-Verification

GOAL: Reference-Title Backfill & Evidence Re-Verification

This is a campaign "/goal" — a durable runbook, not a routine curation project. It defines the objective, the per-disease subagent contract, and the definition of done. Executed via the existing parallel-dispatch skills (superpowers:dispatching-parallel-agents, superpowers:using-git-worktrees, dismech-references, create-pr, superpowers:receiving-code-review, superpowers:finishing-a-development-branch). No new tooling is required.

1. Objective

For every disorder entry that has a reference or inlined evidence item lacking a title, produce a PR that:

  1. Backfills the missing title — reference_title on each EvidenceItem, title on each top-level references[] PublicationReference.
  2. Re-verifies the evidence — re-reads each cited source (cached abstract, and full text / PDF when available) and confirms, in the gestalt of the whole entry and with domain knowledge of the disease, that: - the snippet is an exact quote from the source, and - the snippet genuinely supports (or refutes, per supports:) the claim it is attached to.
  3. Repairs or flags anything that fails re-verification (§5.5).

This is deliberately more than the mechanical reference_title backfill the weekly-compliance job already does. The title backfill is the occasion; the manual re-verification is the point.

2. Scope / worklist

Update (2026-07-25): the title half of the worklist is now empty. A repo-wide mechanical backfill (just backfill-reference-titles) added every remaining missing title in one pass, sourcing each one verbatim from references_cache/ frontmatter. find_missing_reference_titles.py now reports 0.

This does not complete the campaign. The title backfill was only the occasion; §1.2 — manually re-reading each cited source to confirm the snippet is an exact quote that genuinely supports its claim — is untouched and remains the point. The §7 "0 files missing titles" criterion can no longer be used on its own as evidence that a disease has been re-verified. Track re-verification progress by the reference-verification PR label instead, and re-derive per-disease worklists from evidence items that have never been through §5.4 rather than from the (now empty) title report.

Generated by scripts/find_missing_reference_titles.py (run at campaign start and re-run to refresh):

The authoritative worklist is the live output of that script — never a hardcoded list. A subagent must re-derive its file's status against origin/main before starting (§5.2), because other PRs will be shrinking the list concurrently.

Regenerate the worklist:

uv run python scripts/find_missing_reference_titles.py            # summary table
uv run python scripts/find_missing_reference_titles.py --details  # per-file ref IDs

3. Unit of work

One disease = one PR = one subagent. Never batch diseases into a shared PR (shared branches multiply conflict risk on the term/reference caches, and defeat per-disease review). The 6 top-level-title files also get the same full re-verification treatment, not just the title: fill.

4. Isolation & worktree strategy

Each subagent works in a temporary worktree, not under ~/worktrees/ (the user does not want this campaign cluttering that directory):

GOAL_ROOT=/tmp/dismech-goal-worktrees        # or: $(mktemp -d)
SLUG=<Disease_File_Stem>
git worktree add -b goal/refverify-<slug> "$GOAL_ROOT/$SLUG" origin/main
cd "$GOAL_ROOT/$SLUG"
pwd   # MUST print a path under $GOAL_ROOT — confirm BEFORE any file write

Hard guardrail (learned from the curate-next production incident): after cd, run pwd and confirm the printed path is the temp worktree and is NOT the parent checkout (/Users/cjm/worktrees/dismech-* or /Users/cjm/repos/dismech) before any Read/Edit/Write/Bash file operation. For Read/Edit/Write (absolute paths), root every path in the worktree; for Bash, use relative paths after cd. Two of three agents in the prior incident wrote into the parent checkout — do not repeat it.

On completion, remove the worktree: git worktree remove "$GOAL_ROOT/$SLUG" (keep the branch until the PR merges).

5. Per-disease subagent contract

5.1 Preconditions

Dispatched (via Agent, subagent_type: general-purpose, run_in_background: true) with: the disease file stem, its list of untitled reference IDs, and this spec.

5.2 Preflight (re-derive, don't trust stale state)

git fetch origin main
git worktree add -b goal/refverify-<slug> "$GOAL_ROOT/<slug>" origin/main && cd "$GOAL_ROOT/<slug>"
uv run python scripts/find_missing_reference_titles.py --details | sed -n '/<Slug>.yaml/,/^$/p'
gh pr list --repo monarch-initiative/dismech --state open \
  --search "<Slug>" --json number,title,headRefName,url

If the file no longer appears in the worklist, or an open PR already touches it, stop and report the skip — do not open a duplicate PR.

5.3 Backfill titles

5.4 Re-verify every claim (the manual core)

Invoke the dismech-references skill. For each evidence item and each top-level finding: 1. Read the source: cached abstract in references_cache/; full text / PDF when available (repo full-text cache, or PubMed MCP get_full_text_article); fetch if absent. 2. Confirm the snippet is an exact substring of the source text. 3. Confirm the snippet actually supports the attached claim — judged against the whole entry (does this evidence belong on this node? does supports: match? is evidence_source: classified correctly?) and your background knowledge of the disease. Watch for Named Entity Confusion (wrong-disease sources) per the CLAUDE.md NEC preflight.

5.5 Repair-or-flag policy (unsupported findings)

Apply the CLAUDE.md evidence SOP, documenting every content change in the PR body: - Fix the snippet to an exact, genuinely-supporting quote from the same source; else - Swap to a better-supporting reference (fetch + validate it); else - Demote the claim to notes: / remove the evidence block if nothing supports it. - If a claim is only checkable from full text that is unavailable, or the correct fix is a judgment call, leave the content and flag it in the PR body for human adjudication rather than deleting real signal.

Stay in scope: touch references, evidence, findings, titles, and the minimal fixes those reveal. Do not re-curate the whole entry (no new phenotypes, modules, treatments) — that dilutes review and is a different goal.

5.6 Validate + provenance

just validate kb/disorders/<Slug>.yaml
just validate-kb-references kb/disorders/<Slug>.yaml
just validate-terms kb/disorders/<Slug>.yaml
just check-reference-cache-frontmatter
just new-history --kind disorder --slug <Slug> --event REVIEW --outcome changed \
  --summary "Reference-title backfill + evidence re-verification" \
  --agent-tool claude-code --model <model-id> --sections evidence,references \
  --details "Backfilled N titles; re-verified M evidence items; <repairs/flags>."

5.7 Open the PR

Follow create-pr. Targeted git add only (kb/disorders/, references_cache/, cache/, history/, research/ if applicable) — never git add -A. PR body states: titles backfilled, items re-verified, every repair/removal, anything flagged for humans, and validation output. Label the PR reference-verification for campaign tracking.

5.8 Monitor to done — the subagent is NOT finished until:

Conflict resolution is content-changing, do it VERY carefully (CLAUDE.md "Refresh your own branch safely"): git fetch origin && git rebase origin/main; then review git diff --name-status origin/main...HEAD and --stat; stop and fix if you see unrelated deletions, stale reversions, or protected-path churn; if phantom case-sensitivity diffs block the rebase, see the macOS workaround in project memory. This is the subagent's own branch, so rebase/force-push of its branch is fine — never touch anyone else's.

Done = green + approved + mergeable → STOP and report the PR URL. Do not click merge; leave the final merge to a human or the common deterministic PR Shepherd controller. If the reviewer and agent reach an impasse, or a conflict can't be resolved cleanly: assign @cmungall, comment with the specifics, and stop — never a review back-and-forth (CLAUDE.md).

6. Orchestration & rollout

Pilot → scale. 1. Pilot (5–10 files). Pick a representative spread: 2–3 high-count entries (e.g. Cystic_Fibrosis 176, Stargardt_Disease 115), 2–3 top-level-title files (e.g. Urea_Cycle_Disorder, Ataxia_Telangiectasia), and 1–2 structured-source-heavy entries (ORPHA/NCIT evidence). Run the full contract. Human reviews the pilot PRs and subagent behavior; refine this spec. 2. Scale. After the pilot gate, process the remainder with a rolling concurrency window of ~4–6 subagents, refilling as each PR reaches "done". The cap protects the single auto-reviewer, the shared CI queue, and limits merge-conflict churn on cache/**/*.csv and references_cache/ additions.

Progress tracking. Re-run the worklist script to measure remaining files; track PRs by the reference-verification label. The campaign is complete when the worklist returns 0 files and every campaign PR is merged or standing green-and-mergeable awaiting the human merge gate.

7. Definition of done (campaign)

8. Guardrails (do-not-repeat list)