# v3.7.3 contamination_signals backfill — migration guide

**Tool:** `scripts/migrate_literature_corpus_to_v3_7_3.py`
**Issue:** [#105](https://github.com/Imbad0202/academic-research-skills/issues/105)
**Spec:** v3.7.3 §3.2 R-L3-2-B (the deferred batch operation)

---

## When to run

After upgrading to ARS v3.7.3+ if your Material Passport's `literature_corpus[]` was populated by an adapter run **before** v3.7.3 was deployed.

How to tell: open the passport YAML and look for `contamination_signals` under any `literature_corpus[]` entry. If absent on entries you trust were produced by an adapter, you have a pre-v3.7.3 corpus and this tool is for you.

If `contamination_signals` is already present (because bibliography_agent v3.7.3+ produced the entry at ingest time), the tool will detect that and skip — idempotent re-run is safe.

## What it does

For each pre-v3.7.3 entry in `literature_corpus[]`, the tool computes both v3.7.3 advisory signals:

- **Signal 1 (preprint_post_llm_inflection):** `year >= 2024 AND venue ∈ {arXiv, bioRxiv, medRxiv, SSRN, Research Square, Preprints.org, ChemRxiv, EarthArXiv, OSF Preprints, TechRxiv}`. Deterministic; no API needed.
- **Signal 2 (semantic_scholar_unmatched):** Semantic Scholar API lookup per `deep-research/references/semantic_scholar_api_protocol.md`. Returns `True` on no-match, `False` on match.

It then writes `contamination_signals: {preprint_post_llm_inflection, semantic_scholar_unmatched}` plus a scalar provenance timestamp `contamination_signals_backfilled_at` (ISO-8601 UTC) onto the entry.

**Exemptions (per spec §3.2):**

- `obtained_via=manual` entries: Signal 2 is OMITTED (user has vouched; running automated unmatched check on grey literature / books / working papers would surface false positives).
- Semantic Scholar API degradation (network failure / rate-limit exhausted / 5xx): Signal 2 is OMITTED, not set to False. Absence ≠ negative confirmation. Re-run later when API is healthy to fill in the field.

## How to run

```bash
# Always dry-run first
python scripts/migrate_literature_corpus_to_v3_7_3.py --dry-run --verbose /path/to/passport.yaml

# Single passport
python scripts/migrate_literature_corpus_to_v3_7_3.py /path/to/passport.yaml

# Directory scan (non-recursive; only *.yaml at the top level)
python scripts/migrate_literature_corpus_to_v3_7_3.py /path/to/passports/
```

The tool prints a one-line report on stdout:

```
processed=42 patched=37 skipped_already_migrated=5 skipped_insufficient_data=0 dry_run=False
```

`skipped_already_migrated` = entries that already had `contamination_signals` set (idempotency guarantee).
`skipped_insufficient_data` = entries missing `year` (a schema invariant; only reachable if the YAML was hand-edited).

## Idempotency guarantee

Re-running the tool on a migrated passport produces zero changes. The detection rule is "skip if `contamination_signals` is present on the entry"; this rule is keyed on object presence, not on the `contamination_signals_backfilled_at` timestamp, so the first migration's timestamp is preserved across re-runs.

The tool uses `ruamel.yaml` round-trip serialization to preserve comments, key order, and quoting style. Diffs across an idempotent re-run are byte-identical.

## Semantic Scholar rate-limit considerations

The unauthenticated Semantic Scholar API allows ~1 request per second. The migration tool retries on HTTP 429 per the existing protocol (`deep-research/references/semantic_scholar_api_protocol.md` — 2s backoff × 3). For a corpus of 5,000 entries this means ~1.5 hours of API time.

If the migration is interrupted (Ctrl+C, network drop, system restart), re-run the tool — the idempotency rule ensures already-migrated entries are skipped. Entries that got `semantic_scholar_unmatched` omitted due to API failure during the interrupted run will have the field filled in on the next successful pass.

Resumable checkpoint state is **not** implemented in v3.7.3; see #105 design doc §8 for the deferred-scope rationale (will be reconsidered if a user reports a corpus > 10,000 entries).

## What this tool does NOT do

- Does not modify `bibliography_agent` ingest behavior (v3.7.3 already computes signals at ingest).
- Does not touch the orchestrator's cite-time finalizer (v3.7.3 finalizer reads `contamination_signals` directly from each entry).
- Does not validate the rest of the passport schema (other v3.7.x lints handle that).
- Does not migrate `literature_corpus_entry.schema.json` itself (the schema field is purely additive; no breaking change).
- Does not implement OpenAlex / Crossref Vector 3 signals (see #102 for the v3.7.4 plan).

## Backward compatibility

- Pre-v3.7.3 entries lacking `contamination_signals` remain schema-valid.
- v3.7.3 ingest-time entries (which lack `contamination_signals_backfilled_at`) remain schema-valid.
- Migrated entries carry both fields and remain schema-valid.

## Related

- v3.7.3 spec: `docs/design/2026-05-12-ars-v3.7.3-claim-faithfulness-and-contaminated-source-spec.md` §3.2
- #105 design: `docs/design/2026-05-15-issue-105-contamination-signals-backfill-design.md`
- Schema: `shared/contracts/passport/literature_corpus_entry.schema.json`
- Tests: `scripts/test_contamination_signals.py` + `scripts/test_migrate_literature_corpus_to_v3_7_3.py`
