coulomb-social/docs/release/corpus-import-reconciliation-2026-10-31.md
tegwick 80fe387dcc Complete matching uncertain import checkpoints with retained evidence
Assistant: codex
Assistant-Model: gpt-6-astra
Assistant-Session: 01a192e9-c28b-7533-bf25-e2341364c22a
2026-10-31 18:07:34 +01:00

5.8 KiB

Completing a matching uncertain import checkpoint

CSOC-WP-0014-T07 now has an explicit checkpoint-completion operation for an already committed import whose reserved identity and complete source-derived state match. This is source preparation, not a production reconciliation or admission of any cohort. The existing native executor and frozen release packets are unchanged.

Owner procedure

Before any native use, the deployment owner must identify the exact original attempt, stop and independently verify application and external writers, retain the original checkpoint and a coherent paired recovery point, and confirm the source/tenant/identity/grant mapping was admitted. Neither of the commands below verifies an external writer fence or provider backup. The current production image does not contain these new commands; owner integration and release remain required.

  1. Inspect the original manifest and existing private receipt directory using inspect_content_corpus. Keep the result privately with the operation evidence.

  2. Review the selected zero-based unit index, reserved identity, source binding, complete file/grant/metadata comparison and the original operation's outcome. Only reserved_destination_matches_source, with no unattributed .import-* entries, is eligible for the operation below. A match describes current state; it is not proof that no intervening edit ever occurred.

  3. Supply the exact inspection SHA-256 and explicit completion action:

    python manage.py reconcile_content_import /private/reviewed/manifest.json \
      --receipt-dir /private/import-attempt \
      --unit-index 0 \
      --expect-inspection <inspection_sha256> \
      --complete-matching-commit
    

    The command reacquires the existing receipt/content locks and a consistent read-only SQL transaction, recomputes the whole inspection, and refuses a changed digest. It fsyncs the matching destination, rechecks the selected unit, source and checkpoint bytes, then atomically replaces only the checkpoint. The completed unit retains its original reserved identity, ordinary importer fingerprints/counts, prior started unit and full hash/count inspection evidence. SQL rows, grants and content bytes are not modified.

  4. Independently read the checkpoint and inspect again. A lost response or failure may occur after replacement; do not infer failure to complete from the absence of terminal output. The previous inspection cannot be replayed against an already completed checkpoint. An interrupted temporary checkpoint file is retained as evidence and is not automatically deleted or adopted.

  5. If continuation is separately admitted, the existing receipted importer skips the completed unit and preserves later edits and grants. Completion of this checkpoint does not authorize importing the remaining corpus.

Other states remain explicit owner work

Observed state Required disposition
Reserved identity differs from source Preserve later edits/grants; reconcile against the original operation and paired point. This command never overwrites them.
SQL only, files only, or neither Retain the uncertain checkpoint and inspect private staging, transaction outcome and the paired recovery point. No automatic reset to pending, retry, restore or deletion is implemented here.
Foreign identity or legacy unbound started receipt Resolve ownership and identity from independent evidence. Do not adopt a row because its slug matches.
Unattributed staging entries Establish their exact attempt/ownership and disposition separately. Matching published bytes do not authorize cleaning up those entries.
Pending or already completed unit No reconciliation mutation is permitted. Completed identities remain historical boundaries and may have valid later edits.

The helper uses cooperating file locks and PostgreSQL repeatable-read/read-only transactions. These are not an external writer fence. The return explicitly reports that limit, no SQL/content changes, no restore, and no authorization to continue imports. The owner must verify native prerequisites before execution; passing synthetic tests does not supply them.

Validation

The focused suite covers exact completion and evidence retention, unchanged SQL/ content, preservation of later edits on resumed import, stale evidence, changed source/mapping/checkpoint, partial or foreign stores, legacy receipts, unattributed staging, invalid selection, sync failure and drift during fsync. Real child-process exits immediately before and after checkpoint replacement leave either the prior started checkpoint or the completed checkpoint with its reconciliation evidence.

Full make check: 648 passed, 69 existing warnings, zero failures/errors/skips in 150.38s, including required browser journeys, rerun after the interrupted session. Nineteen new reconciliation cases pass; the focused corpus checkpoint/inspection/reconciliation selection passes all 52 cases. Log: /tmp/csoc-import-reconciliation-check-20261031.log.

The isolated PostgreSQL 16.14 rehearsal passes with 243 actual read-only and repeatable-read transaction checks. It uses synthetic data, explicit source and locked dependency overlays, an internal Docker network and no published ports. Its disposable containers, volumes and network were removed. Source-bound receipt: corpus-import-reconciliation-postgres-2026-10-31.json. This is not packaged candidate or production proof.

After resuming, all 12 source hashes in that PostgreSQL receipt were independently checked against the current files and matched. State Hub consistency passed and the T07 description was independently read back with the reconciliation evidence.

T07 remains in progress for the other reconciliation states, supervised owner integration, source/grant admission, native release/import/postflight and productive verdicts.