coulomb-social/docs/release/source-date-journal-review-2026-10-30.md
tegwick 2344b2d18b
Some checks failed
Use-case regression gate / usecase-regressions (push) Failing after 8m0s
Guard creation-date correction with full-store checks and SQL writer locks
Assistant: codex
Assistant-Model: gpt-6.1-sol
Assistant-Session: 01a18e2d-f513-7742-b8e7-12af0b744ff3
2026-10-30 22:07:31 +01:00

7.3 KiB

Creation-date cross-store recovery journal

Prepared source only. No productive date correction, deployment or acceptance. WP-0014-T04 still requires a supervised, backup-bound, quiesced executor and productive postflight. The sealed storage release proposal is unchanged.

date_backfill_journal.py adds a correction primitive to the existing byte-preserving patch compiler. Its caller must first fence all SQL and content writers and verify a fresh immutable paired backup/readback. A backup receipt SHA256 is required and retained in the journal; the primitive does not verify the receipt, its freshness, its content binding or the writer fence. It is not exposed through a management command or product route.

The operation locks the selected SQL row and checks its tenant, slug and observed creation date, plus the exact selected-space page inventory and hashes. It writes an exclusive, private, fsynced intent containing the original index bytes, plan and backup-receipt digest before either store changes. Then it updates only SQL created_at, atomically replaces the index with its compiled patch, fsyncs the file and parent, commits SQL, and marks the journal completed. Replacements preserve file ownership and mode. Symlink destinations, ambiguous page inventories, drift and journal replay refuse. Journal parents must be private; journals must stay outside served content because they contain original private page bytes.

On interruption, keep writers fenced. reconcile_date_patch locks the row and uses its committed date: the original date restores original index bytes, while the corrected date finishes the patch. Both the index and every other observed page must still match their permitted hashes. Unknown SQL state or later edits refuse without overwriting those edits. A pending intent survives an uncertain commit result. Keep the terminal journal as evidence; do not retry the original plan automatically. A malformed/partially created journal requires manual inspection before writers resume. PostgreSQL/file durability depends on the underlying storage honoring fsync.

Validation: tests/test_date_backfill_journal.py terminates actual child processes at durable intent, uncommitted SQL, replaced file and committed SQL. After death, the tests reconcile against persisted SQL and verify untouched page hashes. Other cases check successful correction, unrelated SQL fields, replay refusal, later-edit refusal, unknown SQL date, private permissions and symlinks. Together with the existing date planner/compiler tests, 30 focused cases pass on both SQLite and disposable PostgreSQL 16.14. The PostgreSQL instance used synthetic data, a temporary Docker container and a loopback-only port; it did not connect to any production/recovered database or provider. The full make check gate is recorded in the companion evidence JSON.

The retained independently recovered pair now also passes the journal rehearsal. scripts/rehearse-source-date-journal.py verifies the actual Docker network, database image, unique database alias and absence of published ports, then clones the recovered PostgreSQL database and copies its content. All commits and crash tests run against that disposable clone. Real child-process exits at intent, SQL, file and commit boundaries reconcile correctly. All 13 tables and 82 files match the expected state after every case; the successful correction changes only the selected SQL date and index. Replay refuses. Before/after SQL digests and file hashes prove the retained recovery pair is unchanged. The disposable SQL clone is removed. Evidence: source-date-journal-real-pair-2026-10-30.json. This uses explicit source overlays on the retained candidate image, not a newly released application image.

To repeat against a retained isolated restore, provide private paths locally:

uv run python scripts/rehearse-source-date-journal.py \
  --restore-receipt /private/restore-receipt.json \
  --plan /private/date-backfill-plan.json \
  --candidate-image '<locally retained image digest>' \
  --output /tmp/date-journal-real-pair.json

Complete store operation checks

date_backfill_operation.py now wraps the journal primitive with complete-store preconditions and postflight. It accepts only PostgreSQL execution in the public application schema, takes the existing content-writer lock without waiting, and acquires ACCESS EXCLUSIVE NOWAIT locks on every public table inside the correction transaction. A busy writer or SQL lock conflict refuses. The reviewed baseline binds all table rows and all content-file hashes, including unrelated records, assets and aliases. The expected result is derived by changing only the selected row's creation date and the byte-preserving index patch. Every SQL/file/commit boundary is compared against that result; an error requires the supervisor to keep writers fenced.

Backup metadata must declare the exact app/image/public recipient, immutable version, complete paired-store/ciphertext hashes, matching own-prefix destination and successful readback, with capture/completion ordered and at most ten minutes old for a new operation. The module validates those declarations and retains their canonical digest. It does not contact the provider or establish that the declarations are true; native capture/readback and source-to-backup bookends are the external supervisor's responsibility.

Reconciliation requires the original pending plan and backup digest. It permits that original backup to have aged, reconstructs the reviewed baseline by undoing only the known date/index change in memory, and refuses any unrelated drift. It never substitutes a new capture of potentially inconsistent state. The committed SQL date still determines rollback versus completion. No generic restore, historical data overwrite or automatic retry is added.

The extended isolated runner accepts --backup-receipt to exercise this exact operation on a disposable clone of the recovered pair. Its four real process deaths reconcile correctly, including recovery with the original binding a day later. SQL drift, an unrelated new file and a busy content writer refuse before an intent is created. A second real PostgreSQL connection's UPDATE times out against the operation's table lock. Successful commit and replay refusal preserve all unrelated rows/files; all 13 tables and 82 files are compared. The retained recovery source remains unchanged and the clone is removed. Proof: source-date-operation-real-pair-2026-10-30.json.

This rehearsal uses the historical verified backup receipt and an explicitly historical metadata-check clock. It proves the operation logic on real recovered data, not freshness of a newly captured point or a production writer fence. The candidate image uses explicit source overlays; no image is released here. The focused planner/journal/operation suite passes 50 cases; make check passes 364 cases including protected browsers, with zero skips.

Remaining wrapper requirements: bind exact reviewed source/plan/target and fresh backup/readback hashes; verify the external writer fence and full live SQL/content baseline; invoke this operation in the pinned helper; supervise the privileged action; refuse outstanding journal state; retain the journal on durable private storage; perform full unrelated-row/file postflight and resolve every pending state before resuming writers. Rehearse that wrapper against the independently recovered real pair before productive use.