Rehearse storage migration on recovered data and prepare the supervised release
Some checks failed
Use-case regression gate / usecase-regressions (push) Failing after 2m44s

Assistant: codex
Assistant-Model: gpt-6.1-sol
Assistant-Session: 01a17f16-d99c-7ea3-95f0-8212897fba6b
This commit is contained in:
tegwick 2026-10-29 23:36:02 +01:00
parent 0eb97ba05d
commit dfc9f9ed18
8 changed files with 455 additions and 3 deletions

View file

@ -0,0 +1,34 @@
{
"status": "recovered_data_candidate_verified",
"candidate_image_id": "sha256:512819463a1902b43a615beee14be7bf97e6a05e016385082b9380692ad1a8cd",
"migration": {
"status": "recovered_data_migration_lifecycle_verified",
"business_tables": 12,
"business_rows": 91,
"forward_preserves_all_business_rows": true,
"sql_collision_denied": true,
"rollback_preserves_all_business_rows_and_tenant_constraint": true,
"legacy_refusal_preserves_rows_and_grants": true,
"reapply_preserves_all_business_rows": true,
"global_constraint_present_after_reapply": true,
"production_access": false
},
"application": {
"status": "isolated_application_recovery_verified",
"recovered_spaces_with_content": 1,
"actual_pages_rendered_through_view": 39,
"restored_owner_access_verified": true,
"synthetic_viewer_reads": 1,
"synthetic_viewer_mutation_denials": 1,
"same_tenant_ungranted_page_denials": 39,
"fixture_transaction_rolled_back": true,
"production_access": false,
"external_identity_provider_used": false,
"real_browser_journey_verified": false
},
"content_files_unchanged": 82,
"network_internal": true,
"published_ports": 0,
"production_access": false,
"shared_view_check_sha256": "d1a641bb93e9ba9c5f879b806d81b1de37ba0743c542fc11137dd555a6b9f9ac"
}

View file

@ -0,0 +1,36 @@
{
"status": "read_only_live_release_preflight_verified",
"cluster_uid_verified": true,
"helm_revision": 17,
"release_status": "deployed",
"replicas_ready": 1,
"deployment_strategy": "Recreate",
"pvc_bound": true,
"resource_count": 3,
"changes": [
{
"kind": "Deployment",
"field": "app image"
}
],
"other_manifest_changes": false,
"candidate_render_sha256": "865b6a22b3c50c65a7417bb80de609b7e0290f54a99edebff497b2259bc067a2",
"baseline_helm_manifest_sha256": "da80414e13e7c5fd254803a241f21ade223b4b84e68ed908535e36e854b523a2",
"node_conditions": {
"EtcdIsVoter": "True",
"MemoryPressure": "False",
"DiskPressure": "False",
"PIDPressure": "False",
"Ready": "True"
},
"node_allocated_resources": "Allocated resources:\n (Total limits may be over 100 percent, i.e., overcommitted.)\n Resource Requests Limits\n -------- -------- ------\n cpu 2720m (68%) 40600m (1015%)\n memory 12228Mi (76%) 41026Mi (256%)\n ephemeral-storage 256Mi (0%) 1Gi (0%)\n hugepages-1Gi 0 (0%) 0 (0%)\n hugepages-2Mi 0 (0%) 0 (0%)\n",
"node_allocatable": {
"cpu": "4",
"ephemeral-storage": "196555267123",
"hugepages-1Gi": "0",
"hugepages-2Mi": "0",
"memory": "16377032Ki",
"pods": "110"
},
"live_changes_performed": false
}

View file

@ -0,0 +1,89 @@
# Storage/content hardening release — exact supervised proposal
Prepared, not deployed. Decision 05c9b00f-e4e4-4593-bdeb-c63fe6a4fe15 is pending
founder disposition. Earlier backup/custody/schedule approvals remain in force.
This packet does not reopen them or approve Bubble/apex cutover.
Proposal digest:
`sha256:3389c506d86f1613c6567a2bba5c29ae22b812f19b011ebe454bb08af39a4132`.
It binds 12 capsule members including all node/launcher/probe code, exact chart and
values, dataset/schema expectations and the candidate snapshot-helper source.
Any changed reviewed byte produces another digest and refuses execution.
Deployment code is prepared on railiance-apps branch
`csoc-storage-release-20261029`, commit45d6178. It has not been promoted to the
deployment owner's main branch or applied to the cluster.
## Exact result
Target: `ADMINISTER @ realm:kubernetes/railiance01`, namespace/release
coulomb-social, current Helm revision 17. Candidate product source 17d0eb6 at
`sha256:512819463a1902b43a615beee14be7bf97e6a05e016385082b9380692ad1a8cd`.
The candidate contains global slug constraint 0003, tested content revision/recovery
hardening and strict original-date import guards. Renderer policy is unchanged.
The attended command rechecks cluster/node, exact live spec/PVC/Helm baseline,
headroom, no Argo adoption, data equality and server/client dry-runs. It then runs
the already admitted fresh paired backup and immutable primary readback, checks
the dataset again, and performs Helm --atomic/--wait with Recreate. The three
resource manifests change only the app image; the PVC, configuration, resources
and secret references are preserved. Startup applies the atomic SQL constraint;
it does not rename/merge/delete business rows or change grants.
Read-only postflight checks schema, all business/content aggregate hashes, actual
owner pages and unsaved-outsider denials, plus public health/landing/static bytes.
Only then refresh the root-owned snapshot helper's image pin to the new digest
and repeat its server dry-run. No private key/recovery pair is delivered to the
worker; no email, caller, issuer, DNS, apex, timer or production-data restore.
## Evidence and limits
- Packaged real recovered dataset: forward/refusal/reverse/reapply preserve all
12 business tables/91 rows and82 files; candidate renders 39 pages with owner/viewer/
ungranted checks. Old image also reads the migrated data and denies outsiders.
- Live data hashes match the independently protected recovered point. Current
image 9a646d8e, one Ready replica, revision 17, Recreate, Bound PVC, no pressure.
Requests:68% CPU / 76% memory. Helm lint passes; exact sealed live preflight passes.
- Product make check: 246 passed, including mandatory browser journeys. Ten
synthetic executor cases cover tampering, data drift, replay refusal, staging
immutability, unattended-launch refusal, rollback and capture cleanup timeout.
The fresh capture normally pauses/resumes in about 54 seconds; previous app
upgrades took about 20 seconds. Helm has a five-minute wait plus rollback handling.
Failure preserves post-capture writes and the fresh point. Automatic rollback
restores the prior image/revision, not database contents; the added uniqueness
constraint may remain. Its separate SQL inverse and old-image read compatibility
are rehearsed. A failed/uncertain attempt requires journal inspection, not replay.
No new node/PVC/service capacity is committed. One existing-size encrypted
generation/readback and normal API/storage usage are added. No EUR autopilot
budget or risk grant is claimed; execution remains supervised.
## Founder execution
Run in your own terminal from the prepared deployment checkout:
```sh
python3 /tmp/csoc-storage-release-owner-20261029/tools/coulomb-storage-release.py \
--execute \
--expect-digest sha256:3389c506d86f1613c6567a2bba5c29ae22b812f19b011ebe454bb08af39a4132 \
--output /tmp/csoc-storage-release-result-20261029.json
```
Review the digest and type DEPLOY followed by that full digest when prompted.
The command uses the founder's existing SSH/sudo path and returns metadata only.
No Scaleway login or credential pasted into chat is required. Reply done afterward;
the agent must inspect the node receipt and postflight before recording acceptance.
The durable node journal is
`/var/lib/coulomb-social-backup/releases/<proposal-hex>/receipt.json`.
This is an attended supervisor command, not a new credential/approval service.
The active gated-executor profile is Kubernetes-only and cannot perform the
root-owned node-helper refresh. Its configuration is left unchanged.
The exact-action requirement comes from the-custodian's
docs/agent-environment-orientation.md §4: “Every live change needs the founder's
go-ahead”, and docs/agent-autonomy-decision.md: supervised privileged actions use
an exact proposal; “The agent cannot approve itself.” This image/schema action
has no current exact founder approval. Preparation and all observed checks above
are complete; the app has not been changed by this packet.

View file

@ -0,0 +1,17 @@
{
"business_tables": 12,
"business_rows": 91,
"business_rows_sha256": "1f404b7468011f8e006ed784628ade14bd03b53617282063f6d300df36f5090f",
"content_files": 82,
"content_files_sha256": "e3113909deccdab8135332c9e9fcacbed88f3216d9a1178f6b63da943aca3deb",
"global_slug_constraint": true,
"tenant_slug_constraint": true,
"migration_0003_applied": true,
"duplicate_slug_groups": 0,
"actual_owner_pages_rendered": 39,
"unsaved_outsider_page_denials": 39,
"production_identity_or_grant_created": false,
"status": "old_image_read_compatible_after_migration",
"production_access": false,
"image_id": "sha256:9a646d8e01a61ba5316d3c2eb0273064e17f2d0300ad07af766ccedda65ecb18"
}

View file

@ -0,0 +1,24 @@
{
"schema": "coulomb.guarded-storage-release.v1",
"proposal_digest": "sha256:3389c506d86f1613c6567a2bba5c29ae22b812f19b011ebe454bb08af39a4132",
"started_at": "2026-10-29T22:33:34.428700+00:00",
"status": "live_release_preflight_verified",
"production_data_restored": false,
"timers_activated": false,
"node_request_percent": {
"cpu": 68,
"memory": 76
},
"preflight": {
"business_tables": 12,
"business_rows": 91,
"business_rows_sha256": "1f404b7468011f8e006ed784628ade14bd03b53617282063f6d300df36f5090f",
"content_files": 82,
"content_files_sha256": "e3113909deccdab8135332c9e9fcacbed88f3216d9a1178f6b63da943aca3deb",
"global_slug_constraint": false,
"tenant_slug_constraint": true,
"migration_0003_applied": false,
"duplicate_slug_groups": 0
},
"live_changes": false
}

View file

@ -0,0 +1,222 @@
"""Rehearse a packaged storage candidate against an isolated recovered real pair.
Never opens production or prints rows, pages, identities or raw child failures.
The original encrypted point and private recovered files remain unchanged.
"""
from __future__ import annotations
import argparse
import ast
import hashlib
import json
import os
import re
import subprocess
from pathlib import Path
def run(arguments):
return subprocess.run(arguments, capture_output=True, timeout=180, check=True).stdout
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--restore-receipt", type=Path, required=True)
parser.add_argument("--candidate-image", required=True)
parser.add_argument("--output", type=Path, required=True)
args = parser.parse_args()
if args.output.exists() or args.output.is_symlink():
raise SystemExit("fresh output required")
state = json.loads(args.restore_receipt.read_text())
assert re.fullmatch(r"csoc-recovery-\d{14}", state["database_container"])
assert state["network"] == state["database_container"] + "-net"
network = json.loads(run(["docker", "network", "inspect", state["network"]]))[0]
database = json.loads(run(["docker", "inspect", state["database_container"]]))[0]
assert network["Internal"] is True and database["State"]["Running"]
assert set(database["NetworkSettings"]["Networks"]) == {state["network"]}
assert not any(database["NetworkSettings"]["Ports"].values())
assert database["Image"] == state["postgres_image_id"]
image = run(["docker", "image", "inspect", args.candidate_image, "--format", "{{.Id}}"])
image = image.decode().strip()
assert re.fullmatch(r"sha256:[a-f0-9]{64}", image)
root = Path(state["restored_directory"]) / "content"
def content_hashes():
return {
str(p.relative_to(root)): hashlib.sha256(p.read_bytes()).hexdigest()
for p in root.rglob("*")
if p.is_file()
}
before_files = content_hashes()
source = Path(__file__).with_name("verify-restored-application.py")
tree = ast.parse(source.read_text())
view_checks = next(
node.value.value
for node in tree.body
if isinstance(node, ast.Assign)
and isinstance(node.targets[0], ast.Name)
and node.targets[0].id == "code"
)
migration_checks = r"""
import io, json, hashlib, django
django.setup()
from django.core.management import call_command
from django.db import connection, transaction, IntegrityError
from coulomb_social.apps.spaces.models import Space
assert connection.vendor == "postgresql"
def fingerprint():
with connection.cursor() as cursor:
cursor.execute(
"SELECT tablename FROM pg_tables WHERE schemaname='public' "
"AND tablename!='django_migrations' ORDER BY tablename"
)
tables = [row[0] for row in cursor.fetchall()]
result = {}
for table in tables:
cursor.execute("SELECT * FROM " + connection.ops.quote_name(table))
rows = sorted(
json.dumps(row, default=str, separators=(",", ":")) for row in cursor.fetchall()
)
result[table] = {
"rows": len(rows),
"sha256": hashlib.sha256("\n".join(rows).encode()).hexdigest(),
}
return result
def constraints():
with connection.cursor() as cursor:
cursor.execute("SELECT conname FROM pg_constraint WHERE conrelid='spaces_space'::regclass")
return {row[0] for row in cursor.fetchall()}
def migrate(target):
call_command(
"migrate",
"spaces",
target,
verbosity=0,
interactive=False,
stdout=io.StringIO(),
stderr=io.StringIO(),
)
global_name = "spaces_space_slug_global_uniq"
tenant_name = "spaces_space_tenant_slug_uniq"
assert global_name not in constraints() and tenant_name in constraints()
baseline = fingerprint()
assert Space.objects.count() > 0
slug = Space.objects.first().slug
migrate("0003")
assert global_name in constraints() and tenant_name in constraints() and fingerprint() == baseline
try:
with transaction.atomic():
Space.objects.create(tenant_id="recovery:collision", slug=slug, title="Synthetic collision")
except IntegrityError:
pass
else:
raise RuntimeError("global_constraint_failed_to_refuse_collision")
assert fingerprint() == baseline
migrate("0002")
assert (
global_name not in constraints() and tenant_name in constraints() and fingerprint() == baseline
)
with transaction.atomic():
Space.objects.create(tenant_id="recovery:collision", slug=slug, title="Synthetic collision")
held = fingerprint()
try:
migrate("0003")
except RuntimeError as error:
assert str(error).startswith("Refusing spaces_space_slug_global_uniq:")
else:
raise RuntimeError("legacy_collision_was_not_refused")
assert global_name not in constraints() and fingerprint() == held
transaction.set_rollback(True)
assert fingerprint() == baseline
migrate("0003")
assert global_name in constraints() and tenant_name in constraints() and fingerprint() == baseline
print(
json.dumps(
{
"status": "recovered_data_migration_lifecycle_verified",
"business_tables": len(baseline),
"business_rows": sum(t["rows"] for t in baseline.values()),
"forward_preserves_all_business_rows": True,
"sql_collision_denied": True,
"rollback_preserves_all_business_rows_and_tenant_constraint": True,
"legacy_refusal_preserves_rows_and_grants": True,
"reapply_preserves_all_business_rows": True,
"global_constraint_present_after_reapply": True,
"production_access": False,
}
)
)
"""
command = [
"docker",
"run",
"--rm",
"--network",
state["network"],
"--read-only",
"--tmpfs",
"/tmp",
"--user",
f"{os.getuid()}:{os.getgid()}",
"--entrypoint",
"/app/.venv/bin/python",
"-e",
"DJANGO_SETTINGS_MODULE=coulomb_social.settings.base",
"-e",
"DATABASE_URL=postgres://postgres@recovery-db/coulomb_social_restore",
"-e",
"CONTENT_ROOT=/recovered-content",
"-e",
"FORGEJO_BASE_URL=",
"-e",
"ALLOWED_HOSTS=testserver",
"-e",
"PYTHONDONTWRITEBYTECODE=1",
"-v",
str(root) + ":/recovered-content:ro",
image,
"-c",
migration_checks + view_checks,
]
result = subprocess.run(command, capture_output=True, timeout=180, check=False)
if result.returncode:
diagnostic = Path(state["restored_directory"]) / "private-candidate-diagnostics.txt"
fd = os.open(diagnostic, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
with os.fdopen(fd, "wb") as output:
output.write(result.stdout + result.stderr)
print(
json.dumps(
{"status": "candidate_rehearsal_refused", "private_diagnostics": str(diagnostic)}
)
)
raise SystemExit(1)
rows = [json.loads(line) for line in result.stdout.splitlines() if line.startswith(b"{")]
assert len(rows) == 2 and content_hashes() == before_files
proof = {
"status": "recovered_data_candidate_verified",
"candidate_image_id": image,
"migration": rows[0],
"application": rows[1],
"content_files_unchanged": len(before_files),
"network_internal": True,
"published_ports": 0,
"production_access": False,
"shared_view_check_sha256": hashlib.sha256(source.read_bytes()).hexdigest(),
}
fd = os.open(args.output, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
with os.fdopen(fd, "w") as output:
json.dump(proof, output, indent=2)
print(json.dumps(proof))

View file

@ -8,7 +8,7 @@ status: active
owner: codex
topic_slug: coulomb-social
created: "2026-10-08"
updated: "2026-10-28"
updated: "2026-10-29"
origin: residual
origin_ref: CSOC-WP-0008
state_hub_workstream_id: "70370f4e-7cba-5b0a-853b-fedcd2bd49a5"
@ -20,8 +20,11 @@ Residual acceptance from CSOC-WP-0008, split at the user's request. Existing imp
```task
id: CSOC-WP-0013-T01
status: progress
status: wait
priority: high
needs_human: true
decision_id: "05c9b00f-e4e4-4593-bdeb-c63fe6a4fe15"
intervention_note: "Exact storage/content release is prepared and native preflight passes. Founder attended execution of reviewed proposal3389c506 is required by current deployment supervision rules; run the one command in docs/release/storage-release-review-2026-10-29.md. Prior backup/custody approvals remain valid."
state_hub_task_id: "4d18a390-c82a-5d6b-8509-1b1704975150"
```
@ -61,6 +64,32 @@ as the prepared candidate and includes the strict source-date guard. No producti
migration or deployment; fresh paired rollback/recovery proof remains required.
Evidence: docs/release/storage-candidate-17d0eb6-2026-10-28.json.
2026-10-29: The independently provider-recovered real pair now passes candidate
forward/refusal/reverse/reapply while preserving all12business tables/91rows and
82files; all39pages/owner/viewer/ungranted checks pass. Old9a646d8e also reads the
migrated copy and denies unsaved outsiders. Live data hashes exactly match that
recovered point. Native live preflight proves revision17, one Ready Recreate
replica, Bound PVC, no pressure and68%CPU/76%memory requests. Three-resource
comparison changes only the app image. Exact sealed client/server/Helm dry-runs
pass without app/schema changes. Full make check246passes; ten synthetic release
safety cases and Helm lint pass.
The founder's new exact-action disposition is required by the deployment owner's
standing supervision rules. Prepared rollout includes a fresh already-admitted
paired capture/readback, data-drift refusals, atomic Helm rollout, read-only
postflight, image rollback preserving writes and root-owned backup-helper image
pin refresh. The packet and one attended command are in
docs/release/storage-release-review-2026-10-29.md; proposal3389c506d86f1613c6567a2bba5c29ae22b812f19b011ebe454bb08af39a4132,
pending decision05c9b00f-e4e4-4593-bdeb-c63fe6a4fe15. No new Scaleway login or
credential delivery is needed. No live migration, rollout, SQL date correction,
purge, grants, mail or timer change has been made. T01 waits only at that exact
privileged execution gate; other workplan tasks remain active.
Evidence: storage-real-pair-rehearsal-2026-10-29.json,
storage-rollback-compatibility-2026-10-29.json,
storage-release-preflight-2026-10-29.json and
storage-sealed-preflight-2026-10-29.json under docs/release/.
## T02 — Complete content fidelity and revision lifecycle
```task

View file

@ -8,7 +8,7 @@ status: active
owner: codex
topic_slug: coulomb-social
created: "2026-10-08"
updated: "2026-10-28"
updated: "2026-10-29"
origin: residual
origin_ref: CSOC-WP-0008
state_hub_workstream_id: "db6b85dc-f2f0-5283-ac52-3a6af28eb5b5"
@ -92,6 +92,7 @@ id: CSOC-WP-0014-T03
status: wait
priority: high
needs_human: true
decision_id: "186be722-e2c7-4109-a1e6-d120fe8fca5b"
intervention_note: "Existing cutover decision 186be722-e2c7-4109-a1e6-d120fe8fca5b remains open, after WP-0012 recovery/service and WP-0013 privacy/content returns."
state_hub_task_id: "599b4b6a-829d-5fe5-b52d-ff419e548fa8"
```