147 lines
8.5 KiB
Markdown
147 lines
8.5 KiB
Markdown
# Secret annotation guard: rollout, failed integration and rollback
|
|
|
|
CUST-WP-0073-T03, September 28, 2026. The founder authorized continuing the
|
|
existing workplans. This receipt records an unsuccessful rollout with recovery;
|
|
it is not evidence for promoting the agent to autopilot.
|
|
|
|
## Source and deployment
|
|
|
|
The platform owner source added the native policy/binding and safe maintenance
|
|
helper in `railiance-platform@800cbfa870661d47bee34c747f04f6145875adf1`.
|
|
Application commit `54885ac1589074a68d9607d1be250c84c5c2307a` pinned that revision.
|
|
The AppProject allowlist gained only the two admission kinds. Publication used
|
|
repo-manager; deployment used manual Argo resource-scoped sync, without syncing
|
|
unrelated root changes. The application has no automated sync or finalizer.
|
|
|
|
Policy type checking passed. The first synthetic proof failed on an SSA field
|
|
ownership conflict; its failed receipt is retained. The corrected proof tests
|
|
SSA of the same value followed by a clean update. All nine native checks passed:
|
|
clean create/update/SSA; annotated create/update rejected, including empty
|
|
values; client-side apply rejected; synthetic fixture removed. These checks did
|
|
not establish compatibility with existing controllers.
|
|
|
|
## Actual integration failure
|
|
|
|
ESO v0.16.1 copied the ExternalSecret source last-applied annotation back onto
|
|
its target when no target template existed. Under Deny enforcement, required
|
|
Secret refreshes failed (ten ExternalSecrets observed in SecretSyncedError).
|
|
31 live ExternalSecrets lacked an explicit template. The implementation is
|
|
visible in the [installed-version upstream source](https://github.com/external-secrets/external-secrets/blob/v0.16.1/pkg/controllers/externalsecret/externalsecret_controller_template.go):
|
|
without a target template the controller copies source metadata; with one it
|
|
uses template metadata. Merely removing annotations from the targets does not
|
|
fix this writer behavior.
|
|
|
|
The binding was deleted promptly to restore refreshes. Failed ExternalSecrets
|
|
were explicitly force-refreshed. All 39 became Ready. Source rollback commit
|
|
`6016f72a8db26df1eed7b7b9f3b36c8a0eb9f3b7` removes the binding from kustomization
|
|
and keeps it in `binding.pending.yaml`. Application commit
|
|
`c5d65b0b405be9ce5624f49c6f6df54c12b38735` pins that policy-only revision.
|
|
Both were published using repo-manager; the root was synced only for this
|
|
Application, then the child synced to its policy-only revision.
|
|
|
|
Final read-back at 13:17:32 UTC: application Synced/Healthy, operation Succeeded,
|
|
binding absent, 39/39 ExternalSecrets Ready. See
|
|
`2026-09-28-secret-annotation-rollback.json`. The policy remains installed but
|
|
UNBOUND. A normal sync of the pinned source cannot re-enable enforcement.
|
|
|
|
## Cleanup and limits
|
|
|
|
The recorded passes removed the annotation from 49 distinct Secrets in active
|
|
namespaces. Some were cleaned again after ESO recreated the annotation. This is
|
|
not a claim that all remain annotation-free after rollback. The helper changed
|
|
only that metadata key, never Secret data. ESO recovery performs its normal
|
|
refresh behavior; no credential rotation was performed by this work.
|
|
|
|
`platform-pg-drill/drill-minio` is orphaned: its namespace is absent, so the API
|
|
refuses the metadata patch. A referencing Deployment, a PVC and Service also
|
|
remain without that namespace. No orphan was deleted or namespace recreated.
|
|
Cluster-wide cleanup is incomplete. Disposition remains under existing T03.
|
|
|
|
Kubectl subprocess output was captured and suppressed throughout the helper.
|
|
The old raw presence template failed on absent annotations; its error was
|
|
suppressed rather than exposing a Secret dump. The replacement iterates keys
|
|
and handles absent/empty maps. Orientation §6 now requires the safe helper.
|
|
|
|
## Remaining work in T03
|
|
|
|
Before re-enabling the strict guard, explicitly set target metadata in the 31
|
|
owning ExternalSecret declarations while preserving intended labels/annotations
|
|
and all existing data templates. Verify actual controller refresh and clean
|
|
resulting target metadata, repeat cleanup and synthetic checks, then verify
|
|
ESO refresh with enforcement enabled. No ESO exemption or controller upgrade
|
|
is proposed. The owner source changes and orphan disposition remain unfinished;
|
|
no additional workplan or task has been created.
|
|
|
|
Receipts alongside this file: `secret-annotation-cleanup.json`,
|
|
`secret-annotation-cleanup-retry.json`, `secret-annotation-cleanup-active.json`,
|
|
`secret-annotation-post-binding.json`, `secret-annotation-admission-proof.json`,
|
|
`secret-annotation-admission-proof-final.json`, and
|
|
`secret-annotation-rollback.json`, all prefixed `2026-09-28-`.
|
|
|
|
## Corrected rollout — September 28 follow-up
|
|
|
|
The founder instructed “Good, go on” after the 31-declaration remediation was
|
|
identified. Explicit target metadata was added to 31 ExternalSecrets in 23 files
|
|
across 12 owning repositories. Source labels and intentional annotations remain;
|
|
controller bookkeeping and last-applied are not inherited. Data mappings, data
|
|
templates, store references, creation/deletion policies and refresh intervals
|
|
were unchanged. Server-side dry-run and semantic comparison verified this for
|
|
all 31. A canary refreshed successfully and removed its copied annotation.
|
|
|
|
All source changes were committed and published. Eleven repositories have exact
|
|
primary repo-manager receipts. activity-core's repo-manager registration refused
|
|
pre-existing historical workplan IDs; its documented `statehub fix-consistency`
|
|
path passed with warnings and pushed the exact metadata commit, without changing
|
|
those historical file IDs. See `2026-09-28-eso-metadata-publication.json` and
|
|
`2026-09-28-eso-metadata-changes.json`. Required user-engine checks passed (four
|
|
tests); telemetry pinned-chart fetch/check and family validation passed (one
|
|
pre-existing declaration warning).
|
|
|
|
Thirty non-Argo-managed ExternalSecrets received metadata-only server-side
|
|
apply through the existing SSH admin path. Target Revenue's ExternalSecret used
|
|
a selective Argo sync at `a3a8a27a8cda32ae3c7b55353ee16a131c50a0a0`, declared by
|
|
platform commit `d2631f6112fca8f374b37aa52bc34400c12c95e1`. The operation result
|
|
lists only that ExternalSecret; no migration/bootstrap hook or workload rollout
|
|
was run. Its application is Synced and Healthy.
|
|
|
|
All 39 ExternalSecrets completed fresh successful refreshes before enforcement.
|
|
A complete active-namespace scan checked 257 Secrets and found no last-applied
|
|
annotations; ESO had removed the formerly inherited metadata. The absent-namespace
|
|
orphan remained separately reported.
|
|
|
|
Guard source `7daf7e90675b21c8f9556028e54aa8c0a13fd9f0` includes `binding.yaml`.
|
|
Application commit `db51ec802801ddf85a80bf81980865c9f9239839` pins that source.
|
|
Both were published through repo-manager. Selective root/child Argo sync enabled
|
|
the binding without unrelated app changes. At 14:30:57 UTC the guard was
|
|
Synced/Healthy, the binding was present with Deny, and policy type checking was
|
|
clear at observed generation 1. Nine native admission checks passed again.
|
|
**All 39 ExternalSecrets then completed fresh successful refreshes under Deny.**
|
|
Receipts: `2026-09-28-secret-annotation-enforced.json`,
|
|
`2026-09-28-secret-annotation-admission-proof-reenabled.json`, and
|
|
`2026-09-28-eso-refresh-{before,after}-binding.json`.
|
|
|
|
The old rollout failure remains a failed original proposal requiring refinement
|
|
and recovery. Successful remediation does not retroactively earn unchanged
|
|
execution credit. No agent promotion, credential rotation or interactive-runtime
|
|
cutover is claimed.
|
|
|
|
The remaining orphan is the August 13 Secret
|
|
`platform-pg-drill/drill-minio`, UID `2fcb66df-d90f-4776-8ea0-8ca1a04bd307`.
|
|
The namespace is absent and has no pods. A reviewable UID-bound proposal to
|
|
delete only that Secret is in `docs/changes/CUST-WP-0073/orphan-secret-deletion.json`.
|
|
Explicit deletion approval is pending; the bound 3 GiB PVC and all other resources
|
|
are outside that proposal. No deletion has occurred.
|
|
|
|
### Approved orphan cleanup
|
|
|
|
The founder explicitly approved deleting only the orphan Secret. The API accepted
|
|
the DELETE with UID precondition `2fcb66df-d90f-4776-8ea0-8ca1a04bd307`; a fresh
|
|
identity inventory verified absence. The 3 GiB PVC retained its exact UID,
|
|
resourceVersion, volume and Bound status. No other resources or the namespace
|
|
were changed, and no Secret values were read or archived. Receipt:
|
|
`2026-09-28-orphan-secret-deletion.json`. This resolves the cleanup exception
|
|
and completes CUST-WP-0073-T03; the earlier pending-deletion text is historical.
|
|
|
|
Final complete cluster-wide scan after deletion: 257 Secrets checked, zero
|
|
forbidden annotations, zero orphan exceptions, helper exit 0. Receipt:
|
|
`2026-09-28-secret-annotation-final-scan.json`.
|