Make the replacement harness runnable and target a live issuer
All checks were successful
Build and Publish Container Image / build-and-push (push) Successful in 36s

Closes the runnable half of gap G04. The Scenario B and C scripts could not
execute: absent compose files, binaries sought at src/bin where the Makefile
builds to bin/, --base-dn passed to a generator whose flag is --basedn, and a
hardcoded workstation Go path invoked from outside the module.

Repairing the shell alone would have proved nothing. Both scripts set
KEYCAPE_TEST_ISSUER while the profile suite built its own httptest server and
never read it, so they passed identically whether or not a provider was running.
A harness that cannot fail for the reason it exists is worse than a missing one.

src/tests/conformance targets the issuer named by KEYCAPE_TEST_ISSUER over HTTP:
discovery, the profile authorization surface, published keys parsed under the
runtime's own rules, excluded grants, and -- with credentials -- a real token
exchange verified against those keys. It skips when the variable is unset, so
make test is unchanged.

Run against Keycloak 26.0 rather than asserted to work. Discovery, authorization
surface and key checks passed, and a client_credentials exchange produced a token
that verified against Keycloak's published JWKS through internal/jose.

It also failed, correctly: stock Keycloak advertises the excluded implicit and
password grants, and in Keycloak those are server capabilities rather than
per-client toggles, so no emitted realm removes them. A migrated Keycloak has a
wider grant surface than KeyCape, which substantiates with evidence what SCOPE
previously asserted without it. Scenario B legitimately reports failure today.

Directory migration, credential and MFA preservation and relying-party behaviour
remain unexercised, and Scenario C has never been run end to end, so G04 does not
fully close.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NV9oijZukGyGbRQGGKnK4P

Assistant: claude-code
Assistant-Model: opus
Assistant-Process: 713576@bnt-lap001
Assistant-Session: 384c511d-9bce-4cb8-a676-2aef6c0c8df6
This commit is contained in:
tegwick 2026-09-07 23:32:11 +02:00
parent ebc48e31ae
commit 7534552754
8 changed files with 511 additions and 31 deletions

View file

@ -0,0 +1,130 @@
---
id: KEY-WP-0022
type: workplan
title: "Make the replacement harness runnable and target conformance externally"
domain: infotech
repo: key-cape
status: finished
owner: claude
topic_slug: replacement-harness-and-external-conformance
created: "2026-09-07"
updated: "2026-09-07"
---
Closes the runnable half of gap G04 of
`history/2026-09-05-011726-scope-intent-assessment.md`. The Scenario B and C
scripts cannot execute: they reference compose files that do not exist, call
binaries at `src/bin/` where the Makefile builds to root `bin/`, pass `--base-dn`
to `lldap-to-ldap` whose flag is `--basedn`, and invoke a hardcoded workstation Go
path from outside the Go module.
The deeper problem is that repairing the shell would not prove anything. Both
scripts set `KEYCAPE_TEST_ISSUER`, but the profile suite builds its own `httptest`
server and never reads it, so the tests pass identically whether or not the
external provider is running. A harness that cannot fail for the reason it exists
is worse than a missing one.
## Repair the harness prerequisites
```task
id: KEY-WP-0022-T01
status: done
priority: high
```
Fix the binary paths, the `--basedn` flag, and the Go invocation so the scripts
run from a clean checkout with `make build`: use the module directory rather than
a workstation-specific Go path, and fail with a clear message when a prerequisite
binary or a required environment value is absent instead of part-way through.
Scenario B rewritten: root `bin/`, `go -C src` so the module root is right,
prerequisite checks before anything starts, and a cleanup trap so a failure does
not leave a stack running. Scenario C repaired in place — `bin/` paths, `--basedn`
for `lldap-to-ldap` (`lldap-export` correctly takes `--base-dn`; the two flags
genuinely differ), and the same module-relative test invocation.
## Add the missing stacks
```task
id: KEY-WP-0022-T02
status: done
priority: high
```
Provide `docker-compose.scenario-b.yml` (Keycloak over the migrated realm) and
`docker-compose.scenario-c.yml` (OpenLDAP seeded from generated LDIF, plus
Keycloak). Validate them with `docker compose config` rather than assuming they
parse. Whether a full scenario passes end to end is a separate claim from whether
the stack definition is valid, and only the second is established here.
Both files added and validated with `docker compose config`. Scenario B was also
started for real; Scenario C's stack has not been run.
## Target conformance at an external issuer
```task
id: KEY-WP-0022-T03
status: done
priority: high
```
Add a suite that reads `KEYCAPE_TEST_ISSUER` and exercises that issuer over HTTP:
discovery metadata, the JWKS parsing under the same strict rules the runtime
applies, and — when client credentials are supplied — a real token exchange whose
claims are verified against the published keys. It must skip cleanly when the
variable is unset so ordinary `make test` is unaffected, and it must fail when
pointed at an issuer that does not meet the profile, which is the property the
current harness lacks.
`src/tests/conformance` checks that the issuer advertises itself, the profile
authorization surface (code, authorization_code, S256, RS256), that published
keys parse under `internal/jose` — the same rules the runtime applies, so a
provider publishing keys KeyCape would refuse fails here rather than at first
login — that excluded grants are absent, and, with credentials, that an issued
token verifies against the published keys and carries a matching issuer.
## Prove the suite against a real provider
```task
id: KEY-WP-0022-T04
status: done
priority: high
```
Run the new suite against an actual Keycloak instance, not only against KeyCape,
since the point of the scenario is a replacement provider. Record what genuinely
passed and what remains unproven. Do not describe a partially exercised scenario
as a demonstrated live swap.
Run against Keycloak 26.0. Discovery, authorization surface and published-key
checks passed. A `client_credentials` exchange against a service client created
through the admin API returned a token that verified against Keycloak's JWKS via
`internal/jose`, with a matching issuer claim — the end-to-end property a relying
party depends on, proved against a replacement provider rather than against
KeyCape's own handlers.
The suite also failed, correctly. Stock Keycloak advertises `implicit` and
`password` in its discovery document. In Keycloak these are server capabilities,
not per-client toggles, so no realm the transformer emits can remove them: a
migrated Keycloak presents a wider grant surface than KeyCape. Scenario B
therefore reports a real failure today, which is the harness working. KeyCape
itself was not targeted in this run — the dev stack needs key material absent
from the checkout — so the suite is proved against the replacement provider, not
yet against both sides of the swap.
## Reconcile the records
```task
id: KEY-WP-0022-T05
status: done
priority: medium
```
Update `SCOPE.md` and G04's status with what the harness now does and what
replacement claims remain unproven — in particular that directory migration, MFA
and unchanged relying-party behaviour against a live Keycloak are not established
by this work. G04 does not fully close here.
SCOPE.md and G04's status record both the working harness and the grant-surface
finding, which substantiates with evidence what SCOPE previously asserted without
it. Scenario C has still never been run end to end.