INTENT design decision 8 of 10, unimplemented for six passes. cb-sim had no flag parsing at all, so --replay had nowhere to go. The bundle is manifest + commands.log + initial.snapshot + expected.yaml, dev-only behind the scenarios feature and charged to AM-4b. The command stream goes through the K11 framing built in T05, so a truncated bundle is detected rather than replayed short — the two tasks compose rather than duplicating. The reviewer's D2 correction was real: this was not "a directory of four files". Pass carried only the end state, RunOutcome::Failed was a formatted String, and scenario.rs created an EventLog, appended to it and never read it. All three had to change. The first round trip failed to reproduce, and the cause is worth keeping: state_hash_hex over a serde_json::Value is a different canonical form than over the typed aggregate — Value's map is key-sorted, a struct serializes in declaration order. The bundle was written with one basis and verified with the other. A round trip written to recompute its own comparison value would have PASSED this bug; it failed because the recorded hash came from the producing process, which is control 2's entire purpose. make replay-test implements ADR-0005 §6's four controls, 14/14: a committed deliberately-failing fixture outside the corpus with covers: [] so it neither fails `make sim` nor inflates AM-1; a tampered recorded hash must fail; a log short by one byte and a corrupted length prefix must be rejected; and a mutated manifest seed must fail — which bites only because replay re-derives the initial state from seed+setup and checks it against the recorded snapshot, since restoring from the snapshot alone would leave the seed inert. Plus a control on the controls: the bundle must still replay after every mutation is reverted. AM-7's hash-identical clause is re-earned. The probe records a hash per per-game segment and replays each from its own genesis; folding from the wrong seed now fails. That is the clause ADR-0005 §4 withdrew as mutation-proven inert. The scaling >= 0.9x clause is still unenforced, so AM-7 stays PARTIAL — reported, not rounded up. Kernel coverage 15/18 -> 16/18. facts-check immediately caught the spec's copy of that number going stale, on a number that moved the same hour. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
216 lines
7.9 KiB
Rust
216 lines
7.9 KiB
Rust
//! K10 replay bundles (GameKernel §2.4, MetricsAndScenarios §4).
|
|
//!
|
|
//! K10: *"A replay bundle (`.cbreplay`) contains manifest, command log,
|
|
//! initial snapshot, and failed expectations; the runner's `--replay`
|
|
//! re-executes it bit-identically."* Unimplemented until CB-WP-0006 T06 —
|
|
//! `games/` had no `replays/` directory, no `.cbreplay` reader or writer,
|
|
//! and `cb-sim` had no flag parsing at all, so `--replay` had nowhere to
|
|
//! go. INTENT names this design decision **8 of 10**.
|
|
//!
|
|
//! MetricsAndScenarios §4 states the point: *"A bug report without a
|
|
//! replay bundle is information; with one, it is work an agent can
|
|
//! start."*
|
|
//!
|
|
//! **Dev-only**, behind the `scenarios` feature, so it is charged to
|
|
//! AM-4b (9.4% headroom) and not to the shipped runtime.
|
|
//!
|
|
//! ## Why replay can fail, and must be able to
|
|
//!
|
|
//! ADR-0005 §6 required four controls, two of which the adversarial
|
|
//! reviewer supplied. The two that shape this module:
|
|
//!
|
|
//! * the comparison hash is **read out of the bundle**, written by the
|
|
//! process that produced it — never recomputed in the replaying process,
|
|
//! which would be `assert_eq!(h, h)` and is exactly the AM-7 defect;
|
|
//! * the recorded **seed must reproduce the recorded initial snapshot**,
|
|
//! so mutating the manifest seed makes replay fail. Without that check a
|
|
//! replay restored from the snapshot would ignore the seed entirely and
|
|
//! the control could not bite.
|
|
|
|
use std::path::{Path, PathBuf};
|
|
|
|
use cb_events::store::{parse, LogStore};
|
|
use cb_events::{state_hash_hex, FileLogStore};
|
|
use serde::{Deserialize, Serialize};
|
|
|
|
use crate::scenario::{CommandStep, ScenarioGame};
|
|
use crate::{ScenarioFile, Setup};
|
|
|
|
pub const MANIFEST: &str = "manifest.yaml";
|
|
pub const COMMANDS: &str = "commands.log";
|
|
pub const INITIAL: &str = "initial.snapshot";
|
|
pub const EXPECTED: &str = "expected.yaml";
|
|
|
|
/// `manifest.yaml` — everything needed to re-execute, and the hash the
|
|
/// replay must reproduce.
|
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
|
pub struct Manifest {
|
|
pub scenario: String,
|
|
pub seed: u64,
|
|
pub setup: Setup,
|
|
pub schema_ver: u16,
|
|
/// Written by the producing process. The replay compares against
|
|
/// **this**, never against a value it computed itself.
|
|
pub end_state_hash: String,
|
|
/// Hash of the initial state, so a mutated seed is detectable.
|
|
pub initial_state_hash: String,
|
|
pub commit: String,
|
|
}
|
|
|
|
/// `expected.yaml` — the assertions that failed, expected vs actual.
|
|
#[derive(Debug, Clone, Serialize, Deserialize)]
|
|
pub struct Expected {
|
|
pub reason: String,
|
|
pub end_state: serde_json::Value,
|
|
}
|
|
|
|
#[derive(Debug)]
|
|
pub struct ReplayReport {
|
|
pub scenario: String,
|
|
pub commands: usize,
|
|
pub hash: String,
|
|
}
|
|
|
|
fn err(context: &str, e: impl std::fmt::Display) -> String {
|
|
format!("{context}: {e}")
|
|
}
|
|
|
|
/// Write a `.cbreplay` bundle. Called when a scenario fails.
|
|
pub fn write_bundle(
|
|
dir: &Path,
|
|
scenario: &ScenarioFile,
|
|
initial_state: &serde_json::Value,
|
|
initial_state_hash: &str,
|
|
end_state: &serde_json::Value,
|
|
end_state_hash: &str,
|
|
reason: &str,
|
|
) -> Result<PathBuf, String> {
|
|
let slug = scenario.scenario.replace('/', "-");
|
|
let bundle = dir.join(format!("{slug}.cbreplay"));
|
|
// A stale bundle from a previous run would replay the old failure.
|
|
let _ = std::fs::remove_dir_all(&bundle);
|
|
std::fs::create_dir_all(&bundle).map_err(|e| err("create bundle", e))?;
|
|
|
|
std::fs::write(
|
|
bundle.join(INITIAL),
|
|
serde_json::to_vec_pretty(initial_state).map_err(|e| err("encode initial", e))?,
|
|
)
|
|
.map_err(|e| err("write initial", e))?;
|
|
|
|
// The command stream goes through the K11 framing, so a truncated
|
|
// bundle is detected rather than replayed short.
|
|
let mut log = FileLogStore::open(bundle.join(COMMANDS)).map_err(|e| err("open log", e))?;
|
|
for step in &scenario.commands {
|
|
let bytes = serde_json::to_vec(step).map_err(|e| err("encode command", e))?;
|
|
log.append(&bytes).map_err(|e| err("append command", e))?;
|
|
}
|
|
|
|
let manifest = Manifest {
|
|
scenario: scenario.scenario.clone(),
|
|
seed: scenario.seed,
|
|
setup: scenario.setup.clone(),
|
|
schema_ver: 1,
|
|
end_state_hash: end_state_hash.to_string(),
|
|
initial_state_hash: initial_state_hash.to_string(),
|
|
commit: option_env!("CB_COMMIT").unwrap_or("unknown").to_string(),
|
|
};
|
|
std::fs::write(
|
|
bundle.join(MANIFEST),
|
|
serde_yaml::to_string(&manifest).map_err(|e| err("encode manifest", e))?,
|
|
)
|
|
.map_err(|e| err("write manifest", e))?;
|
|
|
|
std::fs::write(
|
|
bundle.join(EXPECTED),
|
|
serde_yaml::to_string(&Expected {
|
|
reason: reason.to_string(),
|
|
end_state: end_state.clone(),
|
|
})
|
|
.map_err(|e| err("encode expected", e))?,
|
|
)
|
|
.map_err(|e| err("write expected", e))?;
|
|
|
|
Ok(bundle)
|
|
}
|
|
|
|
/// Re-execute a bundle and require it to reproduce the recorded hash.
|
|
///
|
|
/// Returns `Err` when the bundle is corrupt, incomplete, internally
|
|
/// inconsistent, or does not reproduce — all of which are the point. A
|
|
/// round-trip that cannot fail proves nothing.
|
|
pub fn replay<G>(bundle: &Path) -> Result<ReplayReport, String>
|
|
where
|
|
G: ScenarioGame,
|
|
{
|
|
let manifest: Manifest = serde_yaml::from_str(
|
|
&std::fs::read_to_string(bundle.join(MANIFEST)).map_err(|e| err("read manifest", e))?,
|
|
)
|
|
.map_err(|e| err("parse manifest", e))?;
|
|
|
|
// Control: the recorded seed and setup must reproduce the recorded
|
|
// initial snapshot. Without this the seed would be inert — replay
|
|
// restores from the snapshot — and a mutated-seed bundle would pass.
|
|
// Hash the TYPED aggregate, matching how the bundle was written.
|
|
let rebuilt = G::setup(&manifest.setup, manifest.seed)?;
|
|
let rebuilt_hash = state_hash_hex(&rebuilt);
|
|
if rebuilt_hash != manifest.initial_state_hash {
|
|
return Err(format!(
|
|
"bundle is inconsistent: seed {} and setup do not reproduce the \
|
|
recorded initial state ({} != {})",
|
|
manifest.seed, rebuilt_hash, manifest.initial_state_hash
|
|
));
|
|
}
|
|
|
|
let initial: serde_json::Value = serde_json::from_slice(
|
|
&std::fs::read(bundle.join(INITIAL)).map_err(|e| err("read initial", e))?,
|
|
)
|
|
.map_err(|e| err("parse initial", e))?;
|
|
let mut state: G = serde_json::from_value(initial).map_err(|e| err("restore initial", e))?;
|
|
|
|
// K11 framing: a truncated or corrupt command log is rejected here.
|
|
let raw = std::fs::read(bundle.join(COMMANDS)).map_err(|e| err("read commands", e))?;
|
|
let records = parse(&raw).map_err(|e| err("command log", e))?;
|
|
|
|
let mut applied = 0usize;
|
|
for record in &records {
|
|
let step: CommandStep =
|
|
serde_json::from_slice(record).map_err(|e| err("decode step", e))?;
|
|
let (actor, command) = G::parse_command(&step)?;
|
|
if let Ok(produced) = state.validate(actor, &command) {
|
|
for event in produced {
|
|
state.fold(&event);
|
|
}
|
|
}
|
|
applied += 1;
|
|
}
|
|
|
|
// Positive control: a bundle that replayed nothing must not be
|
|
// reported as a successful reproduction.
|
|
if applied == 0 {
|
|
return Err("bundle contains no commands; nothing was replayed".to_string());
|
|
}
|
|
|
|
let hash = state_hash_hex(&state);
|
|
if hash != manifest.end_state_hash {
|
|
return Err(format!(
|
|
"replay did not reproduce: {} != recorded {}",
|
|
hash, manifest.end_state_hash
|
|
));
|
|
}
|
|
|
|
Ok(ReplayReport {
|
|
scenario: manifest.scenario,
|
|
commands: applied,
|
|
hash,
|
|
})
|
|
}
|
|
|
|
/// Read a bundle's raw command log. Used by the negative controls.
|
|
pub fn raw_command_log(bundle: &Path) -> Result<Vec<u8>, String> {
|
|
std::fs::read(bundle.join(COMMANDS)).map_err(|e| err("read commands", e))
|
|
}
|
|
|
|
/// Overwrite a bundle's raw command log. Used by the negative controls.
|
|
pub fn set_raw_command_log(bundle: &Path, bytes: &[u8]) -> Result<(), String> {
|
|
std::fs::write(bundle.join(COMMANDS), bytes).map_err(|e| err("write commands", e))
|
|
}
|