feotest: Statistical Verdicts

feotest is the probabilistic testing framework that ships in the starter. The idea: treat each execution as a Bernoulli trial, pass or fail against a contract, and issue a verdict about the pass rate. It uses Wilson score intervals rather than a raw average, which is what you want at these sample sizes.

Use it for the two rate rows in the acceptance table: hard-braking success and empty-road false alarms. It also fits the protocol grid from the SPEC’s stretch section: one verdict per cell is how Euro NCAP grades its test points, at classroom scale. The latency row isn’t Bernoulli. Report latency response coverage plus the responder median, and quantify uncertainty with a bootstrap confidence interval or another documented distribution-free method. Don’t feed a median into a pass/fail-rate tool and call the result evidence for the median.

The entry point fits the runner directly:

use feotest::model::ContractViolation;
use feotest::probabilistic_test;

#[probabilistic_test(samples = 100, threshold = 0.99, threshold_origin = "slo")]
fn hard_braking_lead_ends_in_brake() -> Result<(), ContractViolation> {
    let outcome = run_scenario(Scenario::HardBrakingLead, Noise::default());
    if outcome.final_decision == Decision::Brake {
        Ok(())
    } else {
        Err(ContractViolation::new("no-brake", "run ended without Brake"))
    }
}

A single failing run counts as one trial rather than failing the test. The verdict is about the rate across all samples, and the threshold’s origin is recorded. Recognized origins are "sla", "slo", "policy", and "empirical"; other strings are stored as unspecified.

The workflow that comes with the tool

feotest’s intended workflow is measure, derive, test: baseline the behavior over many trials, derive a statistically grounded threshold from the baseline, then verify against it with an affordable sample count. This raises the question the acceptance table is really asking: 99% verified with how many samples, at what confidence? The framework offers three ways to answer (threshold-first, sample-size-first, confidence-first) instead of letting “run it 20 times” stand in for an answer.

Its statistical assumptions are stated in its documentation, and you should check them against guardian: trials roughly independent and stationary, binary outcomes, controlled conditions.

Note: feotest is early-stage and its API is marked unstable. Evaluate it the way you evaluated crates yesterday. For today it’s pinned in the starter’s lockfile.