Skip to content

ADR 0005 — Latency benchmark and supply-chain scanning

Status: accepted · Date: 2026-10-04

Context

The project sets an engineering target (sub-100 ms local-API batch for 100 synthetic nodes) and asks for dependency vulnerability / license checks and SBOM generation before any public release.

Decision

Latency benchmark

  • Harness: crates/server/examples/bench.rs, run in release. It drives the full in-process HTTP stack (tower::oneshot: router + strict parse + evaluate + serialize), excluding negligible loopback socket RTT.
  • Metrics: /evaluate-batch (100 candidates), a 100-distinct-node /evaluate sweep (the headline "100 nodes"), and the pure evaluator; warmup + ≥ 1000 samples; nearest-rank p50 / p95 / p99. The harness prints machine + build profile and warns when run in debug.
  • Numbers are published per-machine; a target is not an SLA. On a 12th Gen i5-12600K the 100-node sweep measured p50 4.25 ms / p95 5.96 ms / p99 6.41 ms — ~15–24× under the 100 ms target.

Supply chain

  • Rust: cargo-deny (deny.toml) gates advisories, licenses (permissive allow-list), bans, and sources; wired into core CI. Requires a committed Cargo.lock.
  • Go: go-licenses check gates the dependency license set (fails on non-permissive); govulncheck runs as an informational CI step that surfaces known CVEs without blocking (many are stdlib-patch / pinned-k8s transitive issues).
  • Images: syft emits an SPDX SBOM per image; grype scans each. Run via make supply-chain.
  • Findings are surfaced, not suppressed. Remediation before any real release: rebuild on the latest base-image / Go patch, bump upgradable deps, pin images by digest, attach SBOMs, optionally sign (cosign).

Consequences

  • CI fails on Rust advisories / licenses and non-permissive Go licenses; Go CVEs are visible but non-gating until a toolchain / k8s bump is scheduled.
  • The benchmark is reproducible (make bench) and self-describes its environment, so published numbers are falsifiable.