Benchmarks¶

openspec/project.md states a release-blocking constraint: runtime overhead p50 < 15 ms / p99 < 60 ms per activation, excluding LLM and tool time. The in-pipeline instrument for it is the overhead_ms distribution (docs/metrics.md), but Beam distributions carry no percentiles. The suite here is where the p50/p99 verdict is actually rendered.

Everything runs offline: no docker, no network, no real provider — FakeLLM only. make bench works on a laptop and on any CI runner.

make bench                     # writes bench-results/*.json
make bench-gate                # verdict + bench-report.md
make bench BENCH_ARGS=--fast   # scaled-down sampling, local iteration only

What each dimension measures¶

Benchmark	Measures	Deliberately excludes
`noop_throughput`	One full `process()` drain with a no-op agent — bridge submission, activation, staged commit. The runtime ceiling with zero agent work; the report also derives activations/sec.	Runner scheduling, bundle formation, shuffle
`overhead_50ms` / `overhead_500ms` / `overhead_2000ms`	Per-activation overhead = wall time − the configured `FakeLLM` latency, one activation per sample. The 50 ms tier is the gated one (dense sampling supports its p99); 500/2000 ms prove latency invariance.	The provider's nominal wait — but not the event loop's scheduling slop above it, which is runtime code and stays in the number
`suspension_roundtrip`	One `Suspend`-committing activation plus one admitted `ToolResult` resume, both element hops summed over shared handles.	The effector and the message bus — deployment latency, not runtime cost. Not an end-to-end SLA.
`state_commit_{1,16,64,100}kib` + `encode_*`	Activation cost against committed `MemoryBlob` size up to the 100 KiB blob cap, plus `DeterministicProtoCoder.encode` alone at the same sizes so the curve is attributable to the coder vs. the surrounding staging.	The runner's actual state-backend write (fake handles make it a no-op)
`runagent_per_element` / `runinference_per_element`	Two DirectRunner pipelines over identical zero-latency `FakeLLM` work: `RunAgent` vs `apache_beam.ml.inference`'s `RunInference` with a minimal `ModelHandler`. Reports per-element cost, delta, and ratio.	Nothing — but the DirectRunner's own overhead appears on both sides and largely cancels in the difference

The fake-handle benchmarks drive _AgentDoFn's element path directly. That surface includes everything the runtime owns (bridge submission, activation loop, replay cache, staging, coder encode at state write, commit ordering) and excludes everything it does not. The RunInference comparison's absolute figures come from a different surface (whole pipelines) and are not comparable to the others — and are not budget-gated.

Overhead is the same subtraction the dashboard publishes¶

overhead_sample records wall_time − tier_ms, mirroring _record_commit's max(0, activation_ms − Σ llm_ms − Σ tool_ms) with the nominal tier standing in for the measured call time. The gate figure and the overhead_ms dashboard figure are therefore the same quantity measured two ways.

Percentiles are computed over pooled per-activation samples from every pyperf worker process — never over per-process means, which would hide the exact tail the p99 budget exists to catch. That is why the overhead benchmark runs with --loops 1: one activation per recorded value.

Reading the report¶

bench-report.md (generated by scripts/bench_gate.py, the single reader of the JSON, so gated and reported numbers cannot drift) carries:

the gated tier's p50/p99 against the budget, with its pooled sample count;
per-benchmark medians, plus the derived activations/sec;
the tier-invariance table;
the RunInference delta and ratio;
every gate message (FAIL: lines and ratchet notes);
pyperf's captured environment metadata (host, CPU, Python, date).

The gate¶

make bench-gate renders two independent judgements:

Absolute budget — the gated tier's p50 must be < 15 ms and its p99 < 60 ms. A breach fails, full stop. Noise-robust by construction: expected overhead is low-single-digit milliseconds, so the budget carries about an order of magnitude of headroom over runner jitter, and it needs no reference machine to be meaningful. It is the number project.md promises.
Baseline ratchet — every benchmark's median against benchmark-baseline.toml. Regressing beyond that benchmark's band fails; improving beyond it passes and prints the instruction to lower the committed baseline by hand, so gains are locked in deliberately (the coverage_ratchet.py mechanism). The tier-invariance check and the RunInference delta ride this layer; the comparison's absolutes do not.

The band is the file's global tolerance unless the benchmark names its own in [tolerance_overrides]. One band cannot serve the whole suite: the activation benchmarks run 0.2–3 ms of Python, where 25% clears runner jitter, while encode_* times a single SerializeToString call — microseconds of memcpy-bound C whose median tracks whichever CPU generation ubuntu-latest allocated that night. The encode_* family therefore carries a 60% band, wide enough to ignore the CPU lottery and still fail anything that doubles; the file's comment carries the measurements behind that number. An override naming a benchmark the ratchet does not track fails the gate rather than silently widening nothing.

Missing result files, a declared benchmark absent from a result file, too few samples for the p99, or an unseeded baseline entry all fail — a gate that passes on a missing run has silently stopped gating. There is no xfail/retry/skip-when-red tolerance anywhere in it.

Updating the baseline¶

benchmark-baseline.toml is hand-maintained, exactly like coverage-baseline.toml and mutation-baseline.toml.

Run the nightly bench job on a GitHub-hosted runner (Actions → nightly → Run workflow), or wait for the schedule.
Read the gate output in that job (and bench-report.md in the benchmark-report artifact).
Copy each seed medians_ms.<name> = <value> or lower medians_ms.<name> to <value> instruction into the file, noting the runner generation in a comment.

Never seed the baseline from developer hardware. The gate runs on CI hardware; a laptop-derived number either blocks every nightly or is so loose it catches nothing.

A lower medians_ms.<name> note is not always a gain to lock in. On a benchmark whose run-to-run spread exceeds its band, the note fires on a lucky night, and committing that number makes an ordinary night red. Check the suite's own physics before copying one: tests/benchmarks/test_bench_smoke.py asserts the committed encode_* curve grows with blob size, because a seeded curve that dips is proof that run mismeasured — which is exactly how the 2026-08-03 seed recorded 64 KiB as dearer than 100 KiB.

Why nightly and not per-PR¶

pyperf spawns worker processes, warms up, and calibrates; its own documentation is blunt that results on busy or virtualized machines are noisy, and GitHub-hosted runners are shared 2-core VMs. A per-PR gate on wall-clock numbers from those machines would either cry wolf or be widened until it caught nothing. So the suite runs in nightly.yml with release-blocking (not merge-blocking) semantics — matching project.md's "benchmark regressions on this are release blockers" — plus workflow_dispatch for the on-demand case. An advisory per-PR check that is usually noise trains everyone to ignore it, which is worse than absent.

pyperf system tune is deliberately not used: it needs capabilities hosted runners do not grant, and a gate that requires root to be honest is not portable.

The benchmarks cannot rot silently¶

The modules are pyperf scripts, never collected by pytest (pyperf owns argv and forks calibrated workers, which fights pytest's collection model; a timed run has no business inside a tier with a 30 s global timeout). So tests/benchmarks/test_bench_smoke.py imports every benchmarks/bench_*.py module and executes each timed function for one iteration inside the offline unit tier — a runtime refactor that breaks a benchmark fails the required ci lane, not a nightly three days later. benchmarks/ is also in mypy's file set, held to the same --strict bar as src/ and tests/.

The release artifact¶

The nightly job uploads bench-results/*.json plus bench-report.md as one artifact named benchmark-report. The consumer lives in release.yml's publish job: it locates the most recent nightly run on main whose bench job concluded green (via the jobs API — sibling nightly jobs can be red independently), downloads this artifact by that stable name, and attaches bench-report.md and bench-results.zip to the GitHub Release. It fails closed: no green bench run in the last 30 nightly runs fails the release before anything is published. See Releasing.