beam-agents 0.3.0 vs. Apache Flink Agents¶
Release: beam-agents 0.3.0 · Report status: methodology published, measurement pending CI hardware · Frozen at: publication of 0.3.0
openspec/project.md names Apache Flink Agents as this project's direct
competitor. This is the report that is supposed to put numbers behind that
sentence instead of prose. It publishes, in full and up front, the parts that
determine whether the numbers can be trusted — the workload pairing, the
methodology, the version pins, and every dimension on which the two systems are
not comparable. It publishes no numbers, because none exist yet that this
project considers admissible.
Status: why the results tables are empty¶
This is the honest state of the measurement, stated first rather than buried:
- The benchmark harness that produces the beam-agents leg exists and runs
(
docs/benchmarks.md, C33add-benchmark-harness). - Its committed baselines do not.
benchmark-baseline.toml's[medians_ms]table is deliberately empty, and the file says why: the gate runs on CI hardware, and a baseline seeded from anywhere else is either permanently red or so loose it catches nothing. docs/benchmarks.mdstates the same rule as a prohibition — never seed the baseline from developer hardware — and the reasoning applies with more force to a published competitive comparison than to an internal ratchet. A number measured on a developer laptop and printed in a document that names another Apache project would be exactly the "one unfair chart" the design for this report (D3) says costs more credibility than ten favorable ones buy.
So every measurement cell below reads pending (CI hardware). What unblocks
them is a run of the nightly bench job on a GitHub-hosted runner, whose
figures seed benchmark-baseline.toml, plus the paired Flink Agents leg
executed in the environment described under Methodology. Until
then this document is a methodology and a pairing, and says so in its own
header.
The measurement tables are the only part of this report that is pending. Everything else — pairing, non-equivalence disclosure, freeze rule, publication policy — is final at 0.3.0.
What the two systems are¶
| beam-agents | Apache Flink Agents | |
|---|---|---|
| Agent execution | Python agent inside a Beam stateful DoFn, driven by an async bridge thread |
Agent executed inline inside the JVM runtime with durable execution |
| Side effects | Staged as ToolIntents, committed with the bundle, executed by an external effector, results re-injected on the same key |
Executed inline, made durable by the framework's own execution log |
| State | Protobuf blobs in Beam keyed state, committed atomically with the bundle | Flink keyed state, committed with the checkpoint |
| Portability | DirectRunner, Dataflow, Flink (Spark best-effort) | Flink |
The two rows that matter for the numbers are side effects and agent execution: they are the design decisions the comparison is actually measuring the cost of, and they are the reason a naive head-to-head number would mislead.
The paired workload¶
beam-agents leg: overhead_50ms, from
benchmarks/bench_overhead_tiers.py
(C33's scenario set; the scenario is named, not invented for this report).
Selection rationale, per design D3's "one primary scenario, compared deeply":
- It is the scenario the release budget is actually rendered on — the gated tier. Framing the comparison around a metric we do not otherwise gate would invite exactly the metric-shopping this report is structured to avoid.
- It is an event-triggered, keyed, stateful activation that makes one model call and commits — the shared core of both systems' target use case, and a shape Flink Agents expresses idiomatically rather than by contortion.
- Its measurement is defined as a subtraction — activation wall time minus the
configured provider latency — which is the same "excluding LLM and tool time"
definition the budget in
openspec/project.mduses, and which transfers to the other side without reinterpretation.
Secondary leg: noop_throughput, for the throughput row. It is the
runtime's ceiling with zero agent work; paired against the equivalent
zero-work agent on Flink Agents it isolates per-element runtime cost from
agent cost.
Deliberately excluded: suspension_roundtrip. It measures the
suspend/resume round trip that only the outbox effect model has. There is no
Flink Agents equivalent to pair it against — their agents do not pay it,
because their effects run inline. Reporting it as a head-to-head row would be
measuring the presence of a feature and calling it a loss. It appears instead
in the non-equivalence table below, where it belongs, and its absolute cost is
reported on the beam-agents side alone.
The state_commit_* and runinference_* scenarios are out of scope: the first
measures a coder curve, the second compares against a different Beam transform,
and neither has a Flink Agents counterpart.
Methodology¶
Versions pinned¶
Every component is pinned, and anything not yet pinnable says so rather than being omitted.
| Component | Pinned version |
|---|---|
| beam-agents | 0.3.0 (this release; the exact commit is the 0.3.0 tag) |
| Apache Beam | 2.72.0 (uv.lock; pyproject.toml requires apache-beam[gcp]>=2.60) |
| Python | 3.11 (requires-python = ">=3.11,<3.13") |
| Apache Flink | 1.19 — flink:1.19-scala_2.12-java17, digest-pinned in docker/compose.yaml |
| Beam Flink job server | apache/beam_flink1.19_job_server:2.72.0, digest-pinned |
| Apache Flink Agents | pending — pinned at run time to the project's latest stable release, or to an exact commit with that fact stated, per design D3's reproducibility rule and its open question on the pin |
Both legs run on Flink: the beam-agents leg runs on the Flink runner rather than DirectRunner or Dataflow, so the underlying cluster is the same on both sides and the comparison is as close to like-for-like as the two architectures permit. That choice costs beam-agents the Beam portability layer, which is disclosed below rather than netted out.
The model is a scripted fake on both legs¶
Both legs call a scripted fake model of equal configured cost — FakeLLM on
ours, the closest achievable stub on theirs — so what is measured is what each
runtime adds, not what a provider takes. This is the same subtraction as the
release budget: runtime overhead per activation, excluding LLM and tool
time. A comparison dominated by provider latency measures the provider.
Environment and reproduction¶
- Both stacks run on the same dedicated host, one at a time, never concurrently; the environment manifest (host, CPU, kernel, JVM, Python) is captured by pyperf on our side and recorded verbatim in the results section when the run happens.
- The beam-agents leg is reproduced with
make benchfollowed bymake bench-gate, exactly as the nightly job runs it — noBENCH_ARGS, no scaled-down sampling, since the pinned per-tier sampling constants are what make the p99 meaningful. - The Flink Agents leg's run configuration is committed alongside this report when it is written, per design D3. It is not committed at 0.3.0 publication: writing a run configuration against APIs that have not been executed would be a fabricated artifact, which is a worse failure than a pending one.
Where the comparison is not like-for-like¶
Enumerated per design D3 and this release's spec, with the direction of each difference stated. A disclosure without a direction is one the reader cannot use.
| Dimension | The difference | Which side it structurally favors |
|---|---|---|
| Language runtime | Flink Agents executes agents inline in the JVM. beam-agents executes Python agent code behind the Beam portability layer, through an SDK harness and an async bridge thread. | Favors Flink Agents. Cross-language process boundaries and Python execution are a cost their inline JVM path does not pay. It is a real cost of ours, not a measurement artifact, so it is not netted out. |
| Effect model | Their side-effecting tools run inline, made durable by the framework. Ours are staged as intents, committed with the bundle, executed by an external effector, and re-injected on the same key. | Favors Flink Agents on latency; favors beam-agents on the guarantee. The re-injection round trip is latency they do not pay; effector-side dedup on a deterministic intent_id, and effects that survive outside the pipeline, are properties they price differently. The primary scenario makes no side-effecting call, so this difference does not distort it — which is precisely why the primary scenario was chosen. |
| State backend and checkpointing | Flink Agents uses Flink keyed state and checkpoints directly. beam-agents uses Beam keyed state over the Flink runner's state backend, with protobuf blobs it encodes itself. | Favors Flink Agents. One layer fewer, and no user-level encode. Our encode cost is runtime cost and stays in the number. |
| Portability layer | The beam-agents leg pays Beam's runner-portability layer on top of Flink. Flink Agents does not have one. | Favors Flink Agents. Running our leg on the Flink runner maximizes comparability at the cost of carrying this overhead into every number; running it on DirectRunner would have removed the layer and the comparability with it. |
| Measurement surface | Our figures come from the harness's direct DoFn element path (bridge submission, activation, staging, commit) with fake state handles; theirs come from the closest equivalent their harness exposes. |
Direction unknown — the largest residual risk in this report. If the two surfaces cannot be made to include the same work, the pairing is withdrawn and this ships as a beam-agents-only budget report with that explanation, per the spec. |
No cherry-picking¶
Every completed run of the final configuration is reported — percentile tables, not best-of. Runs are not dropped for being unfavorable. If beam-agents loses a metric, the row is published at the same prominence as a favorable one, and where the loss is the known price of a design decision the row links the decision rather than explaining it away. Configuration changes after the first completed run restart the run set; the report states how many runs the tables summarize.
Results¶
Nothing below is measured yet. See Status.
Primary: per-activation runtime overhead (overhead_50ms)¶
Framed as this release's primary claim — beam-agents meets its stated budget of p50 < 15 ms and p99 < 60 ms per activation, excluding LLM and tool time, under this workload — with the Flink Agents figures as measured context, not a scoreboard.
| Leg | p50 | p99 | Pooled samples | Verdict vs. budget |
|---|---|---|---|---|
| beam-agents (Flink runner) | pending (CI hardware) | pending (CI hardware) | pending (CI hardware) | pending (CI hardware) |
| Apache Flink Agents | pending (CI hardware) | pending (CI hardware) | pending (CI hardware) | n/a — pending (CI hardware) |
Secondary: throughput (noop_throughput)¶
| Leg | Median per element | Derived activations/sec | Runs summarized |
|---|---|---|---|
| beam-agents (Flink runner) | pending (CI hardware) | pending (CI hardware) | pending (CI hardware) |
| Apache Flink Agents | pending (CI hardware) | pending (CI hardware) | pending (CI hardware) |
beam-agents only: suspend/resume round trip (suspension_roundtrip)¶
Reported unpaired, for the reason given under the paired workload: it is the cost of an effect model the other side does not have.
| Measurement | Value | Runs summarized |
|---|---|---|
| Suspend-commit plus admitted-resume, summed | pending (CI hardware) | pending (CI hardware) |
How to read these numbers when they land¶
- The primary claim is absolute, not relative: does beam-agents meet its own budget under this workload? That question has a right answer independent of what the other project measures.
- The comparison is context for that claim, bounded by the non-equivalence table. A reader who wants a single winner will not find one here, and the report is not structured to produce one.
- Every figure is "as of these versions". Apache Flink Agents is a young, fast-moving project; a stale measurement is the expected end state of this document, not a defect in it.
This report is frozen¶
Numbers here are frozen at publication of 0.3.0 and attributable to the exact commits and versions pinned above. It is never edited to reflect later performance changes — a later release publishes a later report. A correction ships as an errata note in a subsequent release, never as a silent edit. The one edit this document will receive is the population of its measurement tables from the run described above, which is the completion of publication rather than a revision of it; after that, the freeze is absolute.
What is still open¶
- The Flink Agents version pin, decided at run time and recorded above.
- Whether the two harnesses' measurement surfaces can be made to include the same work. If review concludes they cannot, this ships as a beam-agents-only budget report with that explanation, per the spec's No fair pairing exists scenario.
- Whether the comparison is repeated per release. Deliberately open; a follow-up change proposes a cadence if the answer is yes.