Skip to content

beam-agents

An agent is a Beam transform.

beam-agents makes AI agents first-class citizens of Apache Beam streaming pipelines. An agent becomes a keyed, stateful, fault-tolerant transform — events | RunAgent(my_agent) — for system-triggered workloads: fraud triage, anomaly response, personalization, IoT reaction, ops automation. Events invoke the agent; its decisions are durable, replayable, and horizontally scalable. It is not built for sub-second interactive chat.

Read the quickstart Browse the examples

v1.0.0a1 · pre-release, not yet on PyPI · Apache-2.0 · Python 3.11–3.12

A runtime, not a framework

beam-agents deliberately owns only what agent frameworks lack: durable keyed memory, event/processing-time semantics, effectively-once side effects, backpressure-aware scale-out, and runner portability (DirectRunner, Dataflow, Flink). Agent authoring belongs to LangGraph, Google ADK, Pydantic AI, or a plain async function — integrated via adapters. There is no prompt templating, no orchestration DSL, and no agent-authoring abstraction here, on purpose.

The shape of a pipeline

The shape of a beam-agents pipeline Three topics — events, tool results and approvals — are flattened onto one stream, keyed by entity id, and fed to RunAgent. RunAgent emits four tagged outputs: dot output, dot traces and dot errors leave the graph to the right, while dot intents leaves downward to an outbox topic, is executed by an external effector, and re-enters the pipeline as tool results on the same key. events tool-results approvals Flatten WithKeys(entity_id) outbox topic effector RunAgent keyed · stateful .output .traces .errors .intents results, re-injected on the same key outside the pipeline

Solid lines are the Beam graph; dashed lines leave it. Beam graphs are acyclic, so an agent that stages a side effect cannot loop inside the graph — it suspends, and the result re-enters as a new element on the same key.

Side effects never execute inside the pipeline: the agent stages declarative ToolIntents, an external effector executes them exactly once per deterministic intent_id, and results re-enter on the same key.

Guarantees, and the gates that enforce them

Every guarantee below is a machine-verified release gate, not an aspiration. A claim on this site that nothing enforces is a defect.

Guarantee Enforced by
Side effects execute effectively once under real worker kills, duplicate sink writes, and full pipeline replay the effectively-once e2e gate (tests/semantics/test_effectively_once_e2e.py)
A retried bundle adds zero provider calls and commits byte-identical intents the retry-determinism gate (tests/semantics/test_retry_determinism.py)
Every adapter exhibits identical lifecycle semantics on DirectRunner and Flink the adapter conformance matrix (tests/conformance/)
Human-in-the-loop timeouts fail closed at both layers the HITL semantics gates (tests/semantics/test_hitl_fail_closed.py)
A failed activation commits nothing, and coverage/mutation scores never regress the ci and quality workflows

Install

Once v1.0.0 is published to PyPI, the install is the ordinary one:

pip install beam-agents

Until then — and always, for working on the runtime itself — install from source, exactly as the repository README describes:

git clone https://github.com/ardada2468/beam-agents
cd beam-agents
uv sync --all-groups

Requires Python >=3.11,<3.13; the source install additionally needs uv. The adapters, the effector, and the other optional pieces are extras (beam-agents[langgraph], [pydantic-ai], [adk], [effector], …) — each page below names the extra it needs.

Start here

The seven example programs are real, runnable Beam pipelines on the real runtime. The code each page shows is included verbatim from examples/ — the same file CI executes in tests/examples/.

Hermetic — no API keys, no network

Four are driven by a scripted FakeLLM, so they run offline with no API keys and (except the console demo's viewer) no docker.

  • Hello, world

    The minimal fast path: one event, one model call, one output.

  • Fraud triage

    Suspension, human approval, and the fail-closed timeout.

  • IoT reaction

    Keyed rolling memory on a stream, with no model calls for quiet readings.

  • Console demo

    One command that exercises the whole error-and-approval vocabulary and feeds the console.

Against the real thing

Three deliberately touch the world outside, because "does it work against the real thing" is a question too.

Then, operating it