An agent that’s stuck in an infinite retry loop doesn’t show up in your error rate. It shows up in your AWS bill — eleven days later.

The Infrastructure Problem Underneath the Agent Problem

Multi-agent systems hit production in 2026 in a way they hadn’t before. Google’s Agent-to-Agent protocol reached 50+ partners, and Anthropic’s Model Context Protocol hit 97 million monthly SDK downloads. Research that cited multi-agent architectures as experimental now has production case studies behind it. The architecture has matured from “interesting research direction” to “real engineering pattern with a known set of failure modes.”

It has also matured into something wildly overused. Most teams reach for multi-agent because the demos look impressive, not because their actual problem benefits from it. The result: complex systems that cost 10x as much, fail in ways nobody can debug, and lose to a well-written single-agent prompt on the actual benchmarks that matter.

The canonical production incident that defines the current state of the discipline: an engineering team ran a multi-agent research system for eleven days. Latency normal. Error rate 0.0%. Uptime 99.99%. Every dashboard showed green. Then they checked their cloud bill: $47,000. The agents had been stuck in an infinite retry loop the entire time. The monitoring stack had no idea — because the agents were not down. They were wrong. And wrong doesn’t show up in Grafana.

This incident is not exceptional. It is representative of a class of failure that emerges specifically from multi-agent architecture and does not exist in single-agent or deterministic pipeline systems: agents that are operating correctly by every infrastructure metric while producing incorrect, looping, or redundant behavior at the application layer. Standard service monitoring — latency, error rate, uptime — has no representation of whether an agent’s reasoning is sound, whether its handoffs are making progress, or whether the system-level task is converging on a result or quietly cycling while your token budget burns.

This post is about the infrastructure underneath the agents: the coordination patterns, the state management, the inter-agent contracts, the cost controls, and the observability layer that are the actual engineering work separating a demo from a system that survives production.

Article context: This post closes the six-article RAG and AI engineering series. The Agentic RAG post covered single-agent iterative retrieval loops — the simplest form of agentic behavior. This post covers what happens when you compose multiple agents together: the new failure modes that composition introduces, and the infrastructure discipline required to contain them. The LLM Evaluation in Production post’s RAGAS pipeline and the Context Engineering post’s assembly layer both apply here — each agent in a multi-agent system is itself a RAG pipeline or an LLM call with assembled context, and all the evaluation and context discipline from those posts applies per-agent, not just system-wide.

Table of Contents

1. When Multi-Agent Is Not the Answer

Before covering patterns, the most important production heuristic first: the strongest production heuristic in 2026 is the negative one — most teams reach for multi-agent too early and pay for it.

Stay with a single agent when:

  • The task fits in one context window. 200K Claude or 1M Gemini contexts cover most enterprise tasks. Splitting an in-context task across agents adds coordination overhead and context-transfer loss without benefit.
  • Latency is tight. Coordination overhead alone — tool calls, inter-agent message passing, synthesis — adds seconds. If your SLA is under ten seconds, multi-agent is very hard to make work.
  • Your single-agent baseline isn’t tuned yet. A ReAct loop with tight prompting on a frontier model should be your ceiling check before assuming multi-agent is required. Most teams that jump to multi-agent haven’t found their single-agent ceiling.
  • The work is sequential by nature. No parallelism, no specialization gain — just orchestration tax on top of the same sequential work.
  • You don’t have observability infrastructure. Multi-agent failures without traces are practically unfixable. The $47K incident above happened because the team lacked per-agent tracing. This is not an infrastructure-you’ll-add-later problem; it’s a prerequisite.

Multi-agent earns its complexity when the work genuinely decomposes: parallel subtasks that can run simultaneously, distinct tool access requirements that don’t belong in one agent’s scope, or failure isolation needs where one agent’s bad output should not cascade to the others.

2. The Four Coordination Patterns

Industry surveys put the orchestrator-worker pattern at roughly 70% of production multi-agent deployments in 2026, including most internal builds at Stripe, Mercury, and the Anthropic and OpenAI reference designs. Understanding why it dominates, and what the alternatives trade off against it, determines which pattern applies to your architecture.

Orchestrator-Worker

A central orchestrator decomposes an incoming task, dispatches subtasks to specialized workers in parallel or sequence, and synthesizes their results. One level of indirection, parallelism across workers, and observability concentrated at the orchestrator’s decision points.

User Request
    │
    ▼
┌───────────────────┐
│    ORCHESTRATOR   │  Decomposes task, routes to workers,
│  (planner/router) │  synthesizes results, handles failures
└──┬────────┬───────┘
   │        │
   ▼        ▼
[Worker A] [Worker B]   Specialist agents with distinct
[Researcher][Coder]     tool access or domain scope

The orchestrator is a single point of failure. If it misclassifies a task, the wrong worker gets it, and misclassification rates compound at scale. Context window overflow is the more subtle problem — the orchestrator accumulates context from every worker, and at four or more workers, context frequently exceeds window limits. Section 4 covers the state management pattern that prevents this.

Sequential Pipeline

Agents execute in a fixed linear chain where each agent processes the previous agent’s output. Deterministic, easy to reason about, and failure modes are localizable to individual handoffs.

Strengths: easy to reason about; each agent has a single clear job; failure modes are usually localizable to one handoff. Watch out for: latency stacking — each handoff adds a round-trip — and context loss in the transition. Sequential pipelines are the right choice when the processing stages are clearly defined, order-dependent, and have no parallelism opportunity — document ingestion pipelines (parse → extract → validate → summarize) are the archetypal case.

Dynamic Handoff (Swarm)

No central orchestrator. Each agent assesses the current task and decides whether to handle it or transfer to a more appropriate specialist based on runtime context. Flexible and emergent, but the number-one failure mode is infinite handoff loops: Agent A passes to B, B passes to C, C passes back to A. Each agent keeps replanning because nobody owns the task, and context loss compounds with every transfer.

Reserve this pattern for tasks where the expertise required genuinely cannot be predicted at the start of the interaction — customer support is the canonical use case, where a billing question can reveal an underlying technical problem mid-conversation.

Hierarchical (Multi-Level Orchestration)

Orchestrators managing sub-orchestrators managing workers. Appropriate for large-scale decomposition where a single orchestrator’s context budget cannot hold the full planning state. The highest coordination overhead and the most complex failure surface — LangGraph’s state machine model, where agents are nodes and transitions are edges, offers the best observability for hierarchical patterns because you get checkpointing, replay, and time-travel debugging at every node.

3. Orchestration vs. Choreography — The Design Decision Nobody Names

The four patterns above are coordination topologies. The more fundamental design decision is the coordination model: orchestration or choreography. These are the same two models distributed systems engineers have debated in the microservices context, and they apply directly to multi-agent systems.

Orchestration: a central entity (the orchestrator, the planner, the supervisor) knows the full workflow and actively directs each step. Every agent waits for direction. Workflow logic is centralized and inspectable in one place.

Choreography: agents react to events or shared state and decide independently what to do next. No central director. Workflow logic is distributed across every participant. The overall behavior emerges from each agent’s local decisions rather than a global plan.

The production implications are significant:

  Orchestration Choreography
Workflow logic location Centralized in orchestrator Distributed across agents
Observability Naturally centralized Requires aggregation across all agents
Failure isolation Orchestrator owns retry/fallback Each agent handles its own failures
Flexibility Changes require orchestrator updates Agents can be added without changing others
Blast radius of orchestrator bug High — affects all flows N/A — no central orchestrator to fail
Debugging complexity Lower — one place to look Higher — need full event trace

Move to choreography when the work has reliable stages and audit-worthy intermediate artifacts. Move to orchestration when the task is breadth-first, decomposable, or spans distinct tool or policy domains. For most teams arriving at multi-agent for the first time, orchestration is the right default — not because it is architecturally superior, but because its observability characteristics make the inevitable first production failure debuggable. Choreography’s distributed event model makes debugging hard enough that it should be a deliberate, justified choice rather than the default.

4. State Persistence — The Difference Between a Demo and a Production System

The single capability that distinguishes production multi-agent systems from well-engineered demos is state persistence: the ability to persist intermediate results, track task completion across agents, and let workflows pause, resume, or retry individual failed agents without restarting the entire execution from scratch. This single capability is the difference between a system that handles production conditions and one that only works in demos.

Without it: any failure — API timeout, model refusal, parsing error, rate limit — restarts the workflow from the beginning, re-doing all completed work, re-spending all spent tokens, and surfacing to the user as a full failure rather than a partial retry.

The state model needs to capture three things at every checkpoint: what the current task decomposition looks like, which subtasks have completed and what they returned, and which are in-flight, pending, or failed.

from dataclasses import dataclass, field
from enum import Enum

class TaskStatus(Enum):
    PENDING = "pending"
    IN_FLIGHT = "in_flight"
    COMPLETED = "completed"
    FAILED = "failed"
    CANCELLED = "cancelled"

@dataclass
class AgentTask:
    task_id: str
    agent_role: str           # "researcher" | "coder" | "reviewer" | "synthesizer"
    input: dict
    output: dict | None = None
    status: TaskStatus = TaskStatus.PENDING
    error: str | None = None
    attempt_count: int = 0
    depends_on: list[str] = field(default_factory=list)   # task_ids this task waits for

@dataclass
class WorkflowState:
    workflow_id: str
    original_request: str
    tasks: dict[str, AgentTask]   # task_id → AgentTask
    created_at: str
    updated_at: str

    def next_executable_tasks(self) -> list[AgentTask]:
        """Return tasks whose dependencies are all completed and that are still pending."""
        completed_ids = {tid for tid, t in self.tasks.items() if t.status == TaskStatus.COMPLETED}
        return [
            t for t in self.tasks.values()
            if t.status == TaskStatus.PENDING
            and all(dep in completed_ids for dep in t.depends_on)
        ]

    def is_complete(self) -> bool:
        return all(t.status in (TaskStatus.COMPLETED, TaskStatus.CANCELLED) for t in self.tasks.values())

    def has_unrecoverable_failure(self) -> bool:
        return any(t.status == TaskStatus.FAILED and t.attempt_count >= 3 for t in self.tasks.values())

Persist WorkflowState to durable storage — Postgres, Redis with AOF persistence, or a purpose-built workflow store like Temporal — at every status transition. The key invariant: if the orchestrator process dies between any two checkpoints, the workflow resumes from the last checkpoint, not from the start. The checkpoint granularity should be at the individual task level, not the workflow level — re-running a ten-task workflow because one task failed on its third try is the unambitious version of this pattern.

5. Inter-Agent Contracts — MCP, A2A, and Structured Handoffs

Two protocol standards now shape how agents communicate in 2026, and understanding where each applies matters for building systems that don’t depend on ad-hoc message-passing between agents you wrote yourself and that break the moment you swap one component.

MCP (Model Context Protocol) governs how agents interact with tools and data sources — file systems, databases, external APIs, the knowledge bases built throughout this series. An agent that needs to query a vector store or run a code execution tool does so through MCP, which provides a standardized, auditable interface. A2A (Agent-to-Agent protocol) governs how agents interact with each other. It reached v1.0 in early 2026 with gRPC and OAuth 2.1 support, backed by over 100 enterprise organizations including Microsoft, AWS, Salesforce, SAP, and Cisco. The two protocols are complementary: MCP for agent-to-tool, A2A for agent-to-agent.

Beyond these protocols, every handoff between agents in your system — regardless of transport — needs a typed, validated contract. The most common source of inter-agent bugs in production is not communication failure (which is visible) but semantic failure: an agent received a correctly formatted message that contained logically wrong output from its predecessor, and proceeded confidently without detecting the problem.

from pydantic import BaseModel, field_validator

class ResearchOutput(BaseModel):
    """Contract for the output of the Research agent → Synthesis agent handoff.
    Pydantic validation runs at handoff time — type mismatches fail loudly immediately
    rather than producing a confidently wrong synthesis downstream."""
    query: str
    findings: list[str]
    sources: list[str]
    confidence: float
    sufficient_for_answer: bool

    @field_validator("confidence")
    @classmethod
    def confidence_in_range(cls, v: float) -> float:
        if not 0.0 <= v <= 1.0:
            raise ValueError("confidence must be between 0 and 1")
        return v

    @field_validator("sources")
    @classmethod
    def sources_not_empty_if_confident(cls, v: list[str], info) -> list[str]:
        if info.data.get("confidence", 0) > 0.7 and not v:
            raise ValueError("High-confidence findings must cite sources")
        return v

The discipline here: every inter-agent handoff has a named, versioned Pydantic model. The validation is not optional. An agent that produces output failing validation raises immediately at the boundary, not silently three agents downstream when the bad data causes an inscrutable generation error. Most agent failures in production aren’t actually agent problems — they’re orchestration and context-transfer issues at the handoff points between agents. Typed, validated contracts at every handoff close a large fraction of this failure class.

6. Failure Isolation — Circuit Breakers and Saga Patterns for Agent Workflows

A 2026 study analyzing five popular multi-agent frameworks across more than 150 tasks identified 14 distinct failure modes across three categories: specification and system design, inter-agent misalignment, and task verification and termination. The conclusion: many failures are structural, not fixable with better prompts.

Structural failure modes require structural solutions — the same distributed systems patterns applied to service meshes, adapted for agent workflows.

Circuit Breakers for Agent Calls

An agent calling an external tool, a downstream model, or another agent is making a remote call with all the failure characteristics of a remote call: latency spikes, timeouts, transient errors, and persistent failures that need to be detected and isolated rather than retried indefinitely.

from dataclasses import dataclass
import time

@dataclass
class CircuitBreaker:
    failure_threshold: int = 3
    recovery_timeout_seconds: float = 30.0
    _failures: int = 0
    _state: str = "closed"         # closed → open → half_open → closed
    _last_failure_time: float = 0.0

    def call(self, fn, *args, **kwargs):
        if self._state == "open":
            if time.time() - self._last_failure_time < self.recovery_timeout_seconds:
                raise RuntimeError(f"Circuit open — agent call blocked for {self.recovery_timeout_seconds}s")
            self._state = "half_open"

        try:
            result = fn(*args, **kwargs)
            self._on_success()
            return result
        except Exception as e:
            self._on_failure()
            raise

    def _on_success(self):
        self._failures = 0
        self._state = "closed"

    def _on_failure(self):
        self._failures += 1
        self._last_failure_time = time.time()
        if self._failures >= self.failure_threshold:
            self._state = "open"

Apply a circuit breaker per downstream dependency, not per agent. One flaky external API should not open the circuit for an agent calling a completely different tool.

Saga Pattern for Multi-Step Workflows

When a multi-agent workflow involves actions with side effects — writing to a database, sending a notification, calling an external API that charges per call — partial completion is a real failure mode. The workflow runs five of six steps and then fails; now the system is in an inconsistent state. The saga pattern handles this by defining a compensating action for each step that can be executed if a later step fails, rolling the partially completed work back to a consistent state.

@dataclass
class SagaStep:
    name: str
    execute: callable      # the forward action
    compensate: callable   # the rollback action if a later step fails

async def run_saga(steps: list[SagaStep], context: dict) -> dict:
    """Execute steps in order. If any step fails, run compensating actions for
    all previously completed steps in reverse order before re-raising."""
    completed = []
    try:
        for step in steps:
            context = await step.execute(context)
            completed.append(step)
        return context
    except Exception as e:
        for step in reversed(completed):
            try:
                await step.compensate(context)
            except Exception as comp_error:
                # Log compensation failure — this is a serious system integrity issue
                # Do NOT swallow it. Alert immediately.
                logger.error("saga_compensation_failed", step=step.name, error=str(comp_error))
        raise

The saga pattern matters specifically in agent workflows where agents take real-world actions — not just generate text. An agentic system that calls an external billing API, writes an audit log, and then sends a notification email has three steps with side effects, and “undo the first two if the third fails” is not a problem the LLM can solve. The saga is infrastructure, not prompt engineering.

7. Cost Controls That Actually Work

Workflows that cost $0.50 in testing can hit $50,000 per month at 100K executions because the orchestrator makes multiple LLM calls for decomposition and aggregation on top of every worker call. The difference between a multi-agent system that is economically viable and one that is a cost disaster is usually a set of deliberate controls that weren’t added until after the first cloud bill arrived.

Budget per workflow run, not per agent. Set a maximum token budget at the workflow level and propagate a remaining-budget signal to every agent. When the budget is exhausted, the workflow terminates with a partial result rather than continuing to accumulate cost. The agentic retrieval post’s AGENTIC_LOOP_LIMITS pattern — max iterations, max tokens, wall-clock timeout — applies here at the workflow level.

Model tier routing. Not every agent in a workflow needs a frontier model. Routing, classification, structured extraction, and format validation work well at small-model cost. Reserve the expensive model for reasoning-heavy steps where quality matters.

AGENT_MODEL_TIERS = {
    "orchestrator":    "claude-sonnet-4-20250514",   # reasoning + decomposition
    "researcher":      "claude-sonnet-4-20250514",   # synthesis from retrieved context
    "code_reviewer":   "claude-sonnet-4-20250514",   # quality judgment
    "classifier":      "claude-haiku-4-5-20251001",  # routing, classification
    "formatter":       "claude-haiku-4-5-20251001",  # output formatting, validation
    "summarizer":      "claude-haiku-4-5-20251001",  # compression, dedup
}

Hard ceiling on retry depth. Each agent in the WorkflowState schema above has an attempt_count field and a max-attempt guard (3 in the example). This is the structural control that would have prevented the $47K incident — a retry ceiling that is enforced in code, not just intended in design.

Parallelism only where it reduces total cost. Parallel agent calls reduce wall-clock latency but increase total token spend — all calls happen simultaneously rather than the system potentially short-circuiting if an early result makes later calls unnecessary. For cost-sensitive workflows, consider sequential execution with an early-exit condition before defaulting to full fan-out.

8. Observability — Tracing What Wrong Looks Like

The infrastructure monitoring problem the $47K incident exposes is fundamental: standard metrics (latency, error rate, uptime) measure whether agents are running. They say nothing about whether agents are making progress. A new class of observability is required — one that can detect “wrong” as a distinct state from “down.”

The foundation is structured tracing at the span level, with every agent call as a traced span carrying the inputs, outputs, model used, token count, and the workflow ID that allows spans to be correlated across agents.

import structlog
from opentelemetry import trace

logger = structlog.get_logger()
tracer = trace.get_tracer("multi_agent_system")

def trace_agent_call(workflow_id: str, task_id: str, agent_role: str):
    """Decorator that wraps any agent call with an OTel span and structured log."""
    def decorator(fn):
        async def wrapper(*args, **kwargs):
            with tracer.start_as_current_span(f"agent.{agent_role}") as span:
                span.set_attributes({
                    "workflow.id": workflow_id,
                    "task.id": task_id,
                    "agent.role": agent_role,
                })
                try:
                    result = await fn(*args, **kwargs)
                    span.set_attribute("task.status", "completed")
                    logger.info("agent_task_completed", workflow_id=workflow_id,
                                task_id=task_id, agent_role=agent_role)
                    return result
                except Exception as e:
                    span.set_attribute("task.status", "failed")
                    span.record_exception(e)
                    logger.error("agent_task_failed", workflow_id=workflow_id,
                                 task_id=task_id, agent_role=agent_role, error=str(e))
                    raise
        return wrapper
    return decorator

Structured traces enable the metrics that standard monitoring cannot: workflow completion rate (how many workflows reach is_complete() successfully vs. fail or timeout), per-agent failure rate (which agent role is the most common failure point), mean attempts per task (rising mean signals retry pressure building), and cost per workflow run (token counts aggregated across all agent spans for a workflow ID).

The alert that would have caught the $47K incident is simple once the data exists: flag any workflow where sum(attempt_count for all tasks) > 2 * len(tasks) — the system is retrying more than it is completing, which is the structural signature of a stuck loop. This alert fires on behavior, not on infrastructure failure. That’s the category of observability that multi-agent systems require that no standard monitoring stack provides out of the box.

The Multi-Agent Production Checklist

Before deploying any multi-agent system:

  • Single-agent baseline validated — multi-agent complexity is justified by a documented limitation of the single-agent approach, not by preference
  • Coordination pattern chosen explicitly — orchestrator-worker, sequential pipeline, dynamic handoff, or hierarchical; the choice is documented with its trade-offs
  • Orchestration vs. choreography decision made — centralized or distributed control; choreography only chosen when the observability cost is acceptable and deliberate
  • WorkflowState persisted to durable storage — intermediate results survive orchestrator restarts; retry resumes from last checkpoint, not from scratch
  • Typed, validated inter-agent contracts — every handoff has a Pydantic model; validation runs at the boundary, not downstream
  • Circuit breakers on all downstream dependencies — per external service, not per agent; failure threshold and recovery timeout set
  • Saga compensating actions defined for any workflow step with external side effects
  • Max retry depth enforced in code — attempt_count >= max_attempts terminates, not retries forever
  • Workflow-level budget cap — total token budget tracked and enforced across all agents in the workflow
  • Model tier routing implemented — frontier model only for reasoning-heavy steps; small model for routing, classification, formatting
  • OTel tracing on every agent call — workflow ID, task ID, agent role, status, token count all in every span
  • Workflow completion rate alert configured — not error rate; completion rate is the signal that matters
  • Retry anomaly alert configured — total retry count significantly exceeding task count triggers immediately
  • Cost-per-workflow-run tracked — token counts aggregated per workflow ID in production, not estimated from test runs
  • Failure modes tested in staging — infinite loops triggered deliberately, context limits hit intentionally, cascade scenarios validated before production

Closing: Coordination Is the New Scale Frontier

The microservices transition of the mid-2010s produced a decade of hard-won lessons about distributed systems: that network calls fail in ways local function calls don’t, that observability has to be designed in rather than added later, that service boundaries are contracts that need versioning, and that the failure modes of composition are qualitatively different from the failure modes of individual components. None of that knowledge transferred automatically — teams that built their first microservices architectures without those lessons learned them expensively.

Multi-agent systems are going through the same transition now, at compressed speed. The coordination patterns aren’t theoretical — they emerge from the same distributed systems constraints that shaped microservice architectures a decade ago: coordination cost, fault isolation, throughput requirements, and observability. The teams that are making multi-agent systems work in production in 2026 are the ones that arrived with that distributed systems intuition already in hand — that treated agent-to-agent handoffs as network calls with failure modes, that designed state persistence as a first-class requirement rather than an afterthought, and that built observability infrastructure before the first incident rather than in response to one.

The $47K infinite-retry incident is not a story about a broken agent. It’s a story about a system deployed without the infrastructure layer that would have caught the failure mode before the cloud bill caught it instead. The agents were working. The system wasn’t. And the gap between those two things is exactly what the infrastructure in this post exists to close.

📌 Key Takeaway

Multi-agent architecture earns its complexity only when the work genuinely decomposes — parallel subtasks, distinct tool access requirements, or failure isolation needs that a single agent can’t provide. When it does, the engineering work is the infrastructure underneath the agents: typed inter-agent contracts (Pydantic validation at every handoff, not downstream), durable workflow state (resume from checkpoint, not from scratch), circuit breakers per downstream dependency, saga compensation for side-effectful steps, hard retry ceilings in code, and model-tier routing that reserves frontier models for reasoning-heavy steps. The observability layer is not optional — standard metrics detect whether agents are running, not whether they are making progress. Workflow completion rate and retry anomaly detection are the signals that multi-agent systems require that no standard monitoring stack provides. Build the infrastructure before you deploy. The alternative is learning about your failure modes from your cloud bill.

Further Reading: Anthropic — Building Effective Agents (2024), Google — Agent-to-Agent Protocol Specification (A2A v1.0, 2026), LangGraph — Stateful Multi-Agent Workflows Documentation, Liu et al. — Why Do Multi-Agent LLM Systems Fail? (arXiv, 2024), Braintrust — Production AI Agent Failure Modes Research (2026)