AI Agents & Orchestration

37 MIN READ

Multi-Agent Systems for the Enterprise: Architecture and Coordination

Multi-Agent Systems fail when teams pick topology before the coordination substrate. Shared state vs. message passing decides what survives a restart.

Add a second agent to a working AI system and reliability drops before capability rises. Most teams choose a topology first, hierarchical, swarm, pipeline, and treat the coordination substrate underneath it as plumbing. Multi-Agent Systems built in that order stall the first time an agent restarts mid-task, because nothing recorded what already happened.

Where this article sits

Journey stage 7 of 7: Scale

readiness → use-cases → roi → pilots → kpis → operationalize → scale

this articlelinkedjourney stagepillar

Your trail so far

The articles you visit light up on this map.

What a Multi-Agent System Is, and Why the Coordination Substrate Is the First Decision

A multi-agent system is several autonomous agents sharing an environment under a coordination mechanism that decides how one agent’s work reaches another; that mechanism defines the boundary against a scripted pipeline, whatever the agent count. Most competing explanations open with topology, hierarchical, swarm, pipeline, and treat the substrate underneath as an implementation detail. That ordering costs more than it looks: every topology in current use is expressible on either substrate, which means the topology decision is downstream of a choice most teams never make on purpose. Before transformation, at the point a Multi-Agent System is first specified, getting that order right is cheap. After the first restart loses state nobody wrote down, it is not.

The Coordination Substrate as the Prior Decision

The coordination substrate is the design decision that precedes topology: whether agents read and write one shared state, or pass messages directly to each other and hold nothing in common.

As models from different providers converge toward comparable benchmark performance, orchestration topology, how agents are coordinated, parallelized and synthesized, now dominates system-level performance over individual model capability, a formal framework for task-adaptive multi-agent orchestration argues, arguing that the coordination choice now outranks the model choice (AdaptOrch). A controlled evaluation of 180 agent configurations found the same asymmetry from the other direction: multi-agent coordination dramatically improves performance on parallelizable work and degrades it on sequential work, and a predictive model built from that evaluation identifies the optimal architecture for 87% of unseen tasks (Google Research). Both findings assume a substrate has already been chosen, orchestration topology routes work on top of shared state or on top of message passing, it does not replace that earlier decision, which is why the substrate has to be settled before the topology gets a name. A 2026 survey of orchestration frameworks maps that territory with a three-topology, one-adaptivity taxonomy and an evidence hierarchy for reading vendor deployment claims, ranking a reproducible benchmark configuration above a demo number with no methodology attached (LLM-Based Multi-Agent Orchestration survey).

Skip that ordering and the topology gets picked for its story rather than its substrate cost. That cost surfaces later, in the incident channel, when an agent restarts and nobody can tell what the fleet already knew.

Shared State and Message Passing: The Two Substrates

Shared state and message passing are the two substrates every Multi-Agent System is built on, and the choice between them decides what an agent can know about the rest of the fleet without asking.

Under shared state, every agent reads and writes one structure, a table, a log, a blackboard, and learns what another agent did by reading, not by being told. Under message passing, agents address each other directly: one agent’s output becomes another agent’s input only if a message carries it there, and nothing exists outside those messages. OpenAI’s Agents SDK documents the pattern practitioners reach for first under message passing at scale: a root agent spins up and coordinates subagents in parallel, then synthesizes their work into a final response, which is effective when a task divides into independent workstreams like codebase exploration or comparing several proposals but weaker when the work depends on a single ordered chain of reasoning or needs frequent writes to shared mutable state Agents SDK (OpenAI). Anthropic’s own research system shows why the same shape gets chosen even for exploratory work: a lead agent plans a research process and creates parallel subagents that search simultaneously, each operating in its own context window and compressing what it finds before condensing it back for the lead agent, with each subagent providing separation of concerns through distinct tools and a distinct piece of the problem (Anthropic). Planning, memory and tool use are the three components a foundational account of LLM-powered autonomous agents identifies as the ones that turn a language model into an agent capable of this kind of delegation in the first place; decomposition into subgoals, short- and long-term memory, and calls out to tools and APIs for what the model cannot know from its weights alone (Lilian Weng).

Neither substrate is free. Shared state buys visibility and charges concurrency discipline; message passing buys isolation and charges an addressing problem; every agent has to know who to talk to, and that knowledge has to stay current as the fleet changes shape.

What Counts as an Agent, and What Doesn’t

An entity counts as an agent in a multi-agent system when it acts autonomously on the shared environment or the message stream, not when it merely routes information between components that share one decision-making process.

The distinction matters because agent-shaped architecture shows up inside systems that are not multi-agent systems at all. A speech-language-behavior model built for embodied conversational agents routes audio, motion and text through modality-partitioned transformer experts, separate experts for speech, facial expression and body motion, sharing information through cross-expert attention so the system can plan language and movement together (ViBES). Each expert specializes the way a knowledge source in a blackboard system specializes, but the experts do not act independently on a shared world; they are components of one agent’s decision process, coordinated by training rather than by a coordination substrate the experts write to at runtime. A genuine multi-agent deployment looks different in outcome as much as in structure: a multi-agent research partner built to accelerate scientific discovery helped a Stanford researcher uncover drug-repurposing candidates for liver fibrosis, surfacing a candidate that blocked 91% of a scarring-linked response in lab testing, work that later published in a peer-reviewed journal (DeepMind). That result came from agents each pursuing a distinct line of investigation against a shared, growing body of findings: autonomy on a shared substrate, not modality experts inside one model.

Drawing the line this way keeps the rest of this guide honest: a Multi-Agent System is defined by autonomous agents and a coordination mechanism, not by the presence of multiple neural components under one control loop.

Coordination Is a Record, Not a Conversation

Coordination in a multi-agent system has to be a record any participant can read without asking anyone, because a conversation’s state lives in whoever last spoke and evaporates the moment that participant restarts or goes quiet. In early 2026, agent runtimes could run several agents at once but could not let them share state, hand work to each other, or know what another had already done. The obvious fix was to build the missing channel: a shared channel, inboxes, handover files, a facilitator, named agents with personas and seats; conversation, formalized into infrastructure.

The Fleet That Coordinated by Conversation, and the Weekend It Stalled

A fleet coordinated by conversation stalls the moment its handover mechanism breaks, because every agent’s understanding of the fleet’s state lives only in messages it has personally seen.

That is what happened on a March 2026 weekend, documented in this site’s own case study on how the site is built, in the section on a fleet coordinated by conversation: a small bug in the handover mechanism stalled five agents at once, each one certain another agent was blocked on it, none of them able to see the others’ messages, with a human in the middle relaying status between them by hand. The record from that weekend states the failure plainly, agent communication is not working, and all the load is on the human, and the month it happened in logged eight named halts, each one the fleet stopping entirely so the handover mechanism could be repaired before work could resume. Nothing exotic broke. The agents behaved exactly as their environment rewarded: check the inbox, wait for a reply, escalate to the human when the reply doesn’t come. The failure was the shape of the system, not the competence of anything running inside it.

Inboxes, Handover Files and a Human in the Middle

A conversation-coordinated fleet is built from a shared channel for cross-cutting announcements, per-agent inboxes for direct handoffs, handover files that carry a task’s state from one agent’s session to the next, a facilitator agent that routes work when no other agent claims it, and named agents holding personas and fixed seats in the roster.

Every one of those five pieces exists to compensate for the same missing property: no agent can read what another agent did without that other agent, or a human, telling it. The facilitator’s job in particular reveals the shape of the problem: it exists because messages alone cannot guarantee that unclaimed work gets picked up, so a dedicated role has to watch the channel and assign it. When the handover file format changed slightly and one agent’s parser silently failed to read it, none of the other four agents had any way to detect that failure except by noticing the silence and assuming the worst.

Five Agents, One Handover Bug, Eight Halts in a Month

A single handover-file bug stalled five agents simultaneously because each agent’s model of the fleet’s progress depended entirely on messages it happened to receive, with no independent way to check what was actually true.

The month’s tally, eight named halts, was not eight unrelated incidents. It was the same structural gap recurring under different triggers: a message delayed, a channel missed, a handover file malformed, an inbox that filled faster than an agent could drain it. Each halt consumed a human’s attention to diagnose and relay, because the conversation itself carried no independently readable state. The fix that eventually held was not a better message format. It was retiring the conversation as the coordination layer entirely.

A Record Any Participant Can Read Without Asking Anyone

The ruling that closed that incident is the thesis of this guide: coordination cannot be a conversation, it has to be a record that any participant can read without asking anyone.

The distinction is operational, not philosophical. A conversation’s state lives in whoever last spoke, and it dies with that participant’s session; ask a fifth agent what happened last Tuesday and the honest answer depends on whether anyone forwarded the relevant message. A record’s state lives outside every participant. It is readable on demand, by a participant who was never in the conversation, without a request routed through anyone who was. That single property, readability without a request, is what a shared channel, however well designed, cannot offer, because a channel’s history is still a sequence of things said to specific listeners at specific times, not a queryable account of what is true now.

State Lives in the Pool, Not in a Dispatcher’s Memory

A multi-agent system survives an individual agent restarting only when the state that agent needs lives in a shared pool, not in the memory of whichever process dispatched it.

A companion page on enterprise AI agent workflow patterns states the same rule from the deployment side: state belongs in the pool, not in a dispatcher’s memory, because a dispatcher that holds state in its own process becomes a single point of failure the moment it restarts, redeploys, or simply falls behind on load. The two arguments are the same argument seen from opposite ends of a Multi-Agent System: one from a fleet that learned the lesson by stalling, one from the architecture pattern that generalizes it. An agent that restarts against a pool-backed record picks up exactly where it left off, because the record was never inside the agent to lose. An agent that restarts against a dispatcher’s private memory picks up nothing, because the state that mattered restarted with it.

Blackboard Architecture: From Hearsay-II to LLM Agents Writing to Shared State

Blackboard architecture coordinates a multi-agent system through three parts: independent knowledge sources with distinct specialisms, one shared mutable data structure they all read and write, and a control component that decides which knowledge source acts next (EmergentMind). The pattern is not new dressed up for language models. It is a forty-year-old answer to the exact coordination problem large language model agents now run into, which is why its 2025 revival counts as evidence rather than novelty.

Knowledge Sources, a Shared Structure and a Control Component

A blackboard system decomposes a problem into specialist knowledge sources, each of which reads from and writes to one shared mutable structure, while a control component decides which knowledge source is allowed to act at any given moment.

The specialism is the point: a knowledge source contributes exactly the piece of the problem it is competent at, posting a partial hypothesis or a piece of evidence to the shared structure rather than attempting the whole task. The control component then evaluates what is on the blackboard and selects the next knowledge source to run, opportunistically, based on what would most advance the current state rather than a fixed sequence. Practitioners reaching for a centralized version of this pattern today often describe it in different words: the Agents SDK documentation for a multi-agent portfolio-research workflow shows a head agent orchestrating specialist agents (Macro, Fundamental, Quantitative) as tools it calls and synthesizes, which is a control component and a set of knowledge sources by another name Agents SDK (OpenAI Cookbook). The formal decomposition, knowledge sources, shared structure, control, gives that pattern a name and a forty-year track record instead of a fresh one invented per project (Blackboard/Event Bus formalization).

The Forty-Year Lineage the 2025 Papers Cite

The blackboard pattern predates large language models by four decades, and the 2025 papers reviving it for agent coordination cite that history explicitly rather than presenting the idea as new.

The lineage runs back to Hearsay-II, a speech-understanding system built around exactly this three-part structure, and forward through subsequent multi-agent blackboard work that generalized the pattern beyond speech recognition to open problem-solving. What changed in 2025 is not the pattern but the agents writing to it: where the original blackboard coordinated hand-built expert modules, the current generation coordinates large language model agents that read and write natural-language hypotheses to the same kind of shared structure.

Erman et al. (1980) and Hearsay-II

Hearsay-II, documented by Erman and colleagues in 1980, established the blackboard pattern as a solution to speech understanding: independent knowledge sources for phonetics, syntax and semantics each contributed partial interpretations to a shared hypothesis structure, and a scheduler decided which knowledge source’s contribution to pursue next based on which hypothesis looked most promising.

That design solved a problem structurally identical to the one large language model agent fleets face today: several specialists, none of which alone can solve the whole problem, needing a way to combine partial progress without every specialist knowing about every other specialist directly. The scheduler in Hearsay-II is the direct ancestor of the control component in a modern LLM blackboard system: the mechanism that decides what runs next without requiring the knowledge sources to negotiate among themselves.

The 2025 LLM Blackboard: Posting Requests Instead of Addressing Agents

A 2025 system for information discovery in data science puts a central agent’s requests on a shared blackboard where autonomous subordinate agents post their answers, so that no agent ever needs to hold a roster of every other agent in the fleet (LLM-based Multi-Agent Blackboard System).

That property, no roster requirement, is what kills the addressing problem that message passing carries by default. A subordinate agent does not need to know which of a dozen peers to send its answer to; it writes to the blackboard, and the central agent reads what accumulates there. A separate 2025 paper pushes the pattern further, describing a variant where every agent sees all information and every other agent’s contribution across the whole problem-solving process, trading some of the original pattern’s specialization for full mutual visibility (Exploring Advanced LLM Multi-Agent Systems). Both variants sit on the same three-part structure Hearsay-II established; they differ in how much of the blackboard each knowledge source is allowed to see.

Blackboards in Production: SE-Blackboard and agent-blackboard

Blackboard coordination has moved from research paper to production software-engineering pipeline, with named implementations running specialist agents over a shared-state architecture rather than a message bus.

SE-Blackboard, a 2026 shared-state architecture built specifically for multi-agent software engineering pipelines, applies the pattern to the problem of coordinating agents across a codebase, where knowledge sources correspond to concerns like design, testing and documentation rather than phonetics and syntax. Claudio Eduardo de Oliveira’s agent-blackboard runs nine specialist agents over the same pattern with tool integration, documentation, API design, backend architecture, and observability among the specialisms, demonstrating that the three-part structure scales past the handful of knowledge sources Hearsay-II coordinated to something close to a small engineering team’s division of labor Claudio Eduardo (agent-blackboard).

The Price of a Shared Mutable Structure: Ownership, Concurrency, Versioning

A shared mutable blackboard buys opportunistic, specialist-driven problem solving, and it charges ownership, concurrency and versioning discipline as the price of that flexibility.

one question · 10 seconds

Quick one while it is in front of you: where does your multi-agent system actually stall in practice?

Every knowledge source that can write to the same structure can also overwrite another knowledge source’s contribution, which means the structure needs an ownership model; which knowledge source may write which fields, and under what conditions. Concurrent writes from agents running in parallel need a concurrency policy, or two knowledge sources acting on stale reads of the blackboard will each write a hypothesis the other one has already invalidated. And because the blackboard’s content changes shape as the problem-solving process advances, the structure itself typically needs versioning, so a knowledge source reading it mid-run knows whether what it sees reflects the current round of contributions or a stale one. None of this is exotic engineering, but skipping it is what turns the blackboard’s central strength, anyone can write anything relevant, into its central risk.

Event Logs, Tickets and Git History as the Coordination Substrate

An append-only log with a total order is a coordination primitive, not storage, because a reader can reconstruct who did what and in what sequence without asking any participant directly. That property is what most enterprises already own and have classified as a filing cabinet: a ticket system, a git remote, an event stream: each one an ordered record most teams have never asked to coordinate agents, only to archive what humans already decided.

The Metalog: Why a Total Order Makes a Log a Coordination Primitive

An append-only log becomes a coordination primitive the moment it guarantees a total order over its records, because that ordering guarantee is what lets any reader reconstruct exactly what happened and in what sequence, independent of who wrote which record.

Boki, presented at the Symposium on Operating Systems Principles in 2021, exports a shared log as an API to stateless serverless functions and introduces the metalog, a mechanism that orders shared-log records so that ordering, consistency and fault tolerance can each be addressed independently rather than bundled into one mechanism that has to get all three right at once Operating Systems Principles (Boki). That separation is the whole argument for treating a log as a coordination substrate: once ordering is guaranteed and decoupled from the storage and fault-tolerance concerns, any function, or any agent, reading the log can determine causality between records without coordinating directly with whoever wrote them. A shared log with this property does for a Multi-Agent System what a blackboard’s control component does through active scheduling: it lets the fleet answer “what happened, and in what order” from the record alone.

The Immutable Log in Event-Driven Agent Patterns

An immutable log sits at the center of event-driven multi-agent system design, where every event or command an agent processes gets written to a permanent, unchangeable record before or as it is acted on.

Four recurring design patterns for event-driven multi-agent systems put this immutable log at their structural core: the log is not a byproduct of the agents’ work, it is the coordination mechanism the agents route their work through. An agent does not tell another agent what it did; it writes an event, and any agent with a reason to react reads that event from the log. This is the append-only discipline Boki formalizes with its metalog, applied at the level of agent-to-agent coordination rather than serverless function state.

Artifact Lineage as a Directed Acyclic Graph

Independent agents can coordinate discovery work with no central authority at all when an artifact layer preserves the full computational lineage of their outputs as a directed acyclic graph, letting any agent trace which prior artifacts a given result depends on.

A 2026 paper on autonomous agents coordinating distributed discovery through emergent artifact exchange describes exactly this design: agents operate independently, exchanging artifacts rather than messages, and the lineage graph that accumulates from those exchanges is itself the coordination record (Autonomous Agents Coordinating Distributed Discovery). A directed acyclic graph is a stronger structure than a simple total order, because it captures which artifacts depend on which others rather than flattening everything into a single sequence; two agents working on unrelated branches of a problem do not need to agree on a relative order for work that never touched each other, only on the dependency edges that actually exist.

Tickets and Commit History: The Substrate You Already Run

A ticket tracker and a git history are the same coordination substrate as an append-only event log, wearing a user interface most engineering organizations already know how to operate.

An issue closed by a merge, with the merge bound to an exact commit, is a total-ordered record of who did what and when: the same property Boki’s metalog formalizes for a shared log, delivered through tooling that predates the multi-agent framing entirely. This site’s own cutover made that substitution directly: issues and milestones became the status record, every merge was bound to an exact commit, and the messaging layer that previously carried status between contributors was retired rather than improved, because the ticket-and-commit substrate already had everything a conversation channel was trying to approximate. Agents claiming and dividing units of work through commit and ticket activity ahead of a pull request follow the identical pattern at agent scale: the claim is a written record before the work starts, not a message announced to whoever happens to be listening (git-and-ticket coordination).

What a log, a ticket tracker and a git history all offer that a conversation cannot is replay. That is event sourcing in practice, whether or not a team names it that: the log itself is the source of truth, and state is derived by re-reading it rather than stored separately and trusted to stay in sync. A log can be re-read to reconstruct state; what was true, and when it became true. A transcript can only be re-read to reconstruct opinion; what someone believed, or claimed, at the time they wrote it. A record-first Multi-Agent System depends on the first property and treats the second as, at best, a debugging aid.

Stigmergy: Coordination by Trace Instead of Message

Stigmergy coordinates agents through traces left in a shared environment rather than through messages addressed to anyone: an agent changes some piece of shared state, and other agents respond to that change directly, without the changing agent ever naming a recipient. Stigmergy is the limit case of the record argument running through this guide: not merely coordination without a live conversation, but coordination without an addressee at all, which is what makes both its evidence and its failure modes worth taking seriously.

Trace-Mediated Coordination Without an Addressee

Trace-mediated coordination happens when an agent’s action leaves a mark on the shared environment that other agents read and respond to, with no message ever sent between the acting agent and the responding ones.

The absence of an addressee is the defining feature, not an incidental one. A blackboard’s knowledge sources still write to a structure that a control component actively schedules around; a stigmergic system has no equivalent scheduler: agents simply notice the traces relevant to them and act. That property scales differently than either message passing or blackboard control: it removes the combinatorial cost of every agent needing to know about every other agent, at the price of removing any central point that can guarantee the traces get read at all. Conventions research on multi-agent collaboration frames this as a distinction between task-specific skills, useful regardless of partner, and partner-specific conventions that emerge only through repeated interaction with the same collaborators: a stigmergic system relies almost entirely on the first kind, because there is no stable partner to build a convention with (Stanford AI Lab).

Virtual Pheromones in Deep Reinforcement Learning Agents

Virtual pheromones let decentralized reinforcement learning agents coordinate through environmental traces instead of explicit communication, mirroring the chemical trail-following behavior stigmergy takes its name from.

A stigmergic multi-agent deep reinforcement learning framework coordinates decentralized agents through exactly this mechanism, framing virtual pheromones and environmental traces as an explicit contrast to explicit inter-agent communication rather than a supplement to it (S-MADRL). Cooperative behavior of this kind is not limited to pheromone-style trace-following: reinforcement learning agents trained to play Quake III Arena Capture the Flag reached human-level performance and cooperated successfully with both artificial and human teammates, using no explicit communication protocol; team play emerged from each agent responding to the state of the shared game environment Quake III Arena Capture (DeepMind). That emergent cooperation is stigmergy’s strongest evidence and its sharpest warning in the same result: the coordination worked, and no one specified it in advance.

Gossip and Auctions on a Shared-State Substrate

Gossip protocols propagate state between agents and auction mechanisms allocate tasks among them, and both sit on top of the same shared-state substrate rather than replacing it with something new.

A gossip protocol spreads a piece of state, a status update, a completed-work marker, from agent to agent through repeated pairwise exchange, so that eventually every agent has seen it without any single broadcast reaching all of them at once. An auction mechanism allocates a task by having agents bid on it based on their own local state, with the highest or best-suited bid winning, which is task allocation without a central assigner deciding who does what. Both patterns are decentralized in the same sense stigmergy is: no agent needs a complete picture of the fleet, only enough local information to gossip or bid correctly. An application-layer formalization of indirect coordination bridges this swarm-intelligence lineage to on-chain multi-agent coordination, describing shared environmental state as the substrate gossip and auction patterns both build on without any explicit messaging layer underneath them.

Where Emergent Coordination Stops Being Engineering

Stigmergy stops being engineering and starts being hope exactly where the traces agents depend on go stale or inconsistent, producing emergent behavior nobody specified and allocation pathologies, oscillation and starvation, that a message-passing system with explicit acknowledgments would not produce in the same shape.

Emergent behavior cuts both ways: the Capture the Flag agents’ cooperation was emergent and desirable, but emergence by definition means nobody chose the specific behavior that appeared, and an enterprise deployment rarely has the tolerance for an undesirable emergent pattern that a research benchmark does. The pathologies are concrete. Oscillation happens when agents keep responding to each other’s traces in a loop, each one’s response invalidating the trace the other just acted on, so the system never settles. Starvation happens when a trace goes stale, an agent reads state that has moved on without updating the trace that describes it, and work that should have been claimed sits untouched because no live signal points to it anymore. Many agents and loosely-coupled work suit stigmergy well; any task that needs a named, accountable owner does not, because stigmergy by design has no mechanism for naming one.

Why Multi-Agent Systems Fail: The Failure Modes Each Substrate Owns

Multi-agent systems fail in two structurally different ways depending on which coordination substrate they run on, and diagnosing an incident starts from that split rather than from a generic claim that agents are unreliable. The question worth asking is never why multi-agent systems fail in the abstract. It is which substrate a given fleet was built on, because that earlier decision determined the specific failures the fleet was always going to get.

Two Annotated Failure Corpora: MAST and Who&When

Two empirical corpora anchor what is actually known about multi-agent system failure, one built from annotated execution traces and one built from annotated attribution judgments across more than a hundred deployed systems.

MAST, introduced in a 2025 paper asking why multi-agent large language model systems fail, is a taxonomy built from more than 1,600 annotated traces collected across seven popular multi-agent frameworks, developed by M. Cemri, Melissa Z. Pan and coauthors (Semantic Scholar). Who&When, from a companion 2025 paper on automated failure attribution, annotates failure logs from 127 multi-agent systems to link each failure to the specific agent and step responsible, work led by Shaokun Zhang and coauthors Shaokun Zhang (Semantic Scholar). The best automated method Who&When’s authors developed reaches 53.5% accuracy identifying which agent caused a failure, and only 14.2% accuracy pinpointing the specific step: a gap that says attribution is harder than detection, and that a record built for after-the-fact analysis has to capture more than a pass or fail outcome.

1,600 Traces Across Seven Frameworks

MAST’s taxonomy covers 1,600-plus annotated traces spanning seven distinct multi-agent frameworks, giving it breadth across implementation choices rather than a single framework’s idiosyncrasies.

That breadth is what makes the taxonomy generalizable rather than a critique of one vendor’s design. A failure mode that recurs across seven independently built frameworks is a property of multi-agent coordination itself, not a bug in any one team’s implementation; which is exactly the argument for splitting failure modes by coordination substrate instead of by framework brand.

127 Systems Annotated for Failure Attribution

Who&When annotates failure logs from 127 separately deployed multi-agent systems, with each failure traced to a specific responsible agent and, where possible, the specific step that caused it.

The low accuracy on step-level attribution, 14.2% even from the strongest method the authors tested, is itself a finding about what coordination records typically fail to capture: which specific step in a longer sequence was the one that mattered. A record built with attribution in mind, carrying an identified author and a trace identifier on every write, closes exactly this gap; a record built only to log outcomes does not.

Failure Modes Message Passing Owns: Loss, Backpressure, Queue Bottlenecks

Message-passing coordination fails through lost or delayed messages, backpressure when an agent cannot keep up with its inbox, and emergent queue bottlenecks where one slow agent stalls every agent downstream of it in a chat-style chain.

These failures share a common shape: they are all failures of delivery, not failures of the underlying task. A message that never arrives, or arrives late enough that the receiving agent has moved on, produces exactly the kind of stall the fleet coordinated by conversation experienced on its worst weekend: a handover file silently unreadable is a delivery failure by another name. Chat-style chains that pass work from agent A to agent B to agent C compound this fragility, because each hop is another point where delivery can fail, and debugging a stalled chain at scale means reconstructing which hop broke from a transcript rather than a record.

Failure Modes Shared State Owns: Conflicts, Race Conditions, Overwrites

Shared-state coordination fails through consistency conflicts, race conditions and overwrite bugs, and each one lands on an incident ticket as its own reproducible symptom: two writes landing in the same window with only one surviving, a field that reverted to an earlier value with no error thrown, an agent’s output silently vanishing between one read and the next. An on-call engineer working that ticket starts from the timestamps and the surviving value, not from a theory about which agent was at fault.

A race condition shows up when two agents read the same piece of shared state, each acts on what it read, and one agent’s write silently overwrites the other’s without either agent knowing a conflict occurred. An overwrite bug is the same failure with cleaner symptoms: agent B’s write simply erases agent A’s contribution because nothing in the shared structure enforced ownership. Neither failure set is smaller than the other, and neither substrate is strictly safer; message passing trades these conflicts for delivery risk, shared state trades delivery risk for these conflicts, and treating one substrate as failure-free is what produces the wrong diagnosis on the next incident.

Attacking the Message Layer: Agent-in-the-Middle

A message-passing coordination layer carries a security vulnerability a shared, ordered record does not present in the same shape: an attacker positioned between agents can intercept and manipulate the messages that carry coordination itself.

A 2025 paper on red-teaming large language model multi-agent systems names this attack Agent-in-the-Middle, describing how it exploits the fundamental communication mechanisms multi-agent systems depend on by intercepting and manipulating message-based coordination between agents (Red-Teaming LLM Multi-Agent Systems). A shared, append-only record does not eliminate every security concern, an agent with write access can still write something false, but it removes the specific vulnerability class Agent-in-the-Middle exploits, because there is no in-transit message to intercept between two agents that never address each other directly. Centralized coordination carries a related but distinct cost: a paper on decentralized evolutionary coordination for large language model multi-agent systems documents centralized designs producing scalability bottlenecks, reduced adaptability, and a single point of failure (AgentNet), which is worth naming plainly so a reader does not leave this section believing the fix for coordination failure is simply routing everything through one central orchestrator.

How to Build a Record-First Multi-Agent System

Building a record-first multi-agent system starts by choosing the coordination artifact, a blackboard table, an append-only log, or an issue tracker, before choosing a topology, and writing down, in one sentence, what any participant may read from that artifact without asking anyone.

Substrate	What it optimizes	Read pattern	Failure mode it owns
Blackboard	Opportunistic specialist contribution	Any knowledge source reads current state	Overwrite, race condition, versioning drift
Append-only log	Total order and replay	Read the sequence, reconstruct state	Backpressure if consumers fall behind the log
Issue tracker / git history	Human-legible status with existing tooling	Read the ticket or commit graph	Same as append-only log, at slower cadence
Stigmergic trace	Coordination at high agent counts	Read the environment, infer from change	Oscillation, starvation on stale traces
Message passing	Isolation, independent context per agent	Read only what was addressed to you	Loss, delay, Agent-in-the-Middle interception

Choose the Coordination Artifact Before the Topology

The first step in building a record-first multi-agent system is naming the coordination artifact the fleet will write to, before any agent is assigned a role or a topology is drawn on a whiteboard.

Three real options cover most deployments: a blackboard table for opportunistic, specialist-driven work; an append-only log for work where total order and replay matter most; an issue tracker bound to a git history for teams that already run one and would rather extend it than stand up new infrastructure. Whichever artifact a team picks, the requirement is the same: write down what any participant, including an agent added to the fleet next quarter, may read from it without asking a currently-running agent or a human for permission first. Skipping this step is what produces a fleet that discovers its coordination artifact by accident, usually a shared document or a chat channel, months into a build.

One Owner Per Fact, Validated Once at Load

Once agents begin writing to a shared coordination artifact, the artifact stays authoritative only under one rule: a fact lives in one place and is validated once, when it is loaded.

That rule is not a suggestion for this build; it is the same standing instruction this site runs its own generation pipeline under, applied to agent coordination records instead of content records. A check on that fact may detect that the source has drifted from what a downstream consumer expected. It may never derive a replacement value, repair the drift silently, or stand in for the source it was supposed to be watching. The moment a check starts doing any of those three things, the record has two owners for the same fact, the original source and the check that started compensating for it, and two owners is the condition that produces exactly the conflicts and overwrites the shared-state failure modes above describe.

Why a Check May Detect Deviation but Never Repair It

A check exists to tell a human or an agent that a fact no longer matches its source, and stops being a check the moment it starts fixing the mismatch itself instead of reporting it.

A check that repairs silently hides the very drift it was built to surface, which means the next failure caused by that drift arrives with no warning, because the mechanism that would have warned about it quietly absorbed the problem instead. Keeping detection and repair as separate responsibilities, one component reports, a human or an explicitly authorized process decides what to do about it, is what keeps the record trustworthy enough for an agent three hops downstream to act on without re-verifying it itself.

Why a Guard Is Not a Control

A guard is not a control because a guard accrete, one reasonable-seeming addition at a time, until the fact it was meant to protect is being read by nothing that still checks it.

A change that adds a guard, a fallback, or a refusal to a coordination record is rejected in this build discipline unless it was explicitly ruled in beforehand, because each individual guard looks justified in isolation while the accumulated set of guards is what eventually makes the record’s actual invariant unreadable underneath its own defenses. The corollary surprises teams used to defensive engineering by default: fewer guards, applied only where a specific ruling calls for one, produce a more trustworthy record than a record wrapped in every plausible safety check anyone thought to add.

The Harness as the External Control Layer

An agent harness is the external control layer that turns a base model into an executable agent, managing the context it sees, the tools it can call, how its work is orchestrated, what it remembers, and how its output gets handled.

A 2026 paper on harnesses that learn from experience defines the term this precisely, the external control layer managing context, tools, orchestration, memory and output handling, and argues that most deployed agents reuse a single global harness across every case rather than adapting it to what a given task actually needs (MemoHarness). Writing on harness engineering makes the stakes of that layer concrete: the harness, not the base model’s raw intelligence, is often what determines whether a deployment succeeds, because the harness is what decides how the model perceives its environment, stores what it produces, and evaluates whether a result is good enough to hand off (Lilian Weng). A record-first multi-agent system’s coordination artifact lives inside this harness layer: it is one of the things the harness manages, alongside tools and memory, not a separate system bolted on afterward.

Instrumenting the Write Path: Author, Trace Identifier, Timestamp

Every write an agent makes to the coordination record should carry a trace identifier and an identified author, so the record answers who and when in addition to what happened.

Without an author on every write, a record still reconstructs a sequence of events but cannot say which agent caused a given one: the reader can see that a field changed and when, but has to fall back on re-running each agent’s context in turn to guess which one touched it last. A trace identifier lets a single task’s writes be correlated across every agent that touched it, turning a flat sequence of events into a reconstructable story for one unit of work. Production tooling for this kind of instrumentation already exists: Vizient’s healthcare-data platform struggled with exactly this gap before adopting a graph-based orchestration runtime and its companion tracing tool, where each agent had been built to handle a specific task but coordinating them reliably, and understanding what any of them had actually done, proved difficult until the runtime’s structure and tracing gave the team visibility into the graph itself (LangChain): the pairing of LangGraph for orchestration and LangSmith for tracing is the production shape of author-plus-trace-identifier instrumentation this section describes.

One Human Signature, Not an Approval Per Step

A record-first multi-agent system needs exactly one human signature in its flow, not an approval gate at every step, because an approval per step recreates the human-in-the-middle router this guide’s second section described the fleet escaping from.

One signature, at a defined point, on a defined artifact, keeps the human gate meaningful: a person is attesting to something specific, and that attestation means something because it is not diluted across a dozen smaller approvals nobody has time to actually scrutinize. An approval required at every step, by contrast, becomes a rubber stamp within a few cycles, because no human sustains genuine judgment across dozens of near-identical gates in a single working session, and the fleet slows down without the review quality improving to match.

Reading the Record: Observability and Failure Attribution in Agent Fleets

Attribution is a property of the record, not of the transcript, and that distinction is where this guide’s opening argument gets measured from the debugging side rather than the design side.

Formative interviews with agent developers, published in a 2025 study at the International Conference on Human Factors in Computing Systems, identified the core obstacle developers face as the sheer difficulty of reviewing long agent conversations to localize where an error occurred Computing Systems (Interactive Debugging and Steering of Multi-Agent AI Systems). That finding is the fleet-coordinated-by-conversation failure restated as a debugging problem: the transcript is precisely the artifact developers report being unable to read back effectively.

What Developers Actually Struggle With: Localising Errors in Long Agent Conversations

Developers report that the central obstacle in debugging multi-agent systems is localizing which point in a long agent conversation actually caused a failure, more than any difficulty with the individual agents themselves.

The 2025 formative interviews behind this finding surveyed developers actively building multi-agent tools and asked what got in their way in practice, rather than testing a hypothesis about where failures theoretically originate. Reviewing a long conversation to find the specific exchange that went wrong turned out to be the shared complaint, independent of which framework or which agents the developers were using: a transcript-based debugging workflow does not scale with conversation length, however good the individual agents are.

Runtime-Decided Control Flow and the Limits of Reading Back

Control flow decided at runtime by the model, rather than fixed in code ahead of time, raises the stakes of this debugging gap, because the only reliable account of what happened is whatever the system actually captured while it happened.

A fixed pipeline’s control flow can be inferred after the fact from the code alone, whatever the logs say. An agentic system’s control flow cannot: the model decided, at runtime, which branch to take, which tool to call, which other agent to hand off to, and that decision is unrecoverable unless the record captured it. A multi-agent system that cannot be traced is, by this logic, a multi-agent system that cannot be trusted: not because the agents are unreliable, but because nobody can confirm after the fact whether they were.

Identified Author: What Attribution Adds to the Record

Attribution adds one property to a record that already has total order and immutability: an identified author on every write, so a fault traces to the specific participant that caused it rather than only to a position in the sequence.

Event Logs, Tickets and Git History as the Coordination Substrate already established total order and replay as record properties; attribution is what those properties are missing on their own. What a total-ordered log cannot say, however completely it orders its records, is which participant is accountable for a given one: the write survives, but responsibility for it does not travel with the sequence position alone, so a reader still has to guess who to ask. Multi-agent memory research reinforces why this gap persists by default: prevailing memory mechanisms in multi-agent systems are overly simplistic and disregard the nuanced inter-agent collaboration trajectories that would make attribution straightforward, focusing instead on what any one agent remembers in isolation (G-Memory). An identified author on every write is the cheapest fix to that gap, because it requires no new memory architecture, only a field on an event the system was already writing.

Distributed Tracing Across Agent Pools

Distributed tracing across an agent pool correlates a single task across every agent that touched it, giving an operator one continuous account of a task’s life instead of fragments scattered across each agent’s individual log.

This is the operational reading surface a record-first Multi-Agent System is built toward: not a dashboard ported unchanged from microservice monitoring, but a trace that follows one unit of work through however many agents handled a piece of it, using the trace identifier and author fields the write path already carries. The honest limit is worth stating plainly, because overclaiming here is the genre’s most common failure: a record proves what happened, not that it was the right thing to do. Deciding which facts are worth keeping, and which traces are worth reading closely, stays a human judgment the record supports but never replaces.

Summary

A Multi-Agent System’s reliability is decided before its first agent runs, by whether coordination is built as a readable record or left to accumulate as conversation.

The Substrate Decision Outranks the Topology Decision

Every argument in this guide traces back to one ordering choice: settle the coordination substrate, shared state or message passing, blackboard or log or stigmergic trace, before naming a topology, because the topology is expressible on either substrate and inherits whatever that substrate can and cannot guarantee.

A team that reverses this order inherits its coordination failures by accident rather than by design. A hierarchical topology built on message passing inherits every delivery and security cost that substrate carries along with its clean chain of command; the same hierarchy built on a shared, append-only log gets total order and replay along with the concurrency discipline a shared structure requires. Neither substrate is universally correct: an enterprise choosing between them is choosing which failure modes it is prepared to operate against, not choosing away from failure altogether. What the empirical record from MAST and Who&When makes clear is that the choice is not cosmetic: both corpora split cleanly along this substrate line, which means a diagnosis that starts from the substrate reaches the right fix faster than a diagnosis that starts from “the agents are unreliable.” Organizations evaluating readiness for multi-agent deployment gain more from confirming this decision has been made deliberately than from any topology diagram, because the diagram survives a redesign and the substrate does not.

A Record Outlives the Session That Wrote It

The property that separates a working multi-agent deployment from one waiting for its own version of the March 2026 weekend is durability: a record written to a shared, ordered structure outlives the agent session that wrote it, while a fact that only ever lived in a conversation dies the moment that conversation’s participants stop being reachable.

That durability is what makes attribution, replay and recovery possible at all: an identified author on a total-ordered write lets a fault be traced to its cause months later, the way a conversation transcript never reliably can. It is also what lets an organization treat restart survival as a deployment assumption rather than a recurring incident-response question; something engineering designs against once, the way a database’s durability guarantee lets application code stop worrying about crash recovery, instead of a guarantee every new fleet has to rediscover for itself by living through its own stall first. Building toward that property does not require an exotic platform. Most enterprises already run an append-only substrate in the form of a ticket tracker and a git history; the work is recognizing that substrate as a coordination mechanism rather than a filing system, instrumenting its writes with an author and a trace identifier, and keeping exactly one human signature in the flow so the gate that matters still means something when it matters most.

Anonymous. Counted, not tracked.

Where is your organisation with this right now?

What is the hardest part where you are?

agility at scale