compositional ai and the end of the monolithic agent

A normal model call is a mapping:

x → y

The system receives an input and produces an output. Quality is judged at the boundary: did the answer solve the task?

An agent is different. An agent is a trajectory:

s0 → a0 → s1 → a1 → s2 → a2 → ... → sT

At every state st, the model must choose an action at: generate text, call a tool, retrieve a document, update memory, ask a question, run code, revise a plan, apply a policy, or stop.

at ~ πθ(a | st)

The quality of the system is not determined by one output. It is determined by the accumulated quality of many state transitions. If the model has local error rates ε1, ε2, ..., εT, then a simplified trajectory success estimate is:

P(success) ≈ ∏t=1..T (1 - εt)

This is why agents are fragile. A model that is 95% reliable on one decision is not 95% reliable across a twenty-step trajectory. Under a crude independent estimate:

0.95^20 ≈ 0.36

The real world is more complex. Some errors are recoverable. Others are catastrophic. A bad retrieval query can hide the right evidence. A malformed tool call can fail silently. A mistaken assumption can become the premise for every later step. But the first-principles point remains: long-horizon systems amplify local mistakes.

This is why frontier models became so attractive for agents. A frontier model is not merely a larger store of facts. It is a broad error reducer. It has lower error across more regions of the state space: ambiguous instructions, long context, code, tables, tool outputs, failed actions, contradictory evidence, user corrections, safety constraints, and final synthesis.

A coding agent may need to read a bug report, inspect a codebase, edit files, run tests, interpret stack traces, and explain the fix. A research agent may search, filter, compare, detect contradictions, and synthesize. A customer-support agent may understand tone, retrieve account state, apply policy, call APIs, and draft a response. These are not independent classification tasks. They are stateful decision processes.

So the industry's preference for frontier models is rational. In an agent loop, the central model must perform many roles at once:

understand the goal
maintain state
decompose the task
select tools
format tool calls
interpret observations
recover from mistakes
apply constraints
write the final answer

If the central policy is weak, every part of the trajectory becomes fragile. Frontier models work because they make the whole loop less brittle.

But this solves reliability by concentrating capability. That concentration creates the next problem: cost and control.

The cost of an agent is not the cost of one model call. It is the cost of the whole trajectory:

Cost(agent) = Σt Cost(model call at step t)
            + Σt Cost(tool call at step t)

If the same large generalist is used for every decision, narrow decisions pay the price of broad intelligence. The system may use a frontier model to decide whether a document is relevant, whether a JSON argument is valid, whether a calculation checks out, or whether a tool call violates policy. These decisions matter, but they do not always require the full capability distribution of the frontier model.

This is the opening for compositional AI. The goal is not to reject frontier models. Frontier models show us the target: coherent multi-step behavior across a wide state space. The question is whether all of that coherence must live inside one model.

Vertical scaling says yes:

make πθ larger so one model handles more states

Compositional scaling says something different:

keep the base model efficient enough to run broadly,
then attach local error reducers around the states
where the base model is unreliable

The problem is not simply capability. The problem is capability allocation. A compositional system tries to find a better frontier:

min E[L(trajectory)] + λ Cost(trajectory) + μ Complexity(system)

The base model should own the global trajectory: the user goal, state, context integration, broad planning, and final synthesis. Specialists should reduce error in specific regions of the state space. A retrieval specialist reduces error when the system is selecting evidence. A math verifier reduces error when checking a calculation. A policy specialist reduces error when a tool call needs approval.

The composed policy looks less like one monolith:

πF(a | s)

and more like a base policy with conditional specialist influence:

π(a | s) = Compose(πB(a | s), g1(s)π1, g2(s)π2, ..., gn(s)πn)

where:
  πB    = base model policy
  πi    = specialist i
  gi(s) = activation strength of specialist i at state s, in [0, 1]

The important term is not only πi. We already have many ways to build specialists. The important term is gi(s) — the activation policy that decides when each specialist should influence the trajectory.

The Existing Stack Already Points Toward Composition

Most of the modern AI systems landscape can be understood as different attempts to reduce trajectory loss. Each method solves part of the problem. None fully solves the question of governed specialist influence.

Vertical scaling reduces error globally

The most direct method is to make the model larger, train it on more data, and post-train it more effectively. Vertical scaling tries to reduce error everywhere:

πsmall(a | s) → πlarge(a | s)

This is why frontier models are useful for agents. They are broad error reducers. But vertical scaling is expensive because it uses the whole generalist for every step. It gives coherence, but concentrates capability, cost, and failure modes inside one model.

Full fine-tuning reduces error on a target distribution

Another method is to update the model's weights for a narrower domain. If the base model has parameters θ, full fine-tuning learns a new parameter set:

θ' = θ - η∇θ Ldomain

This can be powerful. The entire model can adapt: attention patterns, representations, domain priors, tool-use behavior, output style, and reasoning habits. But full fine-tuning creates another monolith. Each specialized copy is expensive to train, store, serve, and govern. A change intended to improve one capability can affect other behavior. Continual fine-tuning can produce forgetting, where the model gains a new behavior while degrading old ones.

Full fine-tuning gives us specialization, but not modularity.

LoRA and adapters reduce the cost of specialization

LoRA attacks the cost problem directly. Instead of updating a full weight matrix W, LoRA freezes the base model and learns a low-rank update:

W' = W + ΔW
ΔW = BA

where A and B are small trainable matrices. The result is a cheap, portable specialization layer. Many task-specific behaviors can share the same base model. Training cost falls. Storage cost falls. Deployment becomes more flexible.

But LoRA mainly solves the representation of specialization. It answers how do we encode this specialized behavior cheaply? It does not answer when should this behavior influence an agent trajectory? In many deployments, an adapter is selected per request. That is useful, but agents are stateful. The relevant capability may change from one step to the next. LoRA makes specialists cheap. It does not define the activation policy.

Mixtures of LoRA move from specialization to composition

A natural next step is to combine adapters. Instead of selecting one update ΔW, the system can compose several:

W' = W + α1ΔW1 + α2ΔW2 + ... + αnΔWn

where αi controls the contribution of adapter i. Adapter influence can be mixed, routed, and made conditional. This is much closer to the compositional idea.

But mixtures of LoRA still operate at the parameter-adaptation layer. They are excellent when the problem is which learned parameter delta should be active? Agentic systems ask a broader question: which capability should influence this state transition? That capability might be a LoRA adapter — but it might also be a verifier, a retrieval model, a policy classifier, a reward model, a code execution signal, or a memory gate.

Mixture-of-Experts proves the value of sparse conditional computation

MoE models address the cost problem from inside the model. Instead of activating the same dense parameters for every token, an MoE model routes each token to a subset of expert parameters:

h' = Σi ri(x) Ei(h)

where Ei is an expert and ri(x) is a router weight. MoE is strong evidence for the basic intuition: not all computation needs to be active all the time. A model can have many total parameters while using only a subset per token.

But MoE experts are trained as part of one model. They are not external, versioned, auditable specialists that a developer can attach, remove, test, or constrain independently. The router is learned during training. The semantic meaning of an expert is not usually a clean human-readable capability like "math verifier" or "clinical entity extractor."

MoE proves the value of sparse computation. It does not solve the product problem of attachable specialist influence during execution.

RAG externalizes knowledge but not capability

Retrieval-Augmented Generation changes the state by adding evidence:

π(a | s) → π(a | s, retrieved_context)

This is deeply aligned with compositional thinking. The system stops forcing all knowledge into model weights. Knowledge can be updated outside the model. Sources can be inspected. Provenance becomes possible.

But retrieval mostly addresses knowledge access. It helps the model see better evidence. It does not decide which reasoning specialist should influence a tool call, whether a calculation should be verified, or whether a domain-specific constraint applies. RAG gives the model better context. It does not give the system a general policy over specialists.

Routing and cascades reduce cost by choosing among models

Model routing introduces a more explicit decision rule. Instead of sending every request to the strongest model, the system chooses a cheaper model when possible and escalates when necessary:

r(s) → choose model m

This is already a kind of activation policy. It recognizes that not every state needs the same amount of intelligence. But most routing systems operate at the request level, asking which model should answer this query? Agentic systems require a finer question: which capability should influence this intermediate decision?

A customer-support agent may need a policy specialist only when deciding whether to call a refund API. It may need a retrieval specialist only when filtering account history. It may need a tone specialist only when writing the final response. Routing the whole request to one model does not capture that internal structure.

Tool calls and subagents modularize execution

The most common way to use specialists today is to call them as tools or subagents:

base model observes state st
base model decides to call specialist i
specialist returns output oi
base model updates state using oi

This is powerful. If the subtask is cleanly bounded, direct delegation is often correct. A retrieval ranker can rank documents. A verifier can check a proof. A policy model can approve or reject a tool call.

But specialist-as-tool has a bottleneck: the base model must know when to ask. If the base model fails to recognize that a specialist is needed, the specialist never runs. If it asks the wrong question, the specialist gives the wrong kind of help. Tool calls modularize execution outside the model. They do not provide fine-grained influence inside the base model's decision process.

Verifiers, judges, and critics reduce loss after generation

Another common method is to generate, then judge:

generate candidates → score / verify / critique → select or revise

This is useful for reasoning, code, factuality, and safety. A verifier can catch a bad answer. A judge can rank candidates. A critic can trigger revision. But many agent mistakes should be prevented earlier — before a tool call executes, before memory is written, before a retrieval query is formed, before a plan is committed, before a false assumption propagates through the trajectory.

Verifiers are specialists. But they still need an activation policy. The system has to know when to verify, what to verify, and how much authority the verifier should have.

The Missing Primitive Is Controlled Influence

The pattern is now visible.

Vertical scaling reduces error globally, but at high cost. Fine-tuning creates strong specialists, but they remain monolithic. LoRA makes specialization cheap, but does not solve activation. Mixtures of LoRA compose adapters, but remain parameter-centric. MoE proves sparse conditional computation, but hides experts inside one trained model. RAG externalizes knowledge, but not general capability. Routing chooses among models, but usually too coarsely. Tools and subagents delegate work, but only when the base model asks. Verifiers and critics reduce loss, but often after a candidate has already been produced.

All of these methods are useful. None is wrong. But they point to the same missing layer.

The missing primitive is not specialization. We already have many forms of specialization. The missing primitive is governed specialist influence during execution.

A compositional system should answer:

which specialist should influence this state?
where should it enter the computation?
how strong should its influence be?
what is it allowed to affect?
how do we measure whether it helped?

Instead of one large policy:

πF(a | s)

or a simple routed policy:

πr(s)(a | s)

we define a composed policy:

π(a | s) = Compose(πB(a | s), g1(s)π1, g2(s)π2, ..., gn(s)πn)

The base model owns the global trajectory. The specialists are local error reducers. The activation policy decides when their expected benefit exceeds their cost:

activate i if ΔL_i(s) > λ Cost_i + μ Complexity_i

This gives compositional AI its economic justification. The goal is sparse activation: spend specialist compute only when it is expected to reduce trajectory loss enough to matter.

It also gives compositional AI its control justification. Each specialist should have a defined surface area. A policy specialist should be allowed to gate tools, not rewrite the user's goal. A retrieval specialist should influence evidence selection, not final tone. A memory specialist should decide what is safe to store, not invent new facts. A math verifier should check calculations, not dominate the whole response. The architecture becomes powerful precisely because influence is bounded.

From Delegation to Influence

This is the difference between tool use and compositional AI.

A tool-call architecture looks like this:

base model → call specialist → receive output → continue

That is useful, but coarse. The specialist only participates after the base model explicitly invokes it.

A compositional architecture looks like this:

state → activation policy → specialist influence → next decision

The specialist may return a score, critique, classification, constraint, retrieval ranking, memory decision, or structured signal. In stronger forms, it may influence decoding constraints, logits, hidden states, tool permissions, memory writes, or intermediate representations. The exact mechanism can vary. The architectural principle is what matters: specialist influence should be sparse, state-aware, observable, and controlled.

This reframes the future of agent systems. The old question was: which model should answer this request? The new question is: which capability should influence this state transition?

That distinction matters because an agent is not one request. It is a sequence of decisions. Some decisions require broad reasoning. Some require narrow verification. Some require domain knowledge. Some require policy enforcement. Some require retrieval. Some require memory. Some require deterministic computation. Treating all of these as the same problem is wasteful and fragile.

The frontier model era taught us that coherence matters. Agents need a central policy capable of carrying state across long trajectories. But the next step is to separate coherence from precision.

Coherence belongs to the base model. Precision can be supplied locally. The activation policy decides when precision should intervene.

The future agent stack will not be built only by forcing more capability into one generalist. It will be built by wrapping a capable base model in a governed membrane of specialists: sparse, state-aware, inspectable, and controlled.

Frontier models remain the reference point. They show what high-coherence agentic behavior looks like. But they are not the only way to approximate that behavior. Some reliability can be recovered structurally.

The model owns the trajectory. The specialists reduce local error. The activation policy governs influence.