An Agent System Is More Than the Sum of Its Parts#

Most agent projects start with the same sketch on a whiteboard:

prompt -> model -> tool call -> answer

That sketch is useful. It gets a demo running. It explains why a model can do more when it has a tool. It gives the first version of the system a shape everyone can hold in their head.

Then the second real requirement arrives.

The retrieval step has to feed the next stage. A tool call needs approval. A run hits a step limit. A teammate asks why yesterday's answer changed. Security wants to know which credential was available. Product wants the same agent behind an interactive UI and a batch job. None of that lives in the prompt.

It lives in the space between the parts.

An agent system is not the pile of pieces. It is how those pieces interact during a run.

The parts are not the agent system#

The easiest way to misunderstand agents is to stare too long at the model call.

The model call matters, but it is rarely where production agent systems become hard to operate. The harder failures show up around governance, execution boundaries, cost, risk, and explanation.

Gartner's June 2025 forecast that over 40% of agentic AI projects will be canceled by the end of 2027 cited escalating costs, unclear business value, and inadequate risk controls [1]. Those are not model-quality complaints. They are system complaints.

The sharper version came later. In May 2026, Gartner predicted that by 2027, 40% of enterprises will demote or decommission autonomous AI agents because governance gaps are discovered only after production incidents occur [2].

The timing is the painful part. The problem surfaces after the system is already doing work.

This does not mean teams have no telemetry. They often have plenty around the model-call layer. Datadog's State of AI Engineering found that about 5% of LLM call spans reported an error in February 2026, and roughly 60% of those errors were rate limits [3]. That layer is visible enough to count. You can see request failures, latency, token usage, and provider errors.

The harder case is the one where every component reports success.

The model returned valid text. The tool returned a valid response. The retrieval call returned documents. The workflow reached a terminal state. The answer was still wrong, unsafe, stale, or impossible to explain. Incident writeups around production agents keep circling this shape: the visible error is not a failed API call, but a run that made the wrong sequence of valid moves [4].

Every component succeeded. The system failed.

A real run may carry provider configuration, model selection, prompt assembly, tool schemas, tool execution, procedural guidance, retrieval, memory reads, memory writes, behavior instructions, phases, outcomes, approval gates, limits, cancellation, hooks, MCP surfaces, traces, reports, and the host app's own UI.

Most systems do not need all of that on day one. Many never need every item on the list.

The point is smaller and more practical: once an agent grows past a prompt and a model call, the interaction between parts starts to matter more than any single part.

A tool is a function the model can reach for at a particular moment in a run. The reaching is the part that varies.

Retrieval decides which source gets queried, when, and how the result survives into the next step.

Memory carries a contract: which state can be read or changed, under which scope.

Approval draws the line between a proposed action and an executed one.

Those relationships are where production behavior lives. If they stay implicit inside one application, the first team can move quickly. The second team inherits a mystery.

An agent run is more than a model call#

Picture an incident-triage agent with two phases.

First it investigates. Then it writes the response.

The prompt for the first phase might be simple: assess the alert, inspect recent evidence, and decide whether the user needs a response. The prompt for the second phase might be simple too: answer from the evidence gathered so far.

The run between those prompts is not simple.

Before the first model call, the system decides which provider is ready and selects a model runtime. Then it assembles the prompt from the user's input, the phase objective, behavior guidance, available procedures, relevant context, maybe previous run state, and whatever runtime controls apply.

It also settles what the model can call in this phase. The investigation phase might allow a log-search tool, an incident-status lookup, a retrieval source, and a memory read. The response phase might allow none of those, because its job is to write from the prior result.

When the model proposes a tool call, the runtime validates the arguments and checks whether the call is allowed in the current phase. It runs the tool, maps the result back into the transcript, records whether the call succeeded, and decides whether a retry is permitted. A failure may fail the phase, abort the run, or ask the model to repair the arguments.

Retrieval has the same shape. The runtime chooses which knowledge source is in scope, which embedding provider or external search backend is ready, what query is issued, what content comes back, and how that content is shown to the model. If the result is missing, stale, or unavailable, the system needs a way to say that without pretending the source succeeded.

Memory adds another layer. Reading state is different from writing it. A run may read customer preferences, append an interaction summary, trigger consolidation, or request a review before persistence. Those are state changes, not prompt decorations.

Then the phase has to end.

That sounds small, but it is one of the most important boundaries in an agent run. Which of the exits defined for that phase did the model take? What output survives into the next phase? Did the phase produce a real handoff, or only a transcript entry that the next phase will never see? Did a limit stop execution? Was approval required before the next phase could begin?

The second phase depends on those answers.

If the investigation phase found that the incident is already resolved, the response phase should be able to answer from that result. If the investigation phase found nothing useful, the response phase should say that clearly. If the tool failed or the source was unavailable, the response should not pretend otherwise.

The model call is one event inside a longer runtime story.

This distinction is easy to ignore when everything lives inside one app. The app owns the tool list, prompt templates, retrieval code, memory writes, approval UI, trace format, and final response. That is the right starting point for prototypes. It keeps the moving parts close together.

The problem changes when the parts become reusable.

One team publishes a tool that another team installs. A retrieval corpus feeds three agents. A memory contract needs to work with a real store behind it. A workflow is reused by a command-line tool, a terminal UI, and an SDK-hosted application. The behavior between those parts needs an owner.

Modularity creates leverage. It also creates a question:

Who interprets the composition?

Without an answer, every host app invents its own runtime semantics. One app handles approvals before a phase begins. Another handles them after a tool is proposed. One app carries prior phase output forward. Another carries the whole transcript. One app treats a limit as failure. Another silently truncates.

The pieces may be reusable. The system is not.

Capability boundaries shape what an agent does#

A phase is a capability boundary as much as a prompt, and the boundary changes behavior.

Tool-scaling research keeps finding pressure in the same direction. RAG-MCP, discussed in The Evolution of Tool Use in LLM Agents, studies tool use across a range reaching thousands of MCP tools and reports degradation as the selection space expands [5].

The sharper result complicates that picture in a useful way. A chance-corrected study asking how many tools an agent should see finds no uniform penalty for a larger catalog — the number that helps depends on the query. What did help was presenting a shorter, query-appropriate list: selection accuracy reached 93.1%, against 87.1% when the agent was always shown a fixed five [6]. Fewer tools was not the finding. A deliberately chosen set was.

Context load shows a related effect. Work on LLM coordination under cognitive load found performance holding above 90% up to roughly 40% context load, then degrading close to linearly past that inflection point [7]. The exact threshold will vary by setup, but the architectural lesson is stable: availability is not neutral. The model's surrounding option set changes what the model does.

The incident-triage example makes this concrete.

Suppose the investigation phase has access to a search tool. It calls the tool, gets a result, and completes with an output: no relevant readiness evidence was found. The response phase receives that output and has a clear objective: answer the user from the prior result.

If the search tool is still available during the response phase, some models will call it again.

That may be correct if the response phase needs refreshed evidence. It may be wasteful if the phase exists only to turn the prior result into an answer. The important point is that the available tool changes the behavior even when the words say "answer from the prior result."

Remove the tool from the response phase and the behavior changes. The model has to use the handoff it was given.

No prompt change produced that. The capability boundary did.

The same principle applies beyond tools.

If a phase can read memory, the model may use remembered state to answer. If it can write memory, the phase can propose a durable change. If it can retrieve from a knowledge source, it may prefer fresh search over prior output. If an approval gate sits before the next phase, the model's proposal has to wait for a human or controller decision.

Capability availability is part of agent design.

This is also why agent behavior can differ across model tiers. A large frontier model may infer from the phase objective that it should not re-investigate. A smaller or faster model may treat the available tool as permission, maybe even as a hint. If you test only on the frontier model, you may never see the repeated call. Your users may run something cheaper.

The right answer is not "hide all tools." That would be as crude as giving every phase everything.

The better rule is to make capability boundaries deliberate. If a phase exists to investigate, give it investigative capabilities. If a phase exists to respond from prior output, remove capabilities that would turn it back into investigation unless re-verification is genuinely part of the job.

What the model can call is part of what the agent is.

The runtime has to make that explicit. A prompt alone cannot.

Agent execution should leave evidence#

When an agent behaves unexpectedly, "the model did something weird" is usually the first explanation.

Sometimes it is true. Often it is where the investigation stops because the system did not leave enough evidence to say anything sharper.

For the incident-triage run, a team should be able to reconstruct the execution after it is over:

• what was installed
• what runtime config applied
• which provider was ready
• which phase was active
• which capabilities were available
• which capabilities were withheld, and why
• what prompt was submitted
• which tools were called
• which arguments were used
• what retrieval happened
• which memory records changed
• which approvals were requested
• how approvals were resolved
• whether cancellation occurred
• which limit stopped the run
• what terminal status was reported

That list can sound heavy until something breaks.

Then it is the difference between debugging and folklore.

Software teams already understand this in other systems. A distributed service call is not considered observable because one database query was logged. Teams want traces, spans, attributes, events, status, causality, and enough context to reconstruct why a request took the path it did.

Agent runs need the same discipline, adapted to agent execution.

OpenTelemetry is moving in that direction. Its GenAI work defines semantic conventions under a gen_ai.* namespace, including operations such as create_agent, invoke_agent, plan, and execute_tool. The OpenTelemetry project moved GenAI conventions into a dedicated repository in June 2026 so they could iterate faster, and its GenAI observability writeup is careful about maturity: model-call and tool-call attributes are usable enough to build around, while agent and multi-agent semantics are still developing [8].

Take that caveat at face value. The industry has not finished standardizing agent observability; it is still converging on the shape of the evidence.

The direction is clear anyway.

A runtime should leave enough evidence to explain the run after it is over.

None of that requires every team to use the same application. The same run semantics can sit behind an interactive terminal UI, a one-shot headless script, a machine protocol, or an SDK-hosted app. The host application can still own its UI, API boundaries, provider choices, storage, approval experience, deployment model, and product-specific workflows.

Shared runtime semantics do not erase application ownership.

They make the execution boundary inspectable.

The execution boundary is where governance becomes more than a policy document. It is where approvals happen, where capability restrictions apply, where limits stop the run, where traces are written, where reports are produced, and where the system can prove what happened.

If the only artifact after a run is the final answer, the system has discarded the evidence it needs most.

How AgentPM Harness makes the agent runtime concrete#

In AgentPM, an Agent is a versioned package installed into a workspace, the same way a library is. AgentPM Harness is the built-in runtime that runs one. It is the proof point that packaged agent artifacts can be interpreted together by one engine without forcing every application to become the same application.

An installed Agent can run through:

• the default interactive terminal UI
• a one-shot --headless command
• a --machine protocol driven by Node or Python SDK hosts

The same Rust engine sits underneath all three, and all three write the same trace format. One installed Agent, three ways to drive it, one account of what happened.

Before a run starts, agentpm harness --json can report readiness: which Agent was selected, which model/provider config applies, which runtime services are ready, and which surfaces are unavailable. During a run, Harness traverses the Agent's installed Loop package: the phases it defines, the exits between them, checkpoints, approvals, limits, and terminal states. After a run, it writes JSON reports and JSONL event traces.

The runtime config lives in agentpm.harness.json. That file maps workspace-specific runtime choices onto the installed package graph: model providers, Hooks, Knowledge runtimes, Memory runtimes, MCP imports and exports, trace behavior, limits, and UI branding. The portable Agent artifacts stay portable. The workspace describes how this environment realizes them.

During the Harness work, we measured a small two-phase shape: investigate, then respond. The investigative tool was left available during the response phase.

Smaller and faster models called that tool again during the response phase. They did it repeatedly, even when the first phase had already produced what was needed and the response objective said not to re-investigate.

Two controls made the result useful rather than anecdotal.

On frontier models, the same setup produced zero repeat calls across a dozen runs. On the smaller models, removing the tool from the response phase stopped the repeated calls entirely.

The tier distinction matters more than the specific models: gpt-4o-mini and claude-haiku-4-5-20251001 showed the behavior, gpt-5.6 and claude-opus-5 did not. One workflow shape, one tool, two model tiers. Enough to be worth knowing. Not enough to be a law.

The lesson is narrow and practical. Capability availability is an authoring decision with observable consequences. This is guidance about capability design, not a defect in a model or a runtime. It became visible because the composition was inspectable from outside the app.

Harness also makes the phase boundary explicit. While a phase runs, its transcript is local to that phase. What crosses into later phases is narrower: the exit that was taken, the output the phase completed with, and a compact summary of what happened earlier in the run. That forced a simple but important runtime rule: if the next phase needs the answer, the current phase has to complete with output that survives the boundary.

The artifact model now has a runtime surface to interpret it:

Tools package actions.
Skills package procedures.
Knowledge packages context.
Memory packages durable state contracts.
Profiles package authored behavior.
Loops package iterative control flow.
Agents compose those artifacts.
Templates bootstrap working systems.
Harness interprets the installed composition at runtime.

The parts become a system when something interprets them together.

A run that completes is the minimum. A run you can explain afterward is the point.

Sources#

1. Gartner. Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027. https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
2. Gartner. Gartner Says Applying Uniform Governance Across AI Agents Will Lead to Enterprise AI Agent Failure. https://www.gartner.com/en/newsroom/press-releases/2026-05-26-gartner-says-applying-uniform-governance-across-ai-agents-will-lead-to-enterprise-ai-agent-failure
3. Datadog. State of AI Engineering. https://www.datadoghq.com/state-of-ai-engineering/
4. Sherlocks.ai. Why AI Agents Fail in Production. https://www.sherlocks.ai/blog/why-ai-agents-fail-in-production
5. The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration. https://arxiv.org/html/2603.22862v2
6. How Many Tools Should an LLM Agent See? A Chance-Corrected Answer. https://arxiv.org/html/2605.24660v1
7. United Minds or Isolated Agents? Exploring Coordination of LLMs under Cognitive Load Theory. https://arxiv.org/pdf/2506.06843
8. OpenTelemetry. Inside the LLM Call: GenAI Observability with OpenTelemetry. https://opentelemetry.io/blog/2026/genai-observability/