Jonathan Asare
← All notes

Systems

The Model Is Replaceable. The Engineering System Is Not.

A coding model should be a dependency inside the workflow, not the foundation of it.

Replace your coding agent tomorrow.

Keep the repository, the task, and the team. Change only the model or agent harness.

What breaks?

If the task, controls, evidence, and acceptance process still work, you probably have an engineering system. If they have to be redesigned around the new agent, the original tool was doing more than generating code. It had become part of the process itself.

That question is useful because it exposes where the engineering actually lives.

Model comparisons draw attention towards capability, speed, price, and context size. Those differences matter. The more durable work sits in how the task is described, which actions are allowed, how results are checked, what state is preserved, and who accepts the outcome.

I would not let the private habits of today's strongest model decide any of those things.

Model quality still changes the result

Replaceable does not mean equivalent.

One model may be better at a broad repository investigation. Another may be faster for a narrow mechanical change. Some agents use tools well but make weak assumptions about the product. Others reason carefully and take too long to finish routine work.

A sensible system keeps those differences visible to routing, evaluation, cost controls, and audit records. It does not pretend that every model is interchangeable in practice.

The boundary is narrower: model differences should not silently redefine the task, weaken the evidence, expand permissions, or move the acceptance decision.

Some provider-specific capabilities are worth coupling to. When they materially improve the work, that can be a reasonable trade-off. I would record the dependency honestly rather than call the workflow portable.

Keep the task and the proof stable

The task should exist somewhere another agent, reviewer, or future run can inspect. A short file containing the outcome, scope, constraints, required evidence, and valid stop states is often enough.

What matters is that the task survives the conversation. The second agent should not need a transcript to discover what the first agent was asked to achieve.

Verification should remain callable independently of the agent. That might be a CI job, a repository check, a test target, or a review workflow. The exact mechanism matters less than both agents being judged against the same evidence.

Product-facing work may still require runtime checks, screenshots, logs, representative data, or a short manual review. Those requirements should remain stable when the agent changes.

I would spend more effort standardising evidence than making two agents receive identical prompts. Prompt differences are inconvenient. Different definitions of success are dangerous.

Keep authority and acceptance outside the model

The workspace, permissions, network access, and credentials policy should come from the system around the agent.

Different tools may ask for access in different ways. They can still inherit the same repository state, directory restrictions, network rules, and approval requirements.

Do not ask the model to reason its way into the correct authority boundary. By the time it is making that decision, the boundary has already failed.

The same applies at the end of the task. The agent may produce the change, run the checks, explain its reasoning, and recommend acceptance. It should not approve its own permissions, merge its own work, or change the definition of success after seeing what it managed to build.

This is the boundary I would be least willing to compromise on.

Leave state that survives the session

The conversation should not be the only record of the work.

Preserve the task, commands, results, changed files, assumptions, known limitations, cost, and stopping reason in a run directory or another structured artefact. A second agent can continue from it, and a reviewer can understand what happened without replaying a private chat.

The record does not need to be elaborate. It needs to outlive the session and use a shape that does not depend on one agent's summary style.

Replaceable Agent, Durable System Several replaceable coding agents connect through an adapter boundary to a durable engineering system containing task contracts, execution controls, evidence, state, and acceptance. Replaceable Agent, Durable System Model differences belong above the boundary. Engineering responsibility remains below it. REPLACEABLE EXECUTION Agent A Model and harness Agent B Model and harness Agent C Model and harness AGENT ADAPTER BOUNDARY Prompt format · CLI or SDK · context packaging · provider configuration DURABLE ENGINEERING SYSTEM Task contract Outcome · scope · stop states Execution controls Workspace · permissions · tools Evidence Checks · runtime · limitations Run state Commands · results · decisions Review and recovery Revise · blocked · escalate Acceptance Authority remains explicit A model upgrade should improve execution, not force a redesign of engineering responsibility.
The agent can change while the engineering responsibilities below it remain stable.

Put model-specific behaviour at the edge

A swap-ready repository does not need a universal agent platform. Until a second agent is genuinely useful, a large abstraction layer is mostly ceremony.

Once a second implementation exists, give the tool-specific details somewhere visible to live. One small version might look like this:

agent-system/
  tasks/
  adapters/
  policies/
  scripts/
  runs/

Most real repositories will not be this tidy, and they do not need to be. The boundary may begin as two scripts in an existing tooling directory.

The adapter can absorb prompt formatting, command syntax, context packaging, and provider configuration. It should not own the task contract, the permissions policy, the verification interface, or the acceptance process.

For example:

./agent-run --adapter agent-a --task tasks/customer-export.yml
./agent-run --adapter agent-b --task tasks/customer-export.yml

These commands are illustrative. The interface could be a Make target, CI job, API, or workflow engine. The folder names are not important. The useful property is being able to change the agent-specific code without redesigning the surrounding process.

Run the Agent Swap Test

Choose one representative, low-risk task and run it through two agents.

Hold these constant:

  • the starting repository state
  • the task definition
  • the workspace and permissions
  • the verification interface
  • the evidence requirements
  • the acceptance owner

Then compare the runs.

MeasureWhat it reveals
Accepted outcomeWhether both runs solved the actual task
Human review timeHow much interpretation the agent left to the reviewer
Verification failuresWhere capability differences affected execution
Retries and escalationsWhether uncertainty was exposed clearly
Cost and elapsed timeWhether the routing choice was worth the resources
Unsupported assumptionsWhere the task or repository context was weak
Changes outside the adapterWhere the process is coupled to the tool

Do not expect identical code. That would be the wrong test. Models are expected to behave differently.

Look for changes to the surrounding process. Did the second agent need a new task format? Did the checks have to be weakened? Did assumptions hidden in the first conversation suddenly need to be written down? Did review become harder because the run left no usable record?

Those process changes are the coupling the test is designed to reveal.

The Agent Swap Test The same task contract and engineering controls feed two agent adapters. Their results pass through the same verification and acceptance process, then are compared for outcome, review time, cost, failures, and coupling. The Agent Swap Test Hold the engineering system constant. Change only the agent boundary. SAME TASK CONTRACT Outcome · permissions · evidence · stop states · acceptance Adapter A Agent and model A Execute in same workspace rules Produce standard run record Adapter B Agent and model B Execute in same workspace rules Produce standard run record SAME VERIFICATION AND ACCEPTANCE Accepted outcome · review time · failures · cost · assumptions Record every change required outside the adapter Differences in results are expected. Changes to the surrounding process reveal coupling.
The swap test keeps the engineering system constant and changes the agent boundary.

Know where portability stops being useful

Vendor neutrality is rarely worth pursuing as an objective in its own right.

A provider-specific capability may save enough time or reduce enough risk to justify deliberate coupling. Small teams may have no reason to maintain several adapters. A specialised agent may require a workflow that does not fit a generic interface cleanly.

I am less convinced that a formal adapter boundary is worth building before a second agent is genuinely useful. Before that point, the abstraction can cost more than the coupling it is meant to remove.

The problem is not dependence. The problem is discovering accidental dependence only when the team tries to change tools.

Take one existing agent task and run the swap. There is no need to adopt the second agent. Record every change that was required outside the adapter, then decide which of those changes belongs in the team's engineering system.