It is tempting to blame the model. Perhaps it misunderstood the request or was not capable enough to handle it.
But imagine that the assistant never had access to the ordering system. Or its update failed, and the software hid the error. Or nothing required it to check that the new address had actually been saved.
Those are different problems. A more capable model does not, by itself, repair a missing connection or make an unavailable error visible.
The software surrounding the model has a name: the harness. It supplies information, connects tools, manages the work and checks the result. Recent engineering reports and research show why changing that surrounding system can substantially improve results without changing the model doing the work.
The lesson is not that models no longer matter. It is that we can mistake a poorly supported model for an incapable one.
The harness turns a model into a working system
A model generates responses from the information it receives. An AI agent uses a model within a process that can take actions: looking up records, editing files, running tests or calling other software.
The harness runs that process. Here, we mean the agent’s working environment and control software, not just the test harness used to score it.
Think of weights as stored data and the harness as application functions
An application combines stored values with functions that make use of them. Having an address in memory does not update an order. A function has to find the order, save the change and check that it worked.
That gives us a useful comparison for AI: the weights are stored parameter data; the harness supplies the application functions and control logic around the model.
Weights are numerical values learned during training, not the original raw training data. They shape how the model responds, rather than acting as a database of records the harness simply looks up. The model’s architecture and inference code perform computations using those values. PyTorch’s model-saving documentation illustrates the distinction between saved learned parameters and the model structure that uses them.
| AI component | Application analogy |
|---|---|
| Model weights | Stored, learned parameter data that shapes behavior. |
| Model architecture and inference code | The computation that uses those parameters to process an input and produce a response. |
| Harness | The functions and application logic that supply context, execute permitted actions and check results. |
For the address-change example, the harness might expose functions such as lookup_order(), update_address() and verify_update(). It calls the model with the request and relevant information, executes permitted tool requests, and returns the results so the model can decide what to do next. Anthropic describes this tool-and-feedback loop in Building effective agents.
Those functions are not the whole harness. It also supplies instructions and manages the sequence, permissions and stopping conditions. But the functions analogy explains its practical value: understanding an instruction and carrying it out are different capabilities.
Without the explicit instructions and context a harness can provide, you are relying more heavily on the model’s raw intelligence—its learned ability to infer what to do from the information available. It has to work out which facts matter, which steps to take and what counts as success. A well-designed harness makes more of those requirements explicit and supplies relevant information, so the model has fewer gaps to fill on its own.
If the update function is missing or its error never comes back, changing the weights does not repair that integration. Improving the functions—and how they work together—can make the application more capable while the model’s weights remain exactly the same.
The same model can perform very differently
The strongest evidence for the harness’s importance comes from comparisons that keep the working model fixed.
In February 2026, LangChain reported improving its GPT-5.2-Codex agent’s Terminal-Bench 2.0 score from 52.8% to 66.5%: a gain of 13.7 percentage points. The changes included verification reminders, better environment context, loop detection and reasoning-budget allocation. The model stayed the same. LangChain’s LinkedIn announcement links the result to Vivek Trivedy’s engineering report.
That is a useful engineering result, not a universal forecast. It is company-reported, combines several changes and includes adjustments to how reasoning time is allocated. “Same model” does not mean “identical compute.” Still, it demonstrates that switching models is not the only route to a meaningful improvement.
An August 27 arXiv preprint, Verify Smarter, Evolve Further, offers a held-out example: tasks the optimization process did not see. With the runtime model fixed, its HarnessLens method improved OpenCode’s result on a challenging BIRD database-task subset from 27 of 72 tasks to 33 of 72. That is six additional successes through harness changes. The same OpenCode experiment showed no improvement on Terminal-Bench 2, so the benefit was not automatic across tasks. HarnessLens paper, Table 1 and Appendix C.1.
A September 8 preprint, Co-Evolving Harnesses and Models, reports a larger, task-specific result. Across seven enterprise task suites, the unchanged Qwen3-Coder-30B-A3B runtime model’s mean success rate rose from 29.2% to 78.0% with an evolved harness. A stronger Gemini model helped design those changes; this was not optimization without stronger-model assistance. Under that same evolved harness, Gemini 3.1 Pro Preview achieved 93.6%. Co-Evolving Harnesses and Models, Table 1.
That last comparison captures both sides of the argument. A harness can unlock much better performance from an existing model, while the model’s capabilities still make a substantial difference. These are specific benchmark results from a recent preprint, not a promise that every smaller model can replace a larger one.
Look at the work, not just the answer
If we only inspect an agent’s final response, different failures can look identical.
“I could not complete the request” might mean the model could not solve the problem. It might also mean the right document was never retrieved, a tool returned an unreadable error, or the agent repeated an unsuccessful action until its time ran out.
The remedies are different. Better retrieval addresses missing information. Clear tool responses expose execution failures. A bounded retry policy can stop an unproductive loop. None of those fixes requires pretending the model has become more intelligent.
This is why the sequence of actions matters. Engineers call that record a trace: what the agent saw, which tool it called, what came back and what happened next.
In a July 7 LinkedIn post, LangChain engineer Vivek Trivedy discusses using traces to find improvement opportunities, turn failures into evaluations and decide between harness changes and model fine-tuning. It is a practitioner’s approach to diagnosis, rather than another controlled performance study. Trivedy’s original post.
The practical implication is straightforward: investigate the failure before choosing the intervention. Improving a model’s reasoning will not necessarily fix the software that supplies its evidence or executes its decisions.
The lesson is not that models no longer matter. It is that we can mistake a poorly supported model for an incapable one.
A better harness is not always a bigger harness
Adding more agents, memory or review steps can create costs of its own. More machinery needs more testing, and a longer process can introduce more places to fail.
Anthropic’s March 24 account of long-running application development illustrates the tradeoff. Its Opus 4.5 example compared a solo run with a planner, generator and evaluator arrangement. The richer application took about six hours and $200, compared with 20 minutes and $9 for the solo run. That is an illustrative result, not an equal-cost experiment. Anthropic’s LinkedIn announcement and engineering account.
The same account describes removing some scaffolding as newer models made it unnecessary. That is an important counterpoint: good harness engineering includes knowing what to take away.
The goal is not the most elaborate workflow. It is the simplest system that reliably meets the task’s quality, safety, time and cost requirements.
A benchmark gain is a starting point, not a guarantee
There is another trap: making a harness better at the examples used to improve it, without making it better at new work.
The September 1 HarnessDev preprint found that improvements during harness evolution were unstable and did not consistently carry over to held-out tasks. Results also depended on the model executing the harness. A successful configuration should not be assumed to travel unchanged between models or tasks. HarnessDev.
An August 27 revision of Rethinking the Evaluation of Harness Evolution for Agents also questions how automatic harness improvements are measured. Under its tested setup, stronger baselines using repeated attempts challenged the claimed advantage of harness evolution. Matching the number of attempts still does not establish equal total token usage or cost. Rethinking the Evaluation of Harness Evolution for Agents.
Neither finding means harnesses are unimportant. They mean we should test whether an improvement survives unfamiliar tasks and a fair comparison.
The arXiv papers discussed here are preprints; citing them is not a claim of peer review or independent replication. Together with the practitioner reports, they support a narrower, more useful conclusion than “the harness is everything”: evaluate the model and its surrounding system together.
What this changes for an engineering team
Our recommended approach follows the same discipline used to improve other software systems:
- Define what completion means. For an address change, check the saved address. For a code change, run the relevant tests. An assistant saying “done” is not enough.
- Inspect the failed run. Establish whether the problem was missing information, a tool failure, reasoning, permissions or the stopping rule. Preserve enough evidence to diagnose it without unnecessarily retaining sensitive data.
- Compare changes deliberately. Start with the same model and change one harness component where practical. If several components change together, do not claim to know which one caused the improvement.
- Test on work the changes were not tuned against. Repeat representative tasks and record failures as well as successes. Measure time and cost alongside quality.
- Enforce boundaries outside the model. Prompts can explain rules; software must enforce permissions, spending limits and required approvals. A persuasive answer must not authorize a dangerous action.
- Recheck the combination when the model changes. A new model may need different instructions or fewer support steps. Retain a component because it helps, not because it helped last year.
These are engineering recommendations, not a claim that any single study validated this entire checklist.
The objective is to find out whether the model lacks the ability to do the work, or whether the surrounding system is preventing it from using the ability it already has.
What this means for CID
For CID, the implication is that a release should be judged by the behavior of the whole AI system, not just the identity of its model.
A change to a prompt, a document source, a tool or an approval rule can change the outcome without changing any model weights. Those changes deserve identifiable versions, relevant tests, review and a recovery plan, just as application-code changes do.
This is the product direction: connect what changed to the evidence used to approve it and the configuration actually running. It is a design objective, not a claim that CID already captures every possible harness component in one release record.
Model engineering and software engineering meet here. Both need a disciplined way to turn a promising component into a dependable system.
Before replacing a model, check what it was given, what it was allowed to do and how the result was verified. Sometimes the answer is a better model. Sometimes the model was never the part that needed fixing.