Staying Harness-Agnostic as AI Coding Models Improve

Replicas is a multi-harness cloud coding agent platform that lets engineering teams run Claude Code, Codex, Cursor, and Opencode through a single entry point, with each task in an isolated Linux VM. Engineers can start work from Slack and get a pull request back without using the harness as their integration layer. Staying harness-agnostic means preserving that separation as models improve, rather than rebuilding workflows whenever the preferred agent changes.

Why the model benchmark you're looking at today will be wrong tomorrow

Frontier coding benchmark scores change from week to week, and the same model gets different scores in different harnesses. A score is therefore a snapshot of a model running through a particular system. The models behind the major harnesses now have benchmark scores so close that changes to harness configuration can make a bigger difference. That makes a benchmark ranking a weak foundation for a long-term integration decision.

Release timing compounds this problem. GPT-5.5 shipped on April 23, 2026, followed by Composer 2.5 on May 18, 2026, and Claude Opus 4.8 on May 28, 2026. A choice made during onboarding can be stale before a team finishes evaluating the alternatives. If that choice also determines task intake, credentials, and result routing, reaching another model requires changing much more than the model. The durable commitment is to the surrounding system. Switching costs accumulate there.

What the harness actually does — and why it outweighs the model in most real runs

LangChain's harness-only swap moved its coding agent from 52.8% to 66.5% on Terminal Bench while keeping the model the same. HarnessX showed an average gain of +14.5%, reaching up to +44.0% across five benchmarks, by composing and evolving harness primitives. Fan et al.'s empirical study looked at 176 matched settings and found that context management, planning configuration, and action space change results even when the model choice stays the same.

Barbaste et al.'s source-code study of eleven systems explains why changes to the harness matter. The harness comprises the code, configuration, and execution logic outside the model, including the loop, tools, memory, safety controls, context management, orchestration, and extension surfaces. These components determine what the model can see, what actions it can take, and how work proceeds across a run. Different implementations can therefore expose or constrain the same underlying capability.

Context management makes the mechanism concrete. Fan et al. found that its value grows as the context-window budget tightens, with much of the benefit coming from preventing context-overflow failures. A stronger model does not remove the need to manage the information passed to it. Teams can improve that management independently of their next model upgrade.

How the major harnesses differ where it matters for teams — not by model, but by architecture

It helps to compare extensibility, orchestration, memory, and how model reasoning is kept separate from execution. Claude Code offers 33 hook lifecycle events, Dynamic Workflows, Skills, plugins, MCP, and CLAUDE.md context files. Dynamic Workflows supports tens to hundreds of parallel subagents within one session. Bun creator Jarred Sumner used it to port roughly 535,496 lines from Zig to Rust at a 99.8% test pass rate in 11 days.

Claude Code also makes specific trade-offs. It is tightly coupled to Anthropic models, and Barbaste et al. describe its durable persistent memory as thin by design. How deeply the agent plans and how it reads files drive token spend before it writes code. Those traits matter in different ways for a deep refactor and a short review, so teams need more than model quality to make the choice.

OpenCode takes a different architectural position. It supports 75+ providers through Models.dev, including local models, with configuration per agent and per subagent in JSON or markdown. Barbaste et al. identify it as having the most thoroughgoing client/server architecture in the studied corpus. Because it has no opinionated model default, teams have control over the choice, and the quality they get depends on model configuration.

Codex provides another relevant comparison through its extension architecture. Its adoption of Claude Code's hook vocabulary and plugin format makes those surfaces less vendor-specific, though it does not establish identical execution behavior. Cursor represents the code-editor-first option. These are different architectural choices.

What harness convergence means — and why it makes switching cheaper than teams assume

Standardizing on one harness does not mean every configuration investment must be rewritten to use another. Barbaste et al. found SKILL.md adoption in 9/11 systems and MCP support in 8/11. Claude Code and OpenCode share the SKILL.md standard, while Codex has adopted Claude Code's hook vocabulary and plugin format.

Behavioral policy has also moved from prompt prose into files such as CLAUDE.md, AGENTS.md, and.cursor. These files make policy readable and diffable, although a common file-based approach does not make every instruction automatically transferable. Loop structure and memory design still differ. Portability lets teams keep useful configuration and integrations when they switch, but they still need to test how the new harness behaves.

The abstraction above these systems already exists. Barbaste et al. analyze Databricks' Omnigent as a meta-harness that places eleven vendor harnesses behind one API. Their study also found hand-rolled async loops and deterministic retrieval rather than general-purpose agentic frameworks in the production harnesses studied. Convergence is happening through conventions. Teams that build well-structured AGENTS.md and SKILL.md files, MCP tooling, and hook-based integrations use those conventions to keep their extensions from being tied to one vendor.

Why the environment the agent runs in is as important as the harness it uses

Portable configuration only goes so far if execution is still tied to a local development machine or shared CI worker. The environment must be separable from the harness so that replacing an agent does not disturb shared dependencies or expose development secrets. The architectural boundary separates the brain from the hands. The process calling the model API runs outside the sandbox and holds credentials, while shell commands, tests, file edits, and package management run inside a locked-down container or VM. That execution environment has no secrets and restricted network access, with actions traceable to specific tasks.

Upsun provides a concrete example. It uses self-hosted GitLab and Linear, and built a custom AI review agent because off-the-shelf integrations did not fit its stack. Webhook events enter a system that classifies requests, triggers CI pipelines, runs Claude agents with sandboxed tool access, and posts results to GitLab comments and Linear sessions. The review agent is read-only, with file write and edit tools blocked. Shell commands run inside a Bubblewrap sandbox whose network allowlist is restricted to package registries and Upsun's GitLab instance.

The rollout sequence begins by moving agents away from development secrets, including.env files, cloud credentials, and SSH keys. Branch protection then prevents pushes to protected branches. A propose/dispose CI pattern keeps the agent job read-only and free of deploy secrets, while a separate job writes changes after a human approves them.

This need for a stable, configurable execution boundary is addressed by platforms such as Replicas, where every run receives a sandboxed Linux VM pre-loaded with the team's dependencies and tooling. Agents can install packages, run services, drive a browser, and verify their work within that environment. Isolation makes harness substitution a configuration decision.

How to trigger Claude Code or Codex from Slack without creating harness lock-in

A Slack message, Linear assignment, GitHub issue, or GitLab webhook can reach an orchestration layer that selects an agent and returns a pull request, comment, or dashboard entry. The harness then acts as a backend. Engineers can keep handing off work in the same way when that backend changes.

Several tools implement this separation. Contrabass pulls issue-driven work from Linear, GitHub Issues, or a local board into git worktrees, with terminal, headless, and dashboard modes. Cyrus watches work assigned through Linear, GitHub, GitLab, and Slack, creates an isolated worktree per issue, and supports Claude Code, Codex, Cursor, Gemini, and Opencode.

Groundcover's June 29, 2026 Agent Mode expansion provides another workflow example. Engineers can direct agents to recommend code, open pull requests, and manage tasks inside Slack, Linear, and GitHub. Reasoning and execution remain in the customer's cloud, and actions are tied to an authorized user operating under that user's permissions. Authorization is thus part of the integration design.

For teams using Replicas, Slack, Linear, GitHub, GitLab, and the dashboard provide task entry points, while Claude Code, Codex, Cursor, and Opencode remain selectable harnesses. The important distinction is where triggers live. A harness-specific Slack app couples intake to that harness, whereas orchestration-owned triggers preserve the workflow when the backend changes.

Measuring what agents actually produce, not just how fast they move code

Pull requests per week, lines of code, and commits are poor ways to measure agent value because agents can produce more code without doing more useful work. A harness that produces code quickly is not necessarily the one that produces the best accepted result. Defects, review time, and rework therefore matter to the comparison. If teams leave those effects out, a gain in throughput can hide extra work for reviewers or fixes needed later.

The measurement framework spans five dimensions, which are adoption, AI code share, complexity-adjusted velocity, code quality, and cost/ROI. Measuring fewer than three gives an incomplete picture and can mislead harness selection. Adoption establishes whether the system is being used, while AI code share describes its contribution. Complexity-adjusted velocity and code quality show whether that contribution helps get useful work done. Cost/ROI then compares the result with the resources used to get it, with token spend counted as a cost.

Attribution is the prerequisite for comparing these dimensions across harness changes. Each merged pull request needs an identifiable agent, harness, model, and credential if a team is to distinguish a better configuration from a different workload. Replicas addresses the attribution layer with analytics that assign every minute to source, harness, model, and credential. That provides an execution-cost record. The decision remains whether a harness improves outcomes for the task class.

How to operationalize harness-agnosticism as a team practice, not a one-time decision

Harness-agnosticism is a practice. Its preconditions are isolated environments, integrations above the harness layer, and measurement that spans harnesses. Teams must keep those boundaries in place as they add automations and new agent features. Otherwise, each convenient integration can quietly restore the coupling the architecture was meant to remove.

The configuration investment should favor AGENTS.md and SKILL.md files, MCP tooling, and hook-based integrations that can travel between harnesses. Routing should then distinguish task classes, including deep refactors, PR reviews, issue triage, and documentation. Evaluate harness fit for each class rather than appointing a permanent winner across all work. Revisit that fit when new models or harness versions ship, using objective acceptance tests where tasks are repeatable.

A routed multi-harness pilot is appropriate when expensive planning consumes a material share of agent allowance and task classes show repeatable quality differences. Staying with one harness remains reasonable when agent volume is low, tests are weak, or nobody owns routing and incidents. Harness-agnosticism requires keeping an evidence-based switch possible in days.

Replicas supports that operating model through per-task harness selection, sandboxed execution environments, and cross-harness attribution, with a new harness added through configuration. The practical capability test is whether the team could route a task to a different harness this week if a materially better option shipped.