Coding Agent Harness Evaluation Criteria for Engineering Teams

Engineering teams testing a multi-harness cloud coding agent platform should score how well it fits each task type, how it behaves in the environment, how it fits the workflow, and the quality of its PR-level output. Teams that want parallel agents without having to manage infrastructure, or Slack-triggered coding agent runs that return pull requests, should test those capabilities too. The deciding evidence is how each harness handles the team’s code, conventions, and review bar.

Why public benchmark scores fail to predict harness performance on your codebase

A patch passes a coding benchmark by fixing the target test, but that result does not establish that a team would merge it. A real review bar can also require no unrelated reformats, no new imports, no broken unrelated tests, and a diff small enough to review quickly. These are different standards of success. A benchmark score that omits those conditions cannot answer how much cleanup an agent’s apparently successful patch will require.

The mismatch extends to the work presented to the agent. SWE-bench supplies a clean repository and a test command, while an engineer may supply a partial diff, a Slack thread, and a reviewer comment. Repository conventions add another source of rejection, including custom ORM wrappers, in-house decorators, and forbidden imports. Generic correctness does not establish local acceptability.

Contamination further weakens the connection between public scores and novel work. When benchmark items predate model training cutoffs, the model may see canonical patches during training, so a high score may say little about how it will do on an unfamiliar ticket. The practical response is to evaluate candidate harnesses on the team’s own code, using the same review bar applied to human contributions.

Why task type is a stronger predictor of harness fit than overall agent rank

Task type is a more useful place to start because different kinds of work take different paths to acceptance. Documentation changes often have narrow scope and visible review criteria. Feature work calls for making sense of requirements, choosing dependencies, testing, and working with other contributors. Each adds an opportunity to produce a plausible change that misses the intended behavior. Combining those tasks into one acceptance rate conceals where a harness helps and where it creates review burden.

A harness’s success on documentation does not establish its suitability for feature work, and refactoring presents different demands again. The category changes what success requires from the agent. Teams therefore need separate evaluation sets for documentation, bug fixes, feature work, refactoring, test generation, and other work represented in their backlog. The goal is a task-specific selection model.

Acceptance rate should be recorded by category, but it is not sufficient on its own. A merged change may still have needed a lot of human editing, more than one review, or a great deal of rework. Merge status hides the route to acceptance. Review time, rework volume, defect escape, test failures, and human editing expose whether apparent success saves effort or merely shifts it downstream.

Those measurements also need a record of the conditions that produced them. Models, prompts, repository conventions, and review practices can change, so an earlier result is not a standing guarantee. Comparability depends on preserving the evaluation context. Each run should record the agent version, task category, repository context, and acceptance criteria, with periodic repetition to reveal changes in fit.

How to build the internal evaluation set that replaces public leaderboards

The last fifty merged pull requests are a concrete set to start with. They contain real intent, reviewer feedback, a passing test suite, and a known-good diff rather than an abstract approximation of the team’s work. For each PR, capture the parent commit, the issue text or commit message, the merged diff, and the test deltas. Keep the tasks separated by category so that the final results preserve the distinctions the evaluation is intended to measure.

Before running a candidate, set the acceptance criteria for each task. The task should state a clear objective, boundaries, relevant files or services, expected behavior, and validation requirements. Those boundaries make it easier to judge the output. They also establish what the agent was asked to do before its response can influence the evaluator’s judgment. Record those criteria with the agent version and repository context for the run.

Next, restore the repository to the parent commit and give the candidate agent the issue text or commit message. Let it run unobserved in a sandbox. Then compare its patch with the merged patch. Exact textual agreement is the wrong standard. A reviewer might accept a different implementation that preserves the intended behavior and follows the same conventions. Score behavioral equivalence and review-bar match rather than penalizing every departure from the historical diff.

The replay should also capture evidence beyond the patch. Tool-call correctness asks whether search, editing, test execution, and commit actions occurred in an appropriate order. Multi-file coherence checks whether the agent changed every call site, import, and configuration that needed to change. Plan coherence tests whether the written plan matched the user’s intent and remained consistent across edits. Rollback discipline distinguishes reverting broken changes from piling further edits on top of test failures.

The evaluation set then needs regular renewal. Each week, promote one freshly merged PR into the set and remove the oldest. This keeps the reference work in line with the repository’s current conventions and leaves the team with a process it can repeat. Periodic runs can then compare candidates and versions under recorded conditions. One successful replay establishes success on one task in one repository under one access policy.

What environment behavior reveals that diff quality alone cannot

A clean final diff can hide how unreliable the agent was during execution. Interactive Evaluation Requires a Design Science argues that judging responses on their own is not enough for systems that act through tools, environments, users, and other agents. Their evidence includes the trajectory through feedback and state. For a harness evaluation, that means assessing process, recoverability, coordination, and robustness separately from the final patch.

Rollback discipline offers a particularly clear signal. An agent that keeps layering changes over test failures has not demonstrated safe recovery, even if some replay outputs look acceptable. Tool sequencing needs the same checks because verification must cover the state being committed. Running tests before editing does not validate the later patch, while committing before verification leaves that change unchecked.

Repository exploration is another observable part of environment behavior. An agent that updates one call site but misses three others has failed to establish the scope of its change. The defect spans reasoning and environment use. Scoring multi-file coherence shows whether the agent found the dependencies it needed to inspect for the task, beyond how convincing the edited file looks.

Access scope makes these runtime judgments consequential for infrastructure risk. A dangerous configuration combines access to private data, exposure to untrusted content, and the ability to communicate externally. Evaluation should establish which of those conditions the candidate enables by default. Private data can include source code, .env files, credentials, and SSH keys, while untrusted material can arrive through issues, web pages, and third-party packages. Network requests, git pushes, and opening PRs provide external communication paths that must be considered with those inputs and permissions.

Support for controls should be evaluated as part of harness fit. Agents can hallucinate package names that attackers register with malicious payloads, so SAST and SCA should block every pull request regardless of the harness. Permission separation provides another concrete evaluation target. Under the propose/dispose pattern, the agent job has read-only contents access and no deploy secrets, while writes happen in a separate job behind required human approval. A candidate’s ability to operate within that arrangement is stronger evidence than a clean diff produced with unrestricted access.

How can a harness fit the team’s existing workflow from Slack to a pull request?

A team using Slack, Linear, GitHub, or GitLab needs to evaluate whether a candidate accepts work there or requires a separate interface. That difference changes the cost of using it. For a buyer seeking Slack-triggered coding agent runs, the trigger-to-PR path must be demonstrated in the evaluation rather than inferred from support for the agent alone. The output must also return to the team’s existing PR review workflow.

A background run that returns a PR joins an existing review queue, while an open interactive session demands synchronous attention. Neither mode is interchangeable with the other. A platform that promises multiple agents in parallel without infrastructure management still needs to be judged on how those runs fit the way people hand off work, review it, and approve it. Parallelism does not establish workflow fit by itself.

Repository conventions must survive that integration across agents. A per-repo file such as AGENTS.md can specify allowed files, allowed commands, coding conventions, boundaries, and required run behaviors. Private chat threads are a fragile substitute. The evaluation should test whether those shared constraints govern each candidate’s work, including when engineers delegate asynchronously. Otherwise, changing the agent can also change the effective rules under which it operates.

Tool permissions and attribution complete the integration assessment. MCP access should begin with read-only access to GitHub, documentation, and issue trackers before write actions are allowed. Tool access needs the same treatment as code permissions, with scope, review, and removal when no longer needed. Analytics should identify which agent ran, on which task, under which model and credential, and what it produced. Without that attribution, neither cost control nor quality comparison has a dependable basis.

The integration decision is therefore also a governance decision. No new write access, MCP server, or autonomous PR workflow should be permitted until an evaluated task shows that the agent can stay inside the team’s convention. The single task establishes the minimum evidence for the proposed integration, while the broader replay set tests whether performance holds across the work the team intends to delegate.

How PR-level output quality connects harness evaluation to review overhead and code comprehension risk

The final pull request determines how much human work remains. When agents produce more output than reviewers can absorb, review becomes the dominant time cost. More PRs are not automatically more useful output. Evaluation should ask whether the harness cuts total human review time or adds to it because its intent is unclear, it makes changes it need not make, and people have to keep correcting it.

PR descriptions offer a concrete place to assess that burden. A description that explains behavior change gives reviewers more useful context than a list of changed files. It should stand on its own. The plan, diff, and explanation of risk should let a reviewer evaluate the change without replaying the agent’s session. Scoring those artifacts tests whether the harness helps people review the code or just hands it over for approval.

(Im)Paired Programming found that coding agents improved initial task completion but harmed users’ code comprehension. The study also found that users understood less when they put little effort into their interactions, including copy-paste prompts and auto-accepted edits. Completion and comprehension can move in opposite directions. That distinction creates an operational risk because engineers must be able to explain shipped code, maintain it when AI is unavailable, and recognize serious errors before merge.

The practical response is to make comprehension part of the fixed review bar. Diff size, test visibility, explanation quality, and the absence of unrelated changes give reviewers a consistent basis for engagement. Tracking rework ratio and defect escape rate with velocity exposes how faster output affects debt and delivery.

A reliable harness evaluation uses a repeatable task, a fixed review bar, and periodic replay to give the team evidence it controls about what each harness is doing to delivery quality.