Coding Agent Harnesses Can Triple Costs While Delivering No Accuracy Gain, Studies Find

Quick Facts

• A study of 176 harness-design settings found heavyweight and minimalist harnesses scored within two percentage points of each other on accuracy benchmarks.
• Cost per task varied by as much as threefold across harnesses running the same underlying model.
• A corroborating UC Berkeley study found Claude Code costs up to twice as much as a minimal harness while delivering nearly identical pass rates.

Coding agents consist of two parts: the model that supplies intelligence and the harness that controls what the model sees, which tools it can use, and how work proceeds. New research shows that investing in a more elaborate harness raises costs sharply but does not improve accuracy.

Researchers at Shandong University tested three production harnesses — Claude Code, OpenCode, and mini-SWE-agent — across five models on SWE-Bench Verified and Terminal-Bench 2.1. Mini-SWE-agent is an open-source harness written in roughly 100 lines of Python that exposes a single bash tool and performs no context compaction.

The results were stark. On a 447-task pool, Claude Code and mini-SWE-agent scored within two percentage points of each other and were statistically equivalent within a five-point margin on both Qwen models tested. A more complex harness bought no measurable accuracy improvement.

Cost told a different story. The Shandong University paper found that on GLM-5.3-Flash, a trial run through Claude Code cost nearly twice as much as through mini-SWE-agent and more than three times as much as through OpenCode. The researchers concluded: "A vendor's own harness on its own model buys no score advantage."

Where the Cost Comes From

The Berkeley research arm traced cost differences to the very first model call. Across all seven models it tested, Claude Code's initial context averaged more than 10 times the size of Pi's, a minimal open-source harness with four basic tools, largely because of longer instructions and tool schemas.

Across shared models on SWE-bench Lite, Claude Code costs about 2.0 times as much as Pi and 1.6 times as much as OpenAI's Codex CLI, using geometric means of cost ratios. On Terminal-Bench 2.0, Claude Code costs about 1.5 times as much as Pi.

Pass rates told a different story. Claude Fable 5 solves 97.8% of attempts in Claude Code, 96.7% in Codex, and 96.7% in Pi. The cost premium for Claude Code produced no meaningful accuracy gain.

Benchmarks Are Harder to Read Than They Look

Public leaderboards frequently compare coding agents running different harnesses, making it difficult to isolate what each harness contributes. Some published comparisons report spreads of 23.8 points on graded task scores and 27.4 percentage points in Pass@1 when only the harness changes.

The Shandong team found that many of those comparisons run each configuration only once. A separate study found that 10 reruns of the exact same setup on SWE-bench Verified produced score spreads of 2.2 to 6.0 percentage points, meaning a single run is not a reliable measure.

The researchers also found a contamination problem in their own data. When they audited early web-enabled runs, 312 of 439 DeepSeek and Qwen trials — 71% — had retrieved the original human-written fix from GitHub. They discarded those runs and blocked network access before publishing final results.

What Harness Components Actually Contribute

The study did identify conditions where harness design matters. Context management becomes valuable as context-window budgets tighten, with most of its benefit coming from preventing context-overflow failures rather than improving reasoning. Staging rule-based filtering before model-based summarization produced the strongest efficiency gains among context strategies tested.

Planning worked differently depending on model capability. For weaker models, planning improved accuracy. For stronger models, it reduced cost without changing accuracy. Bash-capable models achieved lower costs using a bash-only interface compared with predefined tool sets.

A separate longitudinal study fixed the model and varied only the harness across 35 sequential releases. It found that practitioners consistently blamed quality regressions on the underlying model when the harness was the actual source of the change.

What This Means for Buyers

For founders and engineering leaders evaluating coding agents, the research points to a straightforward problem. Vendors market harness improvements as accuracy upgrades, but the data shows cost increases are far more consistent than accuracy gains.

Teams running coding agents at scale should audit which harness they are using and what it costs per task. Switching to a simpler harness may reduce inference costs by two to three times without changing pass rates on most tasks. The research suggests the model choice matters more than the orchestration layer around it.