|

Which harness is best? FrontierHarness leaderboard comparison

Which harness is best? FrontierHarness leaderboard comparison

Author’s note: This article only interprets the final leaderboard after all 26,040 runs of FrontierHarness v1.1 finished — comparing 21 harnesses side by side, broken down by model, and cross-checking against the protocols and prices actually available for the relevant model IDs on APIYI.

If you’re already using coding agents like Codex CLI, Claude Code, OpenCode, or Pi, and you frequently switch the underlying model, then the question “which harness is best” probably doesn’t have a single answer. On 2026-11-10, Guanlan Dai of Runta launched FrontierHarness v1.1, and the question it posed outright was: when you switch models, should you switch harnesses too? As of 2026-11-15, the live leaderboard shows running 0, scored 26,040 — 21 harnesses × 5 models × 62 tasks × 4 attempts have all finished running.

This is not a leaderboard write-up claiming “one harness takes first across the board”. On the final leaderboard, the highest pass rate and the lowest cost per pass don’t belong to the same harness, and the cheapest harness for each of the five models isn’t the same one either. This article breaks these conclusions down by model and explains them cell by cell.

The key takeaway: after reading this, you’ll have a model-by-model comparison of the FrontierHarness final leaderboard, you’ll know which column to look at when you switch models, which old numbers you can’t use anymore, and which protocol you need to verify before integrating.

which-harness-is-best-frontierharness-leaderboard-comparison-en-image-0


FrontierHarness Final Leaderboard Methodology: Check the Rules Before You Decide Which Harness Is Best

All FrontierHarness runs execute inside the Runta runtime and use the same set of resource limits. Each harness × model combination has to complete all 248 runs before it counts as a final result, so rankings during the live stream keep shifting as new results come in. The table below lays out the scoring rules for the v1.1 final leaderboard; all comparisons later in this piece use it as the reference.

Criterion v1.1 final leaderboard setting Why it matters when comparing
Dataset [email protected], 62 tasks All harnesses run the same task set
Scale 21 harnesses, 4 attempts per task, 248 per combination, 5,208 per model, 26,040 total Every cell has the same denominator, so you can compare across the board
Runtime environment 1–4 vCPU, 2–8 GB memory, 32 GB disk, per-task timeout of 15 minutes to 8 hours Eliminates machine differences
Inference settings Inference provider Baseten; all five models use effort max Model-side variables are unified
Pass rate passes ÷ counted attempts; timeouts and reward hacking count as failures, infrastructure errors are excluded Peeking at answers earns no points
Cost list price × reported tokens It’s a leaderboard algorithm, not any platform’s bill
Confidence 95% bootstrap by task Read very close rankings with caution

A few sets of numbers get passed around a lot, but none of them are from the v1.1 final leaderboard. v1.0 tested only 9 harnesses on a single model, Kimi K3, with 12 configurations and 360 runs. Codex’s 66.7% pass rate and $3.47 median cost per task are both v1.0 numbers. In the 2026-11-11 mid-run snapshot, cost per pass ranged from $0.57 to $1.32, and pass rates ranged from 56.2% to 77.6%; that too reflects only the halfway mark. On the same day, someone wrote that “GLM-5.3 does best with KimiCode, and Kimi-K3 does best with Codex.” Once you re-rank by the final leaderboard’s cost per pass, neither claim holds.


Which Harness Is Best: FrontierHarness Final Leaderboard Comparison

The table below brings together the results for 21 harnesses across five models, sorted from lowest to highest cost per pass. All dollar figures come from the FrontierHarness final leaderboard and are calculated using list prices multiplied by reported token counts.

Rank Harness Cost per pass Average cost per task Pass rate
1 Codex CLI $0.937 $0.573 61.1%
2 Pi $0.976 $0.563 57.7%
3 Oh My Pi $0.983 $0.564 57.3%
4 ZCode $1.06 $0.659 61.9%
5 KimiCode $1.09 $0.653 60.2%
6 OpenCode $1.12 $0.646 57.4%
7 Maka $1.13 $0.654 57.7%
8 MiniMax Code $1.14 $0.647 56.8%
9 DeepSeek Harness: Minimal $1.15 $0.661 57.4%
10 Hermes Agent $1.16 $0.692 59.9%
11 Cline CLI $1.19 $0.689 57.7%
12 Grok Build $1.20 $0.723 60.0%
13 Alma $1.22 $0.731 59.7%
14 DeepSeek Harness: Standard $1.27 $0.697 55.0%
15 DeepSeek Harness: Creator $1.27 $0.713 56.1%
16 mini-SWE-agent $1.29 $0.757 58.6%
17 DeepSeek Harness: Code $1.32 $0.732 55.2%
18 Exo $1.36 $0.691 50.6%
19 GitHub Copilot CLI $1.38 $0.787 56.9%
20 Claude Code $1.55 $0.915 58.9%
21 Antigravity CLI $1.63 $0.897 55.0%

The Two No. 1 Spots in the Final Overall Leaderboard

The highest pass rate overall is ZCode’s (61.9%), and the cheapest cost per pass is Codex CLI’s ($0.937). You have to read those two facts together. Codex CLI’s pass rate is 61.1%, just 0.8 percentage points below ZCode, but it costs about $0.12 less per pass; ZCode is the steadier choice when you’re more focused on passing more tasks. The most expensive option, Antigravity CLI, costs $1.63 per pass, about 1.74 times Codex CLI. Pass rates across the entire field range from Exo’s 50.6% to ZCode’s 61.9%, an 11.3-percentage-point gap.

There are two easy misreads. First, the four DeepSeek Harness entries are four configurations of the same version, 0.2.1-rc.3 (Minimal, Standard, Creator, and Code), not four separate products. Second, Alma ranks 13th by cost per pass but 6th by pass rate, behind ZCode, Codex CLI, KimiCode, Grok Build, and Hermes Agent. The November 11 snapshot’s claim that Alma ranked No. 3 with 72.7% is interim data; the final leaderboard’s 59.7% is the number to use.


Which Harness Is Best After Switching Models: the FrontierHarness Five-Model Breakdown

A site-wide summary only tells you the average. What actually drives your bill is the column for the model you’re running. In the table below, “cheapest” always means the harness with the lowest cost per pass on that model’s page, and every pass-rate denominator is 248.

Model Average cost per task (averaged across harnesses) Cheapest per pass Highest pass rate Most expensive on the same model
GLM-5.3 $0.968 Codex CLI $0.803, 57.7% ZCode 58.9%, $1.48 Exo $2.72, 50.0%
GLM-5.3-Flash $0.193 Pi $0.223, 58.1% Hermes Agent 60.9%, $0.328 DeepSeek Harness: Minimal $0.484, 55.2%
DeepSeek-V4.1-Flash-1017 $0.185 OpenCode $0.177, 61.3% ZCode 69.0%, $0.222 Cline CLI $0.860, 63.3%
Muse Spark Open $1.01 Oh My Pi $1.21, 58.9% Grok Build 59.7%, $1.77 Claude Code $3.75, 52.8%
Kimi-K3 $1.12 Oh My Pi $1.45, 57.7% mini-SWE-agent 64.9%, $2.02 Antigravity CLI $2.73, 56.9%

which-harness-is-best-frontierharness-leaderboard-comparison-en-image-1

Conclusion 1: The cheapest harness isn’t the same one across all five models

The cheapest shell in each of the five columns is Codex CLI, Pi, OpenCode, Oh My Pi, and Oh My Pi — that’s four different shells. Codex CLI, the overall leader, only comes out cheapest in the GLM-5.3 column; on GLM-5.3-Flash it costs $0.226, just a bit more than Pi. So when you switch models, you have to re-evaluate the harness — you can’t just reuse your old pairing.

Conclusion 2: The cost gap from switching models is bigger than the gap from switching shells

Looking at the average cost per task by model, it runs from $0.185 for DeepSeek-V4.1-Flash-1017 to $1.12 for Kimi-K3 — about 6.1x. Looking at the overall average cost per task by harness, the lowest is Pi at $0.563 and the highest is Claude Code at $0.915 — a gap of less than 2x. In other words, on the final FrontierHarness leaderboard, which model you choose first has a bigger impact on task cost than which shell you choose first.

Conclusion 3: The vendor’s own harness didn’t win on price for its own model

In the DeepSeek-V4.1-Flash-1017 column, looking at cost per pass, the four DeepSeek Harness tiers are Code at $0.243 / 64.1%, Minimal at $0.256 / 63.7%, Creator at $0.273 / 64.9%, and Standard at $0.292 / 62.1%. It was neither the cheapest (OpenCode at $0.177) nor the highest pass rate (ZCode at 69.0%, 171/248—also the highest pass rate among all combinations in the entire field). Claude Code hit 68.1% in this cell, while Codex CLI was $0.237 / 67.3%. The most expensive in the same column, Cline CLI, used 6.77M tokens per trial and cost $0.860 per pass—about 4.9× OpenCode. That multiple applies only to this pair.

The Kimi-K3 column tells the same story: by cost per pass, the vendor’s own harness KimiCode is $1.84 / 60.1%, worse than Oh My Pi’s $1.45, and its pass rate is also below mini-SWE-agent’s 64.9% and Alma and ZCode’s 64.1%. The lowest pass rate in the entire field is also here: Exo + Kimi-K3, 43.5%, 108/248.


Scenario recommendations: picking the best harness by model

To turn the per-column conclusions above into practical choices, you can break it down by “which model you use + whether you care more about cost or pass rate.”

When you’re using GLM-5.3

  • If you care about cost per pass: Codex CLI, $0.803, 57.7%, about 1.86M tokens per trial.
  • If you care about pass rate: ZCode, 58.9%, but $1.48 per pass—nearly double Codex CLI.
  • One caveat: KimiCode in this column is $1.55 / 58.1%; the earlier interim claim no longer holds.

When you’re using GLM-5.3-Flash

  • If you care about cost: Pi ($0.223) and Codex CLI ($0.226) are essentially tied, and the latter has a slightly higher pass rate at 59.3%.
  • If you care about pass rate: Hermes Agent at 60.9%, $0.328 per pass; ZCode at 60.1%, $0.264 is the middle-ground option.

When you’re using DeepSeek-V4.1-Flash-1017

  • If you care about cost: OpenCode, $0.177, 61.3%.
  • If you care about pass rate: ZCode, 69.0%, at $0.222 per pass—only about $0.045 more than the cheapest. In this column, paying for pass rate is cheap.

When you’re using Muse Spark Open or Kimi-K3

  • Muse Spark Open: Oh My Pi is cheapest ($1.21); Grok Build has the highest pass rate (59.7%); Claude Code uses 9.79M tokens per trial and costs $3.75 per pass—the most expensive entry in this column.
  • Kimi-K3: Oh My Pi is cheapest ($1.45, 942k tokens per trial); mini-SWE-agent has the highest pass rate (64.9%); Codex CLI at $1.78 / 63.7% is competitive on both fronts.

which-harness-is-best-frontierharness-leaderboard-comparison-en-image-2


Decision Guide: Harness Protocols vs. APIYI Model IDs

The harnesses on the leaderboard don’t map one-to-one to platform endpoints. A strong score for Codex CLI doesn’t mean the model ID you’re holding has Responses enabled; same goes for Claude Code — check the Messages column first. Every harness burn tokens when it runs a task, and swapping the harness or the model directly changes your bill. So before you try a given cell, check the model page on APIYI apiyi.com to confirm whether that ID exposes Chat, Responses, or Messages, then point the harness’s base_url at https://api.apiyi.com/v1.

The table below shows default pricing from the APIYI docs as of 2026-11-15 (USD / 1M tokens). These are a separate set of numbers from the leaderboard list prices above — don’t mix the two.

Model ID Input Cache read Output Chat Responses Messages Group
glm-5.3 $1.4 $0.259 $4.396 Supported Not supported Supported ClaudeCode 0.95×, Default, SVIP
glm-5.3-flash $0.15 $0.03 $0.5 Supported Model page: not supported Model page: not supported CodexResponses, Default, SVIP
deepseek-v4.1-flash-1017 $0.3 $0.006 $1.2 Supported Not in summary table Supported ClaudeCode, Default, SVIP
deepseek-v4.1-flash $0.3 $0.006 $1.2 Supported See group Supported One extra: CodexResponses
kimi-k3 $3 $0.3 $15 Supported Not supported Not supported Default, SVIP
Muse Spark Open Not yet available — — — — — —

A few points need to be spelled out one by one:

  • glm-5.3 can go through Messages and has a ClaudeCode group at 0.95×, making it a good fit for Claude Code-style harnesses; that 0.95× applies only to glm-5.3. It can’t go through Responses.
  • glm-5.3-flash has two inconsistencies between the summary table and the model page, and we’re keeping both versions: the group listing includes CodexResponses, but the model page says both Responses and Messages are unsupported; the summary table’s type column says “text”, while the model page says the input includes text, images, and video.
  • deepseek-v4.1-flash-1017 can go through Messages, but the summary table doesn’t show CodexResponses; if you need Responses, look at the other entry, deepseek-v4.1-flash.
  • kimi-k3 only offers Chat Completions, so before using Codex CLI or Claude Code, confirm whether the harness can speak the Chat protocol.
  • The economy version deepseek-v4-1-flash-economy-261017 costs $0.24 for input, $0.96 for output, and $0.006 for cache reads, and only exposes chat. Per APIYI’s live updates, it loads the same V4.1-Flash-1017 weights as the standard version but generates more slowly. Important: FrontierHarness hasn’t tested this ID, so the pass rates on the leaderboard don’t apply to it.
  • Muse Spark Open isn’t available yet; we’ll get it listed as soon as we can. Models and prices are subject to the platform’s model list.

💡 Selection advice: First pick the candidate harnesses from the final leaderboard column for your model, then filter out combinations that can’t connect based on protocol, and finally run a small-scale trial with the same batch of tasks. Using one key on APIYI apiyi.com to switch between IDs like glm-5.3 and kimi-k3 lets you test the “swap model” and “swap harness” variables separately.

Your goal Which column to check first Final leaderboard candidates
Not sure which model to use — you just want a solid harness Overall summary Codex CLI (cheapest per pass), ZCode (highest pass rate)
Model already decided — you want to cut cost That model’s cost per pass See the “Cheapest” column in the five-model breakdown table
Model already decided — you want the highest pass rate That model’s pass rate See the “Highest” column in the five-model breakdown table
You want to shrink the overall budget Average cost per task by model Switch models first, then harnesses

FAQ

Q1: What is FrontierHarness, and how does it answer “which harness is best?”

FrontierHarness is a coding agent harness benchmark started by Runta. In v1.1, 21 harnesses each run 62 tasks across 5 models, 4 times per task, with pass rate and cost measured in the same environment. What it gives you isn’t a single champion — it’s a separate ranking column for each model.

Q2: So who’s actually first — Codex CLI or ZCode?

Both are first, just on different metrics. On cost per pass, Codex CLI leads at $0.937. On pass rate, ZCode leads at 61.9%. It depends on whether you care more about budget or success rate.

Q3: Can I still use v1.0’s 66.7% and the November 11 ranking as a reference?

No — not as a final leaderboard. v1.0 only tested Kimi K3 and 9 harnesses, for 360 runs total. November 11 was a mid-run snapshot of v1.1. The live leaderboard on 2026-11-15 showed 26,040 runs completed, and that’s the version to treat as final.

Q4: What should I watch out for when running GLM-5.3 with Claude Code?

The final leaderboard doesn’t list that particular cell — Claude Code on GLM-5.3 — so you can’t infer its score directly. At the protocol level, `glm-5.3` supports Messages, and on APIYI apiyi.com there’s a ClaudeCode group at 0.95×, so the prerequisites for hooking them up are all there.

Q5: How do I reproduce a specific cell in my own environment?

FrontierHarness offers a self-test entry point, so you can run it with your own model key in the same environment. On the platform side, the approach is: first confirm the model ID exposes the protocol the harness needs, then set the harness’s base_url to `https://api.apiyi.com/v1`. New users can use the free trial quota to run a few tasks first.

Q6: What are the limitations of this final leaderboard?

The dollar figures are list price multiplied by reported tokens, which differs from your actual platform bill. Every model ran at effort set to max, so results may shift at other settings. On top of that, some gaps between ranks fall within the 95% confidence interval — don’t read too much into adjacent positions.


Which Harness Is Best: Key Takeaways

  • No harness wins across the board: The cheapest harness differs for each of the five models, spread across four different ones.
  • Two firsts coexist: ZCode has the highest pass rate at 61.9%, while Codex CLI is the cheapest at $0.937 per pass.
  • The model affects cost more than the harness: Average task cost varies about 6.1x between models, but less than 2x between harnesses.
  • In-house harnesses don’t have an edge: By cost per pass, neither DeepSeek’s four tiers nor KimiCode came out cheapest on their own models.
  • Top cell: ZCode + DeepSeek-V4.1-Flash-1017, 69.0%, 171/248.
  • Check the protocol first: On the APIYI model page, confirm the Chat, Responses, and Messages columns before choosing a harness.

Summary

The final FrontierHarness v1.1 leaderboard’s answer to “which harness is best” is: it depends on the model.

  1. Two overall firsts: For pass rate, look at ZCode; for cost per pass, look at Codex CLI.
  2. Change the model, and you have to reconsider the harness: The cheapest harnesses in the five columns are Codex CLI, Pi, OpenCode, Oh My Pi, and Oh My Pi, respectively.
  3. For budget, look at the model first: The average task cost gap caused by the model is larger than the gap caused by switching harnesses.

When making an actual choice, first find the column for the model you’re using on the final leaderboard, then check whether the protocol connects. We recommend using APIYI at apiyi.com to switch model IDs through a unified interface and run small-scale trials on candidate combinations.

📚 References

  1. FrontierHarness Live Leaderboard: the v1.1 final leaderboard plus per-model breakdown data
  2. Link: frontierharness.org/live/
  3. Notes: as of 2026-11-15 it shows scored 26,040 — this is where every leaderboard figure in this article comes from

  4. FrontierHarness Self-Evaluation Entry: run it in the same environment with your own key

  5. Link: frontierharness.org/evaluate-your-own-harness/
  6. Notes: handy for reproducing a specific harness × model combination

  7. Guanlan Dai’s Announcement Post: the problem setup and scale details for v1.1

  8. Link: x.com/guanlan/status/2120192585235046712
  9. Notes: includes background such as “only 248 runs count as final” and the inference sponsors

  10. APIYI Model Directory: pricing, endpoints, and groups for each model ID

  11. Link: docs.apiyi.com/models
  12. Notes: the pricing data in this article is as of 2026-11-15 09:36 (UTC+8)

Author: APIYI Technical Team
Technical Discussion: Feel free to reach us through the APIYI homepage to talk about harness and model pairing choices

Related Articles

Similar Posts