Author’s note: This article only interprets the final leaderboard after all 26,040 runs of FrontierHarness v1.1 finished — comparing 21 harnesses side by side, broken down by model, and cross-checking against the protocols and prices actually available for the relevant model IDs on APIYI.
If you’re already using coding agents like Codex CLI, Claude Code, OpenCode, or Pi, and you frequently switch the underlying model, then the question “which harness is best” probably doesn’t have a single answer. On 2026-11-10, Guanlan Dai of Runta launched FrontierHarness v1.1, and the question it posed outright was: when you switch models, should you switch harnesses too? As of 2026-11-15, the live leaderboard shows running 0, scored 26,040 — 21 harnesses × 5 models × 62 tasks × 4 attempts have all finished running.
This is not a leaderboard write-up claiming “one harness takes first across the board”. On the final leaderboard, the highest pass rate and the lowest cost per pass don’t belong to the same harness, and the cheapest harness for each of the five models isn’t the same one either. This article breaks these conclusions down by model and explains them cell by cell.
The key takeaway: after reading this, you’ll have a model-by-model comparison of the FrontierHarness final leaderboard, you’ll know which column to look at when you switch models, which old numbers you can’t use anymore, and which protocol you need to verify before integrating.

FrontierHarness Final Leaderboard Methodology: Check the Rules Before You Decide Which Harness Is Best
All FrontierHarness runs execute inside the Runta runtime and use the same set of resource limits. Each harness × model combination has to complete all 248 runs before it counts as a final result, so rankings during the live stream keep shifting as new results come in. The table below lays out the scoring rules for the v1.1 final leaderboard; all comparisons later in this piece use it as the reference.
| Criterion | v1.1 final leaderboard setting | Why it matters when comparing |
|---|---|---|
| Dataset | [email protected], 62 tasks |
All harnesses run the same task set |
| Scale | 21 harnesses, 4 attempts per task, 248 per combination, 5,208 per model, 26,040 total | Every cell has the same denominator, so you can compare across the board |
| Runtime environment | 1–4 vCPU, 2–8 GB memory, 32 GB disk, per-task timeout of 15 minutes to 8 hours | Eliminates machine differences |
| Inference settings | Inference provider Baseten; all five models use effort max | Model-side variables are unified |
| Pass rate | passes ÷ counted attempts; timeouts and reward hacking count as failures, infrastructure errors are excluded | Peeking at answers earns no points |
| Cost | list price × reported tokens | It’s a leaderboard algorithm, not any platform’s bill |
| Confidence | 95% bootstrap by task | Read very close rankings with caution |
A few sets of numbers get passed around a lot, but none of them are from the v1.1 final leaderboard. v1.0 tested only 9 harnesses on a single model, Kimi K3, with 12 configurations and 360 runs. Codex’s 66.7% pass rate and $3.47 median cost per task are both v1.0 numbers. In the 2026-11-11 mid-run snapshot, cost per pass ranged from $0.57 to $1.32, and pass rates ranged from 56.2% to 77.6%; that too reflects only the halfway mark. On the same day, someone wrote that “GLM-5.3 does best with KimiCode, and Kimi-K3 does best with Codex.” Once you re-rank by the final leaderboard’s cost per pass, neither claim holds.
Which Harness Is Best: FrontierHarness Final Leaderboard Comparison
The table below brings together the results for 21 harnesses across five models, sorted from lowest to highest cost per pass. All dollar figures come from the FrontierHarness final leaderboard and are calculated using list prices multiplied by reported token counts.
| Rank | Harness | Cost per pass | Average cost per task | Pass rate |
|---|---|---|---|---|
| 1 | Codex CLI | $0.937 | $0.573 | 61.1% |
| 2 | Pi | $0.976 | $0.563 | 57.7% |
| 3 | Oh My Pi | $0.983 | $0.564 | 57.3% |
| 4 | ZCode | $1.06 | $0.659 | 61.9% |
| 5 | KimiCode | $1.09 | $0.653 | 60.2% |
| 6 | OpenCode | $1.12 | $0.646 | 57.4% |
| 7 | Maka | $1.13 | $0.654 | 57.7% |
| 8 | MiniMax Code | $1.14 | $0.647 | 56.8% |
| 9 | DeepSeek Harness: Minimal | $1.15 | $0.661 | 57.4% |
| 10 | Hermes Agent | $1.16 | $0.692 | 59.9% |
| 11 | Cline CLI | $1.19 | $0.689 | 57.7% |
| 12 | Grok Build | $1.20 | $0.723 | 60.0% |
| 13 | Alma | $1.22 | $0.731 | 59.7% |
| 14 | DeepSeek Harness: Standard | $1.27 | $0.697 | 55.0% |
| 15 | DeepSeek Harness: Creator | $1.27 | $0.713 | 56.1% |
| 16 | mini-SWE-agent | $1.29 | $0.757 | 58.6% |
| 17 | DeepSeek Harness: Code | $1.32 | $0.732 | 55.2% |
| 18 | Exo | $1.36 | $0.691 | 50.6% |
| 19 | GitHub Copilot CLI | $1.38 | $0.787 | 56.9% |
| 20 | Claude Code | $1.55 | $0.915 | 58.9% |
| 21 | Antigravity CLI | $1.63 | $0.897 | 55.0% |
The Two No. 1 Spots in the Final Overall Leaderboard
The highest pass rate overall is ZCode’s (61.9%), and the cheapest cost per pass is Codex CLI’s ($0.937). You have to read those two facts together. Codex CLI’s pass rate is 61.1%, just 0.8 percentage points below ZCode, but it costs about $0.12 less per pass; ZCode is the steadier choice when you’re more focused on passing more tasks. The most expensive option, Antigravity CLI, costs $1.63 per pass, about 1.74 times Codex CLI. Pass rates across the entire field range from Exo’s 50.6% to ZCode’s 61.9%, an 11.3-percentage-point gap.
There are two easy misreads. First, the four DeepSeek Harness entries are four configurations of the same version, 0.2.1-rc.3 (Minimal, Standard, Creator, and Code), not four separate products. Second, Alma ranks 13th by cost per pass but 6th by pass rate, behind ZCode, Codex CLI, KimiCode, Grok Build, and Hermes Agent. The November 11 snapshot’s claim that Alma ranked No. 3 with 72.7% is interim data; the final leaderboard’s 59.7% is the number to use.
Which Harness Is Best After Switching Models: the FrontierHarness Five-Model Breakdown
A site-wide summary only tells you the average. What actually drives your bill is the column for the model you’re running. In the table below, “cheapest” always means the harness with the lowest cost per pass on that model’s page, and every pass-rate denominator is 248.
| Model | Average cost per task (averaged across harnesses) | Cheapest per pass | Highest pass rate | Most expensive on the same model |
|---|---|---|---|---|
| GLM-5.3 | $0.968 | Codex CLI $0.803, 57.7% | ZCode 58.9%, $1.48 | Exo $2.72, 50.0% |
| GLM-5.3-Flash | $0.193 | Pi $0.223, 58.1% | Hermes Agent 60.9%, $0.328 | DeepSeek Harness: Minimal $0.484, 55.2% |
| DeepSeek-V4.1-Flash-1017 | $0.185 | OpenCode $0.177, 61.3% | ZCode 69.0%, $0.222 | Cline CLI $0.860, 63.3% |
| Muse Spark Open | $1.01 | Oh My Pi $1.21, 58.9% | Grok Build 59.7%, $1.77 | Claude Code $3.75, 52.8% |
| Kimi-K3 | $1.12 | Oh My Pi $1.45, 57.7% | mini-SWE-agent 64.9%, $2.02 | Antigravity CLI $2.73, 56.9% |

Conclusion 1: The cheapest harness isn’t the same one across all five models
The cheapest shell in each of the five columns is Codex CLI, Pi, OpenCode, Oh My Pi, and Oh My Pi — that’s four different shells. Codex CLI, the overall leader, only comes out cheapest in the GLM-5.3 column; on GLM-5.3-Flash it costs $0.226, just a bit more than Pi. So when you switch models, you have to re-evaluate the harness — you can’t just reuse your old pairing.
Conclusion 2: The cost gap from switching models is bigger than the gap from switching shells
Looking at the average cost per task by model, it runs from $0.185 for DeepSeek-V4.1-Flash-1017 to $1.12 for Kimi-K3 — about 6.1x. Looking at the overall average cost per task by harness, the lowest is Pi at $0.563 and the highest is Claude Code at $0.915 — a gap of less than 2x. In other words, on the final FrontierHarness leaderboard, which model you choose first has a bigger impact on task cost than which shell you choose first.
Conclusion 3: The vendor’s own harness didn’t win on price for its own model
In the DeepSeek-V4.1-Flash-1017 column, looking at cost per pass, the four DeepSeek Harness tiers are Code at $0.243 / 64.1%, Minimal at $0.256 / 63.7%, Creator at $0.273 / 64.9%, and Standard at $0.292 / 62.1%. It was neither the cheapest (OpenCode at $0.177) nor the highest pass rate (ZCode at 69.0%, 171/248—also the highest pass rate among all combinations in the entire field). Claude Code hit 68.1% in this cell, while Codex CLI was $0.237 / 67.3%. The most expensive in the same column, Cline CLI, used 6.77M tokens per trial and cost $0.860 per pass—about 4.9× OpenCode. That multiple applies only to this pair.
The Kimi-K3 column tells the same story: by cost per pass, the vendor’s own harness KimiCode is $1.84 / 60.1%, worse than Oh My Pi’s $1.45, and its pass rate is also below mini-SWE-agent’s 64.9% and Alma and ZCode’s 64.1%. The lowest pass rate in the entire field is also here: Exo + Kimi-K3, 43.5%, 108/248.
Scenario recommendations: picking the best harness by model
To turn the per-column conclusions above into practical choices, you can break it down by “which model you use + whether you care more about cost or pass rate.”
When you’re using GLM-5.3
- If you care about cost per pass: Codex CLI, $0.803, 57.7%, about 1.86M tokens per trial.
- If you care about pass rate: ZCode, 58.9%, but $1.48 per pass—nearly double Codex CLI.
- One caveat: KimiCode in this column is $1.55 / 58.1%; the earlier interim claim no longer holds.
When you’re using GLM-5.3-Flash
- If you care about cost: Pi ($0.223) and Codex CLI ($0.226) are essentially tied, and the latter has a slightly higher pass rate at 59.3%.
- If you care about pass rate: Hermes Agent at 60.9%, $0.328 per pass; ZCode at 60.1%, $0.264 is the middle-ground option.
When you’re using DeepSeek-V4.1-Flash-1017
- If you care about cost: OpenCode, $0.177, 61.3%.
- If you care about pass rate: ZCode, 69.0%, at $0.222 per pass—only about $0.045 more than the cheapest. In this column, paying for pass rate is cheap.
When you’re using Muse Spark Open or Kimi-K3
- Muse Spark Open: Oh My Pi is cheapest ($1.21); Grok Build has the highest pass rate (59.7%); Claude Code uses 9.79M tokens per trial and costs $3.75 per pass—the most expensive entry in this column.
- Kimi-K3: Oh My Pi is cheapest ($1.45, 942k tokens per trial); mini-SWE-agent has the highest pass rate (64.9%); Codex CLI at $1.78 / 63.7% is competitive on both fronts.

Decision Guide: Harness Protocols vs. APIYI Model IDs
The harnesses on the leaderboard don’t map one-to-one to platform endpoints. A strong score for Codex CLI doesn’t mean the model ID you’re holding has Responses enabled; same goes for Claude Code — check the Messages column first. Every harness burn tokens when it runs a task, and swapping the harness or the model directly changes your bill. So before you try a given cell, check the model page on APIYI apiyi.com to confirm whether that ID exposes Chat, Responses, or Messages, then point the harness’s base_url at https://api.apiyi.com/v1.
The table below shows default pricing from the APIYI docs as of 2026-11-15 (USD / 1M tokens). These are a separate set of numbers from the leaderboard list prices above — don’t mix the two.
| Model ID | Input | Cache read | Output | Chat | Responses | Messages | Group |
|---|---|---|---|---|---|---|---|
glm-5.3 |
$1.4 | $0.259 | $4.396 | Supported | Not supported | Supported | ClaudeCode 0.95×, Default, SVIP |
glm-5.3-flash |
$0.15 | $0.03 | $0.5 | Supported | Model page: not supported | Model page: not supported | CodexResponses, Default, SVIP |
deepseek-v4.1-flash-1017 |
$0.3 | $0.006 | $1.2 | Supported | Not in summary table | Supported | ClaudeCode, Default, SVIP |
deepseek-v4.1-flash |
$0.3 | $0.006 | $1.2 | Supported | See group | Supported | One extra: CodexResponses |
kimi-k3 |
$3 | $0.3 | $15 | Supported | Not supported | Not supported | Default, SVIP |
| Muse Spark Open | Not yet available | — | — | — | — | — | — |
A few points need to be spelled out one by one:
glm-5.3can go through Messages and has a ClaudeCode group at 0.95×, making it a good fit for Claude Code-style harnesses; that 0.95× applies only toglm-5.3. It can’t go through Responses.glm-5.3-flashhas two inconsistencies between the summary table and the model page, and we’re keeping both versions: the group listing includes CodexResponses, but the model page says both Responses and Messages are unsupported; the summary table’s type column says “text”, while the model page says the input includes text, images, and video.deepseek-v4.1-flash-1017can go through Messages, but the summary table doesn’t show CodexResponses; if you need Responses, look at the other entry,deepseek-v4.1-flash.kimi-k3only offers Chat Completions, so before using Codex CLI or Claude Code, confirm whether the harness can speak the Chat protocol.- The economy version
deepseek-v4-1-flash-economy-261017costs $0.24 for input, $0.96 for output, and $0.006 for cache reads, and only exposes chat. Per APIYI’s live updates, it loads the same V4.1-Flash-1017 weights as the standard version but generates more slowly. Important: FrontierHarness hasn’t tested this ID, so the pass rates on the leaderboard don’t apply to it. - Muse Spark Open isn’t available yet; we’ll get it listed as soon as we can. Models and prices are subject to the platform’s model list.
💡 Selection advice: First pick the candidate harnesses from the final leaderboard column for your model, then filter out combinations that can’t connect based on protocol, and finally run a small-scale trial with the same batch of tasks. Using one key on APIYI apiyi.com to switch between IDs like
glm-5.3andkimi-k3lets you test the “swap model” and “swap harness” variables separately.
| Your goal | Which column to check first | Final leaderboard candidates |
|---|---|---|
| Not sure which model to use — you just want a solid harness | Overall summary | Codex CLI (cheapest per pass), ZCode (highest pass rate) |
| Model already decided — you want to cut cost | That model’s cost per pass | See the “Cheapest” column in the five-model breakdown table |
| Model already decided — you want the highest pass rate | That model’s pass rate | See the “Highest” column in the five-model breakdown table |
| You want to shrink the overall budget | Average cost per task by model | Switch models first, then harnesses |
FAQ
Q1: What is FrontierHarness, and how does it answer “which harness is best?”
FrontierHarness is a coding agent harness benchmark started by Runta. In v1.1, 21 harnesses each run 62 tasks across 5 models, 4 times per task, with pass rate and cost measured in the same environment. What it gives you isn’t a single champion — it’s a separate ranking column for each model.
Q2: So who’s actually first — Codex CLI or ZCode?
Both are first, just on different metrics. On cost per pass, Codex CLI leads at $0.937. On pass rate, ZCode leads at 61.9%. It depends on whether you care more about budget or success rate.
Q3: Can I still use v1.0’s 66.7% and the November 11 ranking as a reference?
No — not as a final leaderboard. v1.0 only tested Kimi K3 and 9 harnesses, for 360 runs total. November 11 was a mid-run snapshot of v1.1. The live leaderboard on 2026-11-15 showed 26,040 runs completed, and that’s the version to treat as final.
Q4: What should I watch out for when running GLM-5.3 with Claude Code?
The final leaderboard doesn’t list that particular cell — Claude Code on GLM-5.3 — so you can’t infer its score directly. At the protocol level, `glm-5.3` supports Messages, and on APIYI apiyi.com there’s a ClaudeCode group at 0.95×, so the prerequisites for hooking them up are all there.
Q5: How do I reproduce a specific cell in my own environment?
FrontierHarness offers a self-test entry point, so you can run it with your own model key in the same environment. On the platform side, the approach is: first confirm the model ID exposes the protocol the harness needs, then set the harness’s base_url to `https://api.apiyi.com/v1`. New users can use the free trial quota to run a few tasks first.
Q6: What are the limitations of this final leaderboard?
The dollar figures are list price multiplied by reported tokens, which differs from your actual platform bill. Every model ran at effort set to max, so results may shift at other settings. On top of that, some gaps between ranks fall within the 95% confidence interval — don’t read too much into adjacent positions.
Which Harness Is Best: Key Takeaways
- No harness wins across the board: The cheapest harness differs for each of the five models, spread across four different ones.
- Two firsts coexist: ZCode has the highest pass rate at 61.9%, while Codex CLI is the cheapest at $0.937 per pass.
- The model affects cost more than the harness: Average task cost varies about 6.1x between models, but less than 2x between harnesses.
- In-house harnesses don’t have an edge: By cost per pass, neither DeepSeek’s four tiers nor KimiCode came out cheapest on their own models.
- Top cell: ZCode + DeepSeek-V4.1-Flash-1017, 69.0%, 171/248.
- Check the protocol first: On the APIYI model page, confirm the Chat, Responses, and Messages columns before choosing a harness.
Summary
The final FrontierHarness v1.1 leaderboard’s answer to “which harness is best” is: it depends on the model.
- Two overall firsts: For pass rate, look at ZCode; for cost per pass, look at Codex CLI.
- Change the model, and you have to reconsider the harness: The cheapest harnesses in the five columns are Codex CLI, Pi, OpenCode, Oh My Pi, and Oh My Pi, respectively.
- For budget, look at the model first: The average task cost gap caused by the model is larger than the gap caused by switching harnesses.
When making an actual choice, first find the column for the model you’re using on the final leaderboard, then check whether the protocol connects. We recommend using APIYI at apiyi.com to switch model IDs through a unified interface and run small-scale trials on candidate combinations.
📚 References
- FrontierHarness Live Leaderboard: the v1.1 final leaderboard plus per-model breakdown data
- Link:
frontierharness.org/live/ -
Notes: as of 2026-11-15 it shows scored 26,040 — this is where every leaderboard figure in this article comes from
-
FrontierHarness Self-Evaluation Entry: run it in the same environment with your own key
- Link:
frontierharness.org/evaluate-your-own-harness/ -
Notes: handy for reproducing a specific harness × model combination
-
Guanlan Dai’s Announcement Post: the problem setup and scale details for v1.1
- Link:
x.com/guanlan/status/2120192585235046712 -
Notes: includes background such as “only 248 runs count as final” and the inference sponsors
-
APIYI Model Directory: pricing, endpoints, and groups for each model ID
- Link:
docs.apiyi.com/models - Notes: the pricing data in this article is as of 2026-11-15 09:36 (UTC+8)
Author: APIYI Technical Team
Technical Discussion: Feel free to reach us through the APIYI homepage to talk about harness and model pairing choices
