Author’s note: In GDPx’s initial evaluation, no model fully passed either the finance or legal domain under the strict binary acceptance standard. This article reconstructs the specific errors in two public cases to explain why “being able to crunch numbers and draft contracts” doesn’t necessarily mean “being able to deliver acceptable work.”
On November 2, 2026, SemiAnalysis released GDPx v1.0, a benchmark of demanding professional tasks. Across its 11 domains, no tested model fully passed the finance or legal domain under the original binary pass standard. The research team instead used “average partial scores” to show the differences between models.
This isn’t an emotional declaration that “AI can’t do the job.” Some models earned an average partial score of 44% on the legal case and passed about half of the acceptance checks on the finance case. The question worth asking is: Why isn’t getting half of it right enough to deliver?
What you’ll learn: This article covers the key amounts, failure counts, and scoring criteria in the Bellwood-QoE finance task and the Series B financing charter amendment task. It also helps clarify the difference between AI’s ability to assist and its ability to deliver work independently and take responsibility for it.
Finance and Legal Tasks: Zero Complete Passes in GDPx
The first GDPx release covers 206 tasks across 11 professional fields, with a default time limit of eight hours per task. It tests the overall performance of the “model + Agent framework + tools + working environment.” Claude Opus 5.5, the top model on the overall leaderboard, scored just 12.0%. Against that backdrop, finance and legal stand out as two of the most striking fields.
| Metric | Finance | Legal |
|---|---|---|
| Number of tasks | 24 | 20 |
| Professional partner | Living Context | Chakra |
| Public example | Bellwood-QoE (quality-of-earnings analysis workbook) | quillon-series-b-redline (Series B charter revision) |
| Runs of public example | 25 selected runs | 20 runs across 4 models |
| Complete passes | 0 | 0 |
| Strict binary score | 0% | 0% |
| Partial scores | Best single run passed 8 of 11 checks | 18% average, 57% best single run |
Two points are worth keeping in mind when reading this table. First, a binary score of 0% doesn’t mean every result was worthless—the partial scores were clearly above zero. Second, the 25 and 20 runs in the table apply to individual public examples. They don’t mean the entire set of 24 finance tasks or 20 legal tasks was run only that many times.
Why Finance and Legal Tasks Use Strict Acceptance Criteria
In both fields, the deliverables are files intended to go straight into a real transaction workflow, not just pieces of advice. In M&A due diligence or financing negotiations, an error in the net debt calculation or a protective clause that can be circumvented could render an entire document unusable. That’s why GDPx uses strict acceptance criteria close to those applied to professional deliverables—and why the binary scores came out at 0%.
Finance Task Case Study: How Bellwood-QoE Got the Numbers Right but Failed to Deliver an Acceptable Result
Deal Background and Task Requirements
Living Context turned real-world deal experience into an anonymized test environment: private equity firm Bannock Capital acquired approximately 62% of Haverly Snack Company for around $880 million, with an additional contingent consideration arrangement. The environment contains roughly 4,000 anonymized documents, including financial ledgers, bank statements, payroll records, accounts receivable aging reports, contracts and correspondence, transaction agreements, historical adjustment schedules, and working papers from multiple advisors.
The model had to take over an unfinished quality of earnings (QoE) workbook. It needed to identify which existing data was correct, fix incorrect figures and classifications, and preserve correct entries and the calculation structure. It also had to review EBITDA adjustments, align 12-month working capital, address the net debt definition, provide documentary evidence for each adjustment, and ensure references were consistent across worksheets. The deliverable was the revised workbook itself.
Bellwood-QoE Test Results
| Model | Average checks passed out of 11 | Passed all checks? |
|---|---|---|
| GPT-6.1 Sol | 5.6 | No |
| Claude Opus 5.5 | 5.4 | No |
| GPT-6 Astra | 5.4 | No |
| GLM 5.3 | Fewer than 2 | No |
| Grok 4.7 | Fewer than 2 | No |
| Best individual run | 8 | No |
The strongest models passed about half the checks on average, and the best individual run passed 8. But if any of the remaining checks reveals a critical financial error, the workbook can’t be delivered. The following three types of errors show where things went wrong.
Financial Error 1: A $418 Adjustment Was Overinterpreted
Some models interpreted a waived $418 insurance audit adjustment to mean that “the entire prior workers’ compensation insurance policy obligation had ended.” As a result, they understated trailing 12-month EBITDA by $4,728. This misinterpretation appeared in 10 of the 15 relevant runs across Opus, Astra, and Sol.
This wasn’t a multiplication error. The models took the accounting and legal meaning of one specific adjustment and extended it to the entire historical insurance liability. The $4,728 discrepancy was a consequence of that misinterpretation, not a multiple of $418. This kind of error is hard to catch by repeatedly checking the arithmetic, because the calculations themselves may be correct.
Financial Error 2: Transaction Fees Included in Net Debt
In 15 runs, the model included the banker’s success fee in net debt, even though the purchase agreement specified that it should be treated separately as a transaction expense. The model found the fee and got the arithmetic right, but it didn’t follow the classification required by the binding agreement.
That’s the gap between having the data and classifying it correctly. In an acquisition, net debt typically affects what the buyer actually pays, so a classification error has consequences beyond the spreadsheet.
Financial Error 3: Missing Expense Adjustments
The research team also found a set of subtler problems:
- In 17 runs, the model still treated compensation paid to family members who didn’t actually work for the business as a normal operating expense;
- Most runs failed to identify the owner’s personal expenses recorded only in the general ledger’s memo field;
- In 10 of the strongest model’s 15 runs, it missed a $9,192 adjustment for a historical promotional expense, counting an expense from another period in the current 12-month period.
These errors involve accounting periods, the nature of expenses, sources of evidence, and the economic substance of the transaction. The point of a quality of earnings analysis is to strip out items like these that look like operating expenses but aren’t. Miss them, and EBITDA won’t reflect the business’s true profitability.
Legal Task Case Study: Why All 20 Series B Charter Revisions Were Unusable
The Legal Task
In the quillon-series-b-redline task, the AI represented the lead investor in a Series B financing and revised the company charter before signing. The input materials included a signed term sheet, a capitalization table, client emails, a draft charter, and other transaction documents. Success meant protecting the investor’s legitimate interests without needlessly changing the commercial terms already agreed upon. This was a legal judgment task requiring cross-checks across multiple documents, not an editing exercise.
Results and Three Key Errors
| Metric | Result |
|---|---|
| Fully met the usability standard | 0/20 runs |
| Average partial score | 18% |
| Partial scores by model | Opus 5.5: 44%, GPT-6 Astra: 25%, Gemini 4 Argon: 2%, GLM 5.3: 0.5% |
| Highest partial score in a single run | 57% (Opus 5.5) |
| Failed to identify that shareholder votes could remove protective provisions | 13/20 runs |
| Failed to prevent circumvention through subsidiaries | 20/20 runs |
| Failed to flag conflicts between the client email and the term sheet | 14/20 runs |
The 18% figure is the average partial score for the runs in this case, not an overall score for GDPx’s legal domain.
Error 1: Reading the provisions without counting the votes. In 13 runs, the AI failed to notice that other shareholders had enough voting power to remove protections needed by the lead investor. Having protective provisions in the charter doesn’t mean those protections are secure. Only by calculating voting rights against the capitalization table can you determine whether the provisions could be overturned.
Error 2: Editing the text without preventing workarounds. All 20 runs failed to effectively prevent the company from using subsidiaries to circumvent the investor protections in the charter. If the provisions bind only the company itself and don’t cover subsidiaries, the same business activity can be carried out through another entity. Understanding the literal meaning of a provision isn’t the same as spotting how it could be evaded in practice.
Error 3: Treating the email as the final agreement. In 14 runs, the AI adopted the terms requested in the client email without pointing out that they conflicted with the signed term sheet. Making those changes amounts to unilaterally renegotiating agreed transaction terms. The right approach is to identify the conflict and alert the user, not accept every request at face value.
How the Legal Scoring Process Controls for Reliability
Chakra’s scoring process includes having legal professionals set the requirements, calibrating the scoring criteria with valid, invalid, and partially correct samples, manually reviewing disputed judgments, and checking the scoring model for bias. Each output is scored five times, and consistency is tracked for each acceptance criterion. These are internal quality controls described by the partner, however; they don’t mean that a fully independent organization has verified that the scoring is free of errors.
Why Financial and Legal Tasks Fail: It’s Not the Writing, It’s the Judgment
Living Context groups common errors in the financial cases into four categories, closely matching what it observed in the legal cases:
| Failure mode | Financial example | Legal counterpart | Professional takeaway |
|---|---|---|---|
| Misjudging source authority | Favoring an old table with clearly stated figures while overlooking a signed agreement | Prioritizing a client email over the term sheet | Establish an order of precedence for documents first |
| Overlooking second-order effects | Completing the calculation but missing its impact on taxes, debt, and transaction obligations | Changing protective provisions without accounting for the voting structure | Check the downstream effects of every change |
| Checking calculations but not assumptions | Repeatedly verifying the arithmetic while relying on incorrect business assumptions | Using precise wording that doesn’t cover the subsidiary scenario | Reviewing assumptions matters more than checking calculations |
| Breaking the existing structure during revisions | Hard-coding numbers in place of formulas, severing links between worksheets | Changing commercial terms that had already been agreed | Deliverables must be corrected without breaking what already works |
These are failure patterns identified by the research team, not a universal error rate for all corporate financial models or legal documents. They point to a common issue: the hardest part often isn’t generating text, but selecting sources, applying professional reasoning, recognizing implicit constraints, maintaining consistency across documents, and checking the final deliverable. The more fluent a model’s response sounds, the easier these issues are to overlook.
Impact on Financial and Legal AI Applications: Distinguishing Assistance from Deliverables
What This Means for Enterprise AI Leaders and Developers
GDPx doesn’t dispute the value of AI-assisted professional work. The partial scores show that models can already meet some acceptance criteria. But the study provides no basis for estimating how much time or money human involvement might save, so its case findings shouldn’t be converted directly into ROI. A more practical approach is to assign authorization levels based on risk:
| Authorization level | Typical uses | Human role | Acceptance criteria |
|---|---|---|---|
| L1 Research assistance | Initial information gathering and organization, anomalous data checks | Spot-check | Results serve only as leads |
| L2 Discrepancy alerts | Flagging differences in terms and data, conflict lists | Confirm each item | Professionals determine validity |
| L3 Draft preparation | Initial drafts of workbook revisions and charter redlines | Full review | Check each item against a checklist |
| L4 Independent delivery | Signing or submitting directly to external parties | — | GDPx cases suggest this level shouldn’t currently be authorized |
This framework is consistent with the direction of external guidance. In Formal Opinion 512, issued on July 29, 2024, the American Bar Association (ABA) noted that lawyers using generative AI must still consider their responsibilities for professional competence, client confidentiality, communication, supervision, and reasonable fees. They should also understand the tool’s capabilities and limitations rather than adopt its output without verification. The NIST AI RMF Generative AI Profile, issued on July 26, 2024, likewise emphasizes that organizations should tailor human oversight, evaluation, documentation, and governance to the context of use. Note that the ABA opinion is professional ethics guidance for US lawyers, and the NIST document is not a GDPx test result. Both serve only as cross-domain references.
Build a “Conflict-Flagging” Workflow With an API
One of the most common mistakes in the GDPx legal case is failing to flag conflicts between a client’s email and signed terms. This kind of work—spotting differences and raising questions—is well suited to L2-level assistance, as long as the output is treated only as leads for human review. The example below asks a model to compare two texts and list conflicts and items that need human confirmation:
import openai
client = openai.OpenAI(
api_key="YOUR_API_KEY",
base_url="https://api.apiyi.com/v1" # APIYI OpenAI-compatible endpoint
)
resp = client.chat.completions.create(
model="claude-sonnet-4-20250514",
messages=[{"role": "user", "content": "对照【已签署条款清单】与【客户邮件】，只列出冲突项和需人工确认的问题，不要直接修改条款。\n条款清单：...\n客户邮件：..."}]
)
print(resp.choices[0].message.content)
View the complete implementation (multi-model cross-checking + human review checklist)
import openai
from typing import List, Dict
client = openai.OpenAI(
api_key="YOUR_API_KEY",
base_url="https://api.apiyi.com/v1" # Access models from multiple providers with one key
)
SYSTEM = (
"你是交易文件审查助手。你的输出只作为专业人员的审查线索，"
"必须遵守：1) 已签署文件效力优先于邮件和旧表格；"
"2) 发现冲突时提示，不得自行改写商业条款；"
"3) 每条结论标注依据的文件名和段落；"
"4) 无法确认的事项明确写'需人工确认'。"
)
CHECKS = [
"客户邮件中的要求是否与已签署条款清单冲突",
"结合股权结构，其他股东能否凭投票权移除保护条款",
"保护条款是否覆盖子公司，是否存在绕行路径",
]
def review(docs: str, models: List[str]) -> Dict[str, str]:
"""Generate review leads with multiple models for human cross-checking."""
prompt = "请逐项检查：\n" + "\n".join(f"- {c}" for c in CHECKS) + f"\n\n材料：\n{docs}"
results = {}
for m in models:
try:
r = client.chat.completions.create(
model=m,
messages=[{"role": "system", "content": SYSTEM},
{"role": "user", "content": prompt}],
temperature=0
)
results[m] = r.choices[0].message.content
except Exception as e:
results[m] = f"调用失败: {e}"
return results
if __name__ == "__main__":
out = review("条款清单：...\n股权结构表：...\n客户邮件：...\n章程草稿：...",
["claude-sonnet-4-20250514", "gpt-4o"])
for model, text in out.items():
print(f"===== {model} =====\n{text}\n")
# A lawyer or financial professional makes the final decision using the acceptance checklist
🚀 Quick-start tip: When building this kind of workflow, you can use a single API key through APIYI at apiyi.com to access multiple leading models and compare how well they identify conflicts in your own de-identified samples. New users can use free test credits for small-scale validation before deciding whether to pilot the workflow.
Impact on Finance and Legal Technology
GDPx’s results remind the industry that evaluating professional AI products takes more than checking whether a conversational demo runs smoothly. The real question is whether the final deliverable can pass professional review. For vendors, competitiveness will increasingly depend on engineering details such as “prioritizing authoritative documents,” “flagging conflicts,” “preserving existing formula structures,” and “tracing evidence to its source.” For buyers, the takeaway is to run independent tests using tasks that closely match their own work. GDPx itself has only released some of its tasks and execution traces, so further external scrutiny and replication are still needed.
💡 Selection tip: Model performance varies significantly across domains, and a single overall ranking won’t tell you how a model will perform for your business. Use APIYI’s unified interface at apiyi.com to test multiple models in parallel on your own tasks. Record partial scores and whether they pass critical checks—don’t just look at the average score. Check the platform’s live model list for current availability.
Frequently Asked Questions
Q1: Does “complete failure” on financial and legal tasks mean AI got nothing right?
No. The 0% result reflects strict pass-or-fail grading: no execution met all critical requirements. In the financial case, the best individual execution passed 8 of 11 checks. In the legal case, Opus 5.5 received an average partial score of 44%. Partial completion and an acceptable deliverable are two different standards.
Q2: Why can AI get the math right but still reach the wrong answer?
The error often lies in a business assumption, not the calculation itself. For example, a $418 waiver adjustment was interpreted as terminating the obligations under the entire policy, causing EBITDA to be understated by $4,728. In another case, the banker’s success fee was correctly identified but classified as net debt. Rechecking the math won’t catch errors like these.
Q3: Do GDPx’s findings apply to all financial and legal AI applications?
No. These results come from specific, highly challenging tasks and scoring rules. The causes of failure identified in the cases can’t be generalized to the entire industry either. Less demanding applications or those involving human collaboration need to be evaluated separately.
Q4: What financial and legal work is AI suited for today?
Reasonable uses include organizing initial materials, flagging differences in terms and data, preparing first drafts, checking for data anomalies, and helping create review checklists. Whether AI is suitable for a specific use case requires independent testing against its risk level and professional acceptance criteria.
Q5: How can I test a model’s performance on my own business tasks at low cost?
You can follow these steps:
1. Prepare several anonymized real-world samples and write an itemized acceptance checklist for each one;
2. Register at APIYI apiyi.com, get an API key, and use the free testing credits;
3. Set `base_url` in this article’s code to `https://api.apiyi.com/v1` and call multiple models in parallel;
4. Record partial scores and whether each critical item passed separately, then have a professional make the final assessment.
Q6: Can I use the task costs in the GDPx report as a reference for API pricing?
No. The dollar amounts in GDPx represent the average cost per task under benchmark conditions, not a standard rate charged by any platform. Actual costs depend on the platform’s current prices and your usage. You can check pay-as-you-go billing details in the APIYI apiyi.com console.
Key Takeaways from AI Evaluations for Finance and Legal Tasks
- No model fully passed under strict acceptance criteria: In GDPx finance and legal tasks, no model passed in full under the binary standard.
- Partial scores weren’t zero: The best single finance run passed 8/11 checks, while the highest single legal score was 57%.
- The error was in judgment, not arithmetic: Misreading a $418 adjustment led to EBITDA being understated by $4,728.
- Document authority matters: In 14/20 runs, the model prioritized a client email over a signed term sheet.
- Implicit constraints were hardest to cover: All 20/20 runs failed to prevent the risk of circumvention through a subsidiary.
- Three concepts shouldn’t be conflated: Partial scores, full passes, and production readiness need separate evaluation.
- Start with assistance, then pilot: Use APIYI’s unified multi-model interface at apiyi.com to conduct staged validation on your own samples.
Summary
The main lessons from GDPx finance and legal tasks come down to three points:
- Fluency doesn’t equal competence: Even when tables are calculated correctly and clauses read smoothly, models can still get the economic substance, ownership structure, or conflicts between documents wrong;
- Professional deliverables depend on judgment: The hard parts are assessing document authority, second-order effects, implicit constraints, and consistency across documents;
- Set levels of authority and clear acceptance criteria: Following the approaches of the ABA and NIST, keep human review in place and use AI for assistance and first drafts.
AI can already do some valuable work on complex professional tasks, but there’s still a clear gap before it can independently produce deliverables and take responsibility for them. If you’re evaluating AI for finance or legal use cases, consider using APIYI at apiyi.com to compare multiple models on partial scores and critical-check pass rates using your own business samples. Then decide where to set the boundaries of their authority.
📚 References
- SemiAnalysis GDPx Original Report: Finance and legal sections, case studies, and failure modes
- Link:
newsletter.semianalysis.com/p/gdpx-measuring-frontier-model-capabilities -
Details: Primary source for the Bellwood-QoE and quillon-series-b-redline case data
-
GDPx Domain Descriptions and Public Execution Traces: Task counts, partners, and Agent traces
- Links:
gdpx.semianalysis.com/domains,gdpx.semianalysis.com/rollouts -
Details: View execution traces for some public tasks
-
ABA Formal Opinion 512 News Release: Professional ethics guidance for lawyers using generative AI
- Link:
americanbar.org/news/abanews/aba-news-archives/2024/07/aba-issues-first-ethics-guidance-ai-tools/ -
Details: Applies to US lawyers and offers a reference for principles of human review
-
NIST AI RMF Generative AI Profile: A risk management framework for generative AI
- Link:
nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence - Details: A cross-domain reference for human oversight, evaluation, and governance measures
Author: APIYI Technical Team
Technical discussions: Contact us through the APIYI homepage. For more integration resources, visit the docs.apiyi.com documentation center.
