Author’s note: from acceptance criteria and data isolation to independent review — five steps for designing external verification of deep AI tasks, so you can avoid the false sense of completion that comes from models grading their own work.
The model says “task complete, results are accurate” — and then the whole thing falls apart the moment you try to reproduce it. That’s the wall many AI users hit when they’re working on deep tasks. This article walks through a design method for external verification that runs independently, turning “the model says it’s done” into “the evidence shows it’s done.”
In a practice guide published in October 2026, Claude developer Felix Rieseberg explicitly recommends against letting a model score itself, and insists on external verification. In his view, the ideal form is a tool the model can run on its own and get immediate feedback from. InfoQ published a Chinese translation of that article on November 4, 2026.
This isn’t a piece nagging you to “double-check your work.” The original points a direction; this article builds on it to lay out concrete design steps you can actually implement, covering how to choose verification methods, common failure modes, and ways to cross-check across multiple models.
The core takeaway: by the end, you’ll be able to write a verification contract for your own deep AI tasks — so the model iterates against feedback it can act on, instead of wrapping up with a self-assessment.
Core Takeaways for External Validation of AI Deep Tasks
The key to external validation isn’t adding another checkpoint. It’s making sure the basis for judgment is independent of the model’s own reasoning process.
| Key takeaway | What it means | Why it matters |
|---|---|---|
| Set the standard first | Write down the pass conditions before work begins, and make them as decidable as possible | Keeps the model from reinterpreting the goal after the fact |
| Isolate the data | Test conclusions with data or scenarios the model hasn’t seen | Separates real findings from after-the-fact fitting |
| Prefer tools | Use verifiers that can run and give immediate feedback whenever possible | Lets the model iterate on its own and reduces manual monitoring |
| Keep the evidence | Save run results, logs, and reproduction steps | Lets people review the conclusion instead of just trusting it |
| Review independently | Challenge results with different methods and different perspectives | Catches problems a single verifier would miss |
Why AI Deep Tasks Need External Validation
When Rieseberg talks about “depth,” he means using the tools, techniques, and knowledge of top experts in a field to go deep into the underlying data—not stopping at the surface of a task. The deeper the task, the harder it is for a person to judge intuitively whether the result is right. A churn analysis written in polished language may read convincingly, but that doesn’t prove its causal claims hold up.
He also points to a common phenomenon: when you ask a model to act like an expert, it often just switches to a different vocabulary while its working method stays the same. Self-evaluation rarely catches this kind of problem, because the evaluator and the producer share the same set of assumptions. If the reasoning is biased from the start, the model’s scores can unfold along the same bias.
Ambitious goals make validation even more important. The original article recommends setting goals that seem unreasonable—for example, cutting a website’s load time from 10 seconds to 200 milliseconds—and asking the model to keep trying. Goals like this are usable because 200 milliseconds can be measured. If a goal is both aggressive and impossible to verify, what you’ll likely get is just a more confident report.
AI Deep Task External Verification Quick Start: 5-Step Design Method
The method below doesn’t depend on a specific model or framework—you can pick a task you’re currently working on and apply it. If you can’t implement everything right away, prioritize the first three steps.
Step 1: Rewrite task goals as external verification criteria
“Analyzes comprehensively” and “high code quality” can’t be judged directly—they need to be rewritten as conditions that can pass or fail. For example, rewrite “fix the crash” as “running the original reproduction steps no longer crashes, and all related tests pass.” The criteria must be written before the model starts work; otherwise, the model might adjust the definition of success after it’s already done.
You can write the verification contract below first, then hand it to the model to execute:
| Field | What to specify clearly | Example |
|---|---|---|
| Final Goal | The actual business or engineering result you want | The application stays stable and no longer crashes the same way |
| Source Materials | First-hand data the model should use directly | Crash reports, mapping between versions and Git commits |
| Verifier | What tool or data judges the result | Reproduction scripts, automated tests, held-out data |
| Pass Conditions | Clear thresholds or decision rules | The crash can no longer be reproduced on all three target systems |
| Prohibitions | What the model must not change | Must not delete tests, must not modify held-out data |
| Submission Evidence | Run results that need to be attached | Commands, logs, failure records, and final results |
| Stop Conditions | When to end or hand over to a human | After reaching the iteration limit, submit the blocking reason |
Step 2: Sever the Link Between Conclusions and Validation Data
A model proposing a hypothesis and then proving it with the same dataset isn’t external validation. The original article’s user churn example offers a good approach: have the model look for the earliest churn signal in the logs, then use data it hasn’t seen to prove that this signal really has predictive power. In practice, you can split the data by time—early data for exploration, later data for testing.
Hold-out data is best kept by an independent script or person. The model only gets to see the run results, not the details directly. If you keep adjusting your approach based on the results, the hold-out data will gradually be “learned” too. So for important conclusions, it’s best to keep another dataset that’s only used during final acceptance.
Step 3: Prioritize External Validation Tools the Model Can Run Itself
The closer the validator is to real results and the faster the feedback, the more the model can iterate independently. The table below summarizes several external validation methods for common tasks. You don’t have to chase a single “strongest option”—combine them based on risk.
| Validation method | What it can prove | Feedback speed | Main limitations |
|---|---|---|---|
| Automated tests and reproduction scripts | Specified behavior meets expectations | Fast | Only covers scenarios that are written out |
| Performance measurement tools | Whether metrics reach thresholds | Fast | Environment differences may affect results |
| Hold-out data backtesting | Whether conclusions have predictive power | Medium | Historical patterns don’t necessarily persist |
| Red-team-style method review | Whether hypotheses and evidence chains are weak | Medium | Quality of judgment depends on the review method |
| Shadow running | Whether the approach works in real processes | Slow | Long cycle, higher organizational cost |
| Domain expert spot checks | Whether results match professional judgment | Slow | Hard to cover at scale |
Step 4: Wire external validation into the model’s working loop
Validation tools only produce continuous iteration when they live inside the model’s working environment. The crash-fixing case in the original article is a good example: the author first made it easy for Claude to pull crash reports, know which Git commit each version mapped to, and use Windows, macOS, and Linux machines. Once those conditions were in place, the model could reproduce the problem and verify the fix on its own.
This also shows that external validation often requires workflow changes, not just prompt tweaks. You need to open up the necessary data and tools — say, connecting to data sources through MCP connectors — while keeping permissions scoped to the task. At minimum, the loop design should include three things:
- Failure feedback: return the actual error logs and the specific failing items, not just the word “failed.”
- Iteration cap: set a retry limit and a total budget so the model doesn’t get stuck in a useless loop.
- Protected objects: make tests, validation scripts, and holdout data read-only so the model can’t “pass” by editing the standard.
Step 5: Use independent review and parallel runs to complete external validation
A single validator passing doesn’t mean the conclusion is reliable. The original article suggests bringing in a “red team” whose job is to find flaws — short sellers and forensic accountants, for instance — but the point isn’t to imitate their tone of voice; it’s to use their actual research methods. For example, ask the model to trace every key number back to its original source, or to actively hunt for data that would overturn the conclusion.
For process-level changes, you can also run things in parallel. The article recommends first using historical data to show that the new retrospective mechanism would have caught problems earlier, then running it alongside the existing process for a quarter. That way you can test the real-world effect without taking on the risk of fully replacing the old process from day one.
Integration tip: If you want different models to handle execution and review separately, you can call them all through APIYI. APIYI provides API access to the models mentioned in this article, compatible with both OpenAI and Anthropic interface formats — in most cases you just configure
base_url,api_key, andmodelper the docs. Documentation: https://docs.apiyi.com
Common Failure Modes in External Validation of Deep AI Tasks
External validation can also become a formality. These issues should be considered at the design stage; otherwise, the validation results will only create a false sense of security.
| Failure mode | Typical symptom | Countermeasure |
|---|---|---|
| Moving the goalposts | Deleting failed cases or loosening assertions | Keep the validator read-only and review related changes |
| Validation set overfitting | Metrics keep rising but fail on new data | Hold out a final acceptance dataset |
| Circular reasoning | Using model-generated data to validate the model’s conclusions | Go back to raw logs and primary materials |
| Form-only validation | Format is correct, but the content is wrong | Add business rules and sample-based checks |
| Role-playing | The conversation sounds expert, but the output doesn’t improve | Require execution of specific methods and submission of evidence |
The “second-model review” deserves special mention. It’s better than self-assessment because it can expose some blind spots, but it’s still a judgment-based review and can’t replace testing or real data. During review, it’s best to provide only the task criteria, raw materials, and final output—not the preceding conversation—to reduce the influence of prior reasoning on the judgment.
Examples of External Validation Design for Different Deep AI Tasks
The following examples can serve as starting points; specific thresholds need to be determined based on the business context.
| Task | Self-assessment-only approach | External validation design |
|---|---|---|
| Fixing an app crash | The model explains the fix mechanism and declares completion | Reproduce across systems and confirm the crash no longer occurs after the fix |
| Finding churn signals | The model provides a plausible attribution report | Test predictive power with data not used in exploration |
| Testing churn causes | The model repeats the judgment that “new user onboarding is problematic” | Look for supporting and contradicting evidence in raw data |
| Optimizing load speed | The model lists optimizations and estimates gains | Measure in a fixed environment and compare against target thresholds |
| Reworking a business retrospective | The model argues that the new approach is more advanced | After historical backtesting, run it in parallel with the existing process for a quarter |
To decide whether a task is worth designing this way, use a simple criterion: is the cost of an error higher than the cost of validation? If you’re just rewriting an email, reading it once may be enough. But if the result affects production systems or resource allocation, you should design external validation.
FAQ
Q1: What is external verification for deep AI tasks?
It means checking results against evidence that is independent of the model’s own judgment—for example, automated tests, performance measurements, holdout-data backtesting, and expert review. The core requirement is that the criteria must not be decided ad hoc by the model that produced the result.
Q2: What’s the difference between external verification and model self-checks?
Model self-checks can quickly catch obvious omissions, but the producer and the evaluator share the same assumptions. External verification introduces an independent source of evidence and is better suited to determining whether a result is genuinely valid. The two can be used together, but self-checks cannot serve as the final acceptance criterion.
Q3: As models get stronger, is external verification still necessary?
Rieseberg believes that as models get smarter, this problem may eventually be solved, but he explicitly recommends sticking with external verification at least as of October 2026. This is a practical judgment for the current stage. Until the reliability of model self-evaluation has been demonstrated, high-stakes tasks should still retain independent checks.
Q4: Which scenarios need external verification most?
Mainly the following types of tasks:
– **Engineering fixes**: crash fixes, performance optimization, and data migration.
– **Data research**: churn analysis, causal judgment, and predictive signal discovery.
– **Long-running agent workflows**: tasks that require models to keep working with limited human intervention.
Q5: How can I set up multi-model cross-review through the API?
You can access multiple models through APIYI in a unified way; its API is compatible with OpenAI and Anthropic formats:
1. Follow the APIYI documentation at https://docs.apiyi.com to configure the endpoint URL and API key.
2. Check the available models and their costs on the pricing page at https://docs.apiyi.com/models.
3. Configure different `model` values for execution and review; review requests should not carry the execution-phase conversation.
4. Record review feedback separately from test results, and prioritize tests and data evidence.
Q6: How should I estimate the cost of an external verification loop?
Total cost usually depends on the number of iterations, context length, output length, and number of review models. We recommend running a few rounds on a small sample first, recording actual usage for each round, and then estimating the budget together with an iteration cap. This article does not list specific unit prices; model pricing, billing rules, input/output unit prices, and cache pricing should be based on the actual data on the APIYI pricing page at https://docs.apiyi.com/models.
Q7: What are the limitations of external verification?
Tests can only cover predefined scenarios, backtesting cannot guarantee that historical patterns will continue to hold, and shadow runs take a relatively long time. Passing verification only means the result is supported under limited conditions; it does not mean it is absolutely correct. For legal, financial, and major business decisions, the final judgment should still be made by the responsible people.
Key Takeaways: External Verification for AI Deep Tasks
- Core principle: Don’t treat a model’s self-assessment as your acceptance criteria — judgments have to come from independent evidence.
- Define standards up front: Write down the pass conditions before work starts, so nobody redefines success after the fact.
- Data isolation: Check the model’s findings against data it has never seen, avoiding circular reasoning.
- Tools first: Validators that can actually run and give immediate feedback are the best fit for autonomous iteration.
- Supporting environment: A model needs access to data, tools, and a runtime environment before it can correct itself.
- Layered review: A second model is there to catch blind spots — it can’t replace tests and real data.
- Flexible access: APIYI lets you call multiple models through a single interface, cutting the cost of wiring up cross-review.
Summary
When you’re designing external verification for AI deep tasks, keep these three things in mind:
- Turn the goal into a criterion you can judge: without pass conditions, you can’t tell whether a task is really done.
- Keep evidence independent of the model’s reasoning: isolate the data, protect the validators, and keep complete run logs.
- Combine verification methods according to risk: tools give fast feedback, review catches blind spots, and parallel runs test what actually happens.
As a next step, pick a deep task you have on hand, write a verification contract first, then let the model get to work. This doesn’t just reduce errors — it also lets you set more ambitious goals, because no matter how bold the goal is, there’s always a way to put the results to the test.
If you want to build a multi-model execution and review workflow, you can run invocation tests through APIYI. See https://docs.apiyi.com for integration details; model pricing and billing rules follow the actual data at https://docs.apiyi.com/models.
📚 References
- How to Set More Ambitious Goals: A Practical Guide for AI Users: InfoQ Chinese translation
- Link:
infoq.cn/article/F1ms7x7KI1p7UfocCywh -
Description: Covers practical methods across three directions: height, depth, and breadth
-
How to be more ambitious: A tutorial for humans with models: Original article by Felix Rieseberg
- Link:
felixrieseberg.com/how-to-be-more-ambitious-a-tutorial-for-humans-with-models -
Description: Original source for external validation advice and cases such as crash fixes and user churn
-
APIYI Documentation Center: API integration instructions
- Link:
docs.apiyi.com -
Description: Use this to look up integration configurations for OpenAI- and Anthropic-compatible formats
-
APIYI Models and Pricing page: Model and billing information
- Link:
docs.apiyi.com/models - Description: Model pricing, billing definitions, and cache pricing are all subject to this page
Author: APIYI Technical Team
Technical discussion: Feel free to contact us through the APIYI homepage. For more resources, visit the docs.apiyi.com documentation center.
