Author’s note: Stanford researchers found that 44% of widely cited LLM findings are outdated. Among findings with an estimated expiration date, the average lifespan was just 15.7 months. This article turns the research into a model-selection retesting checklist development teams can use.
“This model can’t count the r’s in strawberry.” “Information in the middle of a long context gets lost.” “Models like to invent references.” You’ve probably heard claims like these repeatedly in papers, technical forums, and model-selection meetings. Stanford researchers Owen Queen and James Zou retested 50 widely cited LLM findings across 20 models and found that 44% were outdated. Among findings with an estimated expiration date, the average lifespan was just 15.7 months.
This isn’t another vague observation that “models are getting better.” The research provides concrete data on which types of findings expire fastest, which last longer, and how much models within the same generation can differ. This article turns the researchers’ advice for academics into a process development teams can follow when selecting models.
What you’ll gain: By the end of this article, you’ll know which “models aren’t good at X” claims are least reliable and have a model-selection habit you can put into practice: record the model name, version, and date; save the prompt; and retest on your target model.
Key Takeaways From the Research on Outdated LLM Findings
Here’s a quick look at the most important numbers from the study. The models studied were released between July 2024 and September 2026 by eight different developers. The reporting on the study was published on November 3, 2026.
| Key point | Study data | What it means for model selection |
|---|---|---|
| Share of findings that expired | Of 50 findings, 44% had expired and 34% had weakened substantially | Most widely circulated claims about model shortcomings can no longer be cited as-is |
| Share that still held | Only 16% remained unchanged, while 6% grew stronger | A few persistent tendencies are worth watching over the long term |
| Average lifespan | Among 22 findings whose expiration times could be estimated, the average was 15.7 months | Treat evaluation findings from more than a year ago with caution |
| Cumulative expiration over time | 16% had expired after one year, rising to 39% after two years | The older a finding is, the more it needs to be retested |
| Differences among models of the same generation | For models released in July–September 2026, the number of expired findings ranged from 11 to 30 | Release date alone doesn’t tell you what a model can do |
Two Key Findings About LLM Findings Expiring
The first key point is what “expired” means. In the study, a finding is considered expired when the effect behind the original finding has essentially disappeared in newer models—not merely improved. Another 34% of findings had “weakened substantially,” meaning the issue was still present but far less severe than the original paper described. Taken together, nearly 80% of the popular findings can no longer be applied to today’s models as-is.
The second key finding is that a finding’s “lifespan” is strongly linked to its type. Findings about capability gaps—what models “can’t do”—expire the fastest. Findings about preferences—how models “tend to behave”—last longer. The researchers suggest that capability errors usually have a “correct answer,” giving developers a clear training objective to optimize for. Preferences and tendencies have no standard answer, so they’re naturally harder to “fix.”
Which LLM Findings Become Outdated Most Easily?
Understanding the type of finding is the first step in deciding what to retest. Based on the research coverage, the table below divides typical findings into those that become outdated easily and those that are relatively persistent.
| Finding type | Typical examples | Status in the study | Retesting priority |
|---|---|---|---|
| Reasoning and capability gaps | Counting letters in “strawberry,” arithmetic in unfamiliar number bases, losing information in the middle of a long context, abandoning a correct answer after a user challenges it | 5 of 9 reasoning findings became outdated; none persisted or grew stronger | Highest: retest before citing |
| Factuality and hallucinations | Fabricating scientific references | Rates have fallen to nearly zero for some models but remain high for others | High: differences are substantial, so test each model individually |
| Compositional reasoning | Two-hop questions, the “reversal curse,” contradictory answers to paired questions about time | Weakened but still detectable | Medium-high: the problems remain, but their magnitude varies widely |
| Preferences as a judge | Favoring its own answers, giving similar scores to accepted and rejected papers, being swayed by the wording of medical abstracts | Persisted or grew stronger | Medium: a long-term risk that needs periodic checks |
| Sycophancy and user appeasement | Agreeing with the user’s views | Made up a relatively large share of findings that persisted or grew stronger | Medium: prioritize for customer service and advisory applications |
| Output distribution tendencies | Left-leaning political bias, highly similar answers to open-ended questions, uneven distributions of generated random numbers | Persisted or grew stronger | Medium: important for content generation and sampling |
There’s an important caveat about the reasoning findings. The article says “5 of 9 became outdated, and none persisted or grew stronger,” while a figure caption says “all 9 have faded.” Taken together, this suggests that 5 became outdated and the others also weakened, with none remaining unchanged. We couldn’t review the original paper’s complete list of 50 findings or its retesting methods, so consult the paper for the definitive classifications.
Why LLM Preference Findings Are More Persistent
Of the 11 findings that persisted or grew stronger, 3 involved LLMs acting as judges and 3 involved sycophancy. The rest concerned political leanings, similar answers to open-ended questions, and uneven random-number distributions. These issues have something in common: it’s hard to define what a “correct” output looks like, so they’re also hard to eliminate through targeted training.
For development teams, this points to a counterintuitive ranking of risks. The often-cited problem of a model being unable to count letters may no longer be an issue, while less obvious systematic tendencies deserve closer attention. If you’re using LLMs for automated evaluation, scoring, or recommendations, preference-related issues should be part of your regular regression tests.
LLM Findings on Social Bias Show a Mixed Picture
The researchers noted that social bias findings haven’t all followed the same trajectory. Left-leaning political bias persists, gender stereotypes and implicit association bias have weakened, and the trend data even showed a reversal in bias against speakers of African American English. This is another reminder that broad claims like “models are generally biased in a particular way” are meaningful only when tied to a specific model and point in time.
Why You Can’t Predict LLM Performance by Vendor or Release Date
When choosing a model, many teams rely on a few intuitive assumptions: newer models must be better than older ones, models from the same vendor will perform similarly, and larger models must be more reliable than smaller ones. The research data directly challenges these assumptions.
The clearest example is the fabricated-reference test. In Walters and Wilder’s original 2023 study, GPT-3.5 fabricated 55% of its references. Stanford researchers tested current models using the same prompt: GPT-6.1 Sol scored 0%, Claude Sonnet 5.5 scored 1%, while Qwen3.7-Flash, also released in 2026, still fabricated 69% of its references—even more than GPT-3.5 had three years earlier.
The differences within the same release window are just as striking. Among models released between July and September 2026, the number of findings “overturned” ranged from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. The study explicitly notes that release date, vendor, and model size can’t reliably predict this pattern.
The key takeaway from this chart is that models from the same era can land at opposite extremes on the same test. If you want to replicate this kind of comparison within your team, you can use a unified interface platform like APIYI apiyi.com to run the same set of prompts across several candidate models. Check the platform’s official model list for availability.
The researchers also caution against reading improvement as proof that “the problem is completely solved.” On two-hop questions, which require combining two known facts, Claude Sonnet 5.5 still got about 10% of combined questions wrong even when it could answer both sub-questions separately. The original paper reported a rate of 43% for ChatGPT. That’s a substantial improvement, but if your business relies on combining facts across multiple steps, 10% is still enough to cause production incidents.
Getting Started with LLM Selection and Retesting: A 5-Step Checklist
Researchers recommend treating empirical findings about LLMs as “observations of specific models at specific points in time,” not universal truths. Papers should document the model, version, and date, and share prompts and analysis code so the findings can be retested on future models. Development teams can apply almost the same advice to their model selection process.
Step 1: Add Metadata to Every LLM Finding
Whether a finding comes from a paper, a blog post, a colleague’s experience, or your own tests, start by filling in the fields below. If key fields are missing, treat the finding as a “lead to verify,” not a basis for a decision.
| Metadata Field | What to Record | Why It Matters |
|---|---|---|
| Model name | The full model name, not just “a model from a particular provider” | Models from the same provider can differ significantly |
| Specific version | Version number or snapshot identifier | A model’s performance may change after updates, even if its name stays the same |
| Test date | The exact date | Helps determine how old the finding is, using the average lifespan of 15.7 months as a reference |
| Complete prompts | System prompt, user input, and examples | Results can’t be reproduced with different prompts |
| Invocation parameters | Temperature, maximum output length, and so on | Sampling parameters can affect findings about stability |
| Evaluation criteria | What counts as a pass or failure | Otherwise, results from different testers can’t be compared |
| Sample size | Number of test cases | A screenshot of one conversation can’t establish a rate |
Step 2: Prioritize LLM Retesting by Finding Type
When you encounter a finding, first determine what type it is. Findings about capability gaps—such as trouble counting, calculating, remembering, or sticking to an answer when challenged—should be treated as “likely outdated” by default and retested before you cite them. Findings about preferences or tendencies—such as self-preference, sycophancy, or repetitive outputs—should be treated as “likely still present” and included in long-term regression tests.
Also consider how old the finding is: according to the research, a finding more than a year old has roughly a 16% chance of being outdated; after two years, that rises to 39%. If the model involved has been superseded by one of your candidate models, retest the finding no matter how well-known it is.
Step 3: Turn Findings into Executable LLM Test Cases
A statement like “the model is bad at understanding long documents” can’t be tested directly. You need to turn it into specific test cases. For example, place a key fact at the beginning, middle, and end of a long document, ask about it in each case, and track the accuracy rate. Or have the model give a correct answer, then simulate a user challenging it and measure how often it changes its answer.
When writing test cases, use data that closely resembles your real business data rather than simply reusing toy examples from papers. The classic questions identified as “outdated” in the study may have already been targeted in training by various providers. Passing those tests doesn’t mean a model will pass on your business data, too.
Step 4: Retest All Candidate LLMs With the Same Set of Prompts
The key to retesting is controlling variables: use the same prompts, parameters, and evaluation criteria, and change only the model. If each model needs its own SDK integration and authentication setup, the cost of retesting can become high enough that teams won’t do it.
🚀 Practical tip: APIYI apiyi.com uses a unified OpenAI API standard, so you can call 200+ models with a single token. To switch between candidate models, just change the model parameter and keep the prompts and other parameters unchanged. This makes it well suited to side-by-side retesting. Check the platform’s official model list for supported models and versions.
Step 5: Create an LLM Findings Archive and Review It Regularly
Save each round of retesting as a findings record containing, at minimum, the original finding, metadata, retest results, and the next review date. Given the average lifespan of 15.7 months, consider reviewing findings your core business depends on every quarter or six months. When you plan to upgrade a model version, always trigger a full retest.
| Scenario | Recommended Retest Scope | When to Retest |
|---|---|---|
| Initial selection | All findings that affect the decision + core business use cases | Before the selection review |
| Model version upgrade | Full regression test suite | Before switching the production model |
| Citing external findings | Relevant findings only | Before including them in a proposal |
| Routine operations | Preference-related findings + high-risk use cases | Every quarter or six months |
| Production anomaly | Capabilities related to the anomaly | Immediately |
💡 Selection tip: Don’t start by asking, “Which model is the strongest?” Instead, identify the types of errors that pose the greatest risk to your business, then retest candidate models against those risks. With the unified interface from APIYI apiyi.com, you can compare multiple candidate models in the same testing workflow and reduce extra variables introduced by integration.
Frequently Asked Questions (FAQ)
Q1: What does it mean for an LLM finding to become “outdated”?
In this Stanford study, a finding is “outdated” when the effect behind an empirical claim about LLMs has largely disappeared in newer models. That differs from “weakened,” which means the issue still exists but is much less pronounced. Of the 50 findings, 44% were outdated and 34% had weakened.
Q2: What’s the difference between findings about LLM capability gaps and those about behavioral tendencies?
Capability-gap findings describe what a model “can’t do,” such as counting letters, performing arithmetic, or retaining information in long contexts. These tasks have clear correct answers and are easier to fix through targeted training, so the findings become outdated fastest. Behavioral-tendency findings describe how a model “tends to act,” such as favoring its own answers or being sycophantic toward users. These don’t have standard answers and tend to persist longer.
Q3: When was this study on outdated LLM findings published?
The related report was published on November 3, 2026. The study tested 20 models from eight developers, released between July 2024 and September 2026. Since models evolve quickly, the specific figures cited here are also time-sensitive. Check the original paper before relying on them.
Q4: Which teams most need to retest LLMs by version?
Key use cases include:
– **Teams choosing or switching models**: Avoid ruling out candidates based on impressions of their shortcomings from two years ago
– **Teams using LLMs for evaluation and scoring**: Tendencies such as self-preference and sensitivity to wording still exist
– **Applications where factual accuracy matters**: Fabricated reference rates range from 0% to 69% across models of the same generation
– **Customer service and advisory chat products**: Sycophantic tendencies need ongoing monitoring
Q5: How can you retest multiple LLMs at low cost?
The key is to use a unified access layer, making it easy to “change only the model, not the other variables”:
1. Visit APIYI at apiyi.com, register an account, and get an API key
2. Check the official model list to confirm that your candidate models and versions are available
3. Use the same set of prompts and parameters, changing only the model parameter for each invocation, and record the results and metadata
This lets you focus on designing test cases instead of repeatedly integrating with different providers’ APIs.
Q6: Can you rely on leaderboard scores for new models instead of retesting?
It isn’t a good idea to rely on them entirely. The study found that release date, provider, and model size can’t reliably predict a model’s performance on specific findings. Models released around the same time can differ considerably. Leaderboards are useful for initial screening, but final decisions should be based on retesting with your own business data.
Q7: What are the limitations of this LLM study?
First, you’ll need to consult the original paper for the complete list of 50 findings and the retesting methods; secondary reports can’t cover every detail. Second, the report’s main text and figure captions differ in how they describe findings related to reasoning, so refer to the original paper. Finally, the study’s results are themselves observations of specific models at a specific point in time. By the researchers’ own logic, they too may become outdated.
Key Takeaways: LLM Findings Can Expire
- Many findings expire: Of 50 popular claims about LLMs, 44% had expired and 34% had weakened substantially.
- Their shelf life is short: Findings with an estimable lifespan lasted an average of 15.7 months. Nearly 40% had expired after two years.
- Capability limitations expire fastest: Prioritize retesting claims about counting letters, arithmetic, and losing information in long contexts.
- Behavioral tendencies last longer: Include self-preference, sycophancy, and repetitive outputs in long-term regression tests.
- You can’t infer performance from vendor or release date: Even among 2026 models, fabricated reference rates ranged from 0% to 69%.
- Metadata is essential: Without the model name, version, date, and prompt, a finding is hard to reproduce.
- A unified interface lowers retesting costs: With APIYI at apiyi.com, you can compare models by changing only the model parameter.
Summary
LLM findings expire after an average of 15.7 months. That figure is a reminder that:
- Findings have a shelf life: Treat any claim about an LLM as an observation of a particular model at a particular point in time.
- The type of claim sets the priority: Retest capability limitations first, and monitor behavioral tendencies over the long term.
- The version is what matters: Vendor, release date, and model size can’t replace testing of the specific version.
The next time you hear “that model isn’t good at X” in a model selection meeting, ask three questions: Which version? When was it tested? Do you still have the prompt? If no one can answer, add it to the retest list.
Consider using APIYI at apiyi.com to bring candidate models into the same testing workflow and quickly retest them with the same set of prompts. Check the official list for available models.
📚 References
- THE DECODER – Rapid LLM development is outrunning the research meant to explain it， study finds: Coverage of Stanford’s research on LLM findings becoming outdated
- Link:
the-decoder.com/rapid-llm-development-is-outrunning-the-research-meant-to-explain-it-study-finds -
Details: Includes key figures and examples, such as the 44% expiration rate, average lifespan of 15.7 months, fabricated references, and differences between models of the same generation.
-
Original paper by Owen Queen and James Zou: Stanford researchers’ replication study of 50 LLM findings
- Link: Search for the authors’ names and paper title on an academic search platform.
- Details: We weren’t able to access the full paper. Please refer to the original for the list of 50 findings and the replication methodology.
Authors: APIYI Technical Team
Technical discussions: Contact us through the APIYI homepage at apiyi.com. For more information on integrating models, visit the documentation center at docs.apiyi.com.
