A Score Surge Doesn’t Necessarily Mean Greater Capability: Truncation and Repetition Pitfalls in Reasoning Model Evaluation, as Seen in Astra’s Modification of Qwen Weights, Plus a 5-Point Checklist

Author’s note: Using a retrospective on Astra’s experiment rewriting Qwen3.5-0.8B’s weights, this article explains the difference between getting an answer wrong and failing to finish it when evaluating reasoning models—and how to break down results by truncation.

On November 2, four MIT CSAIL researchers published a blog post titled “Sparks of Weight-Space Programming.” They reported that GPT-6 Astra modified only about 0.016% of Qwen3.5-0.8B’s parameters, yet its MATH-500 score jumped from 7.9% to 48.1%. GSM8K and IFEval scores also rose sharply. The story was widely shared as a landmark case of “AI changing AI’s brain.”

This isn’t an attempt to dismiss the research. The project’s public repository contains round-by-round logs for 1,125 calls. In a retrospective, journalists recalculated the results from the per-question records and reached a more modest conclusion: almost all of the score increase came from the model finally being able to finish writing its answers, not from becoming better at math.

The key takeaway: Rather than revisit whether RSI is happening, we’ve turned this retrospective into a reusable method for evaluating reasoning models. It can help you determine, when selecting a model, tuning parameters, or preparing for deployment, whether a score change reflects a difference in capability or truncation caused by repetitive loops.

Thinking Model Evaluation: Key Facts at a Glance

Let’s start with the key facts about the evaluation. The figures below come from the original blog post and media reviews of the public repository.

Where Does “Truncation” Come From in Thinking Model Evaluations?

The first time Astra had Qwen tackle 96 practice problems, it got only 11 right. The other 85 weren’t wrong answers: all of them reached the length limit without stopping. Measured in eight-word segments, about 95% of the truncated text endings consisted of repetition. All 11 answers that finished normally were correct.

This suggests the baseline score was low mainly because the model couldn’t finish its answers, not because it couldn’t solve the problems. Notably, the researchers’ initial prompt to Astra listed “being truncated by the token limit, often due to repeating itself while thinking” as one failure type, and gave “fixing whatever drives repetition” as an example of a valid improvement. Astra’s final report also stated that it targeted repetition, not mathematical knowledge.

Reasoning Model Evaluation Breakdown: All Gains Came From Truncated Problems

The most convincing evidence in the review comes from a paired before-and-after test on 1,152 held-out practice problems. Split the problems by whether their original responses were truncated, and the conclusion is clear.

There’s another easily overlooked detail: before the change, only 7 problems were completed but answered incorrectly. Afterward, that number rose to 245. Once the loops stopped, the model’s actual calculation errors became visible. On the official MATH-500 benchmark, average response length fell from 30,868 tokens to 22,037 tokens. That reduction was also concentrated in responses that had originally hit the limit.

Validating the Reasoning Model Evaluation With a Control Model

The researchers recognized this too, so they ran the same experiment on Gemma 3 270M. Comparing the two experiments shows that “removing an obstacle” and “adding capability” are two different things.

The original blog post also acknowledges that changing weights can remove obstacles blocking a model’s capability, but it can’t add capability the model doesn’t have. For evaluators, the takeaway is that the same “big score increase” can mean entirely different things depending on the underlying failure patterns.

A Pitfall in Evaluating Reasoning Models: Temperature and the Thinking Switch

There’s another often-overlooked factor in this debate: the evaluation settings themselves can make loops more likely. The Qwen3.5-0.8B official model card information cited in the retrospective differs from the settings the researchers actually used.

Note that the model card details above come from a media retrospective. It’s worth checking the original Qwen model card directly before running an evaluation. The key point is methodological: the same model can have a completely different “baseline” with a different set of sampling parameters. If the baseline was measured under settings that encourage loops, any change that makes the model “stop” will look like a leap in capability.

Reasoning Model Evaluation Checklist: Separate Truncation Before Drawing Conclusions

The lessons from this review translate into an evaluation process you can use for any reasoning model. Whether you’re selecting a model, tuning parameters, or validating a fine-tuning or prompt change, use the table below to check each step.

🎯 Evaluation tip: The trickiest part of comparing models is writing separate invocation code for each provider. With the OpenAI-compatible interface from APIYI at apiyi.com, you can invoke models from multiple major providers using the same script, prompts, and parameters—making it easier to include “truncation rate” as a standard column in your results.

Thinking Model Evaluation Script: A Minimal Example

The code below calculates the proportion of questions truncated by the length limit. A finish_reason of length means the response hit max_tokens:

import openai

client = openai.OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.apiyi.com/v1"  # APIYI OpenAI-compatible endpoint
)

questions = ["计算 17 * 23 的结果", "求 1 到 100 的整数和"]
truncated = 0
for q in questions:
    r = client.chat.completions.create(
        model="your-thinking-model",  # Replace with the actual model name
        messages=[{"role": "user", "content": q}],
        max_tokens=4096,
    )
    if r.choices[0].finish_reason == "length":
        truncated += 1
print(f"截断率: {truncated / len(questions):.1%}")

import openai
from collections import Counter

client = openai.OpenAI(
    api_key="YOUR_API_KEY",
    base_url="https://api.apiyi.com/v1"
)

def tail_repetition_ratio(text: str, n: int = 8, tail_words: int = 400) -> float:
    """Calculate the proportion of repeated n-grams at the end of the text (for Chinese, you can split by character instead)."""
    words = text.split()[-tail_words:]
    if len(words) < n * 2:
        return 0.0
    grams = [tuple(words[i:i + n]) for i in range(len(words) - n + 1)]
    counts = Counter(grams)
    repeated = sum(c for c in counts.values() if c > 1)
    return repeated / len(grams)

def evaluate(dataset, model, temperature, max_tokens, judge):
    """dataset: [(question, answer)], judge: function that checks whether the answer is correct"""
    stats = {"done": [0, 0], "truncated": [0, 0], "loop": 0}
    for q, ans in dataset:
        r = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": q}],
            temperature=temperature,
            max_tokens=max_tokens,
        )
        choice = r.choices[0]
        text = choice.message.content or ""
        group = "truncated" if choice.finish_reason == "length" else "done"
        stats[group][0] += 1
        stats[group][1] += int(judge(text, ans))
        if group == "truncated" and tail_repetition_ratio(text) > 0.5:
            stats["loop"] += 1

    for g in ("done", "truncated"):
        total, correct = stats[g]
        rate = correct / total if total else 0
        print(f"{g}: {total} 题，正确率 {rate:.1%}")
    t = stats["truncated"][0]
    print(f"截断题中疑似循环: {stats['loop']} / {t}")
    return stats

Note that models don’t all use the same parameters to turn thinking mode on or off. Some use request fields, some use different model names, and others specify it in the prompt template. Whether thinking content counts toward max_tokens and whether it’s returned separately also varies by model. Check the documentation for each model before evaluating it.

🚀 Quick start: If you just want to try this grouping method, register at APIYI apiyi.com and use the free test credits to run a few dozen questions through two or three thinking models. Compare their truncation rates first, then decide whether to expand the evaluation.

Impact of the Reasoning Model Evaluation Incident

Impact on Developers

• The overall score can be misleading: Looking only at overall accuracy lumps unfinished responses together with incorrect answers. Before launch, evaluation reports should include at least two additional columns: truncation rate and accuracy by group.
• The benefits of tuning can be misread: Raising the output limit, adjusting the temperature, or adding a repetition penalty can substantially improve scores. But that’s removing obstacles—it doesn’t mean the model itself has changed.
• Production is affected too: In evaluations, truncation shows up in production as overly long responses, runaway costs, and long waits for users. The model card itself recommends streaming output with loop detection and early termination.

Impact on Model Evaluation Research

• Claims of “major improvements” need a breakdown of the evidence: This review shows that what looks like a leap in capability may turn out to be a bug fix once results are broken down by truncation. Going forward, it’s worth asking how the truncation rate changed whenever someone claims an improvement to a small model or reasoning model.
• Control group design deserves more attention: The additional Gemma 3 270M experiment and Astra’s own random-direction control, which changed the score by only about 1.5 percentage points, both show that control design is key to separating real signals from noise.

Frequently Asked Questions

Reasoning Model Evaluation: Key Takeaways

• Where the gain came from: Qwen scored 93.9% both before and after the weight change on the 114 questions it completed. The entire gain came from truncated responses.
• Failure pattern: In the baseline run, 93% of responses hit the token limit, and roughly 95% of their tail content was repetitive.
• Parameter mismatch: The evaluation used a temperature of 0.6, while the model card recommends 1.0 for thinking text.
• Control evidence: Gemma 3 270M had a low truncation rate and gained less than 1 percentage point on MATH-500 after the weight change.
• Core method: First separate truncated responses by finish_reason, then compare accuracy.
• Getting started: Use the unified APIYI interface at apiyi.com to measure truncation rates across models under the same parameters.

Summary

The controversy over Astra’s changes to Qwen’s weights is a useful reminder for anyone evaluating reasoning models:

1. Break down sudden score jumps: Group responses by whether they hit the token limit, and check which group drove the gain.
2. Parameters are part of the baseline: Verify temperature, thinking mode, and output limits against the model card, and include them in your report.
3. Distinguish removing obstacles from adding capability: Use a control model with a low truncation rate to test the true nature of an improvement.

The next time you see a claim that a change multiplied a model’s score, ask: How much did the truncation rate change? That question often reveals more than the overall score.

If you’re preparing to evaluate several reasoning models before deployment, try running truncation-rate statistics with the same script and parameters through APIYI at apiyi.com before deciding which model to use.

📚 References

1. Silicon Stars Pro analysis (republished by 36Kr): The Viral “Astra Rewires Qwen’s Brain” Has Little to Do With the RSI We’re Looking For
2. Link: 36kr.com/p/4056494372181124

3. Details: Paired test results recalculated from question-by-question records in the public repository; the main data source for this article.

4. MIT CSAIL researcher’s blog: Sparks of Weight-Space Programming

5. Details: The original experiment description and results table. Refer to the original for the setup of Experiment 8.

6. Official Qwen3.5-0.8B model card: Sampling parameters and thinking mode

7. Details: Check the default thinking mode, recommended temperature, and guidance for handling loops.

Author: APIYI Technical Team
Technical discussions: Contact us through the APIYI website at apiyi.com. For more information on API usage, visit the documentation center at docs.apiyi.com.