A Score Surge Doesnât Necessarily Mean Greater Capability: Truncation and Repetition Pitfalls in Reasoning Model Evaluation, as Seen in Astraâs Modification of Qwen Weights, Plus a 5-Point Checklist
Authorâs note: Using a retrospective on Astraâs experiment rewriting Qwen3.5-0.8Bâs weights, this article explains the difference between getting an answer wrong and failing to finish it when evaluating reasoning modelsâand how to break down results by truncation. On November 2, four MIT CSAIL researchers published a blog post titled âSparks of Weight-Space Programming.â They reported…