Abnormal model performance

Author’s note: A model that seems to have “gotten dumber” hasn’t necessarily been replaced. Starting with Anthropic’s recent FlashProve research on proving inference, this article lays out a troubleshooting sequence for the factors that affect API results, from the caller’s setup to the server. “The model solves the same problem on the official website, but…