|

Gemini 4 Neon data analysis benchmark changed to a perfect score; why does Plotly still recommend Hydrogen?

Gemini 4 Neon data analysis benchmark changed to a perfect score; why does Plotly still recommend Hydrogen?

Plotly still recommends Hydrogen, because this score revision only brought Gemini 4 Neon level with Hydrogen—it didn’t surpass it. In the same evaluation round, both earned a perfect score, but Hydrogen’s total cost for the full set was roughly one-third of Neon’s, and its median time per question was also shorter. With the scores tied, the comparison comes down to cost and speed, and Hydrogen leads on both.

One more point: the third-party evaluator Plotly revised its scoring rules here—Google didn’t upgrade the model.

What will you learn from this article?

  • Why Plotly changed Gemini 4 Neon’s score from 42/43 to 43/43 on November 19.
  • The scores, full-set costs, and per-question times for Neon, Hydrogen, and Carbon in the same evaluation round.
  • Which conclusions can be used directly for model selection in data analysis tasks, and which cannot.

Index: The article first uses a table to explain a few key concepts, then briefly summarizes the implications by audience. In the main body, the “Objective Facts” section lays out the original evaluation, the correction, and the results from the same round in order; the “Analysis and Judgment” section explains why Hydrogen wins and the limits of this leaderboard’s applicability for model selection.

gemini-4-neon-plotly-data-analytics-bench-perfect-score-hydrogen-en-image-0


Gemini 4 Neon Score Correction News Lead-in: To Know Concept Explanations

Concept Brief Explanation
Plotly A data visualization and analytics company; the third-party organizer of this benchmark, not a model vendor
DataAnalyticsBench A data analytics benchmark from Plotly; the question set is private and built on synthetic wind-power data
Plotly Studio Embedded The agent test environment used for this benchmark; every model answered inside this environment
Rubric The criteria for judging whether an answer is right or wrong — this correction changed the rule for one of the questions
Synthetic data Artificially generated simulated data, used to construct controllable questions
Single sample Each model completed just one full round, with no repeated runs averaged together
Median time per question The median time across all questions; this measurement includes query time
OpenRouter pricing The basis for costing this entire run, calculated at each model’s OpenRouter price on the day it ran

Gemini 4 Neon Perfect Score: Impact at a Glance for Different Groups

Group Impact
Developers choosing Google models for data analytics tasks Neon and Hydrogen are tied on accuracy; the cheaper, faster Hydrogen remains the first choice
Teams that already chose Neon The correction shows Neon’s analytical ability was underrated by the original rule, but its cost and latency haven’t changed
Researchers who follow evaluation methods This is a case where a question’s rule misjudged a real problem and was later fixed — proof that private question sets also need review
General users The model itself hasn’t changed; the change came from the evaluator’s scoring rule

Gemini 4 Neon Benchmark Correction: The Objective Facts

Plotly’s Original Evaluation: Gemini 4 Neon Initially Reported 42/43

Plotly’s original evaluation article is dated 2026-11-18, and the score published at the time for Gemini 4 Neon was 42/43. Information about the Gemini 4 Neon release came from Google’s official blog, and the evaluation was run by Plotly as an independent third party.

The tests were conducted in the Plotly Studio Embedded agent environment, using a private question set built on synthetic wind power data. The question types cover the following categories:

  • Routine data analysis
  • Ambiguous requests
  • Unanswerable requests
  • Data quality
  • Statistical traps
  • Hidden patterns

November 19 Correction: Gemini 4 Neon Goes from 42 to a Perfect Score

Plotly added a correction to the original article, dated November 19. No specific time or time zone was given.

The problem was in a data-quality question. The original scoring rule treated two genuine issues in the synthetic data as irrelevant, so if a model pointed out those two issues in its answer, it was actually marked wrong. In other words, the model got it right, but the rule graded it wrong.

Here’s how Plotly handled it:

  1. Revise the scoring rule for that question;
  2. Re-score the models that hadn’t passed that question before;
  3. Manually review every score change that resulted;
  4. Manually verify all 43 of Neon’s answers, one by one.

After the correction, Neon’s score was changed to 43/43. Carbon remains at 42/43, unchanged.

gemini-4-neon-plotly-data-analytics-bench-perfect-score-hydrogen-en-image-1

Item Before Correction After Correction
Gemini 4 Neon score 42/43 43/43
Carbon score 42/43 42/43 (unchanged)
Reason for change — The scoring rule for one data-quality question was revised
The model itself Unchanged Unchanged; this was not a Google model upgrade
Correction date Original article dated 2026-11-18 Correction section dated November 19, with no time or time zone specified

Scores from the Same Evaluation Round: Neon, Hydrogen, and Carbon

All the data below comes from the same round of Plotly testing in the same test environment. GPT-6.1 Sol is listed only as a cross-vendor reference and is outside Plotly’s recommendation scope for Google models.

Model Score Full-suite cost Median time per question
Gemini 4 Neon 43/43 $17.03 46 sec
Hydrogen 43/43 $6.36 32 sec
Carbon 42/43 $11.55 93 sec
GPT-6.1 Sol (reference) 43/43 $2.61 18 sec

Here are a few things to keep in mind when reading this table:

  • Full-suite cost is calculated using OpenRouter pricing on the day each model was run. It is not the same as Google’s official unit pricing or the listed prices on other platforms.
  • Median time per question includes query time and is not purely model generation time.
  • Single sample: Plotly notes that each model was run only once. In earlier repeated tests, other models’ scores varied by as many as three questions.

gemini-4-neon-plotly-data-analytics-bench-perfect-score-hydrogen-en-image-2

Sources

  • Plotly’s original evaluation and correction: plotly.com/blog/gemini-4-neon-plotly-data-analytics-bench/
  • Plotly’s full leaderboard and methodology: plotly.com/blog/introducing-dataanalyticsbench/
  • Google product announcement: blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-4-neon/

Why Plotly Still Recommends Hydrogen: Our Analysis

A Perfect Score Only Means a Tie — Model Selection Comes Down to Cost and Speed

After the score correction, Neon and Hydrogen are indistinguishable on accuracy. Based on the same round of data, Neon’s total cost for the full set is about 2.7× Hydrogen’s, and its median time per question is about 1.4×. When accuracy is equal, the cheaper and faster option is naturally the better deal, so Plotly’s recommendation doesn’t need to change because of this correction.

Carbon’s case is even clearer. It scores one question lower, costs more than Hydrogen, and takes nearly three times as long per question — no clear advantage in this comparison.

What the Correction Shows: Neon’s Data-Quality Judgment Was Undervalued

The correction is good news for Neon. The misjudgment landed precisely on a data-quality question, and Neon was marked wrong because it pointed out a problem that genuinely existed in the data. For analysis tasks that require spotting dirty data and outliers, this shows Neon has the relevant detection ability.

That said, the correction only affects accuracy; Neon’s cost and speed are unchanged. Plotly manually verified all 43 of Neon’s answers one by one, which makes this perfect score more credible than an automated grade — but it doesn’t change where Neon sits on cost.

Which Conclusions You Can Use for Model Selection — and Which You Can’t

Conclusion Directly usable for model selection? Reason
For data analysis tasks, prefer Hydrogen among Google models Yes, but with scenario caveats Plotly’s recommendation is scoped to its own tests and to choosing among Google models
Neon and Hydrogen are comparable in accuracy Use with caution Single-sample test; repeat runs of other models have shifted scores by up to three questions
Neon costs about 2.7× Hydrogen Can serve as a relative reference Computed from OpenRouter pricing on the day of the run; actual per-unit prices may differ
Hydrogen is faster than Neon Can serve as a relative reference Timing includes query latency and depends on the test environment
Hydrogen is the best at data analysis across all models No The cross-vendor reference GPT-6.1 Sol in the same round was cheaper and faster
Neon represents a model upgrade No This is a scoring revision; the model itself hasn’t changed

A Few Things to Watch Going Forward

To be clear, the following are predictions and directions to watch, not established facts:

  • If Plotly later publishes results from multiple repeated runs, the sub-one-question gap between Neon and Hydrogen could reorder, and only then will accuracy comparisons be more reliable.
  • Cost rankings depend on pricing. If model prices change, the cost ranking recalculated on the same question set may shift too.
  • Private benchmark sets can also suffer rule-based misjudgments. This correction is a reminder that when reading an evaluation, you should look beyond the total score to whether the evaluator disclosed its corrections and manual review process.

For readers currently choosing a model for data analysis tasks, the practical takeaway is this: Neon is no longer undervalued by the scoring rules, but in Plotly’s benchmark, Hydrogen is still the cheaper and faster option among Google models.

Gemini 4 Neon’s perfect score fixes a scoring error, while the Hydrogen recommendation is driven by cost-effectiveness — the two aren’t in conflict. As the English saying goes: “Price is what you pay; value is what you get.”

Similar Posts