Direct answer: A new prompt isn’t necessarily better—and “it looks better” doesn’t count as evidence. To figure out whether the new article template beats the old one, hold the source material, model, and generation settings fixed, change only the template, and then use Promptfoo assertions to check structure, order, and content quality at different layers. You also need to present results dimension by dimension—you can’t let a writing-style score cancel out factual errors. This article lays out an experimental plan; it doesn’t run a real A/B test, so it gives no win rates, scores, or savings estimates.
Here’s what you’ll learn:
- What Promptfoo’s
contains, JavaScript, andllm-rubricassertions can check—and what they can’t prove. - How to run an assertion file separately against existing model outputs, so you don’t regenerate them when you only change the scoring rules.
- An experimental design for comparing old and new article templates, including variable control, three-layer checks, and dimension-by-dimension reporting.
- The limits of model judges and rule-based assertions, plus the cost issues that come with paid judge models.
Index: We’ll start with a concept table explaining a few key terms in Promptfoo evaluation, then look at what this means for content teams, developers, and evaluation owners. The “Objective Facts” section of the main text only summarizes the assertion capabilities and verification status described in the documentation. The “Analysis and Judgment” section offers experimental design suggestions, configuration examples, and a discussion of limitations—and flags each item as a suggestion rather than a measured conclusion.

Concepts to Know Before Comparing Prompt Templates with Promptfoo
You’ll see these terms repeatedly later on. Most of them come from Promptfoo’s configuration documentation, and a few are general terms used in evaluation experiments.
| Concept | Brief explanation |
|---|---|
| Promptfoo | An open-source tool for evaluating Large Language Model prompts and outputs; you can define test cases and assertions in a config file |
| Assertion | An automatic check rule for model output that determines whether the output meets a certain condition |
| contains | Checks whether specified text appears in the output; this is a deterministic string match |
| JavaScript assertion | Uses a JavaScript expression or function to implement more complex conditions |
| llm-rubric | Has a model score the output according to a custom rubric; this falls under “model-graded” assertions |
| Grading Provider | The model that performs the llm-rubric grading; it can be configured separately |
| Rubric | The criteria given to the grading model, explaining what counts as passing and what gets points deducted |
| LLM-as-a-Judge | Using a Large Language Model instead of a human to review output quality |
| Variable control | Changing only one factor in an experiment (here, the template) while keeping other factors constant |
At a glance: the impact of evaluating old and new prompt templates
The value of this kind of evaluation isn’t how new the tool is. It’s that it turns “I think the new template is better” into a reviewable record of checks. Different roles get different things out of it.
| Audience | Practical impact |
|---|---|
| Content teams and editors | Template revisions now have a reviewable acceptance basis, and the team can catch problems like “the structure is right, but the factual section is mixed with speculation” |
| Prompt engineers and developers | Can break template requirements into rule assertions and rubric assertions, so they don’t have to regenerate every output when changing scoring rules |
| Evaluation owners | Need to report results by dimension and clearly distinguish the evidential value of model scoring, rule matches, and human review |
| Budget owners | Rule assertions add almost no extra invocation cost; if llm-rubric uses a paid model, it incurs judging fees |
Objective facts about Promptfoo’s assertion capabilities
This section only describes capabilities documented by Promptfoo. We checked the relevant documentation pages on 2026-11-17, but the initial release dates of these capabilities are unverified, so this article does not provide specific launch dates.
Promptfoo’s contains assertion: check whether text appears
contains is the most basic assertion type. It checks whether a specified piece of text appears in the model output. It’s deterministic string matching: the same output will produce the same result every run.
To be clear, passing contains only means “this text was matched.” It doesn’t say whether the paragraph containing that text is well written, nor whether the content is correct.
Promptfoo’s JavaScript assertion: implement complex conditions
When the condition goes beyond “does it contain this?”, you can use JavaScript assertions to write custom logic. For example, you can compare the order in which two headings appear, count how many times a section appears, or check whether a piece of text falls inside a specific section. These assertions are still deterministic, and results are reproducible.
Evaluate existing Promptfoo outputs with standalone assertion files
Promptfoo can run standalone assertion files against model outputs that have already been generated, without regenerating all outputs. This matters for evaluation iteration: scoring rules often need repeated adjustment, while generation itself is both time-consuming and incurs call costs.
Promptfoo’s llm-rubric: model-graded rubric scoring
llm-rubric is a model-graded assertion. It passes the output and a custom rubric to a grading model, which decides whether it passes and gives a reason. The docs state that the grading provider that performs the scoring can be configured separately, and the prompt used for scoring can also be customized.
| Assertion type | How it judges | Deterministic? | What it proves | What it doesn’t prove |
|---|---|---|---|---|
| contains | String matching | Yes | The specified text appears | Paragraph quality or content correctness |
| JavaScript | Custom code logic | Yes | Codable conditions such as structure, order, and counts | Semantic quality |
| llm-rubric | Model-graded rubric scoring | Affected by the grading model | The output appears to meet the rubric in the grading model’s view | Factual correctness, and it cannot replace human review |
Sources and verification status
The factual portion of this article is based on the following Promptfoo documentation pages, checked on 2026-11-17:
- Promptfoo assertions and metrics documentation: promptfoo.dev/docs/configuration/expected-outputs/
- Promptfoo model-graded assertion documentation: promptfoo.dev/docs/configuration/expected-outputs/model-graded/
- Promptfoo llm-rubric documentation: promptfoo.dev/docs/configuration/expected-outputs/model-graded/llm-rubric/
This article did not run a real old-vs-new template A/B experiment. The configurations and workflows that appear later are design suggestions, not measured results.
Comparing Old and New Article Templates with Promptfoo: Analysis and Judgment
The following is our interpretation and recommendations based on the facts above. Take the “news article template” as an example. Suppose the new template requires an article to include two sections—”objective facts” and “analysis and judgment”—with facts first and analysis second, and no speculation mixed into the facts section.

Experiment Design: Change Only One Variable When Comparing Templates with Promptfoo
The most common mistake when comparing templates is switching templates while also changing the source material, model, or temperature parameter. That way, even if the results improve, you can’t tell whether the template made the difference or something else did. So the first step is to lock down every variable except the template.
| Variable | How it’s handled | Why |
|---|---|---|
| Input source material | Fix the same batch and share it between the old and new templates | Different source material directly changes the factual content |
| Generation model | Fix the same model and version | Writing-style differences between models can be larger than template differences |
| Generation parameters | Fix settings such as temperature and maximum length | Parameters affect output stability and length |
| Prompt template | The only variable that changes | This is what we’re testing |
| Scoring rules | Use the same set of assertions for both groups | Ensures both groups receive the same checks |
Promptfoo Rule Layer and Position Layer: Check Structure First, Then Order
The first layer is a rule check, using contains to confirm whether the headings required by the template are present. The second layer is a position comparison, using a JavaScript assertion to confirm that the “Objective Facts” heading appears before the “Analysis and Judgment” heading. Both layers are deterministic checks: they’re cheap and stable, making them a good first pass to filter out obviously non-compliant outputs.
Below is an illustrative assertion. For the exact field names, refer to the official Promptfoo documentation:
# Illustrative config: shows the approach; verify field details against the official documentation
assert:
# Rule layer: whether the headings appear
- type: contains
value: "## 客观事实"
- type: contains
value: "## 分析判断"
# Position layer: the facts section must come before the analysis section
- type: javascript
value: |
const a = output.indexOf('## 客观事实');
const b = output.indexOf('## 分析判断');
return a >= 0 && b >= 0 && a < b;
Notice that the position-layer check first verifies both headings exist. If you only write a < b, then when the “Objective Facts” heading is missing, indexOf returns -1, which still satisfies the “less than” condition and causes a false positive. This is exactly why deterministic assertions also need to be written carefully.
The Promptfoo Rubric Layer: Using Explicit Rubrics to Check Whether Speculation Has Slipped into Factual Sections
Having all the required headings and the right order doesn’t mean the factual sections are clean. Whether a factual section contains speculation is a semantic judgment that rule-based assertions can hardly cover—which is where llm-rubric comes in. The rubric needs to be specific: tell the grading model what counts as a violation instead of asking vaguely whether the writing is “good.”
# Illustrative config: both the grading provider and the rubric content are examples
assert:
- type: llm-rubric
value: |
只评估“客观事实”章节。
若该章节出现预测、主观评价或未标注来源的推断,判为不通过,并引用出问题的原句。
不评价文采,不评估其他章节。
provider: your-grading-model # The grading provider can be configured separately
Conclusions from the rubric layer still need manual spot checks. The grading model may miss violations, or it may mistakenly flag a reasonable factual statement as speculation. A safer approach is to review every failing sample one by one, then randomly pull a portion of passing samples for manual review.
Reporting by Dimension: Don’t Let Promptfoo’s Writing-Quality Score Cancel Out Factual Errors
If you combine all assertions into a single total score, an article with fluent prose but speculation mixed into the factual sections could end up with a score close to a tightly structured but somewhat bland article. For news-style content, this kind of “mutual offsetting” is dangerous. So it’s better to report results by dimension and make factual dimensions a one-vote veto.
| Evaluation dimension | Check method | Presentation | Can other dimensions offset it? |
|---|---|---|---|
| All required headings present | contains | Pass rate | No — hard requirement |
| Correct section order | JavaScript | Pass rate | No — hard requirement |
| No speculation in factual sections | llm-rubric + manual review | Pass rate + list of problematic original sentences | No — one-vote veto |
| Expression and readability | llm-rubric | Listed separately | For reference only; not used for offsetting |

When only the Promptfoo scoring rules change, re-score the old outputs directly
In evaluation iteration, what changes most often is usually the scoring rules, not the template itself. Maybe you find the rubric description isn’t clear enough, or a positional assertion misses an edge case. In that case, neither the template nor the outputs have changed, so there’s no need to regenerate anything.
Promptfoo’s ability to run an independent assertions file against existing outputs lets you completely separate generation from scoring: generate once, save the outputs, and from then on every rule change only re-runs the assertions. This saves time and guarantees that both rounds of scoring look at exactly the same text — which is what makes the comparison meaningful.
The boundaries and costs of Promptfoo’s model-as-judge
There are three things worth stating clearly, so you don’t over-interpret the evaluation results:
- A model judge’s score can’t prove factual correctness. llm-rubric only says the output matches the rubric as the scoring model sees it; fact-checking still has to go back to the original sources.
- A
containspass only means the text matched. It says the title appeared, not that the paragraph beneath the title is any good. - Paid model reviews cost money. Rule-based assertions mostly don’t trigger extra model calls, but llm-rubric calls the scoring model once for every output it evaluates — so you can’t call the whole suite “completely free.”
Given these boundaries, our judgment is: deterministic assertions work well as a “gate,” model scoring works well as a “first pass,” and human review is the final authority on factual conclusions. The three have different jobs and can’t substitute for each other.

Directions Worth Watching Going Forward
The following are observations that haven’t been verified yet. They’re just suggestions for follow-up experiments:
- Consistency of the scoring model. Is the conclusion stable when the same output is scored multiple times? Does it flip when you switch to a different scoring model? If it isn’t stable, the rubric needs to be more specific.
- How well manual spot checks agree with model scores. This determines how far you can rely on llm-rubric for initial screening.
- Whether the sample size is sufficient. When you compare only a small number of articles, chance factors can mask template differences. You need enough input material to draw meaningful conclusions.
Until there’s empirical data on these questions, any claim like “the new template has a win rate of X” or “the score improved by Y” should be treated with caution. This article deliberately avoids giving numbers like that, and that’s part of the reason.
What you learn from books is shallow after all; to truly know a thing, you must practice it yourself. — Lu You, Written on a Winter Night to Instruct My Son Ziyu
