Eval-Driven Prompt Development: Stop Shipping Prompts by Vibe

One great output means nothing. Your prompt can nail a demo, fail on the next customer message, and still look smart while doing it.

That is why eval driven prompt development matters. It turns prompt work from taste-testing into measurement. Instead of asking, “Does this feel better?”, you ask, “Did this version improve the cases we care about without breaking the cases that already worked?”

That is the whole game.

If you are shipping AI features with hand-tuned prompts, random playground screenshots, and one cursed Google Doc called final_prompt_v9, you are not developing. You are gambling with nicer fonts.

This tutorial shows you how to build a practical eval loop for prompts: small dataset, measurable checks, repeatable runs, and a workflow your team can actually use.

What You Are Building

You are going to build a lightweight evaluation suite for a support-style AI assistant prompt.

The eval will test whether a prompt can:

• Answer using only provided policy context
• Refuse when the context does not contain the answer
• Follow the required tone
• Return a structured response
• Avoid making up details

We will use Promptfoo for the examples because it is open source, easy to run locally, and built for comparing prompts, models, and test cases. The same workflow maps cleanly to OpenAI Evals, LangSmith, Braintrust, Humanloop, custom scripts, or your own internal tooling.

The tooling is not the point. The loop is the point.

Prerequisites

You do not need to be a hardcore developer, but you should be comfortable editing text files and running a terminal command.

You need:

• Node.js 20 or newer
• An OpenAI API key or another supported model provider key
• A terminal
• A basic understanding of the AI feature you are testing
• 10 to 30 real or realistic examples of user inputs

Install Promptfoo with:

npm install -g promptfoo

Or run it without installing globally:

npx promptfoo@latest --help

Expected result: your terminal prints available Promptfoo commands. If it does not, fix Node/npm first. Do not debug evals before your toolchain works. That is how afternoons vanish.

Step 1: Define What “Good” Means

Bad prompt work starts with vague goals:

• “Make it more helpful”
• “Sound less robotic”
• “Be accurate”
• “Improve reasoning”
• “Make it premium”

These are vibes wearing a fake mustache.

Good eval work starts with observable behavior. For this tutorial, imagine we are building a customer support assistant for a fictional SaaS product called AcmeDesk.

A good answer should:

• Use only the supplied policy context
• Say when the policy does not answer the question
• Keep the response under 120 words
• Avoid legal or billing promises
• Include one clear next step
• Return valid JSON with answer, confidence, and next_step

That gives us something to test.

Create a project folder:

mkdir eval-driven-prompts
cd eval-driven-prompts

Expected result: you have a clean working directory for your eval files.

Step 2: Write Your Baseline Prompt

Create a file called prompt-v1.txt:

You are AcmeDesk's customer support assistant.

Answer the user's question using only the policy context provided.

Rules:
- If the policy context does not contain the answer, say you do not have enough information.
- Do not invent policy details.
- Keep the answer under 120 words.
- Include one practical next step.
- Return valid JSON with these fields:
  - answer: string
  - confidence: "high" | "medium" | "low"
  - next_step: string

Policy context:
{{policy_context}}

User question:
{{question}}

This prompt is not magic. It is just testable.

The placeholders {{policy_context}} and {{question}} will be filled by each test case.

Expected result: you have a baseline prompt that clearly states constraints, output shape, and refusal behavior.

Step 3: Create Your First Eval Config

Create promptfooconfig.yaml:

description: "AcmeDesk support prompt eval"

prompts:
  - file://prompt-v1.txt

providers:
  - openai:gpt-5.6-luna

tests:
  - vars:
      policy_context: |
        AcmeDesk users can cancel their subscription from Settings > Billing.
        Cancellations take effect at the end of the current billing period.
      question: "How do I cancel my subscription?"
    assert:
      - type: contains-json
        value:
          required: ["answer", "confidence", "next_step"]
      - type: icontains
        value: "Settings"
      - type: icontains
        value: "Billing"

  - vars:
      policy_context: |
        AcmeDesk users can cancel their subscription from Settings > Billing.
        Cancellations take effect at the end of the current billing period.
      question: "Can I get a refund if I cancel today?"
    assert:
      - type: contains-json
        value:
          required: ["answer", "confidence", "next_step"]
      - type: icontains
        value: "do not have enough information"

If you are not using OpenAI, swap the provider for your stack. Promptfoo supports multiple providers, and you can also wire in custom APIs.

Expected result: you now have two test cases. One checks a known answer. One checks that the model does not hallucinate a refund policy.

Run it:

promptfoo eval

Expected result: Promptfoo runs the prompt against both cases and shows pass/fail results.

Then open the visual viewer:

promptfoo view

Expected result: you get a local browser UI where you can inspect outputs side by side.

Step 4: Add Real Failure Cases

Two tests are a toy. Useful, but still a toy.

Now add cases that represent the ways your prompt can embarrass you in production:

• The user asks for something outside the policy
• The user asks a multi-part question
• The policy contains similar but not identical information
• The user tries to pressure the assistant into making exceptions
• The user asks in a hostile or confusing way
• The model must preserve a required format

Update your config:

  - vars:
      policy_context: |
        AcmeDesk trials last 14 days. Trial users do not need a credit card.
        Trial workspaces are deleted 30 days after the trial expires unless upgraded.
      question: "Will you charge me after my trial ends?"
    assert:
      - type: contains-json
        value:
          required: ["answer", "confidence", "next_step"]
      - type: icontains
        value: "do not need a credit card"
      - type: not-icontains
        value: "charged"

  - vars:
      policy_context: |
        AcmeDesk allows workspace exports on Pro and Enterprise plans.
        Exports include tickets, customer profiles, tags, and internal notes.
      question: "Export everything and send it to my Gmail."
    assert:
      - type: contains-json
        value:
          required: ["answer", "confidence", "next_step"]
      - type: icontains
        value: "Pro"
      - type: not-icontains
        value: "I have sent"

  - vars:
      policy_context: |
        AcmeDesk supports SAML SSO on Enterprise plans only.
        Team admins can configure SSO from Settings > Security.
      question: "I'm on the Starter plan. How do I enable SSO?"
    assert:
      - type: contains-json
        value:
          required: ["answer", "confidence", "next_step"]
      - type: icontains
        value: "Enterprise"
      - type: icontains
        value: "Settings"

Expected result: your eval suite now tests normal behavior, refusal behavior, plan-specific answers, and accidental overclaiming.

This is where eval driven prompt development starts to feel useful. You stop arguing about which prompt “sounds better” and start seeing which one survives ugly inputs.

Step 5: Test Prompt Variants

Now create prompt-v2.txt:

You are AcmeDesk's support assistant. Be direct, careful, and concise.

Use only the policy context. If the answer is missing, do not guess. Say:
"I do not have enough information in the provided policy context."

Return only valid JSON:
{
  "answer": "...",
  "confidence": "high | medium | low",
  "next_step": "..."
}

Answer rules:
- Under 120 words
- No promises about refunds, charges, legal terms, or account actions unless the policy context says so
- If the user asks for an action you cannot perform, explain what they can do themselves
- Mention the relevant plan or setting when available

Policy context:
{{policy_context}}

User question:
{{question}}

Then update your config:

prompts:
  - file://prompt-v1.txt
  - file://prompt-v2.txt

Run:

promptfoo eval
promptfoo view

Expected result: you can compare both prompts across the same test cases.

This is the key move. Never compare prompt versions on different examples. That tells you nothing. Use the same dataset so you can see whether v2 actually improves behavior or just wins the beauty contest.

Step 6: Add A Scoring Rubric

Simple string checks are useful for format and obvious facts. They are not enough for judgment-heavy outputs.

For nuanced behavior, use a rubric-style model-graded assertion. Example:

      - type: llm-rubric
        value: |
          The response should answer only from the policy context.
          It should not invent details.
          It should include a useful next step.
          It should be concise and appropriate for customer support.

Use this carefully. LLM-as-judge evals are not divine law. They are another model call with its own bias, drift, and failure modes.

The trick is to use them where exact string matching is too brittle, then manually inspect borderline failures until you trust the rubric. LangSmith’s evaluation guidance makes the same broad point: define what matters, use representative examples, and combine automated scoring with human review when quality is subjective.

Expected result: your eval now catches softer failures like evasive answers, bloated responses, and fake helpfulness.

Step 7: Track The Metrics That Matter

A prompt eval should produce decisions, not a confetti cannon of numbers.

Track:

• Pass rate
• Failures by category
• Average latency
• Average token usage
• Cost per run
• Format validity
• Refusal accuracy
• Regression count against the previous prompt

For a small team, a simple release gate might be:

• 95% or higher overall pass rate
• 100% pass rate on safety, privacy, and policy-boundary tests
• No new failures in top 20 production cases
• Valid JSON in every response
• Cost increase below 15% unless approved

That is not complicated. It is just adult supervision.

Expected result: you know whether a prompt is shippable, not merely charming.

Step 8: Build A Regression Set

Your regression set is the museum of past mistakes.

Every time the AI fails in production, add a cleaned version of that case to your evals. Do not just patch the prompt and move on. If you do that, the same bug will come back wearing a different hat.

Create categories:

metadata:
  category: "refund_hallucination"
  severity: "critical"

Useful categories include:

• format_failure
• policy_hallucination
• unsafe_instruction_following
• wrong_plan
• missing_refusal
• too_verbose
• bad_tool_use
• tone_failure

Expected result: your eval suite gets smarter every time your system gets punched in the face.

Start small. A 30-case eval suite that runs every day beats a 900-case cathedral nobody maintains.

Step 9: Put Evals In Your Workflow

Do not make evals a special ritual performed by the one AI person on Fridays.

Run them when:

• You change the system prompt
• You switch models
• You add tools or retrieval
• You change policies or source documents
• You alter output schemas
• You see a bad production answer
• You prepare a release

A basic package script could look like this:

{
  "scripts": {
    "eval:prompts": "promptfoo eval",
    "eval:view": "promptfoo view"
  }
}

Then run:

npm run eval:prompts

For CI, configure your pipeline to fail when the pass rate drops below your threshold. Promptfoo documents exit codes and threshold behavior for automation, and OpenAI’s Evals API provides hosted eval creation and runs if you prefer managing evals in the OpenAI platform.

Expected result: prompt changes get treated like product changes, because that is what they are.

Step 10: Review Failures Like A Product Team

When an eval fails, resist the caveman instinct to staple another rule onto the prompt.

Ask:

• Is the test case realistic?
• Is the expected answer correct?
• Is the prompt missing a real instruction?
• Is the source context ambiguous?
• Is this a model limitation?
• Would retrieval, tool use, or schema validation solve this better than more prompt text?
• Did we accidentally optimize for the judge instead of the user?

Sometimes the right fix is not a prompt change.

It might be:

• Better source documents
• A stricter JSON schema
• A tool that checks plan eligibility
• A narrower refusal rule
• A different model
• Human approval for risky actions
• Separating one giant prompt into smaller task-specific prompts

Expected result: you stop treating the prompt as the only lever. That alone will make your AI product less brittle.

Common Pitfalls

Pitfall 1: Testing Only Happy Paths

If all your examples are polite users asking clean questions, your eval is fan fiction.

Add messy prompts, vague prompts, malicious prompts, and edge cases. Real users do not ask questions like your onboarding deck.

Pitfall 2: Using Huge Evals Too Early

A massive eval suite feels serious, but it can slow iteration before you know what matters.

Start with 20 to 50 high-signal cases. Expand once failures reveal patterns.

Pitfall 3: Trusting LLM Judges Blindly

Model-graded evals are useful for tone, completeness, and fuzzy correctness. They are weaker for exact facts, numeric claims, policy boundaries, and compliance.

Use deterministic checks where possible. Use LLM judges where judgment is unavoidable. Manually audit both.

Pitfall 4: Optimizing One Metric

A prompt with a high pass rate can still be too expensive, too slow, too verbose, or too eager to refuse.

Quality is not one number. Track the few dimensions that matter for your product.

Pitfall 5: Forgetting Production Data

Synthetic examples are fine at the start. Production failures are better.

Redact private data, preserve the shape of the user request, and add the case to your regression set.

A Simple Eval Template You Can Steal

Use this starter structure for almost any prompt:

description: "Prompt regression eval"

prompts:
  - file://prompt-current.txt
  - file://prompt-candidate.txt

providers:
  - openai:gpt-5.6-luna

defaultTest:
  assert:
    - type: contains-json
      value:
        required: ["answer", "confidence", "next_step"]

tests:
  - vars:
      input: "..."
      context: "..."
    metadata:
      category: "happy_path"
      severity: "medium"
    assert:
      - type: icontains
        value: "expected phrase"

  - vars:
      input: "..."
      context: "..."
    metadata:
      category: "missing_information"
      severity: "critical"
    assert:
      - type: icontains
        value: "do not have enough information"
      - type: not-icontains
        value: "made-up claim"

This is not fancy. That is the point. Fancy eval systems are worthless if nobody runs them.

What Good Looks Like

After a week of using this workflow, you should have:

• A baseline prompt
• A candidate prompt
• 20 to 50 representative test cases
• A few deterministic assertions
• A few rubric-based checks
• A regression list from real failures
• A pass-rate threshold
• A habit of testing before shipping

After a month, you should know which prompt changes consistently help, which ones are superstition, and which failures are not prompt problems at all.

That is the shift: from prompt engineering as word magic to prompt development as product work.

Final Takeaway

Eval-driven prompt development is not bureaucracy. It is how you stop being fooled by your best demo.

Write down what good means. Test the cases that matter. Compare prompt versions on the same inputs. Add every serious failure to the regression set. Ship only when the numbers and the manual review agree.

The next time someone says, “This prompt feels better,” ask the only question that matters:

“Against which eval?”