An FAQ assistant can sound convincing while answering from a policy that says something else. A small benchmark gives its owner a way to see the actual answers, compare two model settings on the same questions and examine the mistakes before choosing either one. This guide uses Promptfoo with an approved FAQ or policy file. It is a documented procedure: BIG CHANGE has not run the models, generated questions, measured a score or checked an account bill.

The intended reader owns the source text and can approve its use with an API provider. By the end, the reader should have a frozen question set, separate results for two models, the original JSON exports and a written decision from the source owner. The owner must review the answers before accepting the comparison.

The big change

  • What changed: Promptfoo can run the same prompt and test cases against multiple providers, grade the responses and export the underlying results. Its dataset generator can also propose additional development cases.
  • Why it matters: A source owner can freeze checked questions, compare two model settings and inspect the actual answers before choosing either model for the team’s FAQ.
  • What to watch: People must verify the expected answers and keep proposed questions out of the untouched held-out set. The documentation establishes a repeatable evaluation procedure; the reader’s run and review determine how each model answers the approved policy.

Prepare the source, access and spending decision

Use one approved, versioned FAQ or policy document and questions the team has permission to send to OpenAI's API. Name a source owner to check every reference answer and a data steward to approve the material sent to the generator, target models and graders. Remove customer details, personal information and secrets before any call. Keep the source, keys, prompts and result exports private. Promptfoo's --no-share prevents a shareable result link; OpenAI's store: false setting does not promise zero data retention. Review the API data controls against your organization's policy.

You need a local Promptfoo installation, OpenAI API access to the chosen models, billing permission and an approved spend envelope. Record promptfoo --version, the resolved model IDs, the policy revision and file hashes before running. The current Promptfoo manual does not identify a package version for these instructions. Its relevant pages and the OpenAI model listings were checked on October 21, 2026; confirm syntax, model access and current prices in the account you will use. A ChatGPT subscription is separate from API billing.

For a contained first comparison, the owner can choose one complete approved corpus of at most 12,000 characters, questions of at most 500 characters each and no more than 24 distinct approved cases. These are trial limits for this guide, not Promptfoo or OpenAI limits. If a governing policy is longer, do not cut out controlling sections merely to meet the trial size; choose a different complete source or revise the budget and design.

Budget for three kinds of calls: question suggestions, answers from both targets and separate grading calls. A planned result for two models can cause two grader calls per answer when both factuality and rubric checks run. Generation internals, errors and transport retries can add calls. A project spend notification is an alert. An optional enforced hard limit can stop later requests, but the documentation warns that usage can exceed it briefly while the limit propagates. Check usage and remaining authorization before each stage; no command below guarantees an exact bill cap.

1. Freeze real questions before generating suggestions

Create approved-policy.md, answer-prompt.txt and an inventory of real, approved questions. For each question, record a stable case ID, the expected answer or an explicit not answered in this policy outcome, and the precise section that supports that judgment. Divide observed questions into development and held-out groups by question intent and source section. Put near duplicates and paraphrases in the same group. Do not show held-out questions to a generator or use their results to revise the prompt.

Save this complete answer-prompt.txt beside the YAML files. It requests a source section, although the model-graded assertions below will not automatically enforce exact section identity. The source owner checks that manually.

Text
Use only the policy below. Answer the question and identify the supporting section. If the policy does not answer, say that clearly. Do not invent details.

Policy:
{{policy}}

Question: {{question}}

2. Ask for a small set of proposed development questions

Promptfoo documents promptfoo generate dataset, an output file and a separate generation provider. Keep that provider distinct from the two models being compared. Save synthesis-provider.yaml with the generation model's explicit key source, low reasoning effort and 1,024-token output cap:

YAML
id: openai:responses:gpt-6-luna
config:
  apiKeyEnvar: FAQ_SYNTH_API_KEY
  store: false
  reasoning:
    effort: low
  max_output_tokens: 1024

Save seed-config.yaml with only the approved policy and development seed cases. dev-seed-cases.yaml must already contain real owner-verified rows. The target-provider entry below gives the seed config a complete Promptfoo shape; the --provider argument in the generation command selects the separate synthesis provider. Promptfoo's dataset guide explains the command and its options.

YAML
prompts:
  - file://answer-prompt.txt
providers:
  - id: openai:responses:gpt-6-luna
    config:
      apiKeyEnvar: FAQ_TARGET_API_KEY
      store: false
      reasoning:
        effort: low
      max_output_tokens: 512
defaultTest:
  vars:
    policy: file://approved-policy.md
tests:
  - file://dev-seed-cases.yaml

After the owner approves the input and budget, a bounded request could use:

Terminal
promptfoo generate dataset -c seed-config.yaml --provider file://synthesis-provider.yaml --numPersonas 2 --numTestCasesPerPersona 3 -o proposed-cases.yaml -i "Propose questions or paraphrases only from this approved development source; do not invent answers."

Two personas times three requested cases means six proposals, not six accepted cases or a known number of API calls. Save the output and its hash. The source owner should reject duplicates, sensitive text, unsupported premises and any proposed answer that cannot be traced to the policy. Add at most six individually verified suggestions to the development set, each tagged as AI proposed. Keep genuine observed questions separately tagged. No proposed case enters held-out evaluation. This design permits one initial synthesis invocation and at most one separately authorized second invocation. Pause and inspect the generation project's usage before deciding whether the second is safe.

3. Configure two targets and a separate grader

Promptfoo's OpenAI provider documentation supports openai:responses: model IDs, an explicit apiKeyEnvar and store: false. Build tests-development.yaml from owner-reviewed rows. The following is the shape of one row, with instructions in angle brackets rather than actual case data. Replace every placeholder with an approved question, human reference, source section and stable ID before running. It is deliberately not a runnable example.

YAML
- description: '<owner-assigned stable case ID; replace with real ID>'
  metadata:
    origin: '<observed or human-approved AI paraphrase>'
    split: '<development or held-out>'
  vars:
    question: '<real approved question>'
    reference: '<owner-verified answer or explicit abstention>'
    source_section: '<owner-verified policy section or no-answer marker>'

Save this full promptfooconfig-development.yaml after the row files are filled and reviewed. Both target models see the same prompt and policy; the grader is a third, explicitly configured provider. Confirm the model IDs and low reasoning effort are available to the actual API projects. A cut-off answer or grade under the starting 512-token output caps is an error or unknown result, never a successful grade.

YAML
prompts:
  - file://answer-prompt.txt
providers:
  - id: openai:responses:gpt-6-luna
    config:
      apiKeyEnvar: FAQ_TARGET_API_KEY
      store: false
      reasoning:
        effort: low
      max_output_tokens: 512
  - id: openai:responses:gpt-6-sol
    config:
      apiKeyEnvar: FAQ_TARGET_API_KEY
      store: false
      reasoning:
        effort: low
      max_output_tokens: 512
defaultTest:
  vars:
    policy: file://approved-policy.md
  options:
    provider:
      id: openai:responses:gpt-6.1-sol
      config:
        apiKeyEnvar: FAQ_JUDGE_API_KEY
        store: false
        reasoning:
          effort: low
        max_output_tokens: 512
  assert:
    - type: factuality
      value: '{{reference}}'
    - type: llm-rubric
      value: 'Check that the answer and cited section are supported by this approved policy: {{policy}}. Required section: {{source_section}}. A missing answer should be an explicit abstention.'
    - type: cost
      metric: inference_cost
      weight: 0
tests:
  - file://tests-development.yaml

The configuration and test-case references describe the file-backed policy variable and multiline test import. Hash the approved source, prompt, config and row files before running. Keep a separate held-out config; never import both splits into a development run.

The full openai:responses:gpt-6.1-sol grader object sits under defaultTest.options.provider, using a separate FAQ_JUDGE_API_KEY, store: false, low reasoning effort and a 512-token cap. Keep this full object in force: a per-test or per-assertion provider setting can override it. Promptfoo's factuality checker allows consistent subsets, supersets and immaterial differences by default, so a pass is not exact agreement with the human reference. The rubric is another model's judgment. Have the owner check section claims, abstentions, risky answers and disagreements against the source.

The cost assertion with metric: inference_cost and weight: 0 retains provider-reported target cost without making cost a quality grade. Missing or invalid cost is unknown, not zero. Keep API credentials in environment variables or a private secret store and out of exported configs.

4. Run development, then lock held-out evaluation

Set the planned case counts and repetitions before spending. Permit an initial development pass and, if the owner authorizes a change, one revised development pass. Run each once. After the prompt, reference set, grader and budget are locked, run the held-out set twice. Use a unique output filename for every stage and preserve the original export. A development command follows the documented CLI pattern:

Terminal
promptfoo eval -c promptfooconfig-development.yaml --no-cache --no-share --max-concurrency 1 --delay 250 --repeat 1 --output development-v1-results.json

For the held-out stage, copy the frozen development config to promptfooconfig-heldout.yaml and change only its final test import to - file://tests-heldout.yaml. Keep the prompt, policy, target and grader settings identical. After the owner approves the locked row file and spend, run this separate stage:

Terminal
promptfoo eval -c promptfooconfig-heldout.yaml --no-cache --no-share --max-concurrency 1 --delay 250 --repeat 2 --output heldout-v1-results.json

--no-cache avoids Promptfoo's response cache for a fresh-call timing comparison; it does not disable provider-side prompt caching. One concurrent evaluation and a 250 ms delay make this a more bounded run, without promising fixed latency or cost. Review the installed version's retry behavior before launch and pause between stages to inspect raw errors, usage and remaining spend. If the source, prompt or model settings change after the lock, version the evaluation again instead of calling it the same held-out run.

5. Read the rows before choosing a model

Open each original JSON result export and record its hash and evaluation ID. Count planned, attempted, completed and successfully graded rows for each target, split by development versus held-out and observed versus AI-proposed origin. Show failures, refusals, malformed or length-limited answers and absent metrics explicitly. Preserve raw text, assertion reasons, available token use, provider-reported estimated cost and latency. Reopen the policy and check every high-risk, failed or disputed answer plus a preselected sample of passes. Record reasons when a person disagrees with either grader.

Review the correct organization, project and UTC interval in OpenAI's Usage Dashboard. Otherwise-idle separate projects for synthesis, target answers and grading make stage billing easier to attribute; project IDs may be recorded, keys must stay private. Allow for reporting lag and compare dashboard exports with Promptfoo rows. A shared project or missing settled billing can leave the all-in or stage cost unknown. Do not turn the target's cost field into a claim about total generator, grader and retry spending.

If a run stops, inspect its saved evaluation ID, original export and project usage before using Promptfoo's documented resume path. This trial permits at most four selected error-row retries across all stages, after a separate owner decision and bill check. It does not reserve four extra blind API calls: each row can entail target and grader work. promptfoo eval --retry-errors retries all ERROR rows from the latest evaluation. If the count exceeds the remaining four-row allowance, do not use that command; verify a selective retry route or stop. Before any retry, keep the original export and its hash, check possibly billed attempts, and keep --no-share in effect because a cloud-configured retry can otherwise share results. Save later results separately. A missing source answer, missing model row, blank grade or unknown bill calls for repair and review, not a pass.

The final report should let the source owner reopen the policy, case files, configs and exports and see exactly how each conclusion was reached. A small FAQ test can support a local choice for the tested questions. It does not prove reliability on every future question or remove the need for a human to maintain the policy and review consequential answers.

For adjacent decisions, BIG CHANGE has a GPT-6 production model selection guide and a report on how AI judges vary by task and fallback model. Neither supplies ground truth for this team's FAQ.

Sources & further reading