Prompt Regression Testing Kit

Change one line of a prompt (or switch models) and something that used to work quietly breaks: the refund timeline disappears, the JSON gets an extra field, the sign-off is gone. This kit gives your prompts unit tests. Write test cases in YAML, run them against any model, and see which cases fail, which became flaky, and what regressed since the last run, in the terminal or in CI.

• Test suites in YAML or JSON: prompts (inline or file://prompts/x.txt) with {{variables}}, test cases inline or from a .jsonl, .json or .csv file (a spreadsheet of cases works).
• 17 assertion types: contains, not-contains, contains-any, contains-all, equals, starts-with, regex, not-regex, max-length / min-length (chars or words), is-json, json-schema, json-path, similar (token overlap), latency, cost, and llm-rubric (a grader model judges tone or correctness). Any assertion can take not: true. JSON is found inside code fences.
• Compare prompt versions and models side by side: every prompt × provider × test case, with a pass count per prompt.
• Flakiness: --repeat 3 runs each case several times; cases that pass only sometimes are reported as flaky.
• Baselines: --out baseline.json, then --baseline baseline.json --fail-on regression lists regressions, fixed cases and new failures.
• Providers: OpenAI, Anthropic, OpenRouter, any OpenAI-compatible server (Ollama, LM Studio, vLLM), or any program (command:). Keys come from environment variables and are never written to results.
• Response cache (.promptcheck/cache) so re-running unchanged prompts costs nothing; repeats bypass it.
• Reports: text, Markdown (for PR comments / $GITHUB_STEP_SUMMARY), JUnit XML (for CI test tabs), JSON.
• Python 3.9+ standard library only. A Claude Code skill (skills/prompt-tests) and a GitHub Actions workflow are included.

python3 -m promptcheck init tests/prompts                     # starter suites
python3 -m promptcheck run tests/prompts/support-replies.yaml --provider openai:<model>
python3 -m promptcheck run suite.yaml --provider anthropic:<model> --provider openai:<model>   # compare models
python3 -m promptcheck run suite.yaml --repeat 3 --out baseline.json
python3 -m promptcheck run suite.yaml --baseline baseline.json --fail-on regression
python3 -m promptcheck run suite.yaml --format junit > results.xml
python3 -m promptcheck run suite.yaml --provider "openai-compatible:llama3.2@http://localhost:11434/v1"   # local Ollama

Exit codes: 0 pass, 1 failures (or regressions with --fail-on regression), 2 suite or provider error.

./demo.sh runs everything offline with examples/fake_model.py standing in for an LLM: two versions of a support-reply prompt (v2 adds "keep replies under 40 words" and loses the refund timeline and the sign-off: 2 of 4 cases fail) and a JSON ticket classifier checked against a JSON Schema. ./run_tests.sh runs 8 test groups, including fake OpenAI and Anthropic servers.

providers:
  - openai:<model>
grader: openai:<model>          # only for llm-rubric
prompts:
  - id: reply-v1
    template: file://prompts/reply-v1.txt
defaults:
  assert:
    - type: not-regex
      value: "as an ai|language model"
tests:
  - name: double charge gets refund + timeline
    vars: {name: Priya, message: I was charged twice this month.}
    assert:
      - type: contains-any
        value: [refund, refunded]
      - type: regex
        value: "\\d+\\s*-\\s*\\d+ business days"
      - type: llm-rubric
        value: apologizes once, doesn't blame the customer

json-schema supports type, enum, const, required, properties, additionalProperties, items, min/maxItems, min/maxLength, minimum/maximum, pattern, anyOf/oneOf/allOf. json-path takes $.a.b[0] with value or one_of (or neither: the path must exist). cost takes price_per_mtok: {input, output} (your model's price per million tokens). Suite defaults can set temperature (only sent when set, since reasoning models reject it; a 400 about temperature is retried without it) and max_tokens (default 4096; reasoning tokens count toward it).

• It tests outputs, not prompts: a passing suite only means your test cases pass. Add a case for every bug you fix.
• llm-rubric results depend on the grader model; run it with --repeat and keep text checks for anything exact.
• No web dashboard: results are files (JSON, Markdown, JUnit). No streaming, tool-calling or image inputs.
• similar is token overlap, not embeddings.