Gatecheck — a testing layer for LLM systems

Shipping with LLMs without evals is shipping on vibes. Gatecheck is the testing layer for model-backed systems — and the name is the thesis: check the outputs, then gate the build on them. Building it is the same instinct as the verification work, aimed at a fuzzier target.

The problem

"Prompt a few examples and eyeball the output" doesn't scale and doesn't gate. You need eval cases grouped by capability, checks that are reproducible enough to run on every commit, and a way to fail the build when a prompt edit or model swap regresses quality.

Approach

• Task suites in plain YAML — add a test without writing code.
• Two grader families. Deterministic checks (exact, contains, regex, json_schema) that are reproducible and free, plus an LLM-as-judge scoring 1–5 against a rubric for subjective qualities like faithfulness and tone.
• Regression gating. Snapshot a baseline, then fail (non-zero exit) when a suite's pass rate drops past a tolerance — wired into GitHub Actions, skipped cleanly on key-less forks.
• Provider-agnostic, local-first. One interface, uniform raw-HTTP adapters — Ollama (local, free, key-less), OpenAI, Anthropic, and AWS Bedrock via a hand-rolled, test-vectored SigV4 signer — addressed uniformly as provider:model.

Why it's trustworthy — the gate-vs-gauge line

The core decision is which signals are safe to gate on. Deterministic graders gate CI; the LLM judge does not — it captures real quality but isn't reproducible enough to block a PR, so it runs locally or on a schedule. An explicit verdict rule (a task passes only if every deciding grader passes; pass rate is computed over deciding tasks only) keeps the numbers honest — pass rates ship with 95% confidence intervals rather than bare points — and local/unlisted models are priced at $0 rather than guessed.

The lane it occupies

promptfoo, Inspect AI, and Braintrust are all good tools — and all heavier than many projects need. Gatecheck is deliberately small: a dependency-light, self-hostable, vendor-neutral Python harness (three runtime dependencies, no SaaS, no account, no telemetry) you can read end-to-end in an afternoon and have gating CI the same day. Every artifact it produces is a plain file you can diff and commit. Known gaps, tracked in the roadmap: no end-to-end test against a live provider yet, single-sample judging (variance unmeasured), and partial seed support.