Trust it · AI evals & regression harness
Know your prompts still work before your users do.
Groundtruth is unit tests for prompts and agents. Change a prompt or bump a model, and outputs drift silently until someone complains. This is a cheap, ownable harness — assertions, an LLM judge, snapshot regression, and a CI gate — that catches the drift on the way in.
Assertions that run offline and free
equals, contains, notContains, matches, jsonPath and a predicate escape hatch — deterministic checks that need no API key, so most of your suite runs for nothing.
LLM-as-judge for the fuzzy stuff
Rubric-scored checks for things a rule can't capture — tone, helpfulness. Provider-agnostic (Anthropic or any OpenAI-compatible endpoint), bring-your-own-key, and skipped automatically when no key is set so suites still run.
Snapshot regression
A baseline pins which cases pass. Every run flags anything that silently flipped from passing to failing — the drift you'd otherwise ship.
A CI gate that fails the build
One command exits non-zero when pass-rate drops or a case regresses, plus a GitHub Action and a JSON report for trend history. Quality can't quietly slide.
Straight from the source
A real suite from the kit — swap the classifier for your model call and the harness is unchanged.
export const sentimentSuite: Suite<{ text: string }, string> = {
name: "sentiment-classifier",
task: (input) => classify(input.text), // ← your prompt / agent call
cases: [
{ name: "clear positive",
input: { text: "I love this, it's amazing" },
assertions: [equals("positive")] },
{ name: "tone check",
input: { text: "The support team was wonderful" },
assertions: [contains("positive"), llmJudge({ rubric: "..." })] },
],
};What's in the zip
A real slice of the shipped tree — small enough that you and your agent can read every file.
- src/assertions.tsdeterministic, offline assertion factories
- src/judge.tsLLM-as-judge — provider-agnostic, skips without a key
- src/snapshot.tsbaseline + regression detection
- src/gate.tspass-rate + regression gate logic
- src/runner.tsruns a suite, aggregates results
- src/suites/example.eval.tsa worked suite — the pattern to copy
- snapshots/baseline.jsonthe pinned passing baseline
- .github/workflows/evals.ymlruns the gate in CI on every push
- test/smoke.test.tsthe harness has its own tests
The honest scope
Groundtruth isn't a hosted eval platform with a dashboard and a subscription. It's a small, readable harness you drop into your repo and own forever — the 20% that catches 80% of prompt regressions, wired to your CI. Most of it runs offline and free; the LLM judge is opt-in and only bills when you set a key. Every claim here is true of the code in the zip: it compiles, it's tested, and the example gate actually runs and exits.
In the box
- TypeScript, runs on the built-in node test runner
- Six assertion factories — all deterministic and offline
- Provider-agnostic LLM judge, bring-your-own-key, auto-skips
- Snapshot baseline + regression detection
- CI gate + GitHub Action + JSON trend report
- A worked example suite you copy to start
Questions
- Do I need an API key to use it?
- Not for most of it. The assertion checks run fully offline. Only the optional LLM-as-judge needs a key, and if you don't set one those checks are skipped cleanly — the rest of the suite still runs and gates.
- Which models can the judge use?
- It's provider-agnostic: Anthropic's Messages API or any OpenAI-compatible endpoint, configured by env. Bring your own key and model.
- How does it fit my pipeline?
- Run one command locally or in CI. It exits non-zero when pass-rate drops below your threshold or a snapshot regresses — the included GitHub Action fails the build, and a JSON report captures the trend.
- What licence do I get?
- Unlimited personal and client projects. You can't resell Groundtruth itself as a competing harness.
Own it outright. Extend it with your agent.
New to the fleet? Start free with AgentShip Lite →