← All work2025 · AI Agents · Infrastructure

LLM-as-a-Judge Framework

Behavioral testing for AI agents — judges that converse, verify with tools, and score.

RoleAuthor
StackPython 3.11+ · LLM-as-a-judge · pytest
Year2025

Client work — case study anonymized; architecture and metrics are real.

Judges that do more than opine

The framework's judges are pluggable across three model providers and come in tiers — default at 80, strict at 100, lenient at 60 — so different scenarios can be held to different bars. Crucially, judges can be equipped with tools through a registry, so a judge verifies a claim (does the cart actually contain what the reply says?) instead of just grading prose.

Goal-driven conversations

Beyond single-turn and scripted multi-turn tests, the framework supports goal-driven mode: the judge itself generates the follow-up queries, conducts an adversarial conversation toward a goal, decides when stopping conditions are met, and evaluates the full transcript. Every test captures token counts, duration, and time-to-first-token.

Built with

Every technology this project runs on, and the job it does here. Each one links to the other projects it appears in.

Languages & runtimes

Python 3.11+↗Primary backend language: agent services, ETL pipelines, eval harnesses and CLI tooling. Typed end to end with Pydantic v2 models, checked under mypy --strict.YAML↗Test-suite configuration, judge thresholds and CI pipeline definitions.

AI & agent systems

LLM-as-a-judge↗Behavioural testing with 0–100 scoring, tiered pass thresholds (60 / 80 / 100), judges equipped with tools so they verify a claim instead of opining on it, and goal-driven multi-turn conversations the judge itself steers to a stopping condition.Google Gemini↗2.5 Pro, Flash and Flash-Lite for reasoning and conversation across most agent products; vision models for reading colorimetric test strips from uncontrolled phone photos; image generation for visual try-on.OpenAI↗Judge model in evaluation suites, and one voice in multi-model consensus used to synthesise test cases for other agents.Groq↗The fast structuring pass in two-model pipelines, chosen for time-to-first-token.

Testing & quality

pytest↗879 test functions across 39 files in the infrastructure library alone — custom fixtures, CLI options, and a package designed to drop into any project by editing one import.Eval harnesses↗32 archived, timestamped evaluation runs; controlled A/B of raw model versus agent wrapper on identical queries to isolate framework overhead; time-to-first-token probes across query complexity.

By the numbers

3Judge tiers
4Test types
3Judge model providers