How to Evaluate Your Agents: Trustworthy Quality at Scale with LLM-as-a-Judge

You shipped an agent—but how do you know if it's good? When "correct" is a judgment call and outputs run at massive volume, manual review doesn't scale, and engagement metrics lag reality. We built a reusable evaluation platform that uses an LLM as the judge to turn subjective quality into an automated, continuous signal. This signal runs at three moments of truth: offline to select models and prompts, in CI/CD to catch regressions before shipping, and in production to monitor live traffic. We'll dig into what teams most often get wrong: designing a golden set that mirrors real traffic, why a representative set measures the product while a balanced set validates the judge, and how to "judge the judge" by calibrating against human labels before trusting it to gate. You'll leave with a practical blueprint for evaluating your own agents in production—and the pitfalls to avoid, from judge bias and drift to the data ratios that make or break a golden set.