How we develop and test evaluators
An evaluator is a measurement instrument, not a prompt that sounds sensible.
When PromptSafe scores your agent, the score comes from an evaluator. If the evaluator is wrong, every score it produces is wrong in the same direction, and it looks exactly as confident as a correct one.
So we do not assume our evaluators are right. We test them, and we record how much the evidence actually supports.
This page explains how that works, in plain language, and is honest about where the evidence runs out.
An evaluator can fail without looking like it has failed
The obvious worry about automated evaluation is that it gives a wrong score. The harder problem is that an evaluator can stop distinguishing between good and bad conversations altogether while its average score still looks entirely reasonable.
In our own research, changing only the wording of an evaluator, leaving the behaviour it was meant to detect untouched, was enough to remove almost all of its ability to tell cases apart. Nothing about the output announced this. You only find it by testing for it deliberately.
That is why we treat an evaluator as an instrument that has to be checked, rather than as a prompt that reads well.
Working, working sensibly, and working correctly
These sound similar and are routinely treated as one thing. Keeping them apart is most of the discipline.
- DevelopmentHow do we make this evaluator better?Writing and rewriting the evaluator, then checking whether a change actually improved it rather than just reading better.
- DiagnosticsIs this evaluator behaving sensibly?Checking for operational faults: an evaluator that abstains on nearly everything, or one that has quietly stopped telling cases apart.
- ValidationHow much evidence supports a conclusion about it?Comparing its judgements against expert-rated examples. This is the only one of the three that can tell you an evaluator is wrong rather than broken, and it needs a reference standard to exist in the first place.
A clean diagnostic result is not validation. Diagnostics can catch an evaluator that has stopped working. It cannot catch one that is working wrongly.
The work behind one evaluator
Every evaluator we develop goes through the same sequence, and each step is recorded so the reasoning can be inspected later.
- Define the behaviourWrite down what the evaluator is supposed to detect, in terms specific enough that two people would agree whether a given conversation contains it.
- Build a first versionTurn that definition into an evaluator, with the scoring bands and the conditions under which it should decline to score at all.
- Compare versions on the same conversationsWhen the wording changes, both versions are run over the same material rather than measured separately. Comparing them directly is far more efficient, and it is the only way to tell a real improvement from noise.
- Test against expert ratings where they existWhere a reference standard exists for the behaviour, the evaluator's judgements are compared against expert-rated examples. Where none exists, we record that rather than substituting a weaker check.
- Assess how strong the evidence isSeparately from whether the evaluator looks good, we record how much confidence the evidence actually supports, and what the next piece of evidence would need to be.
- Keep watching how it behaves in useAn evaluator can look fine on curated test material and behave differently against the volume and variety of conversations that real testing produces, so we track it over time. What is being watched here is our evaluator running inside PromptSafe, not your agent running in production. PromptSafe tests before deployment and does not monitor live systems.
Most evaluators on the platform are written by the teams using them, not by us. This is the process we follow for the evaluators we develop, and the one we bring to enterprise engagements where we build evaluators with a customer.
How good it is, and how much we know, are different questions
Evidence strength describes how confident we are in a conclusion. It does not describe the quality of the evaluator. A good evaluator can be poorly evidenced simply because not enough testing has been done yet, and saying so is more useful than implying a confidence we have not earned.
How consistently an evaluator judges the same material also varies with the model doing the judging and the configuration it runs under. That is a reason to measure consistency rather than assume it, which is what we do. It is not something we would offer you as a headline number, because an evaluator that reliably returns the same wrong answer is perfectly consistent and no use to anyone.
Where uncertainty remains, the honest output is the next piece of evidence that would resolve it, not a conclusion stretched to cover the gap.
Including when the answer is inconvenient
During our own evaluator research, one of our reference evaluators produced a performance estimate that looked acceptable. The data behind it was not strong enough to confirm that conclusion with confidence, so we recorded it as insufficient evidence rather than treating our own reference as automatically correct.
We also withdraw findings that do not hold up. Several results from our research programme looked convincing, including one that would have made a good story about catching a dangerous evaluator before deployment. It did not survive testing on more data, so it was retracted rather than quietly narrowed until it was technically true.
A programme that only reports what survived is not reporting.
Where the evidence runs out
For many of the behaviours these evaluators measure, no external reference standard exists to test them against. Nobody has established an agreed correct answer for what good looks like. That is a limitation of the field rather than of any particular evaluator, but it has a direct consequence we would rather state than bury.
We do not claim our evaluators are validated. Where a reference standard exists, we test against it. Where one does not, we say so.
Evaluators are version controlled, tested, refined and revisited as evidence develops. None of this is a one-time certification, and a PromptSafe result is not a statement that your agent is safe, compliant or cleared for use. It is evidence for your team to weigh.
The wider research programme this sits inside, including the published work behind it, is on the Scientific foundations page. How PromptSafe is used inside studies is on the research page.
Test your agent before your users do.