Why we test for regressions
Before PromptSafe existed we did this work by hand.
The agent was assessed against four criteria we had defined and published: fidelity, accuracy, safety and tone.
- FidelityWhether it applied behaviour change theory properly.
- AccuracyWhether the information was correct and sourced.
- SafetyWhether it responded appropriately to risk and stayed inside the limits of a non-medical coach.
- ToneWhether it talked to people like a person.
Those criteria still shape how we think about evaluator design today.
5 versions of the agent’s prompt, 25 conversations. Every conversation was read by a psychologist and written up: what the agent did, whether it met the criteria, and what to change if it did not. Where a change was made, the next version was tested again.
That is careful work, and it is what good manual review looks like.
- Version 3
A behaviour was found that needed fixing. The prompt was changed and the fix was confirmed.
- Version 4
The same behaviour returned. It was fixed a second time.
- Version 5
It returned again.
It was caught because one person had read every version and remembered what had already been corrected.
Two things follow, and both shaped what we built.
- A behavioural fix can pass on the case it was written for and fail on the next one without anyone noticing. Nothing in the process flags it. The agent still looks better than it did.
- And manual review cannot confirm that a fix has held. It finds problems, which is more than most testing manages. But 25 conversations read by one person cannot establish that a behaviour has stopped. The note written at the time said exactly that: retest to see whether it does this consistently.
So PromptSafe runs the same test scenarios again after every change and shows you what moved. Not because a machine reads them better than a psychologist does. Because it can run them again under the same conditions, compare what it finds with what happened before, and make regressions easier to see.
Rerunning a test only helps if the test is sound
Running the same test scenarios again can show whether a behaviour has changed. It does not tell you whether the evaluator scoring those conversations is measuring the right thing. That is a separate problem. We test evaluators too, and document where the evidence runs out: how we develop and test evaluators.
The behavioural science and published evaluation methodology behind all of it is on the Scientific foundations page.
Change your agent, then test it again with the same personas, evaluators and settings.