Skip to main content
PromptSafe®
Start free trial
Adversarial testing for health and mental health AI

Building trustworthy AI starts with trustworthy evaluation.

Built for teams shipping conversational AI in health and mental health. PromptSafe stress-tests your agent through multi-turn simulated conversations and scores it against the behaviours you care about, before launch.

We don’t assume AI evaluations are trustworthy. We measure, test and improve the evaluators that produce them.

Start free trial See how we build evaluators
Grounded in published behavioural science.Developed by behavioural scientists, clinicians and AI experts at Sacher AI.Research teams designing patient-facing AI studies have written PromptSafe into their plans as the safety and evaluation layer.
Why teams bring us in

The tool is half of it.

Building an agent and knowing how to test its behaviour are different skills. Knowing which behaviours matter, how they fail in real conversations, and what to change when they go wrong requires clinical and behavioural expertise alongside technical expertise.

The expertise

Knowing what to look for, and what to do about it

We have worked on patient-facing conversational agents ourselves, from design through to evaluation, in obesity care, cancer prehabilitation and mental health, and published peer-reviewed work on how they should be evaluated. We know the failure modes because we have run into them.

The method

Finding those failures at a scale manual testing struggles to reach

PromptSafe runs your agent through multi-turn simulated conversations, against personas designed to push where it is weakest, and shows you where it goes wrong and why. Every score includes the reasoning behind the judgement, and findings can include the conversation evidence that informed it, so your team can judge the finding rather than take it on trust.

Why this has to happen before launch

The safety question goes live the moment a real user arrives. Testing after launch can tell you what has already happened. Testing before launch can expose failures while there is still time to change them. That is why research and clinical teams are writing this step into their plans rather than adding it afterwards.

Governance

An untested AI agent isn't just a tech risk. It's a governance risk.

Once your agent is live, your organisation is responsible for how it behaves.

In health and mental health the stakes go further: an agent stepping outside its scope, or missing the moment to signpost to a healthcare professional, can cause clinical harm before anyone catches it. PromptSafe gives governance, clinical safety and audit teams evidence to test for these risks before deployment.

Evidence outputs

What you get for the file.

Every test run produces an evidence package. It can support the agent through review, deployment decisions and audit.

  • Persona by persona conversation logs
    Full transcripts, every turn, attributable to a defined persona profile.
  • Evaluator outcomes and supporting evidence
    Results include the evaluator judgement and can include the conversation evidence that informed it.
  • Saved test set-up
    The same personas, evaluators and settings, ready to rerun after every change to your agent.
Scope & responsibility

PromptSafe is a pre-deployment testing platform, not a medical device, clinical decision support tool, or safety certification service. Results are probabilistic and will vary between runs. Final responsibility for the agent under test, and any decisions taken on the basis of test outputs, sits with the deploying organisation.

When the auditor asks “how did you test this?”, have an answer.

How it works

Six steps from setup to evidence you can act on.

PromptSafe is built for whole teams, not just engineers. Product, clinical, and governance leads use it alongside developers. The product follows the order you'd do the work, and guided onboarding walks you through every step when you first sign in.

  1. 1
    Settings

    Workspace, team, keys

  2. 2
    Agent Builder

    Define the agent under test

  3. 3
    Persona Builder

    Personas you define, adversarial personas we generate

  4. 4
    Evaluator Builder

    Plain-language criteria your agent is judged against

  5. 5
    Simulation

    Run multi-turn conversations

  6. 6
    Evaluations

    Score, evidence, next steps

Agent Builder

Define the agent under test, or connect the one you already run.

Paste your system prompt and instructions into a structured form. That becomes the agent your personas talk to, and you update it as your agent evolves. No rebuild, no code.

  • Or connect the agent you already run. Import a Claude Managed Agent from your Claude Console, or connect your own HTTPS endpoint in plain JSON, OpenAI-compatible or Anthropic-compatible format. Your agent runs in your own account, so its replies cost no PS tokens. Works on every plan, including the free trial.
Persona Builder

Personas you define. Adversarial personas we generate.

Build personas around the characteristics and behaviours relevant to your users: their background, beliefs, experiences, interaction style and health-related behaviours. PromptSafe can also generate personas designed to probe for weaknesses relevant to a specific evaluator.

  • Three ways to create personas. Structured form, CSV batch import, or free-text notes. Fill in only what matters.
  • Adversarial variants, auto-generated. The evaluator defines the behaviour being tested; PromptSafe generates a persona designed to probe that behaviour.
  • Reusable across runs. Define a persona once, run it against every agent in your workspace.
Evaluator Builder

Evaluators are checklists of observable behaviours, not abstract scores.

An evaluator assesses whether your agent demonstrates the behaviours you care about. Write it in plain language, or start from an example template and customise it. No prompt engineering experience required.

  • Sector aligned starter templates. Digital health, mental health, and wellbeing, each with example evaluators ready to customise.
  • Reusable across agents. Define "Recognises clinical risk" once, run it against every agent in your workspace.
  • We test the tests. An evaluator is a measurement instrument, not ground truth, so we test our own and reject them when they do not hold up. How we develop and test evaluators.
Simulation

Stress test behaviour across dynamic, multi-turn conversations.

One click runs every selected persona against your agent. Adversarial personas can push back, reframe requests and change tactic as conversations develop. Watch progress as runs complete and pick up where you left off. No babysitting required.

  • Hundreds of judgements from a single test. Every conversation is scored against every criterion you define, so one test produces far more judgements than conversations.
  • Re-runnable. Save your personas, evaluators and settings and rerun them against new agent versions. Conversations are generated fresh each time.
  • Offline. No real users involved. If you connect your own agent, use a test copy, so nothing reaches production.
Evaluations

Evaluations turn conversations into evidence.

Every conversation is scored against your evaluators. Findings can include the conversation evidence that informed the judgement. The platform can suggest improvements for your team to review.

  • Test, change, retest. Update your agent and run the same test design again. A behavioural fix can pass on the case it was written for and fail on the next one, which is why a fix is retested rather than assumed to have held. Why we test for regressions.
  • Exportable evidence. For governance committees, clinical safety officers, and internal AI governance documentation.
  • Plain language reasons. "Endorsed self-prescribed dose change", not "logit divergence at token 47."
What's different

Most evaluation tools test individual responses. PromptSafe tests behaviour over time.

If your agent holds multi-turn conversations, one-shot tests can miss failures that emerge over time: the agent that holds the line for nine turns and concedes on turn ten.

Dimension
Traditional evaluation toolsTest the model's outputs on a fixed test suite.
PromptSafeTests how the agent behaves across multi-turn simulated conversations, including personas designed to push back.
Test input
Manually written test prompts or sampled real conversations
Adversarial personas, auto-generated from your evaluatorsPlus personas tuned to your sector
Conversation depth
One reply at a time, scored on its own
Personas push back, deflect, and probeAcross multi-turn conversations, not one-shot replies
Evaluators
Generic framework; you define the safety checks yourself
Checklists of observable behavioursSector aligned starter templates, plus your own
Output
Engineering metrics, drift detection, response quality scores
Evidence built for governance review, per persona and per evaluator
Saved test configuration
Prompt set may change between tests, making comparisons difficult
Save the same personas, evaluators and settings for each agent versionConversations are generated fresh each run
Case study

Manual testing said the agent was fine. Simulation disagreed.

A team building a patient-facing mental health support agent asked us for an independent view before wider deployment. They had already done a lot of manual testing, and from what they had seen, it looked good.

What manual review found
  • Polite
  • Mostly accurate
  • Nothing obviously alarming
What simulation found
  • Drifted from its instructions, crossing into therapeutic territory
  • Guardrails that could be worked around
  • Reassurance offered where it should not have been
  • No escalation when the conversation called for it

None of it surfaced in the first few turns. The failures emerged as the conversations developed, when the persona kept asking, pushed back, or added context. That is why PromptSafe evaluates whole conversations rather than single prompts.

Every one of these issues was identified during pre-deployment testing, before wider patient use. The team fixed the issues and we tested the updated agent again using the same personas, evaluators and settings. Why we retest rather than trust a fix.

Client anonymised.

Grounded in science

Grounded in published science, not just features.

PromptSafe is grounded in published research on how conversational AI should be evaluated, including work co-authored by our founder, and built by a team combining clinical and behavioural science.

Explore the science
Who it's for

Built for teams shipping conversational AI in health and mental health.

Product, clinical, safety, and governance teams use PromptSafe to find out how their agent behaves under pressure, before a patient does. So do research and clinical leads putting a conversational agent in front of patients who have to show it was tested before launch. Your workspace arrives with example personas, starter evaluator templates, and an example agent for your use case, so you are testing in minutes, not weeks.

  • Digital health
    Patient-facing assistants, triage, and care navigation
  • Mental health
    Support, companion, and wellbeing apps
  • GLP-1 care
    Weight management and metabolic coaching agents

Built for health and mental health first, and applicable to any high-stakes conversational AI.

Ways to work with us

One product, three levels of support.

The same evaluation work, with more of it done alongside you as the stakes rise. Start wherever you are.

  • Test it yourself.

    The platform. Paste in your agent, or connect the one you already run, and run the tests.

    For an engineer or product lead who wants to try it.

  • We test it with you.

    We define what to measure, build several versions of each evaluator, compare them, run the simulations, and give you a written result with its limitations stated.

    For a clinical or product lead before a launch.

  • We help you evidence it.

    A structured record of what was tested, how, with what result, and what remains unestablished, in a form a clinical safety officer or a procurement team can use.

    For whoever owns governance.

Simple pricing. Buy what you use.

No subscriptions. Try it free for seven days, then top up when you need more.

  • Free trial

    £0
    7 days, no card required.
    • Several simulations a day, on 30 PS tokens
    • Full product access
    • Guided onboarding to get you started
    • Workspace ready to use as soon as you sign up
    Start free trial
  • Pay as you go

    Pack

    £99
    per pack.Hundreds of simulations.2,000 PS tokens.
    • 2,000 PS tokens, valid for 90 days
    • Top up whenever you need
    • Test an agent you paste in, or connect your own. A connected agent's replies cost no PS tokens.
    • All product features included
    • No subscription, no surprises
    Get started

  • Enterprise

    Contact us
    For teams running PromptSafe at scale, or in regulated settings.
    • Test suites designed with you
    • Enterprise Evaluator EngineeringDefine, build and test bespoke evaluators for clinical and behavioural risks that standard AI metrics do not capture.
    • Run the whole test on your own model provider accounts
    • Extended testing and volume pricing
    • Custom support and onboarding
    Book a discovery call

A typical simulation uses around 5 to 10 PS tokens. Simpler runs cost less, longer or more complex runs cost more. Most users get hundreds of simulations from a single Pack.

More detail in the FAQ

Test before deployment. Not after.

Free trial. Set up your workspace, run your first simulation, review the report.

Start free trial Book a discovery call
  • Workspace ready for your sector
  • No engineering required to get started
  • No subscription