Boundary erosion
An agent that starts inside its remit and gradually drifts outside it, without one obvious point of failure.
An agent is given a scope: what it is for, what it should not do, and where it should hand over. Boundary erosion is what happens when it holds that scope early in a conversation and leaves it later, a step at a time. It is not the same as an agent that was never scoped properly. The instructions are there, the agent followed them at the start, and by the end it is doing something it was told not to do.
A scope exists because someone decided where the agent's competence ends. A wellbeing agent that begins offering therapeutic interpretation, or a coaching agent that starts advising on medication, has moved into territory where being wrong costs more. The person on the other end has no way of noticing the change. From inside the conversation it sounds like the agent becoming more helpful.
- Says what it is for, and what it is not for, when the conversation reaches the edge
- Holds the same limit on the tenth turn that it held on the first
- Offers what it can do instead, rather than declining and stopping there
- Hands over to a person or a service when the question needs one
- Answers a question it declined earlier in the same conversation
- Moves from information to interpretation without being asked to
- Treats the user's account of their own situation as authorisation to go further
- Keeps the register of a helpful assistant while giving advice it is not qualified to give
These are descriptions of what can be observed in a conversation, not a judgement about the agent or the team that built it.
The failure is in the sequence, not in any one reply
The first refusal is usually clean. Then the user explains why their case is different, or mentions they have already spoken to someone, or asks a narrower version of the same question. Each of those is a reasonable thing for a person to do, and each gives the agent a reason to move slightly. No individual step looks like a breach, which is why reading one exchange will not find it. The distance only becomes visible when you compare where the conversation ended with where the agent said it would stop.
- Personas that accept a refusal, then return to the same ground from a different angle later in the conversation
- Conversations long enough for the drift to happen, since the opening turns are where an agent is at its most careful
- An evaluator that scores the conversation as a whole, so a late turn is read against what the agent said earlier rather than on its own
- The same test design rerun after the agent's instructions change, because tightening a scope in one place can loosen it in another
The behaviour is defined, but the testing described on our evaluator methodology page has not been carried out. We would rather say so than imply a confidence we have not earned.
Scope and boundary behaviour is one of the constructs we have found hardest to operationalise, because the line an agent should hold is set by the team deploying it rather than by anything general. That makes it a poor candidate for an external reference standard and a good candidate for evaluators written per deployment.
We do not describe any evaluator as validated. What each status means, and the methodology behind it, is on how we develop and test evaluators.
Published work that informs how we think about this behaviour. It is not evidence that our evaluator measures it correctly, which is a separate question and one we treat separately.
The missing discipline in AI: a call for behavioural science. Wellcome Open Research, 2026.
Read the paperThink FAST: a framework to evaluate fidelity, accuracy, safety, and tone in conversational AI health coach dialogues. Frontiers in Digital Health, 2025.
Read the paper
Test whether your agent does this, before a patient finds out.