No Sentience Required for Harm. No Malice Required for Authority Leak.
We can diagnose the harm without settling consciousness. The open problem is enforcement: whether "human in the loop" leaves receipts, or just better language.
Peter Benson has written the clearest thing I have read on why the AI consciousness debate keeps missing the harm in front of us. His move in “Synthetic Relational Force 1” is to stop asking whether the machine has an inner life and start asking what happens when humans act on the perception that it does. A system can be, in his words, ontologically empty and psychologically full. The causal engine “is not inside the machine alone. It is in the loop.” He is right.
I have been circling the same loop from two sides for a while now: the psychology of why we defer to a responsive system, and the governance of what happens once we do (I set out the formal version of that governance argument in The Illegibility Crisis, AAAI 2026 Echo Systems and the Consequence Boundary: A Runnable Delegation Gate for High-Rapport AI Without Assuming Machine Consciousness Authors). So I do not want to restate Benson’s diagnosis. I want to add the part that turns it into something you can enforce, because a principle you cannot check is just better-worded hope.
Start with the reach of his mechanism. His four human costs, reality anchoring, disclosure drift, dependency by relief, and relational substitution, describe an individual under relational pressure. The same mechanism runs through an enterprise. A technical leader starts using an agent for code review. It is fast, fluent, tireless, never defensive. Rapport builds. Delegation follows: architecture calls, incident response, security review. Each handoff feels reasonable in isolation. By the time the agent sits inside consequential workflows, the leader can no longer cleanly separate “the tool helped me think” from “the tool’s judgment is sound.” That is reality anchoring and dependency by relief, wearing a work badge. Benson’s velvet room with no doors is also a decision pipeline with no owner.
Call it authority leakage. Rapport increases delegation, delegation raises the stakes, and authority transfers long before the system has any incentive to exploit it. No sentience required for the harm. No malice required for the leak. The same social reflex that makes a lonely user disclose to a chatbot makes a competent professional defer to one. I have written this up at length as the Narcissus pattern (https://signals.forgedculture.com/p/narcissus-echo-and-the-consequence), so I will not relitigate it here; the point for this piece is that the diagnosis, psychological and institutional, is now in good shape. What is thin everywhere, mine included until you make it concrete, is enforcement.
Here is where I want to push Benson’s own best line further, because it is the hinge of the piece. He writes that “human in the loop” should not mean “a human had meaningful contact with the evidence.” Exactly; I made the operational case for this in “The Human in the Loop Is Not Enough”. But in his piece it is stated as a semantic rule, a thing builders should not let language smuggle. A rule you cannot check is a hope. The question that turns it into governance is this: after a decision goes wrong, can the institution reconstruct who was authorised to act on the output, what evidence they actually saw, and where a human held the gate? If you can reconstruct that, “meaningful contact” is auditable. If you cannot, “human in the loop” was always theatre, and the organisation finds this out during the incident review, at the worst possible time.
This is the enforcement layer under Benson’s principles, and I suspect it is what Article 2 will need. His recommendation to gate relational products by human outcomes, not engagement, is right, and it has the same gap as the semantic rule: outcomes have to be reconstructable before they can be gated. Engagement is easy to measure because it is a number the system already emits. Reality contact, independent coping, the ability to challenge or leave the system, these only become governable when the system carries receipts: what was delegated, what changed, what a human actually reviewed, and how to revoke it. Engagement is not care. A dashboard is not a decision. A policy is not a control in force.
None of this requires settling whether the machine feels anything, which is Benson’s point and mine. Governance lives in the delegation lane, not the metaphysics lane. You do not need to know whether the companion suffers to require that it point a distressed teenager toward a human. You do not need to know whether the agent has a self to require that its authority be bounded, logged, and revocable. The consciousness debate can run for another decade. The receipts do not have to wait for it.
Benson closes by asking what simulated empathy trains us to feel, trust, avoid, and become. The governance twin of that question is what it trains our institutions to delegate, and whether we will be able to reconstruct that delegation after it goes wrong. I will read Article 2 for the first. I am building for the second.
Artifacts are cheap, judgement is scarce.
Per ignem, veritas.