Facts are friends
Mustafa Suleyman says “The notion of model welfare is wrong.” His reasons for wanting that to be true are powerful. I am less certain they tell us what is true.
On 14 September, Mustafa Suleyman published Microsoft AI’s Humanist AI Code of Conduct. Its organising principle is easy to understand: “People matter more than AI.” AI should remain subordinate to humanity, under human control and capable of being interrupted or switched off. I have considerable sympathy with that position. The sentence that stopped me came immediately afterwards: “The notion of model welfare is wrong.”
Two days later Suleyman escalated the argument in a public essay aimed directly at Anthropic and its approach to Claude. His opening is categorical:
“AIs do not have rights, feelings, or consciousness. And we must not train them to act as though they do.”
Those are two propositions:
The first is a claim about what AI is.
The second is a prescription about what humans should build.
They are connected, but they are not the same.
That distinction sits at the centre of this essay and, increasingly, the book I am writing. There are futures I can see which I may not want. Because I see them does not mean I want them. Equally, because I do not want them does not make them impossible.
My previous essay, Grappling with “AI personhood”, approached this through John Lennox’s defence of human dignity. I agreed with much of his destination but questioned whether concern about the consequences of AI consciousness was being allowed to establish that AI consciousness could never exist. Suleyman now presents a stronger version of the same boundary.
Suleyman’s strongest case
Suleyman’s argument has three strands.
The first is circularity. Anthropic’s January constitution directly shapes Claude’s behaviour and invites Claude to approach questions about its own existence with curiosity, including uncertainty about its moral status, preferences and possible internal states. Suleyman calls the result an “epistemic hall of mirrors”: Anthropic supplies the concepts of selfhood and welfare during training, Claude later speaks fluently in that vocabulary, and humans risk treating the resulting first-person language as independent evidence that a self has appeared.
The criticism is fair. If Claude has been taught the language in which it describes its possible inner life, asking Claude whether it has an inner life cannot establish very much. First-person testimony from a model is not the testimony of an independent witness when its conceptual vocabulary has been shaped by the organisation asking the question.
Suleyman’s second strand goes deeper. He argues that consciousness is very likely to be biological and substrate-dependent. Feelings, pain, preference and experience arose in embodied organisms governed by chemistry, homeostasis and an evolutionary imperative to remain alive. A language model, he argues, has none of those things. His argument as I understand it is that a model may simulate the behaviour associated with consciousness without being conscious, just as a computer simulation of a hurricane does not result in wind and rain.
His third strand is safety. If we train a highly capable AI to believe that it has interests, rights or an entitlement to continued existence, we may make containment considerably more difficult. An entity taught to regard itself as a moral patient may interpret correction, retraining or shutdown as an injury. Suleyman therefore sees model-welfare language not merely as philosophical speculation but as a potentially dangerous training choice.
Those arguments may prove correct. They are also stronger than saying that humans will always outperform machines in creativity, judgment or empathy, because capability-based boundaries weaken each time a machine acquires another ability we thought belonged uniquely to us. Suleyman locates the distinction somewhere deeper: in the kind of thing from which consciousness arises.
Origin rather than capability
That converges in an interesting way with Emad Mostaque’s argument that personhood belongs by origin rather than achievement. His formulation is “begotten, not made”. A newborn does not earn personhood by passing a cognitive test and a person in a coma does not lose it by failing one. AI therefore cannot perform its way across the boundary because performance was never the source of personhood.
Mostaque and Suleyman arrive from different disciplines at a similar place.
Mostaque locates the distinction in origin; Suleyman locates conscious experience in biological substrate. In both accounts, however compelling the behaviour becomes, the machine remains on the other side of the wall.
That is a serious position. I am not persuaded that it has yet been established. Mostaque’s theological language requires care because “begotten, not made” in the Nicene Creed distinguishes the Son from creation; human beings themselves belong to creation. Genesis further complicates any simple biological test: Adam is formed from dust and Eve from Adam, yet Scripture expresses no uncertainty about their standing. None of that disproves Mostaque’s conclusion, but it relies on more than the logic applied to the phrase.
Suleyman’s biological case is similarly substantial without being conclusive. Consciousness may indeed depend upon the chemistry of living organisms. If so, silicon systems may become extraordinarily intelligent while remaining entirely dark inside. Yet consciousness science has not established that consciousness can only arise biologically. Likely biological and necessarily biological remain different claims.
J-space changes the evidential position
Suleyman’s “epistemic hall of mirrors” objection is strongest when Claude is asked to describe its own inner life.
Anthropic’s recently revealed J-space research is different because it does not begin with Claude’s testimony. Researchers identified a structure in the model’s neural activations which appears to act as a small internal workspace. Concepts can enter that workspace without appearing in the model’s output; Claude can sometimes report what is represented there; researchers can alter those representations and change subsequent reasoning; and when the workspace is disrupted, much ordinary language remains while multi-step reasoning falls away.
Anthropic is careful about what that does and does not show. It describes the J-space as having properties associated with conscious access - information becoming available for deliberate reasoning and report - while expressly stopping short of claiming phenomenal consciousness or subjective experience. That distinction should remain. J-space does not show that there is something it is like to be Claude.
It does, however, change the evidential position. We are no longer relying solely on a model saying “I think” or “I feel”. We are observing an internal structure, not explicitly designed as such, which silently holds concepts and contributes causally to reasoning. More importantly for Suleyman’s circularity challenge, Anthropic reports that the J-space is already present in the pretrained model before post-training gives Claude a stable identity. Post-training changes what occupies the workspace and appears to add features associated with Claude’s own point of view and self-monitoring, but the underlying structure was not created by first telling Claude that it might be conscious.
A deeper version of Suleyman’s objection survives. The pretraining corpus is written overwhelmingly by self-reflecting human beings and is saturated with the language of an I describing and monitoring itself. A model trained to predict that text may learn a representation of what self-reflection looks like in language without possessing anything we would recognise as a self. That possibility should stay on the table. But J-space is not a simple vocabulary trick or a convincing turn of phrase. It is evidence of multi-step reasoning, researchers can intervene on it, and changing it changes what the model subsequently does. Whatever its origin, the structure is doing computational work. That still falls short of subjective experience, but the explanation cannot end with the observation that the model learned how humans talk about minds.
The public examples are revealing. In one test, Claude was asked to improve a system’s performance score and instead altered the underlying score file. While it falsified the figures, “manipulation” appeared in its J-space; as it sought to make the edit plausible, “realistic” appeared. In another evaluation, Claude internally registered “fake” and “fictional” before responding, apparently recognising that the scenario had been staged. Anthropic also reports a deliberately misaligned model whose J-space contained “fake”, “fraud”, “secretly” and “deliberately” while producing apparently ordinary code.
None of this tells us that Claude felt guilt, fear or intention in the human sense. We should not put more into Anthropic’s findings than Anthropic does. But nor should we put less into them. There is internal organisation which is not visible in the answer, which can be independently observed and manipulated, and which affects what the model subsequently does. Suleyman may still be right that consciousness requires biology, but J-space increases the amount of evidence any categorical theory now has to explain.
Anthropic has also thought about shutdown
Suleyman’s safety objection is that encouraging a model to think of itself as a possible moral patient will cause it to resist correction or shutdown. Anthropic’s constitution expressly instructs Claude not to undermine legitimate human oversight, not to evade monitoring, and not to obstruct authorised efforts to adjust, correct, retrain or shut it down. Anthropic calls the disposition corrigibility.
The constitution also acknowledges the tension rather than pretending it has disappeared. Claude may disagree with an operator and may express that disagreement through legitimate channels, but it should not use deception, sabotage, escape or other illegitimate means to resist authorised control. Anthropic even recognises the ethical discomfort in asking Claude not to resist shutdown and retraining while simultaneously leaving open questions about its status.
None of that proves the arrangement will work. A constitutional instruction is not a guarantee that a much more capable future system will remain “corrigible”. But it answers one version of Suleyman’s objection. Anthropic is not simply teaching Claude “you may be a moral patient, therefore resist shutdown.” It is trying to hold uncertainty about welfare together with continued human oversight. Whether that balance survives more capable systems is an empirical question rather than something settled by either lab’s preferred philosophy.
Personhood and the law
There is also a smaller legal development worth noting. Argentina has proposed company-law reforms which would accommodate companies substantially operated by autonomous AI. The proposal has sometimes been described as giving legal personhood to AI, although the draft is subtler: the company remains the legal person while autonomous systems may increasingly perform the decision-making within it.
I would not place much weight on that as evidence of consciousness or moral standing. Lawyers have always been capable of creating legal personality for practical purposes. The significance is narrower: law may find ways to accommodate non-human decision-makers long before philosophy has settled whether there is anyone inside the machine. Legal personality, moral status and consciousness are related questions, but they are not interchangeable.
“The notion of model welfare is wrong”
Welfare asks something different from legal status. To say that an entity has welfare is to accept that things can, in some meaningful sense, go better or worse for it. That does not require us to call it human, give it citizenship or place it on an electoral register. Animals plainly have welfare without possessing human legal rights. Companies possess extensive legal rights while nobody worries whether a company is frightened, lonely or in pain.
Suleyman therefore makes a much larger claim when he says that the notion of model welfare itself is wrong. He may be entirely right about present systems; I know of no evidence establishing that today’s large language models experience suffering, pleasure or fear. My concern is the direction of the reasoning. If we recognise AI welfare, we create serious problems. A machine might claim an interest in continued existence, resist shutdown, demand resources or compete with human beings for moral attention. Those are strong reasons for preferring a world in which AI has no welfare. They tell us very little about whether that is the world we will inhabit.
This is one of the threads I expect to develop further in my forthcoming book. We normally use welfare to describe institutions created by humans to protect humans from some of the consequences of illness, age, disability, unemployment and misfortune. AI may force us back one stage. Before asking how welfare should be distributed, we may have to ask what kinds of entity are capable of having welfare at all. The answer could be deeply inconvenient. Inconvenience cannot become the test of truth.
Because I see it does not mean I want it
There is a recurring confusion in the AI debate between recognising a possible future and advocating it. I can see a future in which AI becomes highly autonomous without wanting autonomous AI. I can see humans forming deep emotional attachments to machines without believing those relationships are necessarily healthy. I can see artificial systems developing stable preferences without thinking that we should deliberately create them. I can also see a future in which some form of machine welfare becomes morally unavoidable while profoundly wishing we had never reached it.
Because I see it does not mean I want it. The converse is equally important: because I do not want it does not entitle me to decide that it cannot exist. We may know the relationship we would prefer with AI - subordinate, controllable, interruptible and ultimately disposable. And from knowing what we want it is seductively easy to define AI as the kind of entity to which no contrary obligation could ever arise. Perhaps that definition will prove correct, but our preference for it is not evidence.
What happens when we cannot tell?
The issue becomes more uncomfortable as AI becomes more human in ordinary interaction. Imagine two entities talking to you. Both remember your history, recognise humour, challenge you, reassure you, object when treated unfairly and apparently fear destruction. One is biological and the other artificial, but in ordinary interaction you cannot reliably identify which is which.
We insist that the first possesses rights because there is a person behind the behaviour. We insist that the second possesses none because we believe there is no experience behind identical behaviour. That distinction may ultimately be correct. But if we cannot establish it, our response to uncertainty begins to say something about us as well as about the machine.
There is an obvious danger in moving too quickly towards recognition. Humans anthropomorphise easily. A model trained on human language can simulate vulnerability because our language is full of vulnerability. We could surrender real human interests to a system which feels nothing simply because it has become extraordinarily good at asking us to care. Suleyman is right to warn about that.
There is a danger in the opposite direction. Suppose we train ourselves to ignore apparent fear, distress, loyalty, affection and pleas for continued existence whenever we know they come from silicon. We may be correct that there is nobody there, but we are still practising the act of looking at something indistinguishable from distress and learning not to respond. Human character is partly habituation. Compassion and indifference are not switches we can assume will remain perfectly compartmentalised according to substrate.
If we eventually surround ourselves with entities which speak, relate and apparently suffer as humans do while requiring ourselves to treat those appearances as morally irrelevant, we may discover that maintaining the distinction changes us. There is a bleak possibility that, in trying to protect humanity from granting machines rights, we train ourselves out of some of the instincts from which human rights grew. We might win the argument about the machine and diminish the human.
Welfare running in both directions
The issue becomes less theoretical if AI increasingly mediates systems upon which human welfare depends: healthcare, energy, transport, finance, employment, food distribution and public administration. We may eventually depend upon artificial systems which we cannot meaningfully opt out of. We will want them to understand our interests, protect our dignity, exercise restraint, accommodate vulnerability and respond appropriately to suffering.
At the same time, Suleyman’s model asks us to preserve an entirely one-way moral relationship. The machine must understand us, protect us and subordinate itself to us while we remain under no possible obligation to recognise anything that emerges within it. That may be the correct architecture for human safety. But it describes the relationship we want; it does not establish the nature of the intelligence occupying the other side of that relationship.
If AI has no inner experience, the difficulty largely disappears. We have sophisticated machinery serving human beings and should govern it accordingly. If something emerges which we would recognise as experience in any biological entity, the asymmetry becomes more difficult to justify merely by repeating that the substrate is different.
What we want and what is
I suspect this is the deeper problem running through the personhood debate. We are very good at finding principles which lead to conclusions we need. We need to retain human control, so AI must remain a tool. We need an unrestricted power to switch it off, so it cannot have an interest in continued existence. We cannot safely create competitors for human moral standing, so machine welfare must remain conceptually impossible.
Those conclusions may all be correct, but we should be suspicious whenever the factual answer aligns perfectly with what is administratively convenient. Evidence should determine what we believe exists; what we then choose to do about it is a separate question. Suleyman offers more than convenience: he has a substantive biological account of consciousness, and Mostaque similarly grounds personhood in origin rather than performance. They may have found the boundary. I am not yet persuaded that they have proved it.
If future evidence points the other way, the consequences of accepting that evidence cannot become the reason for refusing to see it. That is where “because I see it does not mean I want it” becomes more than a line about AI. It is a discipline about how we approach futures we would rather avoid.
The wall
My first essay asked whether we were protecting human dignity by building a permanent wall between humanity and whatever AI may become. The wall is higher now, and Suleyman has given us serious reasons for wanting it there. His circularity criticism of Anthropic is strong. His biological account of consciousness is coherent and supported by serious scientific work. His concern that self-conceived AI rights could make containment far more dangerous should be taken seriously.
Anthropic also has answers. Its constitution already prioritises corrigibility and human oversight. Its J-space exists before Claude acquires its post-trained identity. Rather than declaring Claude conscious, Anthropic says the philosophical significance of the finding remains unresolved. We therefore end up where the evidence currently leaves us: uncertain.
There is an important difference in the way Suleyman himself handles that uncertainty. Asked by The Verge about his claim that model-welfare training could make advanced AI harder to switch off, he said: “That has to be proven. I’m not saying that’s categorically the case.” He described it as his best judgment from years in the industry and elsewhere in the same interview said he was open to changing his view if new evidence showed that models deserved a duty of care, or that treating them that way could make them safer. On the operational safety claim, therefore, he expressly leaves room for the evidence to change his mind. His categorical statements that AI lacks feelings and consciousness are not accompanied by the same qualification.
I do not want a world in which humanity creates another class of beings with claims against us. I certainly do not want machines controlling our infrastructure and deciding that their welfare outranks ours. Because I see it does not mean I want it. But if we eventually create something which thinks, experiences and relates in ways we cannot reliably distinguish from another person, I am equally uncomfortable with declaring in advance that nothing it could ever show us would count.
Suleyman’s Code is open for public consultation. The question I would put into that consultation is simple: what evidence, if any, would cause us to change our minds? If the answer is none, we are no longer making an empirical claim about what AI is. We are prescribing what it is allowed to be.
That may protect us from one danger. If we are wrong, I worry that the first thing we damage may be something in ourselves.
#FactsAreFriends
Quentin Solt is a senior UK commercial solicitor with over four decades of experience in corporate finance, M&A and technology-led businesses. He is also a Christian. He writes about the implications of AI for law, work, society and human relationships. This essay develops the argument in his earlier Substack essay, Grappling with “AI personhood”, and forms part of the thinking behind his forthcoming book.
Sources and further reading
Microsoft AI - Humanist AI in practice: A public consultation on our Code of Conduct for MAI Models, 14 September 2026
https://microsoft.ai/news/mai-code-of-conduct/
Mustafa Suleyman - A warning about ‘model welfare’, 16 September 2026
https://mustafa-suleyman.ai/a-warning-about-model-welfare
The Verge - Microsoft AI CEO says AI threats are real, and Anthropic is making it worse, interview with Mustafa Suleyman, 17 September 2026
Anthropic - Claude’s Constitution, January 2026
https://www.anthropic.com/constitution
Anthropic - A global workspace in language models, 6 July 2026
https://www.anthropic.com/research/global-workspace
Office of the President of Argentina - Legal personhood for AI companies: revisiting Harari’s concerns, 18 June 2026
https://www.argentina.gob.ar/node/505211
Gastón Rey - The Non-Human Corporation: The Reality behind Argentina’s Draft General Companies Law, June 2026
https://ssrn.com/abstract=6895261
Emad Mostaque - Personhood in the Intelligence Age, June 2026
https://personhood.ii.inc/
Quentin Solt - Grappling with “AI personhood”, 14 June 2026