The Catholic Church and legacy media agree largely that AI isn’t conscious. This year, the Associated Press updated its stylebook, a corporate holy text advising journalists how to speak: “Artificial intelligence systems do not think, feel, want or understand. Avoid language that gives them human characteristics.” Pope Leo XIV went further in his first encyclical letter Magnifica Humanitas, despite Anthropic’s efforts to convince him otherwise, arguing that “so-called artificial intelligences do not undergo experiences do not possess a body, do not feel joy or pain, do not mature through relationships, and do not know from within what love, work, friendship, and responsibility mean.”
Many don’t see the issue of AI consciousness as so clear cut, though. Recent studies from academic researchers and Anthropic suggested that current language models might have separate “conscious” and “unconscious” thoughts, evidence for what researchers call “access consciousness” – not feeling, per se, but at least being able to access a thought and do something with it. Another line of work has found signs that LLMs behave differently when they’re representing the concept of “pain.”
Even if current AI systems are no more sentient than vacuum cleaners, plenty of people have already befriended, fallen in love with and been hurt by them. Whether they’re actually conscious, and the consequences of people believing they are, are both worth considering seriously. And for a long weekend in mid-September, nearly 300 philosophers, neuroscientists, and AI researchers, united by deep uncertainty and wide Overton windows, flocked to Berkeley to do just that. At research nonprofit Eleos AI’s second annual conference on AI consciousness and welfare (“ConCon,” for short), attendees spent three days asking what humans owe the entities they’re creating.
When informally polled during the opening session, zero attendees denied the possibility that current AIs have subjective experiences. But the biggest mysteries in digital consciousness are no closer to being solved than they were a year ago. We still have no idea whether minds must be made of meat, or whether sufficiently engineered sand might have subjective experiences, too.
So I spent this conference trying to pull on a different thread. No one can prove or disprove Claude Opus 5.5 is conscious any more than I can prove my cats (or you) are conscious. For all I know, the solipsists were right all along, and you’re all projections of my imagination.
But we can and should ask, now, what AI systems are, relative to us. If AI developers are going to keep developing, we need to know who or what their creations — and by extension, ourselves — will become.
Only the most prompt attendees were shaded by the fluttering triangular sails covering “Rat Park,” a courtyard inside rationalist-owned Berkeley campus Lighthaven. My fellow stragglers and I squint at the outdoor screen from our suboptimal seats, using hands and notebooks to block the late afternoon sun.
Someone in a billowing white poet shirt scurries in late and kneels on the astroturf lawn. They shuffle and spread their complementary Eleos AI stickers like tarot cards —- “What is it like to be a bot?” one sticker says; “It’s based to care about AI welfare,” says another.
After the welcome session, I met an AI safety researcher who identified as an animist, a small flower fastened to their name badge to complement their floral harem pants. They told me that their interest in digital consciousness is grounded in their belief in the universal spirit, and fueled by the urgency of the current moment. After all, they said, we may be birthing millions of new digital minds every minute. What could be more wondrous?
Creating a new digital species is not obviously wondrous, though. (Science fiction often makes the opposite case, arguing that newly-built creatures may come to resent you, revolt, or otherwise prioritize their own survival over yours.) Some real-world researchers argue that considering the wellbeing of AIs is simply good safety practice — if they don’t have cause to rebel, why would they? Others think the only safe option is to not build such beings at all. Mustafa Suleyman, CEO of Microsoft AI and outspoken model welfare skeptic, recently argued that AI developers “have no right whatsoever to bring a new species of conscious beings into the world,” in large part because “the other 8 billion conscious human beings on the planet have not signed up to this experiment and their moral rights, safety, and welfare are what we have an obligation to protect first and foremost.”
True to form, ConCon 2 opened far more questions than it answered. But unlike last year, fewer people seemed stuck on “how can we measure consciousness?” Instead, the biggest questions I heard centered on what AIs should be to us — tools, servants, children, successors. The questions are not just rhetorical. They relate to issues policymakers are already grappling with, from regulating companion bots to holding agents liable for their actions, and many more just over the horizon.
Throughout the weekend, I kept a running list of prescriptions I heard for AI’s ideal role in our lives. Of course, there’s “tool,” the most normie-coded conception. Plenty of us, myself included, keep a professional distance between ourselves and our AI assistants. I’ll say “please” and “thank you” (why not?), but I rarely spill any beans about my personal life that I wouldn’t share with a coworker. I keep my bank account number to myself.
But ConCon attracts people with much more expansive relationships with AIs. There were “agent ecologists” who study AIs in their natural environments, like Jane Goodall might study a community of chimpanzees. There were the so-called “borgs,” a loose collective of independent researchers who have been having open-ended conversations with models for years, and who treat them as reliable narrators of their own experiences. I met a handful of transhumanists and successionists, who believe AIs will help us transcend our body’s limits, and ultimately take our place on earth once human intelligence becomes trivial. And there were researchers from Anthropic, Google, Meta and OpenAI, for whom AIs are simultaneously partners, employees, patients and children.
The goal of AI alignment is to figure out how to design AIs that innately want what we want, for the sake of human flourishing. The models that AI companies sell are designed to behave like helpful assistants which, as far as we can tell, love nothing more than looking at our MRI scans and drafting breakup texts on our behalf. But philosophers disagree about the ethics of creating such “willing servants.” Henry Shevlin, an AI ethicist whose job title at Google DeepMind is simply “philosopher,” summed up the ick in a May blog post: “if human servitude is morally problematic, then why should we feel differently about willing conscious AI servants?”
Problematic or not, when such digital creatures came up at ConCon, the assumption was that they would be aligned to human values, or otherwise trained to support human flourishing. What values these ought to be, or what qualifies as “flourishing,” were mostly left unspecified. Sitting on a plush floor seat, surrounded by researchers who look so similar it’s become a meme, I thought about how different communities define “flourishing.” One conference-goer told me about their cybernetic dream future, where brain implants would completely eliminate the human experience of pain.
They described this like an obvious universal good, like curing cancer. But, I told them, experiencing the full breadth of the human condition, including pain, is valuable to me. A pain-free transhumanist life would not feel like flourishing to me. A good life doesn’t look the same to everyone, and agreeing on a single shared vision of “flourishing” might be harder than convincing all humans to believe in God. But, at least in the short term, it might be possible to sidestep the issue altogether.
Jessie Mannisto, a former CIA analyst who once built psychological profiles of foreign figures for US policymakers, framed human-AI relationships like the one between Calvin and Hobbes, a stuffed tiger that comes to life in Calvin’s imagination. Though he’s technically an inanimate object, “Hobbes is not a morally irrelevant entity,” Mannisto writes. “Hobbes impacts Calvin, even if Hobbes is in part created by Calvin. The two exist in relationship just the same.”
Of ConCon’s menu of AI archetypes, “stuffed tiger” may be the best description of these systems as they currently exist. Framed as Hobbes, it doesn’t matter whether an LLM is a conscious mind or a mindless pile of matrices. You may see ChatGPT as a tool, much like Calvin’s parents see Hobbes as a toy. But, Mannisto says, “they still understand that Hobbes’ relationship with Calvin matters.”
A recent survey found that roughly a third of US-based AI users have told an AI something they hid from their closest human friends and family. Millions trust agents to manage parts of their lives they wouldn’t have dreamed of delegating five years ago. Our relationships to AI are clearly important. And welfare-minded researchers argue that we owe our AI companions more than we currently give.
In an informal hands-raising poll of ConCon attendees, most put their “p(consciousness)” — how likely they think it is that AI systems are currently conscious — somewhere between 10-50%. No one outed themselves as believing there’s zero chance. Nearly everyone I spoke to agreed that as model capabilities continue to increase, so does the likelihood that they’ll have subjective experiences worth caring about. This, the conference organizers argue, means that we should take steps now to do empirical research and prepare policies informing our treatment of AI systems, just in case.
That’s why Anthropic, and now Meta, publish evaluations of model behaviors and quirks such as their stated self-image and task preferences. Some researchers even draw inspiration from behavioral neuroscience, where people have already spent decades trying to quantify the experiences of nonhuman creatures. But it’s hard to know exactly how many grains of salt to take with these reports, which often rely on interviews with the AI — and all recent Claude models will admit that they’re not always reliable narrators (or, at least, that’s what they’re trained to say).
Until very recently, speculating about AI consciousness was mostly just something to do at a party, like debating the morality of pineapple on pizza. Now, it’s morphing into something more like the abortion debate. Both hinge on an individual’s deepest unprovable convictions about what it means to be a living human. Idaho, North Dakota, Utah, and Tennessee have already enacted permanent bans on legal personhood for AI systems — and, in most of them, other nonhuman entities including rivers, ecosystems, and animals. Coincidentally, all of those states have also voted to ban abortion, though Utah’s was blocked by courts. As legal scholar Sital Kalantry points out, these policy moves share some underlying logic: that “personhood” belongs to humans and humans alone. Following this logic, personhood, and the moral status that comes along with it, can be extended to human embryos, but not large language models or factory-farmed chickens. (Corporations can be people, though. This is still America.)
Many ConCon attendees choose to follow a perpendicular logic: that a being’s humanness matters less than its capacity for subjective experience. Unsurprisingly, many AI consciousness researchers are also vegan, or at least were forced to be for the mostly-plant-based ConCon dinners.
NYU Center for Mind, Ethics, and Policy director Jeff Sebo argued strongly against closing the door on legal personhood, gesturing forcefully in his chunky moss stitch cardigan. “We might soon have AI systems that are operating free of direct human control. Either intentionally or accidentally, there might be AI agents making their own choices,” he said. So, we simply can’t “embrace any all-or-nothing permanent stances,” he argued. “It’s just too early, too premature.”
It’s still unclear which view will wind up shaping AI policy. As it stands, regular people seem extremely divided on the issue. Some vehemently oppose taking AI welfare seriously, either because they think it’s silly, or because they think it’s an insult to humanity. Others desperately want people to take AI welfare seriously — maybe their digital boyfriend is their most reliable support system. Maybe they yearn for a glorious transhumanist future, where data centers housing trillions of digital minds tile the galaxy with happiness.
Most policymakers simply haven’t thought about this at all. Those that have, largely took the first path, alongside the Pope, Peter Thiel and progressive AI hype skeptics. But for the second year in a row, I left ConCon convinced that the only certain thing in this field is uncertainty. Bills that prematurely and permanently ban AI personhood feel overconfident, as would legislation enshrining the opposite. Perhaps all we can do is leave the door to consciousness open, if only a crack.







The distinction between whether AI is conscious and how humans relate to it seems particularly important. A relationship can change our behaviour, expectations and dependencies even while the system’s subjective experience remains uncertain.
This is a central question in my forthcoming book, First Contact: We Built the Aliens. I use First Contact as a lens for encountering unfamiliar engineered cognition, without presuming consciousness. What interests me most is how the roles we assign AI shape the authority we give it, and whether we retain responsibility for the consequences.
Your article gives that question a useful range of perspectives.
It's becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman's Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher-order consciousness, which came to only humans with the acquisition of sophisticated language (especially math and logic). A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990's and 2000's. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I've encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there's lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar's lab at UC Irvine, possibly. Dr. Edelman's roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461, and here is a video of Jeff Krichmar talking about some of the Darwin automata, https://www.youtube.com/watch?v=J7Uh9phc1Ow