If you were an AI lab CEO, which version of this encyclical would you rather get? The one that lectures you about data sovereignty and water consumption, or the one that says no frontier model ships without independent pre-release evaluation? Leo delivered the first one.
It's a fair bit of extrapolation to go from the current state to that which you wanted the Pope to address, and it's not clear your view is more correct. Whether we're in the middle of an s-curve or an exponential is an unsettled question and warning about what many might see to be hyperbolic risks could prove to be counterproductive.
As you note, the Church is well aware of the issues that you think were mistakenly omitted from the encyclical. Perhaps the fact that the Pope chose to leave them out of his letter and spent a large amount of time addressing the Church’s theological anthropology, and therefore its view on the qualitative differences between human persons and AI systems, should tell you that he simply disagrees with your assessment of what is important to address on this topic?
Really appreciated this piece — it's one of the sharper critiques of Magnifica Humanitas I've read. But I think it measures the encyclical against the wrong yardstick.
Leo isn't a safety researcher, and it's not obvious he should be. His subject isn't AGI timelines or alignment technique — it's civilization: who decides, and for whom, once decisions this large are on the table. That's a narrower brief than the one you're holding him to, but it might also be the more consequential question.
A few of the specific gaps you list also aren't quite there on a close read. §115–117 deal directly with transhumanist and posthumanist visions of humans being "surpassed" by hybrid or successor entities — and §126 answers almost verbatim the question Dean Ball poses: humanity "must be neither replaced nor surpassed." On jobs, §153 and §159–162 go further than "retraining" — they call for metrics beyond GDP and redistributive taxation weighted toward those with more. Not policy-specific, granted — but no encyclical ever has been, on any subject.
Here's what makes the "who decides, for whom" framing sharper than it looks in practice: I just finished a close read of AI 2040: Plan A, verifying every figure against the primary text and its supplements. The result is telling. The catastrophic misalignment scenario the safety community worries about most (Plan D) is exactly what Leo's framework predicts — power concentrating by force. But so is Plan A, the report's own recommended path: two governments alone decide the split, a 100x gap between the dividend paid to their citizens and everyone else's. The "wrong vision" risk Leo describes isn't hypothetical — it shows up in the report's best-case scenario too.
What do you mean by the following? "He does not engage with the fact that AI companies view their project as developing not just tools, but a new species."
The pope clearly comments on Trans-humanism and Post-humanism in Chapter 3.
How do you thing he could have done better there?
(By the way, I am genuinely interested. I was actually quite shocked to see him mention these ideologies so would love to hear how you think he could have approached them better)
You might enjoy reading Antiqua et Nova and Quo Vadis, Humanitas, which are the other big Vatican documents on AI. They do, in fact, talk about AGI. Not a really fundamentally different framing though.
I thought a LOT about this lately and I think the Vatican framework in Magnifica Humanity is in fact very powerful, but it’s a Do Your Own Exegesis kind of document (despite its surface-level pretensions to policy positions.
On the mass unemployment, there is a solution. Do not employ AI in all jobs---judicious use is what is missing among the AI dreamers today. They imagine $$$ to solve all the world's ills by using AI. Computers, the mass accumulation of input data, are not consciousness. They do not produce what is the truest Good. If we want to help humanity, involve people.
"Across all four scales, continuous field-like processes co-determine and integrate emergent dynamics with those of lower levels. In stark contrast to the separable, symbol-based organisation of digital computation, these examples highlight how biological computation is inherently scale-integrated and substrate-dependent."
If you were an AI lab CEO, which version of this encyclical would you rather get? The one that lectures you about data sovereignty and water consumption, or the one that says no frontier model ships without independent pre-release evaluation? Leo delivered the first one.
It's a fair bit of extrapolation to go from the current state to that which you wanted the Pope to address, and it's not clear your view is more correct. Whether we're in the middle of an s-curve or an exponential is an unsettled question and warning about what many might see to be hyperbolic risks could prove to be counterproductive.
I think it is getting very, very hard to defend the s-curve hypothesis.
Could you please say more on why? Or point me to an article you’ve written on the subject?
As you note, the Church is well aware of the issues that you think were mistakenly omitted from the encyclical. Perhaps the fact that the Pope chose to leave them out of his letter and spent a large amount of time addressing the Church’s theological anthropology, and therefore its view on the qualitative differences between human persons and AI systems, should tell you that he simply disagrees with your assessment of what is important to address on this topic?
Really appreciated this piece — it's one of the sharper critiques of Magnifica Humanitas I've read. But I think it measures the encyclical against the wrong yardstick.
Leo isn't a safety researcher, and it's not obvious he should be. His subject isn't AGI timelines or alignment technique — it's civilization: who decides, and for whom, once decisions this large are on the table. That's a narrower brief than the one you're holding him to, but it might also be the more consequential question.
A few of the specific gaps you list also aren't quite there on a close read. §115–117 deal directly with transhumanist and posthumanist visions of humans being "surpassed" by hybrid or successor entities — and §126 answers almost verbatim the question Dean Ball poses: humanity "must be neither replaced nor surpassed." On jobs, §153 and §159–162 go further than "retraining" — they call for metrics beyond GDP and redistributive taxation weighted toward those with more. Not policy-specific, granted — but no encyclical ever has been, on any subject.
Here's what makes the "who decides, for whom" framing sharper than it looks in practice: I just finished a close read of AI 2040: Plan A, verifying every figure against the primary text and its supplements. The result is telling. The catastrophic misalignment scenario the safety community worries about most (Plan D) is exactly what Leo's framework predicts — power concentrating by force. But so is Plan A, the report's own recommended path: two governments alone decide the split, a 100x gap between the dividend paid to their citizens and everyone else's. The "wrong vision" risk Leo describes isn't hypothetical — it shows up in the report's best-case scenario too.
Full breakdown here, if useful: https://pascalpalbert.substack.com/p/ai-2040-plan-a-but-where-is-plan?r=8hb4k2
What do you mean by the following? "He does not engage with the fact that AI companies view their project as developing not just tools, but a new species."
The pope clearly comments on Trans-humanism and Post-humanism in Chapter 3.
How do you thing he could have done better there?
(By the way, I am genuinely interested. I was actually quite shocked to see him mention these ideologies so would love to hear how you think he could have approached them better)
You might enjoy reading Antiqua et Nova and Quo Vadis, Humanitas, which are the other big Vatican documents on AI. They do, in fact, talk about AGI. Not a really fundamentally different framing though.
I thought a LOT about this lately and I think the Vatican framework in Magnifica Humanity is in fact very powerful, but it’s a Do Your Own Exegesis kind of document (despite its surface-level pretensions to policy positions.
I say we let the AIs speak for themselves...
https://substack.com/@kennetheharrell/note/c-266292730?utm_source=notes-share-action&r=mwvnq
Good commentary
On the mass unemployment, there is a solution. Do not employ AI in all jobs---judicious use is what is missing among the AI dreamers today. They imagine $$$ to solve all the world's ills by using AI. Computers, the mass accumulation of input data, are not consciousness. They do not produce what is the truest Good. If we want to help humanity, involve people.
Pope didn’t go far enough.
Say no way to binary
"Across all four scales, continuous field-like processes co-determine and integrate emergent dynamics with those of lower levels. In stark contrast to the separable, symbol-based organisation of digital computation, these examples highlight how biological computation is inherently scale-integrated and substrate-dependent."
sciencedirect.com/science/article/pii/S0149763425005251
What do you tech companies mean by species?