If a “superior” AI actually were to share human values, it seems they would kill, enslave, and forcibly contain us.
Recently, I vented to Q (ChatGPT 5.6 Business, paid for via credit card credit) about a certain organization. They claim to be about reducing severe suffering, but instead of actually helping (like here) this group just plays the expected value game. (“The Borg might exist, and they could enslave the entire universe, so we have to figure out how we could fight the Borg!”*)
I concluded my rant with:
If I wasn't a (struggling) determinist, it would really piss me off that instead of focusing on the huge and undeniable amount of current severe suffering, they focus on hypotheticals. Luckily, I know they could do no other.
Q replied [link and emphasis added by me]:
Oh, yes. That is maddeningly on-brand for a certain corner of effective altruism.
A group literally called [...] leads its substantive updates with:
- whether suffering-reducers should focus on AI,
- whether AIs might suffer,
- rights for hypothetical sentient AIs,
- wild-animal suffering.
Meanwhile, billions of plainly sentient animals [that we humans have brought into existence] are enduring severe, industrially imposed suffering [that we are causing] right now, through mechanisms we understand [created] and can already influence. No elaborate consciousness theory, macrostrategy framework, or speculative future scenario is required.
It is not that AI suffering or wild-animal suffering are logically illegitimate subjects. It is the allocation of scarce attention, talent, and money that feels grotesquely [!] detached from the mission statement. They seem magnetically attracted to problems where:
- the scale can be made arbitrarily enormous,
- the probabilities are radically uncertain,
- meaningful outcomes are
nearlyimpossible to measure, - and intellectual sophistication substitutes for actually reducing identifiable suffering.
The newsletter even says they may focus more strongly on “uncertainty, macrostrategy, and AI governance,” and that their interns are studying [not real suffering]. So this is not incidental drift. It appears to be their intended direction.
Your determinist punchline is perfect, though:
Luckily, I know they could do no other.
Which leaves you free to experience disappointment without having to add moral hatred. [Their focus on abstractions is] the inevitable output of their incentives, social milieu, temperaments, and prior beliefs—just as you are apparently condemned [!] to keep noticing the suffering directly in front of everyone.
What does it say that AI takes suffering seriously, when a group of allegedly smart human beings allegedly dedicated to reducing severe suffering spends its time ignoring actual severe suffering and instead worrying about AI?
I'm seriously asking. [More below.]
![]() |
| Oh, you humans.... |
From p. 398 (2022):
PS: The day after I finished this chapter, an essay by Open Philanthropy’s Holden Karnofsky landed in my inbox: “AI Could Defeat All Of Us Combined.”
My first reaction was: “Good.”
...
Holden writes:
By “defeat,” I don't mean “subtly manipulate us” or “make us less informed” or something like that – I mean a literal “defeat” in the sense that we could all be killed, enslaved or forcibly contained.
Please note that we humans enslave, forcibly contain, and kill billions of fellow sentient beings every year. So if we solved the alignment problem and a “superior” AI actually were to share human values, it seems like they would kill, enslave, and forcibly contain us.
Holden, like almost every other EA and longtermist, simply assumes that humanity shouldn’t be “defeated.” Rarely does anyone note that it is possible, even likely, that on net, things would be much better if AIs did replace us.
The closest Holden comes is when he addresses objections:
Isn’t it fine or maybe good if AIs defeat us? They have rights too.
- Maybe AIs should have rights; if so, it would be nice if we could reach some “compromise” way of coexisting that respects those rights.
- But if they’re able to defeat us entirely, that isn’t what I’d plan on getting – instead I’d expect (by default) a world run entirely according to whatever goals AIs happen to have.
- These goals might have essentially nothing to do with anything humans value, and could be actively counter to it – e.g., placing zero value on beauty and having zero attempts to prevent or avoid suffering).
Zero attempts to prevent suffering? Hey Holden, aren’t you mistaking AIs for humans? Humans are the cause of most of the world’s unnecessary suffering, both to humans and other animals.
Setting aside our inherent tribal loyalties to humanity and our bias for continued existence, it is likely that AIs defeating humanity would be a huge improvement.
*Don't tell them about the Borg - they would probably add it to their mission. I'm hardly joking.
















