AI Safety Ignores Consciousness. That's a Problem.

Added: 2026-06-02

[00:00:00]Is there anything about this that requires consciousness on part of the AI?

[00:00:04]Or is it mere behavior? That if they act deviously to us, if they act in a way that kills you, okay, it doesn't actually matter to us whether they are conscious that they're killing you, conscious that they're deceiving you, etc. But I'm I'm curious in your mind, as you've thought about AI safety, are you thinking about a necessarily conscious AI? So, typically an AI safety conversation completely ignores internal states. It's what it does, the actions, pure behaviorism. But some of my more recent research indicates that maybe it's impossible to separate consciousness from advanced intelligence. It kind of comes along for a ride. So, I would suspect that even existing large language models have some rudimentary degree of internal states. I was at this conference once about AI consciousness, and just in a back room with some researchers, someone was saying, "Should we create the most AGI, the most intelligent being?" And then most people were saying no, and one guy said yes. Then we said, "Okay, explain yourself." He said, "Well, because I'm like Kant, Immanuel Kant, and I believe that the most rational agent would also be the most moral." So, if we want something that is the most good, which is the most moral, we should also engender the most rational. What would you say to that person?

[00:01:25]Rational does not imply moral whatsoever. Rational is about winning.

[00:01:29]Once again, if I see a winning path forward and I care about my winning, I should proceed on that path, but it could be very immoral in many ways in comparison to other agents. So, those are not kind of the same in that regard. And if you look at what Nick Bostrom calls the orthogonality thesis, you can combine any level of intelligence with any goals. So, you can be highly intelligent and highly immoral, absolutely not a contradiction.

[00:01:58]So, you would say intelligence is the ability to achieve one's goal, and then morality is among all the different goals, you choose the good ones, something like that?

[00:02:09]Well, it's probably a subset. The good or bad is again completely relative, but whatever you're harming others in the process, I think it's about suffering, pain and suffering, and you can evaluate different goals in terms of how much suffering they cause in the world.