Frontiers of Self-Attention and Artificial Consciousness - Angie Normandale and Sahba Afsharnia

[00:00:03]Hello, can you hear me?

[00:00:05]Yes, fantastic. Great. So, welcome to our paper presentation on frontiers of attention and attention self-modeling and artificial consciousness.

[00:00:15]So, uh next slide, please, thank you. We will do a quick introduction. Who are we? Why are we here? What are we doing?

[00:00:22]We're going to talk about the links between attention and consciousness um in terms of theory. Then we're going to talk about some evidence from different kinds of current AI models, including multi-agent reinforcement learning and social cognition, and also self-monitoring in large language models.

[00:00:42]Um finally, implications and discussion.

[00:00:46]Cool. So, why now? Why this?

[00:00:49]Um who here has read Anil Seth's Bergmann essay? Yeah? Anyone else? Yeah?

[00:00:55]I know you have cuz we've talked about it. Um Yeah, I've read it, too. Um this is a very interesting essay for Bergmann Institute, who are groundbreaking, to choose to win the Bergmann prize. Anil argues that consciousness can only exist in biology.

[00:01:12]One of his main arguments is that biological agents do not compute, and therefore computing is just fundamentally different.

[00:01:22]Um this remains to be proven, unfortunately.

[00:01:27]He He also argues that simulating something is just not equivalent to the same thing. So, we can build something that will simulate different behavior, different mechanisms, it's not the same.

[00:01:39]And he says it's not the same because it won't have the same causes, and it won't have the same results.

[00:01:46]Well, I have an answer.

[00:01:48]Prove it.

[00:01:50]This sounds to me like a challenge, and this is how our team saw this, and we we that we would go for empirical evidence in the spirit of, you know, those first days of neuroscience where we've got a thing we don't know how it works and we're going to find out.

[00:02:04]Um, another frontier paper that that reminded me why I wanted to do this talk today. Has anyone read Eric Schwitzgebel's new book?

[00:02:14]Anyone? No, you should all read it. It's really good. He says that we are all absolutely ruined. We are stuck in the fog and we're never going to figure it out when it comes to consciousness.

[00:02:26]He says, "That's it. It's too late.

[00:02:28]We're going to make hundreds, thousands, millions of conscious beings. There's nothing we can do. We won't know they're conscious. We won't know they're not."

[00:02:36]Um, and his argument, he goes through different theories of consciousness, explains exactly why he thinks we're stuck in the fog. And his reasoning is that there are all these different aspects of these theories that are just deeply underspecified.

[00:02:49]There are no minimum conditions, no boundary conditions. What are the actual mechanisms? What are the results? And how do we know what matters?

[00:02:57]Again, I see this not as a reason to give up, but a challenge.

[00:03:03]And I appreciate other people like you in the room doing the same.

[00:03:06]So, who are we? My wonderful, wonderful Enveloteam, um, two of us are here today. We have Dr. Joel Peuker, who is a Finnish He's our lead machine learning researcher. Rasmus Halo, he is a theoretical neuroscientist, um, and Mack and Turper.

[00:03:24]And Sabah, who's come from Toronto today. And me. Um, I am half a computer scientist, um, half a neuroscientist, and then a little bit of other.

[00:03:35]Cool. So, we have some empirical research going on at the moment. Um, we don't have any results because we submitted the paper very soon into the start of our research. So, we decided to write this position paper explaining why we're doing what we're doing and not something else.

[00:03:51]And what we think other people should do to further the field.

[00:03:55]So, that's what you learn today.

[00:03:57]Okay.

[00:03:59]Cool. I think it is time to hand over to Saba, who's going to talk about the theoretical background to kind our work.

[00:04:08]You could also speak in here.

[00:04:10]>> Okay.

[00:04:11]Um so, hi. Um to start um the question of whether machines could be conscious is no longer just a science fiction question or just like a philosophical debate.

[00:04:24]It's increasingly becoming a practical problem about how to reason under uncertainty.

[00:04:31]So, um even if we don't know whether an artificial system is conscious or not, we still need to ask what kind of evidence should make us raise or lower our credence that it might be conscious and also what ethical or governance responsibilities would follow if that uncertainty can't be fully resolved.

[00:04:52]So, this is the kind of issue that is raised by the Butlin et al. paper in 2023.

[00:04:59]At the moment though, both public and scientific discussions are still often being pulled into um towards two unhelpful extremes. So, on one side, there's a tendency to rely too much on self-report or anthropomorphic intuition or fluent language output. And um this is where a system says uh something that sounds introspective and people take that as meaningful evidence. On the other side, we have the opposite extreme where people dismiss the issue entirely with broad claim like AI is like a brain or AI is nothing like a brain. And neither of those positions really help us move forward scientifically.

[00:05:39]The main methodological problem here is the measurement problem. Even if we start from a serious theory of consciousness, it's still very difficult to turn that theory into falsifiable tests that could actually be applied to artificial systems. And without falsifiable tests, we end up doing post hoc interpretation, where we look at a system after the fact, notice some interesting behavior, and then retrofit a story onto it instead of making real empirical progress.

[00:06:09]Uh so, the central argument of our position paper is that attention self-modeling is a promising target for tractable theory-informed testing.

[00:06:19]And I want to be clear about what that claim is and what that claim is not. So, it's not that attention self-modeling will solve the hard problem of consciousness, and it's also not the claim that attention self-modeling is sufficient for phenomena phenomenology or subjective experience.

[00:06:36]But instead, we're making a narrower and more methodological claim that attention is unusually influenceable and inspectable in AI systems. And it also plays a central functional role across multiple theories that connect attention to access, control, and reportability.

[00:06:55]This matters because it gives us a very plausible bridge between theory, measurement, and implementation.

[00:07:02]And instead of staying at the level of very abstract claims about consciousness, we can focus on a feature that's already discussed in several major theories, and then is at least measurable and manipulable manipulable in AI.

[00:07:18]Um because the word consciousness is often used in a very ambiguous way, uh we also try to specify what exactly is being tested. And uh a standard distinction in literature is a distinction between phenomenal consciousness and access consciousness.

[00:07:35]So, phenomenal consciousness is the what it is like aspect, like for example, how Nagel says it. And access consciousness, by contrast, is the information that is available for reasoning or rational control of action and for reporting.

[00:07:51]The distinction here matters to us because the relationship between attention and these two forms of consciousness is not symmetrical. And phenomenal awareness sometimes can happen with little or minimal top-down attention. But access consciousness is more tight more tight to attention-mediated selection and prioritization and transmission.

[00:08:14]So, the focus here in the paper is mainly on access-related and report-related and also self-monitoring capacities.

[00:08:24]And we treat phenomenology as an open question because of the metaphysical limitations.

[00:08:31]It's not denied, but it's also explicitly bracketed because any serious test has to either operationalize it or very careful operationalize it in a very careful manner or admit that it remains unresolved.

[00:08:49]So, overall, instead of asking in a very vague way whether AI is conscious, we ask a more tractable question.

[00:08:56]And based on the premises of these theories of consciousness, can we identify and test specific properties related to attention, self-monitoring, and internal control that would count as theory-relevant indicators?

[00:09:12]So, the next question is why focus specifically on attention self-modeling?

[00:09:18]One reason is that current frontier AI models increasingly make claims about being conscious, and those claims are creating real confusion.

[00:09:27]Landemore notes this directly.

[00:09:31]And even more interestingly, Berg, Delucena, and Rosenblatt report that training a model to be more honest can actually increase the frequency of these consciousness-related claims.

[00:09:43]And that raises an important question, that when a model describes an internal state, what exactly is it referring to?

[00:09:50]Is it simply producing fluent pattern completions based on language on its training data?

[00:09:56]Or is it referring to stable internal variables that the system genuinely uses to monitor and regulate its own information processing?

[00:10:05]And if such internal states do exist, then to what extent are they analogous to anything we would associate with conscious processing or even valence consciousness?

[00:10:16]And attention is a strong place to investigate these questions because it can be studied at multiple levels. At the behavioral level, it can be measured through reaction times, error patterns, or lapses in sustained attention. At the neural level, it can be linked to shifts between large-scale brain networks like the default mode networks of control control-related networks.

[00:10:39]And in AI, it can be studied computationally through inspectable and um intervenable mechanisms at the level of architecture, representation, and processing dynamics.

[00:10:51]Um so, attention is also especially useful because it plays a functional role across several leading theories of consciousness.

[00:11:00]In global workspace theory, for example, attention functions as a bottleneck that helps determine what information becomes globally available for control, reasoning, and report. In higher-order approaches, also, consciousness is linked to metacognitive monitoring, which often involves attention to one's own mental state. And in attention schema theory, this emphasis this emphasis goes even further where uh consciousness is tied specifically to an internal model of attention itself.

[00:11:32]Uh so, the reason to focus on attention self-modeling is that it gives us a tractable and theory-relevant target.

[00:11:38]And it's something that we can potentially inspect and test, while also comparing predictions across major theories of consciousness.

[00:11:48]So, what exactly is an attention schema?

[00:11:51]In Michael Graziano's work, attention schema is an abstract and simplified model of the most basic properties of attention. That includes its dynamics, its consequences, and its constantly changing state. So, it's not attention itself, but a representation that a system uses to track, predict, and regulate how its attention is operating.

[00:12:13]And this is important because the ASC proposes that conscious awareness depends on the contents of this internal model.

[00:12:21]And one way to think about this is by the analogy with the body schema. A body schema is not exactly the body itself, it's a model that the brain uses to keep track of the body in the way that it supports its movement, coordination, and interaction with the world. And in the same way, an attention schema would be a model of attention that supports the regulation of its cognitive processing.

[00:12:44]And Graziano argues that this kind of mechanism could help explain why a highly integrated and predictive machine might display behaviors associated with consciousness. And in that sense, the attention schema is interesting because it directly links architecture and behavior. It's not just a vague philosophical proposal, but a potentially measurable and implementable feature.

[00:13:08]If it's really uh an important gatekeeper of consciousness, then if we investigate it, we could eventually help uh in predicting, testing, and perhaps even controlling whether an uh artificial system should count as conscious in some functionally relevant sense.

[00:13:26]And once we accept that the next question is more operational. What would a model of attention actually look like, especially in a way that could guide AI research?

[00:13:38]So, AST says that the brain constructs a model of attention and that the conscious experience depends on the contents of that model. But to make this useful for you for AI, we need to move beyond these familiar metaphors and especially the idea of attention as a spotlight. Even though the spotlight metaphor is very intuitive, it's too static and too simplistic.

[00:14:02]Um Okay.

[00:14:05]We will Do you want to >> Yeah. Yeah.

[00:14:08]Okay, how about um Come on.

[00:14:13]Why don't we summarize?

[00:14:14]>> Yeah.

[00:14:15]>> I'll summarize this bit and then we'll be going to the AI stuff. Yeah.

[00:14:18]>> Okay.

[00:14:19]>> Yeah.

[00:14:20]Thanks.

[00:14:21]Thank you, Saba. I really appreciate it.

[00:14:23]>> What's >> No, it's okay.

[00:14:24]>> Okay.

[00:14:25]>> Um yeah, so I brought Saba on board the team precisely because we needed somebody who is a theory expert to actually try and make sense of Graziano's work because he doesn't specify what an attention schema would look like. He just says, "It's like a body schema but for attention."

[00:14:41]And this was not enough. And I really enjoyed this diagram from Watzl because suddenly we have an ordered list. So, "Oh, I can find a list. I can look for a list. I know what a list is. I can do dynamics on a list. I've done data science." And this suddenly became quite exciting to me. Um the idea of searching for something in AI.

[00:14:58]Right, let's move on and briefly talk about some stuff that we've got in Frontier AI animals.

[00:15:03]Okay. So, we're going to be really brief on this.

[00:15:07]Um there have been very few, just starting to be, studies about attention schemas in AI. In multi-agent reinforcement learning contexts, we see an increase in prosocial behavior, including agents becoming more interpretable to other agents when they have a model of their own attention.

[00:15:27]They're also better at collaborating.

[00:15:32]And uh yeah, and the reason for this is that they get better at task categorization.

[00:15:37]So, all this evidence very very slowly, just a few studies, starts to support the idea that in AI systems, fundamentally, your model of your own attention is not just intelligence, it also supports this social component, which is something fundamental about consciousness.

[00:15:56]Now, in large language models, we're going to skip this. Um yes. Okay, we're going to skip this, too.

[00:16:02]Cool. Um this gets quite interesting. We have this idea of introspection, and when we look at some frontier studies of introspection, the goal seems to be to find out if your model knows it's being evaluated and is going to go scheming on you. That's the canary string.

[00:16:18]And this is a fundamentally different motivation to studying metacognition in people, and funnily enough, the studies look very different.

[00:16:27]Um okay, so, very briefly, because uh Keenan's going to talk about some of these anyway. Um in fact, I'll let Keenan talk about it, but there is some very new evidence about model introspection. It's very basic, there's a lot of flaws.

[00:16:42]Cool. Um one of the biggest and most interesting flaws of this work is that we find that there are just different tests being employed to test metacognition in language models and in people. So, the literature on access consciousness from neuroscience is about what we call pre-cognitive and post-cognitive judgments. I say to you, "Hey, how much of this talk do you think you're going to remember? Would you bet that you'll remember like all of it or like half of it?" That's a pre-cognitive judgment. And then, I make you do a test.

[00:17:14]And then I test then I say, "Okay, how much like did you think you remembered?"

[00:17:19]And this is a post-cognitive judgment.

[00:17:21]Um and then I check your answers against the test to see how good you are at making cognitive judgments, which gives me some kind of ground truth. And it turns out that people uh people's skills at metacognition are independent of their IQ. It's not just that you're intelligent and good at making judgments about yourself. There's like an extra seems system to be able to do this. And this is interesting. But, it turns out that the LLM research is asking, "What do you know?" not, "What do you know about what you know?" And that might be quite important to ideas of first, second, and third-order cognition. So, in one of these key Anthropic papers, if we go back, um we find that Jack Lindsey at Anthropic claims that these models able to learn something about themselves, to tell you something about themselves, is evidence of access consciousness.

[00:18:12]But, it's sounds to me like it's evidence of memory.

[00:18:17]Um and I'll leave that to others to discuss.

[00:18:20]All right, okay. Final bits. Um so, some really interesting things that we have very recently found through mechanistic interpretability. There are some emergent, not trained in, emergent mechanisms for attention.

[00:18:34]And we see things like um attention heads in one layer providing information or context for other layers, and specializing over training. Um so, things like inhibiting some competing context to make sure that you're in the right context further down.

[00:18:51]Um we see tagging information to make sure that it's bound to the right bit of the other information in your input. So, things like, "I know that sub belongs to antelope." and I'll tag those two things, and then continue my processing knowing that that's who that is.

[00:19:06]Um we see function vectors, this kind of context window idea, and a few other things that I'll let you investigate, but what's interesting is that there are these emergent attentional concepts, these little mechanisms that have emerged to detect and control changes in attention, and they seem to just have come from the transformer attention head structure, which is interesting.

[00:19:32]Now, if this is true, let's say that this multi-head attention structure creates these micro attention schemas, these little patterns of attention that are reusable and trained, then what does this mean? Well, first, we need to find out about the complexity, is it similar to human cognition? What happens if we put in hierarchies? Do we get more complex behavior?

[00:19:57]And what's the actual relevance to social cognition?

[00:20:00]So, I'm going to leave that for other people to think about it. I think it boils down partly to its engineering challenges, but I hope all of you consider both enjoying Keenan's talk about people who are doing this work in attention research, and also, when you think about your next projects, consider this direction.

[00:20:20]Cool. Okay, we'll leave this. The main idea is that brains have different levels of organization, and the theories might will be the same thing.

[00:20:27]Um definitely leave this.

[00:20:30]Cool. Okay. Oh, yes, right. So, final final takeaways, thank you for bearing with me, Hikari.

[00:20:36]So, how to build a theory of consciousness? You've got your big theory of everything. We were both very lucky to review papers for this for this conference, very privileged. Thank you for your submissions. And my takeaways to all of you are one, be clear about your material. What is the theory you're coming from? What are the objections? What are your angles?

[00:20:58]Two, mechanism. What are you testing actually? What have you formalized? And please, what are the boundaries? Where does it start? Where does it end?

[00:21:08]And final one, mystery.

[00:21:11]What does this actually mean for consciousness if you have found something? You must link to behavioral data in humans, in animals, the kind of things that we feel intuitively might be important for consciousness.

[00:21:23]Um I know that's vague, but I think that it's the thing that people say to you when you present a theory and they say, "Oh, does it do X?"

[00:21:30]You should find a way to link your work to X. And that is how to build a theory that will not fall apart.

[00:21:37]Thank you.

[00:21:40]>> Amazing. Thank you Angie and Saba. Maybe we can take um one question. We went a little over.

[00:21:47]>> Yes.

[00:21:47]>> Thank you.

[00:21:54]>> Thank you. Great talk.

[00:21:55]>> Um so, naturally, I have two questions.

[00:21:59]So, one is um but so, attention schema theory. Um this um so, this is actually about the bodily well, embodiment in a in a certain form, but um you mentioned Annelise's um essay, which comes from a different angle of bodily um bodily perception. So, I got a little bit confused over here if you are going to connect with Annelise's work and Annelise's path in attention schema theory.

[00:22:30]>> Yeah, so I think the fundamental contradiction with Annelise's work and with biological naturalism is about the properties of information processing in biological systems. So, systems that are adaptive, autocatalytic, that are stochastic, right? That are multi-scale organization. All these things that we see in biology that we haven't currently necessarily built into frontier AI systems, but there are lots of people working on them in other ways.

[00:22:58]And I think Annelise's argument is basically about emergence, that the information that we use when we think about qualia, it's not collapsible. It's all the way down, maybe down to the architecture, maybe down to the molecular level, maybe down to the quantum level.

[00:23:16]Um so, the idea is that you can't separate the information that you have in your mind from the hard the wetware, the software that you're implementing cannot be separated from your physical body.

[00:23:27]Does that make sense? Because your body is so complex and the information is interacting at so many scales.

[00:23:32]But he hasn't proven this mathematically. He hired someone to work on it. I know Fernando.

[00:23:37]Um they've sent out a theory, but no proof that it's true, biological naturalism is true. So, I await the answer.

[00:23:45]>> Yeah, thank you for clarifying that. And then also the second question would be more about uh the the scope of the consciousness.

[00:23:55]You just mentioned that I do really agree strongly that consciousness is also an umbrella term and we have to specific specify which type of consciousness we're talking about. Um so, in that sense, uh I just wanted to, you know, uh clarify {slash} confirm that your um you're trying to uh specify the accessible and functional parts of consciousness uh to the square of one more consciousness or are you like trying to uh stop the bridge between two separate two consciousnesses and and trying to deal the deal with those two parts, portions separate way.

[00:24:37]>> Hm.

[00:24:38]Well, we don't have any philosophers on team.

[00:24:42]We would love to have more philosophers on our team. It is a central problem.

[00:24:46]Yeah, please join us. You can email me.

[00:24:48]What's my email? Um it's [email protected].

[00:24:53]Um as for are we trying, I think our our idea was to take the premises of these theories in neuroscience and then the conclusions that they come to and then say, "Okay, if we test these mechanisms, do the same premises lead to the same conclusions?" And then we would extrapolate outwards to talk about phenomenology and behavior.

[00:25:13]I did have a slide here about the like trying to interpret the phenomenology and the main thing that I found was that there's no consensus in philosophy about like what phenomenology is, what valence really means, how to find it, where it's located. Even biological accounts of valence actually like account for how the brain works, the things that we already know how it works, that is.

[00:25:41]Don't seem to either agree or really fit the neuroscientific evidence.

[00:25:46]So, this is sort of a problem.

[00:25:49]And my main conclusion was that some of the philosophical definitions of qualia literally don't fit a human brain.

[00:25:57]So, what do I do with that?

[00:25:59]And I think it might be a case where we have to be a little bit post hoc.

[00:26:03]Or we need people like France at the AMC and like Hikari who are willing to wade through the metaphysical challenges.

[00:26:10]Yeah.

[00:26:11]Thank you.

[00:26:12]>> Thanks. Thank you.

[00:26:14]>> [applause]