History of Neuroscience and AI: Visual cortex and the Birth of Modern AI

Added: 2026-05-26

[00:00:00]so I left off the last lecture as sort of a cliffhanger we got to the Rosen Blatz perception which was directly inspired by the way the biological neurons are capable of computing and it even implemented a rudimentary sort of learning algorithm so that the network can not only um accomplish tasks um it can also learn to correct from its own mistakes if you told it and to go in the right direction but then it hit a wall with the well-known XR problem funding was pulled people were disappointed um things didn't really happen for a long time so what happened in the time since is that a small number of people toiled away and then kept working on this kernel of the idea of building artificial neuron networks that are capable of doing certain things so what happened in the next three decades is that artificial neuron networks um was not popular um in that most people didn't hear about it most popular uh certainly wasn't in the media was no longer in the New York Times after bis and blats uh promises were not fulfilled but what happened is that neurobiology kept progressing and so did the the development of the computation and Mathematics of artificial neuron networks so what happened in biology during those uh couple of decades is that people started being able to understand a lot more not only about single neurons and how they function but how networks of neurons are connected to each other the architecture of the nervous system and what the amp computational implications of those things are so this is most well studied in the visual system um people know know a lot about others but I'm just going to focus on the visual system just because that also has the most directly connected in terms of inspiring the development of neural networks in artificial neuron networks and so in uh the visual system you have neurons in your retina in your back of your eyeballs that are photo detectors it's capable of responding to physical stimuli such as photons hitting the very back of your eyeballs so the circuitry of what happens after it hits the retina and the retina actually already has a bunch of neurons in it is that it gets passed on to a mid-brain area called the thalamus in particular the lateral nucleus lgn it then goes to the back of your brain to the aital cortex right here in the very back of your brain V1 stands for visual cortex one the first place it goes and then it goes to a bunch of very creatively named places called V2 V3 and V4 etc etc now I will note that this diagram is a vast simplification but this is sort of the picture that was emerging in those early decades when people were tracing these tracks looking at the anatomy of how the neurons in your eyeballs are connected to the rest of your brain um and uh this is all in the very bottom of your brain for uh just just just for Simplicity sake so that's actually um a picture there of the bottom of your brain like if you took it out and then looked at it as if you were like staring up through your mouth that's what the bottom of your brain would look like it's really cool looking um in fact it's really cool here because this is uh the optic nerves are actually cut off right here these are your factory bulbs right here like right behind your nose this is what's called called the aptic kosm it's where the tracks all the nerves from your left and right eyes cross because the left eye sees left and right hemifields and has to go to the opposite side of your brain why it crosses like that really fascinating story that's a totally different topic but it does cross Okay cool so this is what's emerging in visual neuroscience at that time okay and furthermore not only were people doing track tracing and looking at the architecture of how the nervous system all these neuros circuits are connected to other they were also characterizing the computation implementations and computational implications of what each of these neurons at these different stages were capable of representing the technology that mostly drove this Innovation at the time was what's called an extracellular reporting now I told you in the beginning about how neurons are electrically active they don't it's not wires and electrons right these are movement of um charged ions sodium potassium for example in and out of cells but nevertheless that is electrical activity and so we can use an an P um a wire stick it into the brain this does require brain surgery um and be able to listen in on what cells are saying we don't necessarily know what language they're speaking but that's what people were about to find out in the 1950s 60s and70s okay so a lot of Technology development at this time that then accompanied much further understanding of what is happening in the visual system how does it detect photons and how is that information being transduced and understood into your perception ction of the visual world that is distinct from what would be gotten from a CCD from a photo detector and so what uh one of the key disc discoveries and this is a subject of a Nobel Prize by huban a weasel is that single neurons in the primary visual cortex can already have pretty sophisticated computational capabilities and here's an early example of this one neuron in V1 of which you have like hundreds of thousands of them and it is able to detect a oriented box that is moving in One Direction but not another what we're looking at here is a Trace by an electrode that's stuck into the cat's brain of voltage as a function of time as it goes to the right and each of those little vertical lines here that's one action potential a single action potential so you can see here that this particular cell is talking a lot as long as the bar is oriented at this 45° angle and going up but the same bar going down made the cell not talk very much at all and the closer the orientation of the bar is to that angle the more the cell tucked but not if it were moving in the opposite direction so the computation here that underlies this direction selectivity is what was discovered in neurobiology and so people were doing recordings and making computational mathematical models of how this could possibly work and so this emerged the idea of what is called a receptive field which is that cells in the visual system neurons in the visual system they don't respond to everything and there's certainly nothing like pixels okay that all I showed you that was directly sensitive to a to a to a moving bar not at all like a pixel that you would get off of a camera and it's because the visual system is nothing more it's like nothing like a camera even in your retina in the very back of your eyeballs and so you have these photo receptor cells that are capable of being active they're activated they can detect photons coming in okay and then there is already quite a bit of connection and a network just in the back of your eyeballs that's making computations I'm going to simplify and walk you through some of the computations and the connectivity there and show you how you can actually infer and model from the connection the function of these cells so what's happening here is that there's a canonical circuit The Motif that gets seen over and over where the photo receptor cells there's a bunch of them in the middle that uh gets activated into what's called an on cell so the more these are active the more those are active and then it's surrounded by these off cells okay these off cells are inhibitory and they all get summed remember addition subtraction summing into this uh what's called an on-center retinal gangan cell now importantly this retinal gangan cell is the only one that leaves your retina all the rest of these cells do not directly talk to the rest of your nervous system the retinal gangline cell is the only one that gets a say okay this goes to the rest of the nervous system and so by this architecture here you can already tell the the only way to get the retinal gang glean so to say anything is that there's a spot of light in the middle and there's no light on the surround so it's called a center surround cell okay so if everything's active it's actually you know they cancel each other out right so the most active that this retinal ganging cell is capable of being you can infer this from the connectivity is that if there's a spot of light in the middle of the correct size and there's Darkness on all the different sides of it so that's a Oncenter field cell if it or the opposite if there's a dark spot in the middle and uh and light on the outside that would be even worse right it would be even less active in that configuration and so this concept that you can infer this receptive fuel the function of the cell and that it's already sumarize the information from lots and lots of different inputs and making a decision and that this effectively it's a pattern matching right there's a little filter here that effectively convolves the input and then makes an output is a theme that we see in neural computation over and over and over this is basically the simplest example of it that we see in the retina but this theme Here gets repeated over and over okay so what we can see here is that uh this is basically because that's one type of cell that exists the opposite also exists what's called an off center retinal gangan cell where it's uh it wants the cell it wants a a dark spot in the middle that's surrounded by light okay why do you want to do this well it's actually really good for contrast and for change direction because you don't actually want to pay attention to everything everything that's in front of you equally you kind of as an animal would like to know when something is changing when there's edges those might be important they might be trees they might be predators and you want to know when things are changing in time as well and that's a different circuitry that is responsible for that um there's also um what's called lateral inhibition so there's actually more cell types here that I'm totally glossing over and that helps uh reinforce this contrast selection this contrast enhancing property of these retinal circuits okay these are all filters that we can have uh in computers in Photoshop right but your I already does it so by the time it leaves the your retina all of this is already done okay the consequence of this as we keep going down is that you have a bunch of these retinal gangan cells let's say there are a bunch of of on on Center on Center cells they then project to the midbrain region in the thalamus called the lateral geniculate nucleus okay and then they converge onto V1 and this is the known circuitry of the system people have track traces you can basically uh inject chemical tracers into these cells and you can kind of see where they go this is where they go so retina goes to lgn and then they get a bunch of lgn cells and project onto the same V1 cell V1 is where those Direction selective cells that Hub AO saw were okay now notice What's Happening Here is that if you do it this way if there's a bar of light that hits the center of a bunch of these retino gangan cells the V1 cell will turn on that will be maximally active if it is a bit off okay then it's it's going to be off again so only if the the bar were right here and if it were at the correct angle would the V1 cell be responsive okay so you basically built for yourself a convolution filter by the architecture and the connectivity of these retinol gangan cells furthermore if there is just broad illumination from everywhere similarly the V1 cell will be off so it's really selective for a bar at this orientation that's the only way you can get it to fire okay so that's the implementation of the convolution and what's happening here is that there's a bunch of convolutions that convolution filters gets into the thalamus and then they converge once again so remember there was already converging that goes onto here right for The receptors horizontal cells goes to those things you convolve again there's a pooling here I'm using these words on purpose and the reason I'm using these words on purpose is because this is directly what started what inspired the first generation of convolution neuron networks the first convolution neur Network a CNN this is a thing that underlies all of image recognition Okay this is a thing that recognize faces and calls out people's names this is the this is the technology that makes it so you hold up your phone camera and immediately recognizes people's faces and even suggests that is that your friend Luke over there this is it okay this is the first one that was ever built by Fukushima back in 1980 he was still working on it even though lots of other people had given up hope Fukushima did not give up hope and inspired by the insights were coming out of visual neuroscience around those couple of decades about these like pooling and convolutions right the different types of cells that have been described in V1 um he built this architecture it had multiple layers so it's like a perceptron in that it had units that were neuron neurons each of these had many neurons but then he had layers of them and we know that he was directly inspired by visual Neuroscience because he called these layers s and C S is for simple cells and C is for complex cells these were the cells that have been described in V1 around the time when this came out and so this was directly inspired by the insights that were coming out of visual science and people trying to figure out how it is that your visual cortex works and how is it connected and how is it able to detect things that are happening in the visual scene like it is okay now that's all well and good but I wanted to point out something that is uh super interesting at least to me which is that I told you before that this this this diagram is a vast simplification okay if you trace all the neurons this is this does happen okay these neurons do talk to those and they talk to those and they talk to those and they keep going and so on and so forth but the reality is uh quite a bit more complicated um but we're going to ignore that for now because like I said in in the beginning all we really want to do is we want to be Picasso everyone wants to be Picasso right you want to look at a cat and be like you know we're not going to draw every hair it's not actually important we're just a skip over it and only draw the things that captur the essence of the cat so between 1980 the first Neo cognit Tron okay the first convolutional nerk that was made and down here to 2012 with Alex net 2012 is when I would consider this to be the the the the the the the beginning of the modern AI age okay between 1980 and 19 and 2012 approximately 30 years a lot of things happened but not very quickly but here are a couple of key things that happened Yan laon at all published well lons net um that capable of uh solving problems like uh handwriting recognition this was a version of the convolutional neural network okay and he developed also a algorithm a really really efficient algorithm for training neuron networks that we still use today concept is called back propagation okay um and then in N uh 2009 F Fe Lee and colleagues they uh decided to collate um the most gigantic at the time collection of data about natural images that they found on the internet but then they did the work the really really hard and needed work of coating it and labeling it so the availability of the CNN technology this idea of having multi-layers of architectures of convolution and and doing that many many times stacked in a row with a efficient algorithm for training convolution neural networks some mathematics and computational insights had to be worked out and then the availability of the largest data set that was high quality and curated of images at the time okay that came out all of that had to happen along with the development of GPU technology so graphical processing units at the same time these couple of decades were just getting stronger and stronger and getting more and more better faster smaller cheaper and that was not driven by science it was not driven by obscure scientists toiling away uh at conferences that were that were not nearly as well attended now as as they are now but that was driven by the video game industry um and so uh at that time this is a this has also been called the hardware Lottery hypothesis right all of this would not have happened would not have culminated in the success of of 20 2012 if the algorithms the insights from neurobiology and the hardware from the video game industry hadn't all converged at exactly the same time to make this Insight possible and so in summary what happened in these 30 years was that a lot of things to happen at the same time right we needed lots of data we needed a lot more compute we needed better Network architectures and better optimization algorithms after all of that happened the rest essentially is how we entered the modern era of artificial neuron networks after 2012 Alex net essentially blew every other image recognition algorithm out of the water like it wasn't even close right you go to these conferences and year after year Sometimes the best paper is like 0.1% better than the previous year and so people are making progress learning things doing lots of great great work and then neuron Network came along after all of this and was just way better than anything else that anyone else is doing and within a couple of years no one else in image computer vision was doing any type of research besides neuron networks everyone else had to Pivot and abandon what they were doing and work on neur networks instead because it was just a way better approach to doing computer vision and now lots of other things right but I did want to point out that it took 30 years okay and it took 30 years from the first CNN and it took 20 years before that before the modern comput convolutional neuron Network came from the first perceptron okay so it took a really long time but the history was there the groundwork was there and it took a lot of work by a lot of different people who were communicating who were talking to each other and who were taking inspiration from biology um as well as contributing new tools for the understanding of biology for all of this to culminate in the story that starts the modern era of AI in 2012 so we'll go into it a little bit more because the story is actually a little more interesting than that right we talked a lot about inspiration from uh neurobiology to the development of convolution neural networks but then people started looking at okay so what the heck are these convolution neural networks doing and it turns out that that had a lot of coincidence in common with the um not only the architecture and the connections of the visual system but what the early visual system is doing as well so we talked about how the retina goes to the lgn lgn goes to the V1 and what happens when you have all of this pooling right all of this convolution and pooling is that uh you go from um detecting spots to detecting to detecting um uh spots of light with contrast and you can put those together to detect bars and then you can put those together to detect oriented bars that are moving stuff like that right it turns out that that's exactly what happened with the CNN that were being trained so if you look at the earlier layers here are the effective filters that the CNN were learning this has by the way it was inspired by Vision but it didn't actually use the knowledge I just said it was detecting things like oh look at that this is like an off-center surround cell right it's got a dark spot in the middle and it has a ring to it and you see all of these like stripey things and things that have a curves to it so that's what was detected in the first couple of layers of the CNN if you keep marching forward you'll see things that are more Composites you're seeing patterns like these little hexagonal circular patterns here you're seeing things like uh intersections that are being detected and you're seeing things like this one here which is kind of like an oriented bar or sometimes um little stripes with different spatial frequencies that's what's being detected in layer three if you keep marching forward once again we start seeing things that are known to be what's in the later and later parts of the visual system so after you get past V1 you get to V2 V3 V4 etc etc and a lot of those higher visual areas for example we have ones that detect faces like a part of your visual system that totally lights up anything anything looks like a human face including if it's just a rock in the mountain you're like face over there I see it dog okay you get dog neurons you get lip neurons you get Circle neurons you get like feather neurons okay and these are things that looked heck of a lot like what is happening in the Maman visual system almost as if it were an inevitable consequence of the architecture and the way that the network have been trained and so these patterns of commonalities can be over interpreted but no one can deny that it was really really interesting that this happened that way and in fact neuron networks the story that I've been telling you I think is a really great example of how life imitates life uh life imitates art far more than art imitates life and so there's a lot of inspiration from biology that got into the development of modern Anns but people are also using a Ann's as a way of understanding biology as well in fact this is one of the most modern indispensable tools in the study of Neuroscience as it exists right now using artificial neuron networks as a tool as one of many many tools to help us understand what is happening in the nervous system here's a couple of examples of that so this is a paper by Dan yans and Jim De Carlo where they use deep learning um artificial neural networks of many layers as a model of understanding sensory cortex how it is that we can go not only from the retina to uh algan to V1 here you can kind of keep going right all the way down to behavior because after all the nervous system in the brain don't just go they don't just sit there perceiving things it also generates action it decides what to do next okay and what you're doing next has a direct influence on what you perceive next because if I decide to walk a little bit over here it changes the input to my retina at the very next instant okay so using deep learning as a as a way of understanding sensory cortex but also then using deep reinforcement learning to understand what's happening in action generation because this diagram is still too simplistic in real it what happens is that there's a bunch of recurring connections the only one that is truly feed forward is that there is in fact only Fe forward connections between the retina and the thalamus this doesn't go back okay every other Block in this diagram has connections to almost every other diagram the arrows are not all equally thick okay but they exist and this is how biology works okay the thing that is uh really the the thing that blew my mind the first time I heard about it is this observation here we have this canonical picture of the thalamus projecting to V1 okay but if you count the number of wires from Thalamus to V1 versus the the number of wires that are going back there are far more of them going back than there are going forward and so the approximation that this is a feed forward system that information just gets passed down in the CNN is not only a simplification is kind of the opposite of what's true and even this diagram is still a vast simplification of what is actually happening so really embracing the recurrence of the neuron network is something that is uh very much a part of active research for example the use of deep reinforcement learning to understand brain and behavior to really embrace the fact that this also happens in feedback not only with other parts of the neural circuit but with the environment and so a lot of really cool stuff going on in deep reinforcement learning both in terms of uh artificial intelligence development of computational tools with ties to Robotics and neurobiology and lots and lots of other different fields as well so here we are 2024 when I'm making this video I don't know what's going to happen next year it could be lots of different things but I did want to say that uh to me this is one of the really interesting things about trying to predict the future is just trying to look backwards and realize how unpredictable things are right so there was the rise of formal logic in neurobiology the the building of these artificial neuron networks the first ones over promise underd delivered and kind of didn't do much for you know and didn't do much people worked a lot but it wasn't really popular and really live up to that promise until 50 years later okay and 2012 is the first time when people started really talking about artificial neuron networks as a credible tool going forward and the the the the progress that we have seen in the decade since has been absolutely astonishing I wouldn't even pretend to venture a guess about what's happening next except to tell you that I'm incredibly optimistic that this is a technology where we have not even begun to see the end of what it could possibly do the scale of the system nowhere approaches the complexity of biological neuron networks we have so vastly simplified lots and lots of things that we know biological agents capable of doing in the way that we compute has not yet been implemented in artificial neural networks I'm not suggesting that we built a virtual fact simile of the brain not at all like I'm not saying just add in all of the known biology and things will be better in fact quite the opposite however I do have a lot of Hope in this technology and the fact that not only are we uh are we going to embrace it but just knowing that it didn't come out of nowhere that it actually came from this almost hundred-year history a scientific discovery and Technology development to me makes it grounded in the context of of engineering and technology in a way that gives me hope that we can proceed forward and uh and do what's coming up next