Titus Winters Talk: From Syntax to Taste

[00:00:02]All right, welcome everybody. This is uh a fantastic uh talk we're going to have today from Titus Winners. Uh my name is Bill Kernney. I am the chair of the ACM2Y. I'd like to talk about the ACM2i briefly for those of you that are not familiar with it and then we'll turn it over to Titus for a eagerly anticipated talk. So the ACM2Y is a organization.

[00:00:24]It's subgroup of the ACM for people that are interested in all things two-year college. Uh, community colleges, junior colleges in Europe, they're called level five or short cycle education. And, uh, you don't have to be like a community college instructor or anything like that to join it. Uh, just if you're interested in the issues surrounding two-year higher education, you're more than welcome to join. The link to join is acmy.acm.org or you can scan the QR code there. I'll leave it up for a second. And uh yeah, a bunch of our members are fouryear uh they teach at four-year institutions or just interested people. It is free to join and uh we do community events, we do workshops like the uh uh talk uh today. Uh we present at conferences and we advocate for all things tier education. uh sometimes uh a lot of the focus in higher education is on four years and so we like to um you know promote the uh the two-year side of things as well. So uh Titus Winters is a senior principal scientist at Adobe. He has served on C++ standards committee.

[00:01:36]Uh he chaired the working group for the design and evolution of the C++ standard library. He served on the ACMI E AI CS2023 steering committee helping set curriculum guidelines for computer science undergraduate educa education focusing on the requirements for software engineering. Bitis was a thought leader at Google for many years focusing on C++ software engineering practices technical debt and culture.

[00:02:00]He's the lead author for the book software engineering at Google which is available at a fine bookstore near you.

[00:02:07]So without further ado, I present to you all Titus winners.

[00:02:12]>> Thank you so much. Uh book is also freely available online uh just because the the goal is more so than anything else uh trying to get the ideas out there. Uh rising tide lifts all ships as it were. Uh all right, sharing my screen. Slides look good.

[00:02:33]>> All right. Uh I think this one will be a slightly short talk uh because I really wanted to leave a lot more time for Q&A.

[00:02:41]Um so we'll see how this goes.

[00:02:44]So surprise, I want to talk about type setting and layout.

[00:02:50]A good friend of mine, George Fairbanks, writes a regular column for it e about software architecture and design. And I used to teach classes with him trying to instill mid-career Googlers with some of the hard one lessons about testing and design. And it's been a few years since we were teaching together, but I still think a lot about one of his lessons about design architecture constraints and the efficiency of our tools. Now, as an Adobe engineer, the example resonates even harder. And the question is this, which would you rather produce a book in, Photoshop or Latte? And I'm sure that you could do it in Photoshop, but that sounds incredibly painful. Or similarly, what would you rather do use to do magazine layout? MS Paint or Adobe Illustrator? Same here. Technically, anything that lets you modify a pixel could do this type of work, but it is probably not the right level of abstraction or specification. As it turns out, we've learned many, many times over the years that more specialized tools provide a mechanism to express higher levels of intent, but they will simultaneously constrain what you are allowed to do. And a libertarian slant might look at that constraint and say, "Heck no, I don't want to deal with your handcuffs. I want full power to control absolutely everything."

[00:04:13]But for most of us, the more pragmatic at least, there are two intertwined and powerful lessons here. One is that constraints aren't a bad thing. Bad constraints are a bad thing. Constraints that you don't understand are frustrating, but well-designed constraints tend to be a powerful enabler and are actually the path towards efficiency. Consider our own experience as humans or computer scientists. Do you want to drive through a city with no traffic lights or stop signs? Do you want to operate a computer with no memory protection or separation between kernel and user mode? I don't think so.

[00:04:52]Progress in design across most domains is a matter of expressing higher level intent with more powerful abstractions.

[00:05:00]It is not a matter of controlling all of the fiddly details. It's a matter of saying what it is that you want done and then trusting the underlying system to implement what you have asked for but in a better way than you would have done it yourself. And that's not necessarily better in absolutely every dimension.

[00:05:19]We're usually talking about essentially talking about quality attributes here in a design vocabulary sense, right? We're looking at the illities. Are you more efficient, more secure, more maintainable, easier to edit and maintain the code? Or at minimum, we need to be talking about better economic incentives, faster to complete or cheaper to build. Historically, if you can give someone a higher level way to solve a problem that is better in some important dimension, that is progress and it will probably not go away. Again, we see this everywhere. Declarative build systems are a higher level abstraction. GitOps, Latte. Heck, I gave a talk a few years ago at CPPCON 2021 that was all about declaration of intent as the critical insight for configuration and design. And we see it absolutely everywhere from my kitchen faucet to thermostats and maybe even the popcorn button on your microwave.

[00:06:18]And we've seen the same thing with programming and programming languages over the past 50 years. In assembly, you can do whatever you want. In early C, we codified the differences between stack and heap. And soon we started requiring that you label the types in your function signatures, which allegedly, although it was before my time, fur uh infuriated some early programmers of C because of how they were being constrained.

[00:06:45]Java forces bounds checking. Rust forces borrow checking. We get these higher and higher levels of abstraction and intent and some previous patterns might become hard or impossible but it becomes easier to express the higher level pattern.

[00:07:00]Going back to Rust as an example, it is truly difficult n impossible to write a doubly linked list because of the way that the borrow checker operates. But in exchange for giving up doubly linked lists, which we shouldn't care about anyway, we're also given powerful guarantees about aliasing, thread safety, object lifetimes, and optimization. All of the above.

[00:07:23]This, I would argue, is a pattern of invention and abstraction that was started by Grace Hopper in the 50s. But this newest development, AI, has some novel differences. So, I'm going to talk about AI and specifically the use of LLM based tools in programming and software engineering, focusing on what skills are now required and how I think that that influences CS or software engineering education.

[00:07:48]First up, let's motivate the attention.

[00:07:51]Why is do we have all of the fuss around these tools? It's just a new tool. How can I possibly merit this buzz? Here is an entirely playable Sudoku solver entirely in HTML, CSS, and JavaScript based on those 54 words of prompt. And it generates about 700 lines in a single file. The color and styling might not be to your liking and it doesn't allow for pencil guesses, but it is a pretty serable Sudoku player. That said, Sudoku app isn't exactly rare in the training data of, you know, LLMs have read the entire internet. I didn't have to describe Sudoku at all. It understands the rules and a lot of the ideas in Sudoku apps are pretty common and well understood these days. Those are partially although not fully represented in this.

[00:08:42]A few years ago, I stopped kind of playing Sudoku myself and I uh my default logic puzzle is nonograms. In nonograms, uh, you have rows and columns of a grid where every grid cell is either filled or empty in the final puzzle and starts out as kind of an in between unknown state. And through deduction, you ought to be able to use only those run length constraints of uh each row and column has one or more runs of filled cells and you should be able to use those run length constraints to fully determine the grid state. So in this example that's supposed to be a two color pixel or twocolor pixel art image of olives in the context of AI. The fact that I have a lot of experience playing nonogs but it's not popular as sudoku has nice benefit for me because it's not as well represented in the training data and there's more substantially more variation in UI quality. It feels like a better test case for me to play with.

[00:09:41]But this was 172 words, 15 minutes of LLM processing, and a few dollars worth of tokens that generated me a very reasonable result. Is it high quality?

[00:09:55]Nah, it's okay. Is it buggy? Yes. Are those bugs my own fault? Absolutely.

[00:10:03]For example, in some cases, the model will get what you meant. Like I didn't explicitly say how many rows or columns are in the input file format. And I didn't say that calls is short for columns. It just intuited what I meant because it was a reasonable shorthand.

[00:10:19]But at the same time, I said once the constraints for a row are met, fill any remaining unknown cells with empty. And it didn't generalize that to row or column. And so in my particular implementation here, if you meet the constraints for a column, the behavior is entirely different than meeting the constraints for a row. And that's on me and my poor specification when I kicked this off. Now, in all of these cases, I feel a little bit of dissonance. It feels like magic that any of this works, and it goes against the instincts that I've built up around computing for the last 35 years. Remember the peanut butter sandwich? like ask, you know, give directions for how do I actually make a peanut butter sandwich? Yeah, that just works now. And it's magic that you can do the equivalent of make me a sandwich and it will fill in the steps.

[00:11:10]But on the other hand, as magical as it is, your sandwich is going to usually be kind of mid. The code being generated is nothing to write home about. And if you don't put in the effort to specify exactly the results that you care about and find a mechanism to give automated feedback to the machine about correctness and quality, then the results are going to be a bit basic and maybe and likely a bit buggy.

[00:11:38]But we certainly can't argue about the efficiency of this style of development.

[00:11:43]There is absolutely no world where even the best programmer available can produce a web game of this quality or complexity in 15 minutes with $5 or even $50 worth of extra expenses.

[00:11:55]LLM based development with no further training and a oneoff prompt gets you good not great quality throughout but we can refine that with additional time and energy and cost scaffolding harnessing and and context. The important point is that your expertise in the problem area and in the process of building software itself still pays off substantially.

[00:12:23]And it's also important to remember these tools are still getting better very quickly. Some of you might have seen the talk I did at SIGN back in February. I've been using Alphabet Page as a litmus test for AI image generation. In the last few months, that has gotten much better. Uh, here's an image that was generated by Gemini and Ano Banana this spring. It got a little panicked late in the alphabet when it needed to pack in a pattern breaking number of letters. So, in this instance, S is a little squished. T is kind of weirdly formatted and U is stretched for some unknown reason, but generally it's is pretty good. I mean, still not perfect, but we have all the letters now and they're all actually letters. as opposed to versions of this same exercise from last spring and summer, which oof, if you haven't seen this particular image, like I fully believe you can pause this talk and spend 10 minutes looking at this image and not find all of the things that are wrong and buggy about this. But this demonstrates how rapidly all of this is improving and showing why the tools are by themselves pretty exciting. So I want to pivot a little bit and look at two major streams of thought in the industry discourse.

[00:13:40]One is that the prompt, the spec, the plan is the only thing that matters and that we're all going to stop looking at the code soon. A few months ago, I thought that that was unlikely. And I would use as an example uh Anthropic's C compiler stunt which they generated a C compiler in just two weeks mostly handoff or hands off and generated that entirely with a swarm of agents. No human really looked at the code and that is a pretty flashy headline and the blog post is really worth looking at. But when you dig into the details, you start seeing things like it's a compiler, but it doesn't have an assembler or linker, and it generates worse code than unoptimized GCC. The implementation of the compiler in Rust is nowhere near expert quality, etc. And the blog post details a lot of the interactions and framing that were needed in order to get it even to that level of quality. it would not have worked at all without lifting the unit tests wholesale from existing compilers and that this is the most specified and most well understood systems component in history but importantly and I think that this is really loadbearing a lot of the limited human involvement here is from an AI researcher not someone with any particular expertise in how compilers work this tells me at least that that the domain veins where output quality is critical, it's probably not just a single prompt and it's probably not entirely without human involvement.

[00:15:15]But on the other hand, a majority I've seen a majority of my trusted colleagues, especially senior engineers that are moving to a prompt first approach primarily not focusing directly on the code or sometimes looking at it at all. And I think that the difference there is how much expertise they bring to understanding both the problem area and software process itself because I don't think that they're wrong. The prompt is not enough. We need that human taste, that expert judgment, experience in the problem domains that we are trying to operate in. And to the greatest extent possible, we need to scaffold and automate and constrain the behavior of the bots. This is discussions about harnesses or scaffolding or AI factories.

[00:16:04]We are moving towards a world where the AI is going to do the coding. But one simple prompt is not nearly enough. We will probably spend hours of time generating examples and test cases and specification constraints uh in order to trigger all of these things at a sufficient level of rigor to get a good result. The basic prompt by itself without all of that additional complexity will probably get you functional correctness because AI can essentially write code in the same way that we have been training students for years. Meet the specification, pass functional tests, do the meat grinder autograder sort of work. But quality is too hard to define, to specify or to prompt for. I think it is demonstrabably clear that for the last six nine months AI can program reasonably but it takes substantially more than that to make the results useful and fit for purpose especially in a professional capacity.

[00:17:04]We still need expertise especially in design architecture verification validation. Some of that will be humans, some of that will be machines, but no part of that is humans are going to look or operate on every line of code themselves.

[00:17:20]What we increasingly see that is it is not those simple prompts that get the job done. We need detailed plans. We need an architectural breakdown, a test strategy, review of our dependencies, specification of the dependencies. We need implementation details for the tricky bits. We need sample test cases.

[00:17:38]If you have enough software knowledge and problem domain expertise to scaffold the problem properly, it is increasingly possible to get very good results on pretty substantial software components.

[00:17:51]current industry benchmarks and leading uh efforts in AI native development say you can probably have a swarm of agents that are running for 8 n 10 hours with little or no human intervention whatsoever producing substantial very large complex software components but only if you can do that planning upfront your communication skills and your expertise in software and the problem domain at hand are are up to snuff. If you don't know enough to plan upfront, that's also fine. But your results are going to be exploratory, right? You need to try things out, work in a more iterative interactive fashion.

[00:18:32]And in both cases, the trick is that we need to bring to bear a sufficient amount over eventually of three things that is your ability to specify the design constraints, the architecture, etc. All of the inputs, what do we need to build? How do we know that we are building a reasonable thing? And then secondly, we need the ability to specify test cases, testing strategy, functional and quality requirements. It is not enough to say what does it take for this to compute the correct things. We also need to specify requirements about latency and maintainability and security and all of these sorts of things and find mechanisms to test and enforce those. And finally, I think maybe most interestingly and importantly and surprisingly, we need a reasonable and growing amount of understanding in the specific problem domain that we are trying to operate in. Now, we may get to the point that the best tools can generate highquality code and perfectly match a specification without spending extra hours in the planning and prompting. I do not think that we're there, especially with unscaffolded AI.

[00:19:42]No harness, no factory. The models themselves have encyclopedic knowledge in the same way that Google search has, but when you ask them to do some work, they are still pretty lazy if you're not rigorous about telling them exactly what you're looking for. Sometimes that rigor is problem specific. But a lot of the time it's not. It's just standardizing skills, tools, context, and background knowledge, which is going to be a common investment for the software industry starting this year. There are a lot of companies that are talking about AI factory work these days. to highlight the gap on unscaffolded AI. Here's an example. A while back, I tried implement a JPEG parser in C++. And while it is incredibly impressive that it gets close to something useful, it's still not good without some additional assistance. Like so many examples, it only gets us part of the way there. From that initial prompt, the first time that I fed it a JPEG to decode, it can clearly parse out the basic structure. Like I can tell that that's a photo of Sam Alman, but there are a lot of artifacts and glitches. This is way beyond what you expect from just a lossy compression format.

[00:20:51]I'm reminded in all of this a little bit of the Galman amnesia effect, an idea from journalism. Whenever you read an article in the paper, you think, "Oh, neat. That's great reporting. I learned something." And then you read the one article about something that you have firsthand knowledge of, and then you're just annoyed and angry. They got all the details wrong. That's missing the point entirely.

[00:21:11]I feel like the AI era is entirely full of that sort of feeling. All of this stuff is convincible. It's believable, but if you don't put in a ton of extra effort to constrain it and guide the it towards quality, it's never quite right.

[00:21:27]And the same is true with the code.

[00:21:29]Here's the meat of the interface that it generated for this JPEG example. It's workable. It's believable, but it's kind of weird. like who is parse result for?

[00:21:40]What use case is there for passing back huff table or quant table? I'm pretty sure those are internal decoder state.

[00:21:48]So this uh strruct feels like a mix of internal state and user visible results.

[00:21:53]And having an image object with the comment only valid if decode was called and a boolean to track that. That should be an optional. Everywhere you look at this, there's kind of a fractal level of that's kind of weird. That's a little wrong. The result is functionally correctish. It gets you 80 90% of the way there, but you have to put in a bunch more effort and find ways to force it if you want something more highquality and more useful than that.

[00:22:22]And that is a novel skill that I do not think that we have been training students in. When I talk about quality in all of this, I mean this in the abstract way that it's used in the design literature. What quality attributes do we really care about? Most of the training and most of our specification is about functionality, not quality. Quality is harder to define. At SIG last winter, I was talking about efficiency as a quality proxy, mostly because it's easy to measure and subjective, and I didn't feel like the audience would fight me on it. But in other domains, like image parsing, security probably matters more as a critical quality attribute. So, let's look at that. We look at my new JPEG parser and try to invoke it on malformed inputs. and the results are just as bad as you would expect. It's incredibly easy to crash the code that was generated by that single prompt. But if you got tokens to burn and you know what you're doing, you know the magic words and the concepts, it does not take any real effort to hook up coveragebased fuzz testing on the outputs to wind up with a robust parser. You gather a couple dozen sample inputs. You define rough match tolerances to around allow for rounding differences in your image compression results. And you give it a test suite that enforces that the decoder images match your reference decoding within those tolerances. And with the fuzzing included in that, getting to the point that my custom JPEG parser produced good outputs, had good tests, and had an interface that looked like it was written by a grown-up surviving against coverageguided fuzzing. All of that took me a grand total of four hours. And most of that time I was in meetings doing other things. Which is to say with a moderate amount of background in image handling, a pretty substantial amount of knowledge of C++, some understanding of modern testing approaches, and awareness of which tools exist, the ability to say the words coverageguided fuzzer, one can write a reasonably secure JPEG parser in the course of a couple hours without ever writing any code directly other than maybe giving it some inputs about the API surface. Without AI, we would be talking days or weeks best case, and that's with 30 plus years of programming history under my belt.

[00:24:39]GraceHopper's vision three generations ago was that we could get to English specification of computer behavior. And so far, all of the levels of abstraction that we've used to build towards that vision have been fairly incremental, and they've all had a similar flavor. We operate in the next higher level of abstraction with a more general declaration of intent coupled with the trust that the provider of that abstraction can both immediately and over time as things evolve accurately and historically deterministically translate that action into intent. The accurate and deterministic parts of that have usually meant that we build one pretty well understood layer at a time.

[00:25:24]But that trick of specifying the intent has always been critical. Even when you get a better abstraction, you have to know exactly what you're trying to express. And you have to be able to step back from controlling all the details, which requires a measure of trust, which may or may not be entirely wellplaced today until we get better scaffolds and safeguards and AI factory ideas in place. But that's progressing very quickly. We've got months or maybe a year or two tops before that is kind of a given.

[00:25:55]I also want to point out that we don't care that much about determinism. Like the argument that this isn't just another higher level specification of intent doesn't entirely satisfy me because let's be honest, we rely on optimizing compilers all the time. And that's a transform that most of us cannot reason about mapping intent to behavior without necessarily giving the same outputs from version to version and run to run. Increasingly I see AI that way. When it's well scaffolded we can give it the right feedback signals on quality and correctness. It is very much like an optimizing compiler. It maps from declaration of intent to behavior non-deterministically.

[00:26:38]Doesn't matter if it generates the same code so long as the generated code matches that intent matches those requirements for both functionality and quality. Continuing with that thread of abstraction and intent. Think back to Latte. If you're trying to get it to look your way, you're going to have a horrible fight on your hand. But if you're willing to take your hands off the wheel and trust the ancient rules on type setting that were handed down, yay, on those stone tablets, it will be a lot easier on everyone. That's a feeling that we're going to have to become more and more comfortable with as we move into AI assisted programming.

[00:27:16]Generative AI coding has a lot of that flavor. The abstractions are not as well defined and the trust in quality is not 100% there yet. It's got just as much squishiness and nuance as English. And there is just no list of terms or keywords that are necessarily going to be understood. I didn't have to give it rules for JPEGs or Sudoku or nonograms.

[00:27:39]But on the flip side, homegrown formats or new puzzles would require a lot more very precise explanation and specification. An engineer or a student that's equipped with these tools is limited mostly in their ability to communicate clearly. Your writing skills are going to matter. It's limited in the ideas that they can bring to bear, their creativity. It's the vocabulary that they know, the concepts that they are willing to bring to bear both in terms of internal architecture design specification as well as which ideas and concepts and tools and technology are they willing to use or uh in order to verify the resulting behavior.

[00:28:20]Coding used to be the bottleneck or at least we think so. I'm not 100% sure that was true, but that is demonstrabably not the case today.

[00:28:30]Nobody believes that. The emerging critical skills include the ability to design, architect, specify, prompt. Do you have the communication skills to operate at the right level of detail and specificity? Do you have the experience to guess whether the AI is going to get you? Have you put in the time and effort and research to understand the problem that you are trying to solve? In a very Fred book, Brooks, be prepared to throw the first one away sense. We need to learn the true nature of the problem in order to specify the results. And I think that that by itself is going to lead us towards a little bit more of iterative work with the AIS.

[00:29:10]Generative AI tools also have a huge amount of variability in whether they actually get the job done at a level of quality that is acceptable and fit for purpose. And I mean that in terms of the underlying code quality, which matters some. I've seen some very interesting talks. Uh there's one recently by Adam Tornhill of Codescene.org codcene.com uh that is studying what is the additional token consumption as a function of how messy and bad your code is. Uh and that is a fairly compelling argument and number. Uh he also studies what is the likelihood of AI introducing additional defects based on how messy your code is. And that's just as bad as you might expect as well. So, we're not saying that code quality doesn't matter.

[00:29:56]We're just saying it's a little bit more complicated than it used to be. Uh, we definitely need to look at the generated interfaces here when we're producing infrastructure components or anything that might be reused or interfaced with other projects. And we want to look at the final quality of the output product.

[00:30:17]Knowing the ideas, tools, tricks, concepts, vocabulary that a professional might bring to bear to test a new component is now a loadbearing difference between getting something that kind of looks right versus something that actually works. It may be the case that the AI factory approaches in the next couple of years make that less critical, but for the time being, those skills matter a lot. If you can listen to someone describe the types of problems that they are facing with software quality and suggest tools or process or approach that would be useful in helping them, then you might be a successful engineer. Without this sort of structure, LLMs and generative AI are an awkward form of magic. Think of it as a machine that gets 80% accurate results for basically anything that you ask it to do. But it's incredibly lazy and it's only going to try the approach that would be used by the average engineer on the internet. After all, it has trained on us, right? It's the 80s drug PSA. You all right? I learned my laziness by watching you. For example, my oneshot JPEG parser didn't come with any unit tests out of the box, nor did the tool ask me to go gather some images to use as test cases. Right now, I think the trick is to give students taste and the right level of skepticism because just because it looks like it's doing the right thing does not mean it actually is. give them the tools and vocabulary that's necessary to pin down all of those sort of squishy behaviors and let the machine iterate that default 80 or 90% accuracy towards something that is actually going to be acceptable with its own feedback. And I mentioned just there taste we need to figure out how to train up taste quickly. there was a really wonderful talk by a engineer a developer advocate at IBM uh Tahas uh I'm blanking on his last name uh but uh he's been speaking on the research literature on taste and one of the points that he makes is if you are trying in traditional disciplines to teach taste somales or the people that identify counterfeit currency you do that by showing them infinite examples of good results and then slowly potentially introducing bads so that they have something to compare against. But what I want us to all be thinking about is how much of the time, if any, have we been showing our students and letting them luxuriate in genuinely good code larger than what fits on a slide. I think we're going to have to re-evaluate that in particular.

[00:32:56]I've been saying for a year or so now, design education and testing education are AI education. Everything that goes into specifying what needs to be done and everything that could be done to validate the outputs, those are what we have to focus on. Now, there is no value in focusing on syntax or how to implement a data structure. It may be the case that we are all accelerating towards the singularity in AGI and that we'll have perfect production quality software at the drop of a hat with no need to provide any architectural details, test strategy, test cases. And if that is the case, we got nothing to do educationally, right? We're shuffling deck chairs on the Titanic. I don't think that is actually the case. And I think that the wise choice here is to assume that there is still a need for expertise and for education. But we have to pivot hard. There is no value at all in thinking about or implementing linked lists. The skills going forward are about substantially higher level communication of intent and architecture and design and about learning all of the various ways to validate some iffy results to ensure that the functionality and quality are what we actually need.

[00:34:10]Even if this metaphorical version of Latte isn't drawing from ancient rules or reproducible precision, it is still time for us to step back and focus on that higher level intent because continuing to teach MS Paint isn't going to get the book written. Thanks.

[00:34:34]All right. Thank you so much for that.

[00:34:37]If anybody wants to put uh the questions in the chat, um we've got uh 24 minutes for for questions for Titus.

[00:34:55]>> I'll just start with an obvious one. Uh what what tools do you like to use right now?

[00:34:59]Uh, everyone around me is using Claude and I'm kind of boggled at how fast that has gone from a I will use it once in a while if I'm sitting down to do a coding project, which is not my primary job most of the time. Like that kind of past that point in my career. Uh, to now like I use it on a daily basis for literally everything. Like if it's a thing that might be slightly annoying, I will spin up claw and ask it to help. Like I came out of 13 years at Google. We did not use Git. We had better version control systems. And I will say Adobe's uh authentication systems and interactions with GitHub are sometimes a little janky.

[00:35:47]>> Uh it is so much easier for me to just be like, "Hey Claude, go check that out." like and it'll like fall back through several different like oh that didn't authenticate oh that did oh this worked now okay great like great I don't need to spin up any of that knowledge we're just done so I use it a lot for that sort of automation of tasks that I don't really want to deal with I use it a lot for uh helping me think initially but not usually for generating outputs uh unless it's a report or a proposal or something that like is pretty run-of-the-mill and doesn't need to sparkle at all. Um, I try very hard to make sure that everything that I produce that matters still has my own voice and uh like yeah, it's um it is a tempting nuisance in a lot of ways when you uh accept any outputs blindly.

[00:36:51]So, uh, we have a question from Nibil Ner. Um, how do you think student learning is going to change for computer systems courses like the ones that I teach?

[00:37:01]Thank you.

[00:37:04]Um I think that systems courses like most things uh we have to assume that a whole lot more can get done and that we need to focus a lot more on highlevel concepts and I would say ideally group work. Um, one of my big pet peeves right now is so much, and I think this is probably slightly less true in systems, but so much of our assignment load is autograder meat grinder sort of exercises because of scale and because of all of the, you know, entirely reasonable that how we got here. But um it does have the nasty unfortunate side effect of suggesting to students that there is a correct answer to most development tasks. And I will just flat out say like once you have a job there is no right answer. there's trade-offs and constraints and in many cases we just haven't encountered what the correct answer is to start with and giving them the false impression that there's a right answer uh is not doing them a whole lot of favors. Um I think the AI really exacerbates that and gives us also the opportunity at the same time for okay let's actually listen to what we've been saying for decades like this is a craft focus on the craft give them open-ended things have them critique their own work their each other's work both the implementation as well as the final you know out uh you know user look userfacing product of it, right? Oh, that was cool, right? Oh, that's a clever way to have done that. Like, focus on those types of things and not just, you know, something easy meat grinder. Um, so, >> uh, Roses, he's actually one of my students, uh, jumped in on the the talk here. Um, I would like to know what what you think about Lennis Stroll's comparison between AI and compilers and what you think about Jensen Wong's statement that he'd be terrified if an engineer did not spend hundreds of thousands of dollars on tokens.

[00:39:31]>> Uh, I mean, I think Jensen Fuang has uh some incentives that we need to look at. Uh, so, you know, take that one with a huge grain of salt. Um, I think that all of the discussion around token maxing and stuff is entirely insane.

[00:39:49]Like, like that is the literal opposite of efficiency.

[00:39:54]And like I was reading a pretty good uh essay today about the mass psychosis of the industry and the people who are like, I can do this job quite well with moderate AI assistance, but my management chain wants me to use a ton of tokens, so let me just burn a bajillion tokens on stuff that looks kind of plausible and then just do my job and pretend that AI did it for me.

[00:40:23]Um, this will pass. I think as soon as the bubble bursts and we start paying a actual fair market price for those tokens, uh, there will be a substantial reset. I also think that as we talk about AI factory stuff, there's some really fascinating ideas that are floating around here. Um, I'm really big on the idea of if you use tokens as a proxy for human effort and cognitive load, suddenly everything that we've ever thought about best practice in software can be empirically verified.

[00:41:03]>> You could build an eval suite of software tasks and vary, you know, prompt, context, model, all, you know, whatever. So long as you have quality and functional uh acceptance criteria identified and if you believe like I do for instance that a more robust nuanced strategy around unit testing is efficient and best practice. You had better be able to do an AB test of for this set of software tasks when I do it with and without that additional prompting scaffolding, you know, behavioral modification in place. If I save tokens, that is indeed best practice. Congratulations. And if not, you probably just wasted everyone's time for 20 years, right? And the idea that we can like software engineering never used to be empirical or experimental like because who would spend you know 10 or 20 software teams building the same component in order to run reproducible experiments and get numeric results out of it and no two people and no two pieces of software are really the same.

[00:42:22]So like those are pretty sketchy comparisons to make in the first place.

[00:42:27]That's not the case anymore, right? Like very soon I anticipate people are going to start running mass experiments. Uh the Adam Tornhill example that I was giving earlier like he's effectively measuring token consumption based on did you clean up after yourself? Um, and so going back to Jensen and and Lenus, like, yeah, I think that there's a lot that can be done by just hitting it with the hammer if you have infinite token budget, but I also think that there's a lot of engineers that I see that are getting incredibly good results with substantially more modest token cost uh, just by being able to specify and prompt and harness the things nicely. ly and knowing what they're trying to do in the first place. If you don't know anything about what you're trying to do and you just set up these complicated agentic loops that are like, you know, urabos snakes eating their own tail, like yeah, you can burn through hundreds of thousands of dollars of tokens in no time at all. Uh I would recommend that we don't. And you know, because of global warming, I would plead that we don't. Um, to be completely honest and blunt, like I desperately wish all of this would just go away, but it's not going to. So, the best we can do is try to be efficient and use this with some attention to detail and quality. So, uh, Nadim said, "How to navigate equity of access to LMS for students? If an institution doesn't provide, i.e. pay for claude for example to students. It is hard to make sure that everybody has the same platform to work with as we instructors try to rework our curriculum.

[00:44:10]>> Yep. And this is going to be an outstanding long-term concern. Um I think that at some level we are unused to the costs that are associated with you know consumables in our domain.

[00:44:25]um chem labs and you know physics experiments might be a you know source of comparison although I strongly suspect that most chem lab expenses are dramatically less than what we are potentially looking at here. Um I I don't have a great answer to this yet.

[00:44:48]uh I'm hopeful that the big AI labs will continue down this path with you know reasonable funding for students and academic use. The flip side of that is we certainly don't need to be using the most expensive frontier model tokens in most cases and if you've got onrem compute available or you have you know a nice allocation from AWS because their large s has not stopped yet the open source models are pretty good like they tend to be somewhere between six and nine months behind the the frontier but you know it's been seven months since uh we crossed the watershed where even the skeptics like me had to admit like, "Yep, that's starting to work." So, I anticipate that the open source models that are coming out through the rest of the year are probably plenty good to do anything that a student would need. And so, it stops being about AI tokens and becomes do you have some GPUs and some compute available to to run those things for local inference. um that's going to take some infrastructure to keep that up and running and you know it's not free but it it might be substantially cheaper.

[00:46:03]>> Uh Marcus Gesler from uh my one of my good friends it's an amazing guy he says many CS instructors argue that if we don't teach old skills how should students be able to design and test systems which at the end of the day produce old style code which needs to be evaluated and tested. What would you tell those instructors?

[00:46:24]>> Uh I do not believe that we are going to read every line of generated code period. Like I think we are going to focus on the things that matter the most. Uh, I was reading earlier today a preprint from Meta about a system that they're using for evaluating whether a given change is actually risky or whether or not it's straightforward enough and understandable enough that they can just have AI review it. And their results are a wild reduction in code review time obviously uh but a uh it's 150th of the roll back rate and onethird of the production outage like problem rate of the things that they judged to be risky. Uh, and that that by itself tells us plenty, right? Like we are not in a world where we're going to look at every line of generated code, period. It's just it's just not going to happen going forward.

[00:47:25]Um to the rest of the instructor skepticism, I think that we have known at some level for 20 or 30 years that we spend a phenomenal amount of time on reading code and almost no time on understanding anything beyond the like CS1 level. And if this was literature or any other sort of creative discipline, like you don't get a lit degree by writing great English literature. You get a lit degree by studying English literature and writing a little bit. Right? We need to substantially focus on how do you build taste? How do you instill students with a spidey sense of wait that doesn't feel right? Uh, one of my favorite tricks instead of reviewing all of the code is to ask Claude, give me a quiz on what we have built together.

[00:48:25]>> Uh, focus on the highlevel like design, the architecture like where the data is, data flow sorts of things. I don't need line by line programming language quiz but I do need architecture quiz and that does two really important things for me. One is uh it reinforces like do I understand what we have done enough that I can stand by it because I'm it's still going to be my name on this poll request. I'm still responsible for whatever happens. But two, very often, like in most cases, during that quick quiz, something comes up where I'm like, "Oo, that that doesn't smell right." Like, "Spidey sense tingling." When we're done with this quiz, come back and we're going to re-evaluate that set of design decisions and then like fix up a thing that could have been a mess in the future. Um, the goal is not sit down and write every code, every line of code.

[00:49:23]The goal is not even understand every line of code. The goal is use human attention which is a rare and precious resource far more valuable than tokens.

[00:49:32]Use the human attention where it matters most.

[00:49:36]And the way to use that effectively is to build up taste. And it is an open question how fast we can power level people through how do you build taste when it is no longer a matter of we're going to sit down and make you hack on reversing a linked list. Right.

[00:49:58]>> Thank you. Uh Katie Cunningham says, "I'm curious about the case of nonCS majors. I'm intrigued by the idea of vibe coding 101. I can see different versions, maybe one of which is helping non-technical majors get to a place where they can share a prototype of a business idea or run a data science pipeline.

[00:50:15]>> Yeah.

[00:50:16]>> Thoughts on the most impactful background to provide to nonCS majors?

[00:50:21]>> Yeah, I think that we're going to have to try a bunch of things and I really really agree. Like I really like the idea of CS0 as a like I can build a prototype. I can build a little app for my own use cases. Like I can adjust the computing environment, my personal computing environment. Like that I think is an incredible democratization and and incredibly powerful and a thing that like we should be actively looking forward to. I don't want the rest of the world to be exposed to that vibecoded slop. Uh but a problem for solving your own problem is great, right? Um, so like we'll need to lean into that distinction a little bit. But, um, as a guess, as a thing that is intrigues me, I would love to see a CS0 that focused on intro to programs like what is the shape? What is the UX? What is the design? Like what are the design affordance? Like who has been explained the inferred like implicit semantic differences between a button, a menu item, a checkbox, a radio button, right?

[00:51:35]Like uh when do we use command line flags versus positional parameters, right? Like let's brain dump into noviceses like what the standard shape of programs looks like.

[00:51:53]uh so that they're empowered to just try to build whatever right for their own purposes uh because I think that you know sets them off on the right foot gives them some vocabulary that they can you know prompt the machines and get the results that they might be uh might be well served by. Um, but honestly, like we don't have time to know the correct answer before we try things and we're not collectively well set up to do things that we can't prove ahead of time. So, we're in this nasty catch 22. So, I I beg I implore anybody that thinks that they can get away with an experiment, do the craziest thing that you can think of and report back because it's it's going to be complicated.

[00:52:45]Uh next up we have Erin Joy Basc uh teaching computational thinking. What would you say is the right level of expertise students should have before we can assume they have sufficient foundation to start using Genai support and what are your thoughts on how to develop those foundational skills while ensuring the students are not dependent on AI? I figured it might relate to the idea of developing taste and an understanding of design first, but I'd be grateful to hear your thoughts. Thank you.

[00:53:13]>> So, I suspect that if we rolled back time, there would be echoes of a question like that about what should we do about compilers and how do we in, you know, allow students to use this new fangled technology without losing the ability to operate without it.

[00:53:33]This is not going away. Like even when the bubble pops, we're not the genie is not going back in the bottle. Like even at the point where we're paying substantially more for this, that's going to stop the token maxing, but it's not going to stop that people just reach for this, you know, when it's a sort of day-to-day sort of programming task. As far as what background students need ahead of time, my like I'm very interested in the work that uh Leo Porter is working on. Like I'm easily convinced that somewhere between halfway through CS1 and maybe halfway through CS2 uh is probably the point where you stop just writing code by hand. Like what we need is for them to be able to read code and understand data flow and algorithmic thinking.

[00:54:29]And the faster that we can get them into understanding algorithmic thinking and the ability to read code coherently, uh there's there's no point in being not AI scaffolded beyond that point. Like it's >> Leo Porter actually just emailed me asking if I was going to record this talk. So, I'm sure you'll be happy to see his name just come up right there.

[00:54:52]>> Y having a discussion with him at CRA in a couple days.

[00:54:58]>> Uh Leo Porter's at UCSD for people that that don't. Um yeah. Uh Den says, "As great as a tool that AI is, do you feel it is important for new up andcoming programmers to keep up with the advancements within AI even if there may be a loss within human creativity and problem solving skills?"

[00:55:16]>> I don't know. Like I think I don't think that anyone needs to necessarily be worrying too much about how this stuff is implemented. And in most cases, I'm not too worried about the differences from one model to the next, especially at the level of the the problems that undergrads are going to engage in. like I suspect you get a good enough model. It's got plenty of intellect to do these sorts of tasks.

[00:55:51]Don't worry about it too much. I don't think that it is a reduction in human creativity and problem solving. Uh, I think it enables you to solve vastly more interesting and more complicated problems so long as your instruction has included enough guidance and scaffolding and vocabulary and concepts for you to understand like what is possible and how to keep things on the rails. Um, one of the examples that I'm kind of a little nervous about, uh, there's a very good school that I won't call out on tape, uh, that has historically done, uh, projects day at the end of their first year. And you know, your project for projects day is whatever you want.

[00:56:40]And last year they had to cancel projects day because so many students were almost done with no real notion of like whether they were actually almost done because they just kept reprompting and reprompting and it kind of sort of worked and they didn't have the software engineering skills to put in unit tests, pin down behaviors, you know, all of that. And then also they didn't have the general, you know, vocabulary and concepts when it came to design for things like, okay, I'm implementing a game. I'm almost done. The client all works. I just need to implement a server. I don't know what that is.

[00:57:18]Right? like the if we pivoted so hard away from syntax and for loops into what are all of the words, what are all of the concepts that actually matter in our domain? What are the things that we talk about that gives like those are the new keywords, right?

[00:57:40]If you know the magic words for server and client and you know singleton and visitor and database, schema, message stream, unit test, integration test, fuzz test, properties, right? Like you have to know those concepts and those words. And then soon as you know what those things are good for and have a flowchart of what's appropriate, you're unlimitedly powerful.

[00:58:11]Uh Logan asks, "So where do you believe this leaves CS students like us?" He's actually the president of the local ACM student chapter.

[00:58:20]>> Yep.

[00:58:20]>> Uh so where do you believe this leaves CS students like us who are in for all intents and purposes in a transition period in a new era of CS? Do you think that our education is lagging behind the neck break speed in which AI is advancing? And if so, what should we do to keep up? And what skills do you think will be the most important?

[00:58:40]>> I think much as before, the more that you can play and experience and get some time in on time on task on your belt, like the better off you will be. Um, a couple years out of date, but related example, like we talk about, you know, elite code interviews and, you know, hacker rank sort of problems.

[00:59:06]The industry does not care about those problems, right? Like you you are never going to be asked to reverse a doubly linked list. Like you are rarely going to be asked maybe once a decade implement a bread for search puzzle, right? Like it just it does not happen.

[00:59:22]The reason why we were focusing on in elite code questions for you especially junior interviews is because it was signal of those are students that practiced more and that practice is valuable because there's only so many hours in a traditional you know uh curriculum.

[00:59:43]practice is still going to be valuable even if it is currently the case that nobody your your your major curriculum might not be fully aligned with exactly what we need it to be going forward.

[00:59:56]Everyone's figuring it out. Uh that said, I think from the stats that I've seen, hiring is still substantially like biodal. like new grad and intern hiring is large partly because especially if you can get into those internships, we get substantially better signal on do we want you around uh than we would for any other, you know, hiring mode. And then the other mode, the other bump is the very senior people that have enough of a resume and reputation that that uh we we know that they know what they're doing.

[01:00:30]Um, so practice, find stuff to play with, tinker, like experiment, come up with, you know, the things that you like, tell your professors what lessons are interesting and important, share your results. Uh, because, uh, one of the scary things is everyone has been reset to the same level of knowledge and making it up as we go.

[01:00:56]uh you you you don't have to deal with it's just received wisdom from above anymore. Like go figure it out yourself and tell the rest of us please. I would love to be uh given more inputs to to help make sense of all this.

[01:01:12]>> Uh Fred Martin says, "I still don't see how we can develop the sense of taste, smell, spidey sense, all great metaphors without years in the trenches of debugging and building a deep understanding of how things actually work. I realize that we won't be writing linkless code ribbon reading in detail AI output. I'm trying to turn this into a question. What do you see as the progression from beginner to being effective in prompting scaffolding genai and having advanced taste?

[01:01:38]So, I mean that I I want I'm guessing that we want to spend a lot more time on open-ended projects, uh playful results, group work, uh uh mutual critique of both like implementation as well as out like final result. Uh, and I think that that's probably the best that we're going to do through the through the CS curriculum.

[01:02:09]Uh, I remains to be seen how much we can uplevel that taste, but I think like spending some time on ideas like all right, which of these is better and why?

[01:02:21]And have them, you know, try to enumerate the way like this is good. I this feels a little weird. like that's, you know, try to try to sus out and build up a sense of like can you recognize elegant? Can you recognize quality?

[01:02:40]Um, I don't know exactly how to do it, but I suspect that it's doable and we should be running some trials on that once it gets into uh the professional domain. I think I've seen several experts and thinkers all come to the same conclusion. So I'm relatively confident we're going to push a lot harder on pair programming.

[01:03:04]Um so for an interesting substantial piece of development we want the senior people to be working with the new grads and the junior folks and probably have the junior folks sitting in the hot seat driving uh so that as we are reading through the output from the LLM when the time comes like you can share in the experience of skimming through that getting a sense of like what's going on and like ah that that feels that feels sus like we need to dig into that more deeply.

[01:03:39]Historically, the arguments about pair programming have really boiled down to two things. One being a manager might not think that having two people sitting on it working together is worth the time. Uh given that we currently have no other mechanism that I'm aware of to build juniors into seniors, I think that that argument disappears. And the second being it's exhausting to be focused and paying that much attention for that extended period of time. But you're already exhausted having that much focus and attention when you're doing the high agentic like AI development anyway because you have too many tabs open doing too many things. So why just do that thing and that that's best current estimate for how we mentor juniors. So >> one last question if you have the time for it.

[01:04:32]>> Yes sure again says if I were to teach an upper division undergraduate course on agentic coding what industry ready skills missing from traditional four-year CS curricula should be prioritized >> testing code review. Like there's definitely a part of me that thinks CS and traditional programming just died and everything that matters in software engineering is all that's left. Um but yeah it's design architecture testing code review and especially the vocabulary and concepts therein like if you can talk intelligently about the connections between design by contract and particular unit tests approaches like cool like you you probably connected the right dots and are going to be well suited to to tackle things. So there was a ACM tech talk that should be on the ACM YouTube channels uh where I talked about uh design is testability several years ago before all the AI stuff popped off. Um that type of material is actually I think incredibly relevant right now.

[01:05:47]Well, that is the end of the questions and I would really just like to express uh the thanks of everybody here for your time and taking you know time out of your busy day to give this talk.

[01:05:58]>> Thank you so much for having me.

[01:06:00]>> Um yeah again just thank you. Thank you.

[01:06:03]Thank you. It was wonderful.

[01:06:04]>> Good luck.

[01:06:09]>> No it's an absolute pleasure and and I really do mean like nobody knows the answers on all of this. We are figuring it out. Take some big swings. So yeah, we'll we'll get there.

[01:06:22]>> Awesome.

[01:06:22]>> Thanks all.

[01:06:24]Catch >> you later. Cheers. Y