SecTor 2025 | How Secure is Your Base Image? A Live Security Test of Popular OSS Containers

本站添加: 2026-05-21

[00:00:02]I have an intimate room here with you folks, so excited to be here and try to make this somewhat interactive. The goal today is pretty straightforward. I'm I'm John, CTO and co-founder of Root. I spend most of my time thinking about software supply chain security, container image security, and and and basically open source software security. And in today's day and age, container security is some in some ways the the the the essence of software supply chain security. It's the place where stuff the developers build become software that runs in production, and inside of those container images embodied a whole bunch of open source software, which is under attack a lot and often today because supply chain attacks are on the rise. And and anyone who is interested in security like hopefully all of us are here at this show, I'm going to kind of help you bridge the the space between app sec developers and and making software that you run in production something that's inherently secure and show you how to look for the kinds of risks there are. So, my core question, how secure is your base image?

[00:01:12]I'll be looking at three very popular open source images that I see every day.

[00:01:16]Our our product our SaaS service processes thousands and thousands of containers a day for our customers, and we see the good, the bad, and the ugly.

[00:01:25]There I picked a few that I noticed were pretty ugly, but also possibly and and as I've seen it in in production and which is boggling my mind, but I'm going to show you how to look for the issues there beyond just looking for CVEs cuz that's just the kind of the surface of looking at these things, but to go deeper into the Docker files and understand the manifestation of these issues and and then what you can do about it. If you're security practitioner, often it's true that you you don't always have have have the ability or time to go digging into code.

[00:01:59]Well, I'm going to show you how to do that if you're interested and then in my in my presentation, you can download it later. There's all examples with, you know, command prompt stuff. You can go and run and see the see this for yourselves and and even try it out. So, um that's what we hope to do today. And let's let's get started. So, let's see if this thing works.

[00:02:21]Nope.

[00:02:22]It's okay.

[00:02:23]I'll I'll do it the old-fashioned way.

[00:02:26]So, why is this important now?

[00:02:27]First, you know, we live in the agentic era, right? And and the ability to write code, exploit code, build attacks, detect vulnerabilities has is is moving from human speed to agent speed. And of course, it's always been done programmatically, but today exploits arrive in hours or minutes and and typically defenders still work in in human processes, processes that are defined in in maybe days or certainly sometimes hours, but days and weeks, which means there's a massive lag between the processes we use to kind of fix things, find exploits, etc., remediate, and respond is not matching with the speed at which the bad guys can operate today to do exploits in open source to to exploit vulnerabilities in open source code. That process lag is is is getting faster and faster all the time. In fact, one of the CVEs that I'll highlight in this presentation is something on the CISA KEV list. This is a you know, security organization in the US. They have a list called the known exploitable vulnerabilities list, which basically marks vulnerabilities that are either critical or high. Um they call them something as as as as a KEV when they reach three criteria. One is they have a CVE allocated to them. Two, there's a known exploit in the wild with a proof of concept. And three, there's an active patch for that so that you can actually fix it, and they make you fix it as fast as possible. Very short time windows. Uh Canada has some similar things, but certainly more things are emerging where uh the kind of pressure will mount uh here as well, but in the US there's a lot of pressure to fix these.

[00:04:10]I'll show you an example of of one that is kind of a interesting example in that the exploit happens so fast that uh it it's way faster than anybody could patch, and therefore really doesn't um isn't something we can really respond to well, and I'll show you maybe how that was responded to. Anyway, these mismatched timelines, the higher uh compliance pressure, critical pressure, and the um and the need to respond to agent speed is causing a massive gap, and um and and we need a way to do something better about it. And and the way to do that is to be proactive. It's to not let attack surface make it to production, um and I'm going to try to help you learn how to do that here. Um so so one big problem that happens with container images is that of course developers build the software using base images or their own images, and developers aren't always security experts, right? And it's up to AppSec and maybe even development leadership to build a secure development life cycle that bridges the concerns of both sides. And I really think that, you know, in some ways container images are the intersection between developers' needs and security people's needs in that they're often embodying code that just shouldn't ever be sent to prod. And why is that true? Well, developers' needs vary, right? From from AppSec.

[00:05:27]They need to build code fast.

[00:05:30]They need to have the tools they have to they they need to make their apps run easily. They don't want to spend a lot of time fixing dependencies and such.

[00:05:37]They also want to make sure their software will run run successfully. And sadly, that's at odds with I want small images with a low surface area and really secure code in there. So, it's like when do those choices become made? And in my experience, I see tons and tons of what I would call developer images in production workloads. So, so so engineers want convenience over security generally because they need to ship fast and help the top line, right?

[00:06:08]They um they they also, I noticed, don't always upgrade or or maintain these images. And sometimes that's the job of a golden image team, but you know, not all organizations can afford to have such separation of duties where they have special teams for engineering base images, they have special teams for building software. Mostly, it converges to developers build and own their own containers. And in some ways, that means they're responsible for infrastructure later. And that's certainly containers are part of infrastructure.

[00:06:39]So, so what happens is container images age like milk, not like wine. The longer they're on the shelf, uh not being adapted, the more insecure they get. The passage of time does not equate to more security, as we all know.

[00:06:52]So, we really you know, the enterprise reality is we've seen the same patterns and the same images, these insecure bloated images winding up in production.

[00:07:01]And most compliance mandates drive towards CVE counting, not core security concerns, which means that you can have low CVEs, and I'll show you example of that, but still have a very insecure image. And you need to be able to know how to spot that. And your engineers need to know how to spot that. And really really drive the the knowledge of how to make secure software as containers something that's kind of an industry-wide, I mean, excuse me, an organizational-wide knowledge. So, we're going to see what's inside containers and and how we can understand, identify insecure patterns. And I picked three container images that I see all the time.

[00:07:36]Uh And in fact, I see these exact ones in production workloads, and that's why I pointed them out. One uh and they're all based on Debian, so that's that's the first thing you should know that uh the Debian is the most popular base image for production images that I see uh and and and you can check some other industry sources, they'll tell you the same. Um and you know, Debian has lots of flavors. You can get full Debian, effectively full Debian Linux embodiments, and you can get slim and secure versions.

[00:08:06]But sadly, a lot of the most popular images, at least if you go to say Docker Hub and say I want the latest, will come with full images. And I think that's really designed for convenience for developers, right? Um and and of course, they leave it up to to to the professionals, so to speak, to uh to know what to do to get from that developer-centric image to something more secure. And it can be done, but lots of people don't know, so we'll show you how.

[00:08:34]And uh and and and then Golang, Python, and of course, we're in the agent time, so I'll show you one that keeps cropping up, this MCP server. Anybody heard of MCP servers? You know what those are?

[00:08:44]Yeah, they they are the way that machines talk to or agents talk to tools. It's kind of a modern emerging design pattern. I see it more and more in uh in customer applications of mine. So, these three are a prominent um lots of the world of SaaS is built on uh on these languages and agents are talking more and more to NCP gateways.

[00:09:08]For the Golang, um we'll point out some really critical vulnerabilities there.

[00:09:12]We'll talk about um how the design of that and some of the key vulnerabilities are problematic and why.

[00:09:18]For Python, there are some interesting structural problems with that image, some some design choices. Um the way it's constructed has a lot of development residual development tools there, and um and and and it runs into this problem where Debian decides to not fix things, but not give you a path to how to for how to fix it. Meaning there are unpatched vulnerabilities all throughout the Debian ecosystem and a lot of deference to future releases. And I'll talk to you about how to get around that and what to look for. And then the MCP gateway is a is one that has this kind of very problematic core architecture to it. Several architectural patterns and software in there that make it a kind of an interesting target and make it hard to deal with. So and so we'll go through each of these.

[00:10:05]You'll see here in this first part in the slides take the same pattern. I've got these commands which you'll be able to copy if you get the slides and and get the results that I have. It just uses open source stuff so you need Docker.

[00:10:19]You need to be able to pull things from Docker Hub and then you can run a tool called Trivy.

[00:10:26]And then for for all of these in the next slide I show a snippet from the Dockerfile. Dockerfiles are basically the programming language for the Docker build system and and they let you take these Dockerfiles which are basically instructions and turn it into a working image. But almost every Docker container you find in public sources has a GitHub repo that goes along with it where you can pull the Dockerfile and that's where I found most of these. You can just search for it. You have the tag by the way. If you look at the top of this screen you'll see Docker pull.

[00:11:00]That's the tag and that will come straight from Docker Hub for you guys to download. And then you can also look right on that the home page in in Docker Hub and you'll be able to see where to get all the source code that I use. So this first one Golang has a lot of CVEs, right? That's like the first thing that jumps out with you to you if you run a scanner. And and and a thousand is a lot.

[00:11:22]If you're in a working for a company that is is under any of say say PCI DSS or HIPAA or in the US FedRAMP or or or any of these compliance regimes where you have an SLA for how how fast you have to fix CVEs, this would be really ugly. You have some criticals and highs.

[00:11:42]Criticals usually are demanding you need to fix them within 7 days.

[00:11:47]Um that's the typically the SLA. Uh usually something like 15 days for or 14 days for highs and 30 days for um for for mediums. That's fast, right? And and if you've got 100 of these containers running, you've got 100,000 vulnerabilities. Um and lots of these kinds of containers have lots and lots of vulnerabilities. So that's that's alarming. And and it's got this one particular CVE uh 2025-48384, which is come something called a Sissa Kev. And um and and this is a particularly interesting one. This Sissa Kev, it was um it was detected in around July.

[00:12:28]Um within days, uh it was it was registered as a CVE. Um and in within days, there was an active exploit. And and by the way, the the average time today between uh a CVE uh registered with the with the NVD and and an active exploit for that CVE for criticals and highs is on the order of 7 to to 10 days. That's about how fast it happens now. And and proof of concepts happen very quickly as well, which means from proof of concept to exploit, it's it's really pretty quick.

[00:13:01]And so um but the average patching time is is much longer than that. So in this particular case, this was exploited within days of it being of of the CVE being um uh noticed. And and and there was no fix even today within Debian for this because it falls under this thing I mentioned as no DFA, which means it's a deferred fix.

[00:13:22]Uh this is in Bookworm, which is one of the long-term release versions of Debian.

[00:13:27]But long-term release uh with Debian, one of the things they do is often say we're not going to fix it in Debian, we're going to put a fix in Trixie, which is their next version of the release. And there's lots of these. And so you're stuck if you're running on on Debian Bookworm with a vulnerability that you can only fix by upgrading to the latest version of the operating system, which isn't that stable yet. At least I wouldn't use it in production applications.

[00:13:50]But it is one of it is their latest official release. And and it can be it can be fixed there. There are backporting things you can do, opportunities. In fact, there are vendors that do that including Root that make backported fixes for these kinds of things. But that's one that that requires a significant amount of consideration. And if you were you were subject to sysdig cav, you'd be scrambling to get that done.

[00:14:15]Can show you some methods again later about how you'd fix these kinds of things. But that's a certainly a timeline pitch for a pitch for anyone who's who's using this image.

[00:14:23]This is the Dockerfile. You'll see these kinds of syntax like from for instance from line is something called the base image. It's where you get all of the base operating system bits and even tools that you build an image from. Images have this hierarchical structure. It's kind of a derived from One thing you can learn very quickly here with this Dockerfile is this is a builder image. This image is meant to be used to build images or build things that you put in other images. And builder patterns typically have a lot of tools in them.

[00:14:57]That's one you'll see here. As a matter of fact, I know that a significant amount of the vulnerabilities that come along for the ride with this with this Docker image come from the builder tools that live inside there. And and and and you see like GCC, G++, libC6-dev.

[00:15:14]These patterns, these are all things that aren't needed for runtime.

[00:15:17]Ever. Like I've never seen it used in runtime. And if it is, it's a bad Docker design. So, for instance, you want to strip those out before you ever get to this production. I'll show you some patterns that do that for you easily.

[00:15:28]There's something called multi-stage builds you can use in Docker. And these multi-stage builds are designed to build something, copy from that build stage, and move it into a production image. And then I have an example of that later.

[00:15:39]Another thing here is that there's some flaws in in in the design and security features of this. So, for instance, there is no run as root. Uh there Excuse me. There is no user here, not root. I'm I'm from root. There is a um there is no user defined. Uh Docker images run with permissions. And And you can set up a a user to be sort of the the user in command of the operation of the of the code in these images. If you do it by default, it runs as root. And we all know we shouldn't be running things as root in production workloads, right? So, that's an easy thing to miss because it's a fault of omission, right?

[00:16:16]You just didn't do anything and got the wrong result. It again, Docker optimizes for developer experience. It's a developer tool, right? But But the images turn into a production experience. And uh and we need to make sure we safeguard that. Again, not no fault of the of the designer of this image. They just If you see this in production, you've got a problem, and which I do see them in production often in in in real world applications. The other is is in this this is a go um a a go image, and it has of course go binaries that are intended to run in it.

[00:16:47]Somewhere later, there's a copy in the in in the Dockerfile that would copy into that path. But you see this chmod R 777 gopath? That sets That sets like world rights to to the to everything in that image. They can You can pretty much do anything. And then that's in the gopath. So, if an attacker gets in this image, all they need to do is put stuff inside that image, and it effectively runs as root. Or put something in that path, and it effectively runs with full permissions. So, this is you know, that's kind of a theoretical attack plane, but it's something you don't want to have in there. And it's easy to fix.

[00:17:17]Just change the permissions to something else. So, really basic Linux security manifested in Docker commands, and and and we need to be thinking about that when we build these images cuz in the end in essence, you're building running images here.

[00:17:29]So, let's keep moving. I I I got uh let's see, time check. I'm doing okay.

[00:17:34]So, >> [snorts] >> I'm just going to switch. Python.

[00:17:40]Similar interesting patterns. Again, here is the commands you can run. Also, lots and lots of vulnerabilities.

[00:17:46]Same thing, if you're in a compliance you know, role, this is really um you know, a big job. This is also one of these um uh full full-powered images that have all of the Linux bits in them. It It didn't come from a from a builder pattern.

[00:18:05]You'll see it in a second. I might as well switch over to the um to the to the good Oh, yeah, it does It is a builder pattern. Excuse me. Yeah, it it's a builder pattern as well. So, same thing. Um these these are rampant throughout. And and um and this one has some uh has some packages in it that are really interesting. So, one one such package in this image, you'll see in the fine print down there, is Linux lib C dev. And And before we saw lib C6 dev, these um these are again used for compilation. But specifically, I see this this this library in tons and tons of images. And And they have like hundreds and hundreds of vulnerabilities.

[00:18:47]The interesting part about this library when it it is that it's simply a header file. So, literally ships with no code in it. It just overlays kernel libraries that are already in your kernel of your operating system. So, the actually the vulnerabilities of those, if they exist at all, are living in the kernel of the host system that the Docker daemon's running on top of cuz Docker inherits kernel functionality from the from the underlying operating system. So, in this case, all the security scanners find it, but it's not really an exploitable in in the form that it's being flagged. It's not a container vulnerability. It might be a host operating system vulnerability, but it but it's probably not because most of the modern kernels don't have this problem. So, in essence, it's a tool someone used to create headers for compiling C code inside of the container, but didn't strip this out when they built the image at the end.

[00:19:42]And it's really easy just to take it out in another layer of the container, and it can be gone. And I see this time and time again where it's problematic. Um so, it's a pattern that you you know, folks should sort of watch out for.

[00:19:53]Uh let's look at the Dockerfile now. So, again, build build it's a build pack, right? Dependency bookworm, full-fledged bookworm, all the tools. In this case, this looks like an image that you'd use in like CICD building with Python. You're often having to pull a lot of packages, that's why you see curl coming a lot with Python. Curl has some really notorious vulnerabilities. It has in the past, so you really need to be careful with the version of this, and get, and build essential. All bringing in tons and tons of tools. Um and a bunch of dev headers.

[00:20:23]Again, these kind of dash dev things.

[00:20:25]These are all again these dot H files effectively that are used to overlay lower-level libraries. Should be not there. One other specific security kind of issue I see in this one is pulling of keys. Sometimes with Ubuntu, um you you you want to pull keys for certain libraries that you want to install. I'm not sure why it's coming from Ubuntu. I'd have to look again. But in this case, it's talking to a key server, but there's no um trust on first use capability here. So, this does not define a way to verify that the source of that key is actually the one you want. Um you know, a a better pattern would be to use some kind of authentication method first or or some other secure key exchange pattern that would do that. Um I I wouldn't do that. So, I think uh it's something to look out for. It's just another kind of common example of security flaws in the setup stage of these of these um of these containers.

[00:21:19]And then the third is this MCP gateway, uh which is kind of different than the others. The others were kind of flaws of configuration and maybe even flaws of of what to pull in during builds. This one has um just five vulnerabilities, right? If you're in compliance mindset, you think that's awesome. I have five vulnerabilities. I'm pretty much done with that. But then there's some really, really bad vulnerabilities. So, the first thing uh it's got it's got three of interest to me. Um, this uh it's got CVE 2025-47907, which is in this case there's a Go database being brought into a library.

[00:21:54]And and that database has a race condition that causes corruption of of of memory buffers and it could accidentally leak stuff into other memory spaces. So, so this is and and and the point here is, you know, not all vulnerabilities look alike. And when you have a small number of vulnerabilities in an image, you have time to look deeply into what they are, right? So, so a lot of noise creates a lot of kind of indirection. We got some signal here now, so you trace into all of these.

[00:22:24]It's kind of the the deeper look at these vulnerabilities. And you see that that one really should be upgraded and and it's actually one of the the upgrade types where you can only fix it because it's a Go binary by changing the Go compiler to a higher version, which brings in a better version of that of that library. So, so in Go, for instance, a very common language, fixing is often recompilation. Uh and that needs to be done in a place where the code lives and the compiler lives, right? Uh bumping versions of the Go um compiler.

[00:22:58]That's a countdown, great. So, um another one, this uh GHSA in in the Docker client library. So, that's a an interesting vulnerability cuz that one has this case where you where you shut down the Docker daemon sometimes, it it forgets to load the firewall rules when it comes up the next time, which is pretty bad, right? It say you have a a service and you think that that ports are closed when they're not, but you fire it up and it and they're open.

[00:23:29]It's kind of hard to know that uh, especially since you know, containers can be quite ephemeral and have all sorts of start stop patterns. So, in this case, you're digging deeper into that CVE is is one that's super critical and you'd want to fix that as soon as possible. At least I wouldn't want that running production for me. And then there's another map structure leak vulnerability in this one, V2. And and this one has a corruption where under high load it can actually leak sensitive information, secrets, etc. into into log files.

[00:23:59]Um, again, all three of these are I think pretty serious risks, operational risks, and and security risks and should be fixed. So, so here's a case where a small vulnerabilities doesn't mean low risk, which is often not you know, kind of kind of a dichotomy, right? Like we we often we want to fix vulnerabilities urgently for compliance reasons that Kev would drive that behavior when lurking underneath the hood here is something kind of really interesting and bad. Um, which means you know, again, this this one this one I I would call this the most insecure image, but at first glance you might not think that for these for these reasons. Now, there's also some interesting structural problems with this with this image from a production perspective. Some of them border on operational risk and some of them border on on security risk. One is if you see this from scratch as MCP gateway DIND.

[00:24:56]That DIND is a common pattern you see in containers that's Docker in Docker, which means there's a Docker demon instance running inside a container.

[00:25:07]Code's using it to run Docker you know a containers inside the container which runs on a Docker demon. There's a lot of interaction there of Docker demons running on Docker demons, which means one, it's kind of complicated to debug an environment like that. Where's the problem? It's really can be difficult. And two, it's heavyweight. It's bloated, right? There are better patterns you can use to run things like that. One's called a sidecar pattern, which you could go explore.

[00:25:34]But there's ways to engineer around that and fix it. The second one is when you do that, you need the Docker CLI which is running on this container. You can see it says the run APK add Docker CLI.

[00:25:47]But there's no user set. So that Docker CLI will run as root. You know, if you were like a crypto miner kind of person who run writes those kinds of malwares, this would be a great thing to find because you could just run your applications here directly and have this Docker in environment at your disposal, which is a little scary. I mean if I pass the firewall bug then you can enter, right? So that's a problem and and then it's got some tools here that are really interesting staging tools like Socat is like cat on steroids. It lets you understand networking and find your way around between interprocesses.

[00:26:22]And and so this has a ton of staging tools. It's got a problematic architecture and it's got and it's got this this awesome toolkit if you were trying to trying to stage something on. Not a great pattern.

[00:26:36]Again, this is clearly a developer oriented image which makes it super easy to run things on, which is also something you don't want to have in prod. But I have seen this running in prod in places, which is kind of scary to me.

[00:26:48]So the pattern kind of repeats itself.

[00:26:51]I've seen you know literally hundreds of instances of this. I I I tend to try to find like a few bad containers to to evaluate every week. This is kind of broken architecture pattern both from a kind of a vulnerability and risk surface issue like I showed you where you've got really development issue development images masquerading as production images. And and that can be cleaned up through some better Dockerfile design.

[00:27:17]And then you've got these these kind of important CVEs, these exploits, these patterns in Docker containers that are really security risks, not just bloating kind of risks, but really security risks that are foundational. And into me those are actually more more scary.

[00:27:35]But yeah, but you know, in the end combining vulnerability analysis management with Dockerfile understanding and understanding how to look at these patterns in Dockerfiles can really help you engineer ways around all of this.

[00:27:50]Um, remediation options. So and I'll just go high-level on on most of these.

[00:27:55]The first is I would reorganize all of these images, especially the first two, as something called multi-stage builds.

[00:28:01]Which means if I'm using builder tools to build things inside of an image, and it's super convenient to build software inside of a container image. Why is that? Well, Docker images are portable, first of all. So if I make a builder image that knows how to build my software, right? Um, then I can move it anywhere and build that software, mean which means it's I can move it from from a developer environment to a CI/CD environment, and really have it be self-contained. And what is atomicity in in you know, self-containment is effectively atomicity. Container images are super cool because I can run them on on this laptop, and then I can run them on an a server instance in Amazon, and environmentally it's highly portable, which means those images can just work there. They're kind of guaranteed to work if If have the Docker daemon underneath them. Uh which means I don't have to worry about environment variables and dependencies and all the stuff that happens when you try to move software from one place to the other. So, Docker images are super good at that. That's why folks like to build on them uh build software in them all the time.

[00:29:01]The problem is you shouldn't mix the building stage of software with the packaging stage of software. So, containers are also packaging for cloud-native applications. It's the thing you ship to the cloud to run.

[00:29:13]Developers like to build things in them.

[00:29:14]If you merge those two concerns, you say one Docker file, I'm going to build all the things, then I'm going to package the things I just built inside of the container I just built the things, you get these bloated products.

[00:29:25]You just do it separate them and and multi-stage does that. I'll show you an example in the next one. But that and that gets rid of a whole bunch of stuff.

[00:29:31]For instance, those lib C debs I talked to you about that just header files, if you do that pattern, well, then you can leave the header files in the builder part of the container and you can move the binaries to the deployment side other container. It's two containers at this point. Um and now you've got this kind of perfect balance of portability, production readiness, and build ease.

[00:29:52]And that's kind of the nirvana of of merging the two concerns.

[00:29:56]The second is I would um also change some some things. So, for that Kev, right? If you're on Debian and that's what you like to be on in this case, the only way to fix that in Debian Bookworm, which is the most popular variant of Debian, is to make a backported patch for that.

[00:30:11]Backported patches mean I take I take software from future release of the of of the operating system. I find the bits that changed and then I move that software back. I rebuild the library that has the vulnerability. I make a new patch for that and I take that patch and I apply it in a patch library.

[00:30:29]Um this is like uh you know, double black diamond skill, right? You need it to do this stuff. So, uh most of the other things I talked about you could probably get away with on the green on the green on the green on the green slopes. This is double black diamond um but folks do it. Um and uh and and and actually it's one of the things that that I do a lot of. So, that's one way to fix it. Um it it it beats switching to a new operating system, by the way. Switching to new operating systems can be a really heavy lift and software tends to break.

[00:30:58]So, that's one thing you should do there.

[00:31:00]Um for the for the Python one and and all of these, um just define a a user, right? And make sure that you have least privileges mindset when you go to build these things, right? Root user um is bad.

[00:31:13]Uh and in fact, if you're uh subject to some of these more high scrutiny security uh and compliance regimes, you have to um not do that, right? With uh Linux hardening and and and OS hardening kind of things. So, um you want to take that out of there. And then in the MCP gateway, it's uh if I were to turn that into a production image, I'd do a lot of things differently. I would not have Docker in Docker, for sure. I would use the sidecar pattern, which means really kind of moving um the the Docker part of that outside of the image that's actually running and making it a supporting feature, or just design the container in a way where it runs directly on top of the Docker which underpins it and not have to live in this kind of nested inception loop that's in there. Um we would bump Go uh in a build stage, which means when I say Go build, if you own the code, uh that's easy. You got a repo somewhere.

[00:32:05]You go build it, right? If you don't If it's open source software and you need to fix it, say it that's not fixed by the maintainer, right? What do you do?

[00:32:13]You go upstream to wherever that library was built, by whoever built it. You find that code, you pull it pull the you clone the repo, and you rebuild it with a new version of Go. And then of course, you got to run all kinds of tests. It's not that easy, but uh but you can do it. And you can do it uh and and lots of folks do.

[00:32:30]And then uh and then really just in all of these, upgrade to the nearest best semantic version. Semantic versions is is what we you know, if you've got a library XYZ live 1.2.3, right? It's upgrading to semantic versions that change in that third number of that version 1.2.3 to 1.2.4. Those are almost always just maintenance and security patches. Pretty safe. Uh do that. Do that as much as you can. Uh semantic versions that change by the second after the dot 1.2 to 1.3, functional changes, likely breaking. Hard to know.

[00:33:07]Uh you can do so by inspection. There's lots of ways you can do that. Um Um and then, you know, buyer beware on any higher level semantic versions. But semantic version difference in upgrades really keen to understand those changes.

[00:33:20]And I wouldn't ship uh what I would call attacker staging tools to production, either. Uh this is an example of this multi-stage build kind of stuff. Um before single stage, very um kind of esoteric, but from build pack depths bookworm, I said that's a builder base image, right? And And in this design, what we do is we And here's the kind of the key line.

[00:33:45]From from Debian bookworm slim, now that's a more of a production grade image Debian bookworm slim. It's uh it's not perfect. It's got a lot of vulnerabilities in it, too. But it's commonly used in production images. Um and we know how to fix them.

[00:33:58]Um but then uh copy from builder, if you'll notice, we labeled it as builder on the other side.

[00:34:06]Um from the builder stage, copy user local go to user local go. So, when the building part of the go builder container finished, it puts the binary image that it built there in the in in that in that directory. What you're What you're saying is I'm going to use the builder container when it's finished as if it's a file system that I can copy from. Now, Docker lets you do do these fancy things. Docker says, you know, I know what container image can run. It's running software, but it's also just a file system. If you look inside of a Docker a Docker image, it looks like a bunch of zip files.

[00:34:42]The Docker daemon knows how to interpret that and like put the software in the right places. It It's something called an overlayed file system, which means each layer of the image modifies all the can modify the layers before. So, but in the end, it's just a file system. And it's sitting on the disk. A kind of complicated zipped-up file system, I'd say. I built tools that traverse all that stuff. So, so the Docker daemon knows how to use that as if it's using a hard drive. Like you can say, "Hey, I got that image over there. I want to copy those parts to my image." And that's how multi-stage works. You basically label something you built, the image and parts of it as what you want, put things in directory structures, you build a new image that says, "Copy it from over there." Well, over there, all I'm pulling is a binary. So, I didn't pull the headers. I didn't pull the builder bits. I didn't pull the stupid tools that I don't need, right? I just copied the go.

[00:35:33]I made a user called golang, which is now a a lower privileged user. And And we go on our merry way. And so, that container I would I would estimate the one on the other side's roughly a gig in size.

[00:35:44]I would estimate the one on this one's something of a 100 meg or less in size.

[00:35:48]And you can trim down slim a lot from there. So, Docker bookworm slim, you got a small image, tight, no security flaws.

[00:35:56]And you can literally copy that pattern for the first two images, and you would get yourself a long way. With Python, it's a little different. You don't really need a builder for Python.

[00:36:03]Sometimes you do, but but not always.

[00:36:06]Python's a more of an interpreted language. And And usually you can just pull the the bits you need. So, that's a a win a win-win, right? So, you get security benefits, no more of those uh in this case uh kind of toolings and patterns that don't work. You get compliance benefits cuz this thing's going to have, my guess, if I built it just like that, I I can remember what I did what it was, but it's probably less than 100 vulnerabilities when the other one was 1,000 or more, which means you can focus on the ones that really matter.

[00:36:33]And um and operational benefits, smaller image, simpler runtime, lower attack surface, and really just few lines. Uh in fact, men I think that's pretty much all you need, maybe a little bit more environment variables to get a real working image. Um if anybody uh wants to see some of those, you can hit me up after.

[00:36:51]So So you know, we talk about compliance, right? Um if you you know, I mean most of us are security people here. What uh what auditors want out of this whole thing, right, is they want assurance, they want evidence, they want to know that what you built is secure. So scan evidence, of course, I've shown some of that here.

[00:37:07]They want it to be detailed, um and they want it to be annotated, they want to know exactly what's in there. Supply chain documentation is becoming more and more popular, so so after you look at all these images, they want SBOMs, usually CycloneDX or SPDX are the common ones. There's a couple of other formats that are interesting.

[00:37:22]And they want vulnerability disclosures.

[00:37:25]More and more I'm seeing folks asking for um I want you to tell me which of these vulnerabilities really apply to that image. Um in in in in in FedRAMP, which is the US government compliance regime, they want to know specifically, do those vulnerabilities matter in these images? And that's telling them about exploitability. VEX, if you haven't heard of it, is a format for doing that.

[00:37:45]It says, this vulnerability is actually affecting this this image and here's how, or not affecting to get rid of false positives. Architecture evidence uh Dockerfile diffs, documenting like how you designed it, are often important in the higher regimes. And then something called attestations, which means I know where this was built, I know it was built securely, and in fact, here's evidence of how I built it, and you might want to be able to repeat that. And this is kind kind of emerging in software supply chain security, being able to tell the person downstream who's using your software how you what you did to secure it, and how you know it's secure when you built it. This is stuff that's coming out of this. Uh so when you think about image security, you can think about that. There's um There's a playbook, right? And if you think about what I did in all these, it's pretty straightforward. One is I always look for the most obvious worst things in any of these images. I've I've I've I've gotten past this kind of like the biggest number of CVEs wins my attention. I I don't always think that.

[00:38:42]I really look for real security issues, and sometimes these are at odds. But, holistically, I think vulnerability remediation is really important. It's really important for a number of reasons, including just operational efficiency and getting noise out of the system and lowering future risk of unknowns, right? Um and these are known vulnerabilities mostly we look at. So, that's uh something you should just be able to do easily, and there are paths to doing it easily. But, triage priority, right? Like, you want to get to to a small number of vulnerabilities that matter and really put a lot of effort into looking at those. So, we want to automate away all the vulnerabilities. You want to build images with with this kind of proper build separation. You want to build your building infrastructure for doing this to be something that, by definition, best practices just relieve you of these problems. So, so really having this best practices for multi-stage builds, having this best practices for what kinds of things you will allow in images. The get, you know, get remediation, the sys cave is an example of a special case. Uh it's a high-priority vulnerability that has no patch.

[00:39:39]Um most organizations will be faced with this, and and at some point you'll be faced with it in a way that's urgent.

[00:39:44]Um You know, you think of any of the headline vulnerability in these exploits that came out, right? Um Log for J. So, these kinds of things um are often a scramble. You can stay ahead of it by by really getting your hands around What do I do about vulnerabilities that have no fix? I don't think doing nothing's the answer.

[00:40:01]I think you need to fix them, but you need to find a path to doing some of the things I mentioned, like like finding a way to make these these uh these back-ported patches.

[00:40:11]Doing toolchain updates, upgrades in general are good, right? Knowing how to upgrade so that it doesn't break your stuff is hard. I think it's really hard, especially at scale.

[00:40:22]Um, and finding a path to do that is important. Policy enforcement, um, organizations aren't like that great at this, right? Like making sure that this is done everywhere. Uh, I see a lot of variability because, well, basically, we want to give engineers and development teams autonomy so they can go fast, but at the same time burdening them with too much process slows that down. So, how do you move this around?

[00:40:43]Automation is the only way. DevOps maturity and automation. You got to do this stuff in an automated way. Do not Shift left is kind of a broken experiment, right? You know what shift left is, right? You push things back at the developers. Well, they got a job to do, especially for security, right? It's hard for them to to to to get their arms around that. So, we got to make it easy for them and and making it easy for them is not shift the work left to them.

[00:41:04]Shift it out of the organization. Turn it into something automated that just happens. That's the best part. And then, of course, for practical reasons, you got to document stuff, especially processes, but certainly for compliance reasons and such.

[00:41:17]Um, the company I work for does a lot of these things. They can do it fast. If you want to learn about that, let me know. I won't talk any more about that, but we can solve a lot of this automation and such that I talked about.

[00:41:27]Um, I'm John, CTO. Been doing security a long time.

[00:41:31]A pleasure to stand here with you guys.

[00:41:33]If you want to LinkedIn with me, um, I can I could even give you the scripts that I just showed you, uh, or anything, or show you more stuff, or if you just want to talk about a Dockerfile someday, we'll get on a Zoom and I I I love to geek out about this stuff. So, anytime is fine with me.

[00:41:47]Questions? Comments? Thoughts?