Beyond the Basics: Production Serverless Patterns for Extreme Scale • Janak Agarwal • GOTO 2025

Added: 2026-05-09

[00:00:13]Uh I'm Janak.

[00:00:16]Janak Agarwal. Uh at AWS, I lead product management for the service called AWS Lambda.

[00:00:21]Uh by way of a quick show of hands, just a quick sense of how many of you have heard about Lambda?

[00:00:28]Cool. Maybe use Lambda in an application?

[00:00:32]Really cool. Lots of cool people here.

[00:00:34]You know, heartwarming to see.

[00:00:36]Uh We're going to talk about a really interesting topic in this session today.

[00:00:41]It's scale.

[00:00:43]Specifically, how serverless makes it easy for you to scale.

[00:00:46]Uh those of you who've been building applications for a while, maybe operating applications for a while, you know how difficult scaling is. And serverless makes it easy. And I'm going to try to cover two patterns, but we'll see if we have the time.

[00:00:59]Uh in this uh session, you know, we're lucky that we have a good topic.

[00:01:04]Even if it's a hard topic, because otherwise in this after-lunch session, with my monotonous voice, I promise you I could have put you to sleep. So, to make it more interesting, I'm going to ask you a series of questions in the middle of the presentation. Don't let me down. Work with me. Uh this is to make sure you're all paying attention, okay?

[00:01:23]So, let us start.

[00:01:26]Okay, let's dive into what type of scale we're talking about today.

[00:01:29]So, bear with me, you know, close your eyes for a second. I'll not steal your wallets and backpacks, promise.

[00:01:36]Close your eyes, okay. You you you're in the first first row, okay. Okay, close your Imagine uh you know, your favorite artist, you know, they've announced a a concert which is driving distance to your home.

[00:01:49]You've been wanting to see this artist's concert uh for a while. The presale goes live at 9:00 p.m. What do you do?

[00:01:59]If you're anything like 5 million people in the US recently, you know, you would go at 8:30 and do F5 on your browser.

[00:02:05]Try to refresh it. Uh that is uh that is some level of scale.

[00:02:12]And then if we move on, uh you know, there's a big sports events, uh like let's say the Cricket World Cup, you know, India-Australia, that will bring back bad memories, India-South Africa.

[00:02:23]Uh 500 million people, you know, log in at around the same time to view that stream. So, that is the kind of scale that we want to target, you know, and how to solve today, you know, with serverless and how serverless makes it easy.

[00:02:37]>> [clears throat] >> Uh let's get some terminology in order.

[00:02:40]So, this is what a consistent workloads look like.

[00:02:44]If you see from the start time, you know, 7:25 to 7:31, the end time, you know, there's a gradual increase in traffic, sure.

[00:02:51]Uh there's a good amount of uh peaks and troughs, that is also fine, but it's you know, it's consistent by and large.

[00:02:57]Uh the the peak to mean ratio is fairly low.

[00:03:00]Okay, most workloads fit this category.

[00:03:02]No problems there. They work well. Uh you we like them.

[00:03:08]On the other hand, your F5, maybe when your kids' exams or results are due, you also F5ing with them, or your concert uh you know, your Cricket World Cup when Sky caught that catch, first ball, last over, you know, it spikes. And those spikes are unpredictable or predictable, but they spike, and we want to discuss how to solve for these kind of workloads.

[00:03:35]So, how do we solve for them?

[00:03:38]A more traditional approach would be, you know, you obviously have some compute, you're managing the VM, you're patching it, uh you're deploying your application, and so on. Uh you're maybe using some sort of an orchestrator system, maybe Kubernetes and containers.

[00:03:53]Uh you're applying to apply auto scaling behaviors, you know, fine-tuning it to reduce your costs.

[00:03:59]But, you know, you you pay a price for a good amount of idle costs.

[00:04:05]In picture, uh if you provision uh for as provisioning means you're provisioning your compute for let's say around the orange line, uh then what happens is uh when the sudden spikes come, the auto scaler is not able to keep up.

[00:04:22]And oftentimes, it results in an outage, which means your end user is not able to purchase tickets, view their kids' exams results. Maybe that's a good thing, but you know, you don't want that end user to experience that kind of uh behavior.

[00:04:37]On the other hand, if you provision for the absolute peak, uh you have some complaints.

[00:04:43]Uh that complaints come from your wallet. It gets lighter, because you know, overprovisioning leading to empty idle capacity could be expensive. Depends, but yeah, it could be seriously expensive.

[00:04:57]So, you know, we don't want to do that.

[00:04:59]>> [clears throat] >> Now, how does a serverless approach help you?

[00:05:04]First, uh serverless services like Lambda will automatically patch your host, your uh micro VM, and also your runtime. So, when I say patch, it's both for security and for performance. You know, so uh any patches that come in open source, we apply them seamlessly for your application, and life is good.

[00:05:24]And then, they scale to zero. What that means is if there's not a single request that your end user, you know, is sending to you, then you're not paying for anything. It's it's idle capacity is never charged. Uh it scales to zero in that sense, and again, your wallet says thank you.

[00:05:41]But at the same time, it auto scales up and down really really fast.

[00:05:45]And that is what we'll see next.

[00:05:47]In response to incoming events, you know, serverless services typically uh scale the fastest amongst all forms of compute.

[00:05:55]With me so far?

[00:05:59]Let's dive a bit deeper. This is where the questions come.

[00:06:03]So, to see how, you know, Lambda helps you with scaling, we're going to first learn about uh how does our concurrency work.

[00:06:14]So, the way Lambda works is uh events trigger a function invocation.

[00:06:20]On receiving the function invocation, uh Lambda creates a execution environment.

[00:06:24]We call it sandbox execution environment. You'll hear me use that term interchangeably, so bear with me.

[00:06:29]Uh we create an execution environment uh for the first time the request comes.

[00:06:33]That is the cold start period. You might might have heard of a cold start in uh in serverless parlance.

[00:06:40]That's what a cold start is. And then we process the event as fast as possible and give the result back to you.

[00:06:46]Okay? This is how it works in practice.

[00:06:49]Invoke comes, uh execution environment is created. This is invoke number one.

[00:06:54]Uh After the cold start, the request is processed uh and the result is written back to you.

[00:07:02]And each execution environment supports one uh request at a time.

[00:07:08]Now, uh we are at time uh of the red line.

[00:07:11]Let's say the request is done processing. We do not take this environment back.

[00:07:16]We keep it around for a while.

[00:07:18]Any guesses why?

[00:07:25]Perfect. You know, uh so, this is to give you an interactive experience. We don't want to take it away. Uh you know, in in a short time, if another invoke comes, the cold start reduces, and you know, life is good.

[00:07:36]There's no uh latency.

[00:07:39]Now, let's see if at this time, you know, time is always moving, not constant, uh the second invoke comes in.

[00:07:46]The same execution environment is reused. We don't create a new one, because you know, it was right there.

[00:07:50]It's again, you know, optimizing for latency.

[00:07:53]Now, the good thing is you're not paying for, you know, this gap, this idle period between one and two.

[00:07:58]Uh this is free for you. This is the cost that we bear in order to give you a good customer service.

[00:08:07]Now, let us say that an invoke three comes, and this is what the situation is. The second invoke is going on.

[00:08:13]Uh who can tell me what happens? Is a new sandbox created? Not?

[00:08:24]Okay, how about this? Uh who believes uh new sandbox is created?

[00:08:31]And the rest are paying attention or no?

[00:08:33]Uh so, let us see.

[00:08:37]You're right, because you know, the second uh execution number two, invoke number two is still going on. Uh next invoke comes, you know, your uh process, your application needs to scale, we automatically do it for you.

[00:08:49]Let us make things a little more interesting.

[00:08:53]Here, let's say an invoke four comes.

[00:08:55]What happens?

[00:08:56]Do we create a new one?

[00:09:00]You guys are going to make me finish in like 15 minutes.

[00:09:03]Uh okay, good. You are right. You see, you know, we're using the first one.

[00:09:09]Time moves on, doesn't stop for anyone.

[00:09:11]What happens if invokes five and six come in at this time?

[00:09:15]Do we create new?

[00:09:19]Fantastic.

[00:09:21]I want to see some other people also, like, you know, speak. It's the same five, six people speaking.

[00:09:26]I wish I knew names, and I could cold call.

[00:09:29]What you're right, you know, it's five and six.

[00:09:32]Now, time again moves on. We have three more invokes now.

[00:09:39]Fantastic.

[00:09:43]And now it's the most interesting of all.

[00:09:46]Here, four more invokes come.

[00:09:50]What happens?

[00:09:53]And you get it. Uh I think you should see four more and there you go.

[00:09:59]So, this is how Lambda's concurrency works. You know, pictorially uh in the idle times, you will see that, you know, where graph is zero. You know, there's no uh uh invoking progress. So, you're not paying for anything. And then, you know, as you scale up, you get more concurrency up to your limit of the account and you know, life is good.

[00:10:20]Application scales automatically.

[00:10:23]Now, the reason why, you know, serverless or Lambda helps you to scale really easily, quickly is here.

[00:10:30]Um you get thousand concurrency. You know, those one, two, three small numbers we're playing at.

[00:10:36]Uh imagine this, you get thousand execution environments right off the bat.

[00:10:41]And then, you get uh a thousand more every 10 seconds.

[00:10:45]And this limit is per function.

[00:10:47]So, let's say if in your account, you know, you have two functions, you know, we'll call it two workloads. One is uh you're authenticating when a new user joins that work upstream.

[00:10:57]And second is, you know, after you authenticate, um you have to retrieve, you know, user preferences or, you know, there what what ads to show. You need some preferences for that user to to show them, you know, target them with the right ads.

[00:11:11]So, you know, both those workloads scale independently of each other.

[00:11:15]Okay?

[00:11:18]With me so far? This is important.

[00:11:21]This is another graph. In a little over 3 minutes, you can get 16 to 18K requests. Uh not requests, you know, execution environments.

[00:11:30]Uh and depending upon the length of your requests, you know, this can easily translate to tens of thousands of uh requests per second, you know, instantly.

[00:11:41]So, let's come back to our workload.

[00:11:44]You saw what we did in a server full or a traditional uh setup. What do you think you have to do if you've built an application on serverless? Like, what do you have to do uh to scale it?

[00:11:56]It's a one-word answer.

[00:11:57]>> [sighs] >> You really have to do nothing uh because, you know, you you saw the scaling rate. Uh it automatically scales up, scales down. You're not paying for any idle times and you know, life is good.

[00:12:10]Put on your sunglasses and then we go forward.

[00:12:13]So, what happens after the big event?

[00:12:15]You know, your costs are not going away.

[00:12:17]The application still has to stay.

[00:12:19]Uh that is what we'll quickly see next.

[00:12:24]So, quieter periods here. The quieter period is, let's say, you know, 5:18 to 7:13, you know, first half of 7:13 in the graph here.

[00:12:34]How do you optimize for costs there?

[00:12:37]Uh while serverless does offer scale to zero, you know, it's not really a reality for most applications. You know, most production applications, you know, there's always someone watching cricket even if there's no World Cup. Uh you know, not every shoe or not every concert is a sellout. You know, so on. So, there's always some traffic.

[00:12:56]So, to optimize for that quiet period is what we're going to try to see next.

[00:13:06]So, I uh Lambda has this feature called provisioned concurrency.

[00:13:11]It is really useful for steady-state workloads or workloads with uh you know, where you see these quiet periods or with low peak-to-mean ratios.

[00:13:21]Uh it uh provisioned concurrency, the feature is like there's no cold starts. We keep some certain amount of capacity pre-initialized for you to serve invokes that come in.

[00:13:32]So, literally zero cold starts.

[00:13:36]Then, there's two types of auto scaling we support there, scheduled and target.

[00:13:40]And we'll see uh how it works.

[00:13:44]The pricing structure is important here to understand. Uh in in on-demand the function model, you know, you have two dimensions, requests and duration. In the provisioned concurrency model, there's three dimensions. Requests and duration exists, and there's a third dimension, provisioned concurrency. This is a nominal charge for the amount of capacity you can configure at base.

[00:14:03]Now, if you if your uh duration uh you know, the important thing to note is like there's a 16% discount in duration if you're driving good utilization in the provisioned concurrency model.

[00:14:16]And there's ways to do it. We will cover that in auto scaling in a second.

[00:14:22]So, scheduled auto scaling.

[00:14:25]You know, the name really gives it away what it does. Uh if you have a workload like, you know, top of the hour, you have to do some data processing work, you know, like a cron job, uh it will scale up in advance. Let's say if the jobs needs to start at 9:00, you can set it to scale at like 8:55, 8:57. And after the top of the hour process, you can scale down. It automatically scales down.

[00:14:47]This is also pretty good for like deal of the day. This kind of workloads you can predict uh when do you want to give out that deal, you know, 9:00 a.m., 10:00 a.m., and so on.

[00:14:56]So, it's fairly simple.

[00:15:00]And then, target tracking is a little interesting.

[00:15:04]Really interesting. Don't go by what I understand the feeling. But uh on the left, you see static provisioned concurrency and and uh allocation. This is really to drive the difference between what target is.

[00:15:15]Uh in on the left, you know, you just set up like a here is the level I want to auto scale at.

[00:15:21]And beyond this, you know, I will uh there's this concept that after provisioned concurrency, you know, if there's more requests, you spill over to on-demand, the previous model, the default model.

[00:15:31]So, any requests that uh require more capacity than what you've configured, you know, go to the on-demand model.

[00:15:38]Uh capacity uh is is kind of provisioned here. You know, you only have like less requests than you're paying for some some idle.

[00:15:46]On the right, you have Actually, you're right. Yeah. Yeah, you're right. So, on the right, you know, you have a dynamic provisioned concurrency allocation.

[00:15:55]Uh this allows you to set up targets. Like, you can say like some metrics uh like bytes transferred or CPU utilization.

[00:16:05]You can set an ideal and then, you know, reactively, the auto scaler will try to meet that ideal. You know, so it will spin up new instances or new capacity or, you know, spin down uh to try to maintain your ideal uh target that you think is right for your workload.

[00:16:22]There's some other cost optimizations that are useful to consider.

[00:16:26]A lot of you um you know, for reasons good or bad, but a common practice that we've found is like some uh folks prefer to write two functions with the same uh sort of uh business logic. One is for test, one is for production.

[00:16:43]Now, just like, when at least I don't when I used to be a developer, I didn't wrap in die equal to zero in try-catch.

[00:16:50]It that thing is just supposed to work.

[00:16:52]You know, so you don't have to test provisioned concurrency scaling, you know, beyond a point. The first time maybe you want to try out the feature.

[00:16:59]Uh so, on test workloads, actually, you know, turn it off. It It's It's one of those things that, you know, you have to just do it to to save costs.

[00:17:08]The next is power tuning.

[00:17:10]So, power tuning is a very underrated uh tool. It's an open-source tool that allows you to configure the right amount of memory for your function.

[00:17:18]And why is that important?

[00:17:20]The more memory you configure to the function, the more CPU you get.

[00:17:24]So, let's say if you have a complex data processing job, then you really want more CPU. You should configure higher memory. But if it's, you know, just like moving a record from that you read from S3 to another S3 look location, you don't need like, you know, too much CPU or too much memory uh without sacrificing any performance.

[00:17:44]So, you know, power tuning will help you to understand what is the right level of memory that you need to set for your function without sacrificing any performance. And that's the key piece.

[00:17:54]And then, we also support Graviton or arm-based instances. Uh they're priced competitively at a 20% discount, leading to an overall 34% better price performance.

[00:18:05]So, do it. You know, no real excuse to not do it.

[00:18:07]And then, uh for large-scale one-off events, you know, AWS has a mechanism called countdown.

[00:18:14]So, a lot of you have account teams here, you know, coordinate with your account team, and they will help you to coordinate with every AWS service your application touches for you to have a seamless, you know, fully seamless experience.

[00:18:27]So, remember this, you know, countdown.

[00:18:32]Now, let's say you've set up successfully provisioned concurrency, and uh you know, you still get an unexpected traffic spike. After all, you don't know when, you know, Virat hits a six or, you know, someone takes a cool catch, you know, you you just don't know.

[00:18:47]So, how do you do it? What do you do?

[00:18:50]Let us see.

[00:18:53]How would you handle it? Let's say you've you've applied your provisioned concurrency for the quiet period, and there's a spike.

[00:19:00]What would you do as a serverless developer? It's again a one-word answer.

[00:19:05]Yeah.

[00:19:06]Nothing. Because uh you know, remember like we spill over to on-demand, and on-demand again has that fantastic scaling rate, you know, thousand every 10 seconds. So, you really have to do nothing.

[00:19:19]Uh when requests exceed provisioned concurrency, there's no availability drop.

[00:19:24]So, long story short, you know, I've I've tried to simplify it, but scaling in general is hard. It's a really hard engineering problem to solve. Serverless helps you to simplify it and abstract it all the way uh with like pretty good large limits.

[00:19:39]Just a quick recap.

[00:19:40]Um automatic scaling, like that's what we do. That's what serverless specializes in. Really fast scaling.

[00:19:48]Fast capacity to you.

[00:19:50]The fastest among the uh compute portfolio.

[00:19:54]And then provision concurrency, you know, use it to handle your quieter periods. And you know, remember those gotchas like make sure to turn it off for test workloads and use power tuning to you know, get the best outcome.

[00:20:08]Uh provision concurrency spills over to on-demand, so you're still covered with spikes. You know, no no problems there.

[00:20:14]And then with serverless and the promise of no security host patching, no infrastructure management still is true.

[00:20:22]And you know, it should always remain true.

[00:20:28]With that, that was the the talk.

[00:20:32]>> [applause]