Building SHERPA-K: AI-Powered Kitchen Manager

Hinzugefügt: 2026-05-15

[00:00:00]We are back from a grocery trip and we got a bunch of things. Just before putting them away, I'm just going to quickly scan all of them with this little cute QR scanner device.

[00:00:17]It is 1000 p.m. right now and I'm craving something sweet. So I asked Claude code what I should eat and it is saying we have 4% milk fat cottage cheese with raspberry jam or orange marmalade. It's amazing that it knows what's in our pantry and fridge and can give me recommendations at this time of night. So I'm going to go stack. I'm the founder of Agrid Intellect and instructor for this boot camp. With every cohort, I try to do a project along with the participants so that I can show them how our methodology works for building custom agents as well as trying out our methodology and improve it.

[00:01:03]Let's talk about the problem statement and how my understanding of it changed over time. So initially when I started this all I had in mind was what if I help the users to go from a set of meals to a grocery list. But very quickly through problem validation we learned that the problem is a little more sophisticated. The first layer of complication was that we need some sort of a source of truth for recipes because some people have their own set of recipes. The next thing that came up was that usually people when they think about their meals and grocery list, they're doing that for the series of meals that are coming up, not one meal at a time. And the second thing was that usually this grocery list has to also be checked against what do we have in the home in the pantry or fridge. The most interesting part came from the user research interviews that I've done in which I learned the biggest complexity is that the burden of planning falls on one person and that person has to take into account a lot of different complications and preferences and constraints for various members of the family as they're doing this planning and purchasing of the groceries.

[00:02:17]How did this evolution happen in a systematic way? So this is the setup that I used here. As you can see, there is an MCP server that we provide to all of the participants in our boot camp. This server contains a series of workflows. Think about them as cloud skills. These connect with your cloud code environment where you're building up your solution or understanding the problem more deeply, brainstorming and cloud code writes reports on your file system or develops the code for you.

[00:02:49]Just to give you a sense of what these workflows are, we go through a process of discovery, development of the product and then the deployment of in the discovery we do ideation, user research and create evaluation data set. Then in the development, we do the implementation. A lot of energy spent on context management, designing the architecture and the user experience.

[00:03:11]For the deployment part, you're going to think about how you're going to do your first deployment, how you're going to demonstrate your agent. what I'm doing right now and how do you do evaluation at a scale.

[00:03:24]Once the planning sessions are done using the workflows that I described a few moments ago, then we turn those specifications that come out of those braining sessions into a series of issues workflows or cloud skills that turn the specifications into GitHub issues so that we have a backlog of work that we want to go through. Then we can prioritize these set of issues using cloud skills.

[00:03:48]And then there's a process of development which follows a test-driven development process. Essentially writes tests for what we're adding to the software that we have. Then it does the implementation for it. And then it refactors. Then it reviews the code for any opportunities to improve the quality. And then finally wraps it up.

[00:04:08]And part of this process of wrapping up is also checking against specifications to see if any documentation needs to be improved. And then you rinse and repeat.

[00:04:17]You pick up another issue and go through this whole process again.

[00:04:22]So after all of this, what I ended up with was an architecture like this. On the user layer, the user interacts with the system currently through a CLI, but very soon through any web version of cloud or chat GPT or Gemini. I think the most convenient mode of communication with the system is SMS. This is already implemented, but also a barcode scanner to be able to quickly keep track of what's in the fridge and pantry. All of these provide information to cloud code, which is the main orchestrator here. And then quad code as the main agent in the system essentially uses a set of tools that are available in its service layer managing the pantry and fridge and the shopping list planning meals and then all of these handled by a very detailed tracking of state in the system. And then there's a data layer. There's the information about the state of the kitchen currently. So, what do we have in a kitchen and what we're planning to do soon? A set of recipes that we care about, any planning history that we have, and a database of common grocery items with about 4 million items in it.

[00:05:37]Okay, so the outcome of this is two components. There is this visualization that shows the state of the kitchen. So, I can very quickly see who's involved in this household. What kind of preferences and constraints do they have? I can see information about our fridge, information about the pantry, what shops do we go to usually? What are we doing in our yard? What kind of pets do we have and what are some nuances about them? What's my current shopping list? So, this is this gets populated from natural language interactions with the system as well as reading all the SMSs that I'm sending to the app and a meal plan. Okay, let me now quickly show you a typical conversation. This is actually quite a long conversation that at the end of it I said just summarize this conversation so that I can present it. I asked it to fetched all the SMS notes and it did and then it sorted them. It said oh this is about things we need to buy. This is about a meal that we want to make but it pointed out that we don't have a recipe for this one.

[00:06:39]Then I provided a screenshot of the recipe. Then it added that to the database. Then I provided a voice note of everything else that we needed to buy and it processed it and added to the shopping list. Then I said check the shopping list against what we have in the pantry and fridge and remove anything that we don't need. So it took care of that. Then after the trip I provided the barcode of everything that we bought in the trip as well as a voice note that said any item that didn't have a barcode just ginger and tomato and whatever. And then it processed and added all of those. This is where it told me that that Japanese curry is ready to make. Then I said, given our preferences and habits we are aiming for, tell me what should we put on our next shopping trip. And it provided some suggestions to me.

[00:07:31]I want to end this part quickly on a piece that we did last night that I thought was fun. So, my wife was craving Korean food and was suggesting to order Korean food. And I said, "What if we asked the app to check if we have ingredients for for some sort of a Korean dish?" And I looked through the ingredients and suggested a few things.

[00:07:51]And then we landed on this fusion madeup dish, which is a Korean sloppy joe. He gave us what ingredients we can pick from our pantry and fridge and gave us the recipe to make it. And that's exactly what we did.

[00:08:05]We had some beef in the freezer which I defrosted which I'm browning right now.

[00:08:11]I'm going to add my onions and my garlic. And once those are softened, I'm going to add this sauce which is a mixture of some goch jang and some tomato paste, ginger, rice vinegar, and a few other ingredients that we had in the cupboard. Another criteria that we had was that I wanted to be able to cook this quickly because it's pretty late at night and we're pretty hungry actually. I put the garlic and the onions in about 5 minutes.

[00:08:44]Thawed the beef in the microwave while that was happening.

[00:08:49]Heated up my pan. At this point, it's been about 8 minutes since I started cooking. All right. Now, it's been about 12 minutes. My garlic is nice and brown in the center. So, I'm going to stop that browning off. I'm going to add my sauce and we'll mix it in and turn my heat down a little in a second and let it simmer until it's nice and thick.

[00:09:24]Okay, so now we've plated. These are my fried green tomatoes with a little bit of onion relish. Got some pickled carrots on top of our Korean sloppy joe.

[00:09:35]And I finished this at the time in 19 minutes to plate, which is pretty good.

[00:09:44]Also, we didn't follow the recipe to a tea. We're not letting the AI make all of our decisions, right? But really, the basis was there. It gave me enough inspiration to pull off dinner. So, pretty happy with it.