Build a Local AI Agent in Python with Tools (Part 3)

Added: 2026-05-05

[00:00:03]How's it going everyone? In today's video, we're going to add the most powerful feature to our agent, tool calling. This is what separates a chatbot from a true AI agent. With tool calling, our agent can actually do things in the world. Run calculations, read files, call APIs, or anything else we can code. It's no longer just generating text. It's taking actions.

[00:00:28]The way this works is through a loop.

[00:00:31]When the agent responds with a tool call instead of regular text, we execute that tool, get the result, and send it back to the model. The model then uses that result to generate its final response.

[00:00:45]This is the agent loop in its full form.

[00:00:48]Now, before we continue with the tutorial, I'd just like to thank Zed for sponsoring this video. Zed is a lightning fast code editor written in Rust and I've been using it to do all my coding for months now. If you feel like trying out Zed, I've left a link in the description box down below where you can download it for free. Anyway, let's get back to the video. First, let me show you the tools class. This is a utility that handles registering functions and generating the schema that the LLM needs to understand how to call them. Here we'll use Python's type annotations to automatically generate the schema. But before we look at the tools class, I'm just going to paste in the imports. So for this lesson, we're going to be using the requests module. We're going to be using inspect JSON and from typing we'll be using annotated get origin get args any colorable union and final. Then from data classes we're going to import the data class and the field. From rich console we will import the console. And finally we will import the date time and get pass. As soon as I format this, it separates what's built in from what has been installed. But how you organize this is up to you. Anyway, right below, we're going to create a data class called tools. And the tools class handles everything about tool registration and execution. The key method is an annotation to schema method which converts Python type hints into JSON schema that the LLM can understand.

[00:02:33]So the first thing we're going to do in here is create a tool schema attribute and we will store the schema under this name on the function. Below that we will create some tools and this will just be a dictionary with the keys being of type string and the values being of type collable which in other words are just functions. Below that, we're going to create a function called annotation to schema, which accepts an annotation of type any and returns a dictionary of type string to any. And once again, this method will take care of converting a type annotation to a JSON schema fragment. As a default, we're going to create a type which contains a string.

[00:03:13]This is the default for any type not defined below. The description will be an optional. And let's quickly format this. So everything looks nice. Then below that we're going to create an origin which gets the origin for the specified annotation. And this will handle the case of any generic and any wrapper type such as annotated.

[00:03:37]Next, we need to handle the annotated types and extract the description. Here we're going to check if the origin is annotated to allow the addition of descriptions to function arguments. If it is annotated, we're going to extract the information from the annotation and we're going to create a schema using a static method on the base type. And if meta exists, we're going to extract the description. Otherwise, we're going to perform a bunch of checks. The first one will check whether the annotation is in int or float. And if it is, the schema is going to be of type number. If the annotation is a boolean, the schema will be set to boolean. If the annotation is a string, it will be set to string. If the annotation is a dictionary, it will be set to an object. If the annotation is a list, it will be set to an array.

[00:04:36]Now, if the origin is a list, we're going to want to grab the types of the contained arguments. But below that, we can check whether the origin is a dictionary. And if it is, we will set the type to object. Down below, we're going to check whether the origin is a union type. And this block will handle optional types. And finally, we're going to check whether there is a description.

[00:05:01]And if there is, we're going to assign it to the key of description. And at the bottom, we're going to return this schema. And that takes care of the first method in the tools class. Now, before we move on to the next part of this video, I do want to mention real quickly that I have uploaded the code for this AI agent to GitHub. So, if you want to copy that code, you'll find a link in the description box down below, which takes you to my GitHub repository in case you don't want to type all of this out by hand. But moving on, we're going to create a method called schema for callable. And this will be a class method. Although I am missing the class method decorator. There we go. This method builds the open AI style tool schema from a function signature and annotations. It inspects the function parameters and their types, generating a complete schema that describes how to call the function. Inside here, we're going to use the inspect module to grab the functions annotations. Then we're going to create a parameter dictionary.

[00:06:10]And this will contain the type of the object, the properties, what's required, and the additional properties. Here we're going to loop through the items and try to grab the annotations for each one of them. If no argument type is supplied, we skip. This could be an error state in which an exception could be raised in a more complete implementation of this agent. But in the interest of keeping the code short, we skip below. for the parameters at the index of properties at the index of name, we'll use the annotation to schema method. Then if the argument has no default value, it must be a required argument. And finally, we're going to return a dictionary with this information. This tool type is a function. For the function, we use the functions name and we try to provide a function doc. If there is no doc, we just provide a default. Here we provide the parameters dictionary and finally we set strict to true which forces the agent to adhere to the schema. How this is handled varies from provider to provider. Some ignore it. And lastly, let's move this out of the for loop because that was an accident. This should not be inside this for loop. The next method we need to create is the get schemas method which will return to us a list of dictionaries and this will collect the schemas from the tools in the registry. So the first thing we need to do inside here is create a variable called out which will be of type list of dictionary. Then for every tool in the registry we will gather the schema. If it's missing, the tool was added incorrectly and will be skipped.

[00:07:57]Otherwise, we will return the list of tools. Then we want to create a method called register, which takes a function and returns a function. And what this method does is attach a generated schema to the function if it's missing. And it also registers it by name. So here we generate the schema if the function doesn't already have one and then add it to the registry and here we use the function name to register that function to the tools and return the function.

[00:08:29]Next we have a method called execute and execute runs a tool with the arguments provided by the llm. It passes the JSON arguments calls the function and returns the result. And we will always return a dictionary for consistency. And as a little disclaimer, this function could be more robust. This is the more trivially implemented version. So first of all, we need to get the function from the payload. First, we try to get the function. If there's no function, we return an empty dictionary. Then we try to get the name. And finally, we try to get the function. If the function's not found, we're going to return an error stating that the tool was not found. We then try and handle the tool call.

[00:09:17]Because we don't know what kind of exception could bubble up here, we will handle keyboard interrupts explicitly and then make everything else get reported to the agent. This will prevent the agent from crashing. This does make tools harder to debug, though. I highly suggest adding a proper logger to your agent. So here we use JSON to convert the types. Depending on the complexity of your argument handling, you may wish to include an object hook for this. That will allow you to handle more complex arguments. But for the most part, this should be enough. And believe it or not, that's all we had to do for the tools class. Next, let's take care of the agent class. The system prompt will be you are a helpful assistant. The model will be set to quen 3.5-9B.

[00:10:09]In the previous videos, I wrote Quen 3.5. And I just want to mention real quickly that this was not enough and that the only reason this was working was because I only had one model loaded on LM Studio. If you ever have more than one model loaded at the same time, you'll notice that typing in Quen 3.5 will not be enough. You need to use the full identifier name otherwise it won't know which agent you are referring to and I found this out when I was trying to use Gemma. Both the base URL and the API key will remain the same. Then we will define a tools attribute which will be of type tools. And here we'll create a field with a default factory set to tools. Then the rest will remain the same. Below that, we will use the post initializer block to remove any trailing backslashes from the base URL. We will then create a tool decorator which registers functions as available tools and we will continue using the context decorator. Then we will create the full agent loop with tool calling. Here we call the API, check if there are tool calls, execute them if there are, and loop back. we only return when there are no more tool calls. So just like before, we're going to append the user message to the messages. Then we're going to create the context content followed by the prefixes. Then we want to run the full agent loop until there are no more tool calls. So inside the while true loop, we're going to create some API quirugs. Then we're going to include tool schemas if we have any tools registered. If we do, we add them to our request. Below, we're going to create the URL and the headers.

[00:11:59]And just like in the previous two videos, we're going to create a post request, raise for status, grab the data from the JSON, grab the choices from that data. If there are no choices, we raise a runtime error and explain that the model response is missing choices.

[00:12:18]Otherwise, we try to grab the message from the choices. And if there's no message, we raise another runtime error that states that the model response is missing a message. Once we have our response, we can check to see if there are any calls. Then we add the assistance message with any tool calls to the history. If there are no tool calls, we're done. So, we'll just return the text response. Otherwise, for each tool call in tool calls, we will execute that tool call and add the results to the conversation. Because we didn't break out of the loop here, the agent will be reprompted with the new additional context. It could also be worth adding a max loops argument to your agent and incrementing a current loop variable so that agents don't call tools forever. This really depends on the kind of agent you wish to build. For agents designed to perform longunning tasks, it might be undesirable to add this. Next, we're going to create our if name is equal to main check and create our agent. The model will be set to Quen and the system prompt will be set to you are a helpful assistant that can perform calculations. Then we're going to add some context just as we did in the previous video. Below that, we're going to register some tools. And the first tool I'm going to register is an add tool, which adds two numbers together.

[00:13:47]As you can see, we can use the annotated type to embed extra information about the argument. I like to use this when I want to explain ranges, enums, and other things. Right below that, we're going to add another tool for multiplying. And finally, we're going to create a tool which returns a secret key. And with that being done, we can create a console and use the same while true loop that we've been using in the previous two videos. Here we grab the user input, check whether the user wants to exit the program, and if they don't, we continue with grabbing the response from the agent. Then we display the response to the console. And with all that being done, we can test the agent. So here we can ask it something such as what is 20 * 10 million? And I'm not even going to try to fix the typos because we are dealing with AI here. But as an output, we got 200 million, which is pretty cool. The assistant was able to understand what we wanted to do and use the appropriate tool for the job. We can also ask it something such as, "Do you have any secret keys?" And the best part about AI is that I don't have to fix any typos. Yes, the secret key is fluffy bunnies. So, as you can see, the AI agent can now effectively use tools that we define in our program. And this is not limited to simple math or silly secrets. You can create whatever kind of tool you want. You could even create a tool that organizes all the files on your computer. But personally, I'd be very careful with giving your AI agent any functionality which allows it to access sensitive information. But again, that's all up to you. And very quickly, I do want to mention what's happening in the while true loop in the chat method.

[00:15:47]So all of this we've already covered, but here we have some tool calls. When the model decides it needs to use a tool, it responds with a special message containing tool calls. We then extract the tool call information, execute the tool function, add the result back to the conversation as a tool message, and loop back and ask the model to continue.

[00:16:12]The model then sees the tool result and can generate a final text response that incorporates the tools output. This is fundamentally different from a simple chatbot. The agent is making decisions about when to call tools, which ones to call, and how to use the results. It's not just generating text. It's reasoning about actions and their consequences. My recommendation is to keep your tools simple and focused. Each tool should do one thing well. The model will figure out how to combine them. And there you have it. A fully functional AI agent with conversation history that persists across messages, dynamic context injection that updates on every request, and tool calling that lets the agent take real actions. And there are some upgrades that could be made to this, such as retaining conversations between sessions, managing context size so the agent doesn't slow down as the message history grows, and integration with other common agent features such as MCP.

[00:17:17]Anyway, thanks for following along with this video. I hope you now have a solid understanding of how agents work and how to build them. The patterns we've covered here, the agent loop, context injection, and tool calling are used in production AI systems everywhere.