Multi-Agent AI Book Dropped on Github: Here’s How to Build It Locally

[00:00:00]If you want to build advanced multi-agent systems, then this repository will hand over every development utility to you. It was built by an ex-Microsoft researcher, and it covers everything from local deployment to multi-agent swarms without holding anything back. And using this repository, I'm going to be building my own local agent system, which will use the three main engines from the repository. Hello everyone, and welcome back to Build with AI series, where I cover local and free AI tools to build practical AI projects. So, now let's first cover what makes this new repository so powerful. This open-source repository is actually based on the book, which is AI Agents in Depth, which is a brilliant book for designing principles and engineering the AI agents in real life. And this project contains the full text of the book, and it also gives you the accompanying example code, and everything is fully open-source for you to try. And this repository is built on a simple formula that any AI agent is just an AI model plus the context that we give to it and the tools that we add to the AI model so that it can connect to external sources. And this repository contains all these agent fundamentals in detail, actually comparing the traditional reinforcement learning with the LLM-based AI agents. And here they highlight multiple agents that you can directly use, like the web search agent, which uses the Gemini 2 model as the agent and has basic deep search capabilities to search the web application for any information it needs. Then there are other agents readily available as well, like search code gen, which is based on GPT-5, then context, which takes the context application study given in the book, and can take multiple LLM providers to see how the different context modes actually change the behavior in the agents. Next, we dive deep into the context engineering, where the context defines the upper bound of our agent capabilities, and these tools actually delve into the context structure of the APIs. And here it also shows you how to design the KV cache, where it shows the impact of different context management modes. So, by incorporating these tools, you do the context design in a proper manner, which significantly reduces your API cost. Or if you are using local AI models, then it will reduce the memory requirements of those models. Then there are ready-made security tools as well which check the prompt injection, where it uses deterministic rules to calculate all the tax successes rates. So you will see how layered kind of defenses actually save your agents from the harmful injunctions. In chapter three, they focus on the user memory and the knowledge bases, where they enable the agents to remember users across multiple sessions, and also cover the basic rag pipelines and deep AI knowledge organizations. So here everything is covered like embedding, which basically converts the raw text into some numbers vectors, which the AI can then use to check the different meanings and the closeness of words and sentences. Here you will see the implementation of agentic rag systems, where it compares the non-agentic rag with the agentic framework, and the agent leads iterative information retrieval using the react patterns. React just means like the agent should reason first before acting on the problem statement, so it can properly answer our queries or do the necessary action. Then there are advanced context retrieval techniques as well, which is contextual retrieval, where it generates prefix summaries containing the core text context, which reduces the retrieval failure rates by a massive margin. So you can see here that they reduce the retrieval failure rate by around 49 to 67% in the AI agent pipelines. Then we will focus about the building and using the agentic tools, where tools are essentially the working hands of an agent, and it discusses the tool classification and the general design principles to add these tools to the AI agents. So we get to build the perception tools MCP server, which provides the capabilities for web search and the multi-model understanding, and most of the features are based on free and open APIs. It specifically uses open APIs like DuckDuckGo, Open Meteo, Yahoo Finance, which require you to provide no API keys at all. And there are tools to build asynchronous agents as well with support parallel execution of the agents. Traditionally agents used to execute their actions in sequence. So stage two waited for the stage one to complete its execution even when both of these were independent. But with asynchronous agents we can run multiple stages in parallel. So let's say if a user is providing an image and in the query they are asking some web queries, then both the task can run in parallel where one agent will understand the image context and the other agent will fetch the web queries. Then there are other chapters as well like how the coding agents are built and how we can use the code to generate those agents by itself. So in a way these are meta capability of a general purpose agent.

[00:04:29]There one production grade coding agent is used as an example to do the complete implementation of the same powerful general tool. Then there are tools to evaluate the final agents, how to do the post training of the LLM models, how to turn on their multimodal and the real-time interaction capabilities, and finally how to build multi agents team that can collaborate with each other. So in the main repositories all these tools are mentioned in the sub folders for different chapters. So you can explore any of them and get the ready-made Python utilities yourself. For me I'm going to be building my own agent harness system which uses three of the key utilities from this repository. For my coding environment I'm going to be using Google antigravity. If you don't have it installed just go to antigravity.google/download and download the installer according to your own platform. Then to integrate those concepts to a local AI brain I'm going to be using the local LLMs from Ollama. To download the software just go to ollama.com/download and download the installer according to your own platform. And once we are in Google antigravity, the key concept of my project would be that I'll be incorporating three different tools from the repository. First is the agent engine which will have the react tool loop which I've already shown would be an agent which would reason first before acting on the problem. Then there will be a context manager which will show how to manage the KV cache which handles the user conversation history with the AI agent using the same design principles that we have mentioned in the repository. And last but not the least, I'll be checking the memory design patterns where we'll use our RAM combined with the rag architectures to save the user and agentic interaction summaries. And finally, the outputs given by all these systems would be reviewed by a proposal reviewer system, which is a two-stage safety loop again according to the design that is specified in the repository. So, the backend of the application would be handled by Python. All the API requests to our application will be passed through the fast API and the unicorn servers. And for the front end, I have some HTML and CSS templates which have some ready-made designs that we can incorporate. Then for the official repository given by Voy J Lee, we are going to incorporate 16 different pure Python tools that will be used in different sections of our project. And all these tools are directly mentioned in this agent code.py file. We have We are going to copy the entire repository given by Voy J Lee and then going to incorporate all the agentic tools that we need using Python sys.path.insert.

[00:06:45]So, any Python classes and the modules that are defined inside that repository will be directly registered to the Python environment so we can later on use it in our codes. For example, we can use a system state class, the tool registry class, and all of these are not inherent in Python because we are copying them from the official repository. And using the tool registry, we'll get all the 16 Python tools that we'll be needing down the line. Now, all the LLM requests will be passed on to the Ollama models. Here, we'll just take the user prompt that is given in the application, define the payload using the Ollama model that we have defined on top. Here, I'm just taking a 1.5 billion parameter model from a Q12.5 family, but you can replace any of your model name here and just install it by running Ollama run and the model name. So, all the AI requests will be sent to these Ollama models, and we'll use the request libraries to send all those requests to our local Ollama URL. Then we are defining the different components. Like first was the agent engine which uses the react pattern and will incorporate the context and tools inside our local LLM model. So, for the local LLM model, it is taking the Ollama model that we have defined. For the context, it is taking the current user history. And for the tools, it is taking all the tools that we have incorporated from the repository. So, now it will do the react reasoning and the tool selection using the Voyager 16 tools. If there are any grep and search tools, then we'll directly execute them to do any kind of web searches that is needed for the user query. Then, if the user wants to list any workspace files, we'll use the list tools to list all the workspace files using the official LS tool. And otherwise, we are going to directly prompt the model for the direct reasoning response. So, all the tools are now added. And now we need to add the context using the self.history, which will save all the user and assistant conversation history. So, here whatever response is given by assistant will be added as role assistant. And finally, we are going to save the result in a markdown file as well. So, we can check out the full agent response along with all the tools that it has used. In the markdown file, there will be simple headings like these are the Ollama LLM and the tool metrics. The core formula is that the agent is using this LLM model with the context and tools. The active model is this from Ollama local inference. And these are the official tools that are loaded from the repository. Then, using the react execution, we are going to take the user query, show the full reasoning or the thinking trace of the AI agent, and finally display the act part, which is the direct action, which is our final response. So, finally, we'll just open the output.md file in a write mode and write all the contents in this file only. Then, in app.py file, we'll have our main application logic with the title of application as the deep agent engine. And the front end will be loaded using the ginger templates on the HTML files that I've kept in this template subfolder. So, using the main landing page, which is this index.html, I'm going to directly launch this as an application in my local browser. And once the user gives any query, we are going to launch the agent.process_turn, which will load the agentic pipeline that we have defined in the agent code dot py file. And once the user executes this app.py file, the application will be launched in our local browser in this URL and host port. Now, to launch this application, you need to run some simple commands in the terminal of the Google Antigravity. First, we need to fetch the official repository from Voyager AI. So, our backend code can access all the tools that are defined in this repository. So, I'll just copy this command and paste it here and press enter. For me, as I've already done this cloning, the official repository is already there in my project folder. And we are going to install all the dependencies, which will be needed to launch our final application. So, again, I'll paste this command here and press enter. And now we can launch our final application using simple python app.py command. Now, our server has started on this local URL, so you can just press control and click on this URL. So, now my application has launched in my local browser. I can give any of the query that I require, and all the engineering is based on the AI agents in the depth book with the LLM model selected as given 5 1.5 B. And there are 16 active tools, and all the entire reasoning trace will be done using the react technique. Now, you can give any query that you want as an interaction to the AI agent. For example, I can just use the grep tool to search all the coding related tools. So, just press send here, and now it has given me the full reasoning trace that it has connected to the local Ollama model. Then it has invoked the official grep tool from the Voyager AI's repo. And using the agentic pattern using the grep tool, it has found 16 tools registered in the chapter 5 coding agent repository. So, for now, I've just shown the agent workspace section, but similarly, there are other sections as well added in the code, which will show you how to handle the context using the KV cache design, how to save the dual layer memory of the users and agents, and finally, the proposer reviewer system for reviewing the final agentic output. The same way you can add more chapters and more tools from the same official repository. Like here, I've just added the major tools from these four chapters. And I can also check that the output.md file is also saved in my local folder, which shows the core formula that we are using using the Qwen 2.5 model. And the official tools loaded are the 16 void J tools active. And the user query and the thinking trace and final response and output are also given here. In the readme file, I've also listed the different use cases where we can use this kind of project and some future features that you can add on top of it.

[00:11:53]Let me know in comments if you want me to cover any of these use cases or features or to cover any more chapters in detail from this repository. But for now, this was a detailed deep dive into the AI agents in depth repository and my custom engine harness that I built using it. All the code files and the readme file with step-by-step instruction on how to set up the project will be available in GitHub link in description and pinned comments. So, you can directly copy them and get started with the project easily. Let me know in comments if you have some suggestions for me to build more local AI projects.

[00:12:23]And follow the channel for more such AI related updates and workflows.