Agentic AI at Scale: Lessons from Enterprise-Level Implementations

追加: 2026-05-21

[00:01:06]Okay, great. Uh Um hi everyone. Um thanks for being here. My name is Agnesha and my today's presentation is about agent AI at scale.

[00:01:19]So um at the beginning I would like to tell you a few words about agents in general because I assume that you come from different walks of life and um your uh knowledge about AI and agents can be quite uh different in your group. Um then I will go into um doing agents on Azure and generally on Microsoft stack.

[00:01:43]There are quite a few spoiler alert quite a few ways to do agents using Microsoft stack and I would like to um create a bit of clarity on what is possible and which uh tools to use. But I will focus on Azure which is the the framework which is the platform that uh I've been focusing on. And I also assume that you're um developers mostly or in some technical roles and Azure is Microsoft's cloud platform public platform uh public cloud and um that's why it's the most relevant aspect for you right um and then I will focus on addressing the most important blockers I've been uh witnessing and and and I've been helping a lot of customers enterprise customers mostly in um implementing the uh agentic platforms and the agentic solutions and the blockers that they have been experiencing are mostly the same um they have the same questions and um they raise the same doubts. So I would like to focus on that towards the end of this presentation. We have around uh 47 minutes currently.

[00:03:03]At the end, I hope there will be some time uh for you to ask questions. Uh most of you probably know me. Uh I work at Microsoft. I've been here for almost four years. I'm a data and AI architect with focus mostly on AI recently. I also do classic ML and I find that it frequently makes sense to combine the classic ML methods for example our own models with Gen AI. uh strategies and solutions. You can contact me by writing on this address or via LinkedIn. I publish quite a bit of news concerning Azure AI and related.

[00:03:46]Okay. So let us start generally um a lot has changed in the last months hasn't it? It's not like most companies currently work on their PLC's and mostly implement simple solutions like like chat bots that simply answer questions.

[00:04:05]We are moving from a single model model towards a multimodel uh solutions and we are moving from uh solutions from software that just answer questions um that answers questions that users provide by writing towards something more participatory, something that is makes a better impression and uh is easier also to to Also um just like two three years ago we frequently had this human in the loop approach and whereas it's still important and plays a role especially when stakes are high. We currently expect already our solutions to do a lot on themselves and only involve us when this is necessary. Right? We are moving from conversational interfaces from chat bots. At the beginning when Genai appeared on our agendas around three years ago, a bit more than that, every company wanted to do a chatbot. And currently we are moving towards operational intelligence. We are moving towards solutions that do things, not just chat, not just deliver response to to what we write. Right?

[00:05:26]So that's why we are currently focusing on agents because this new word are agents and agents are uh entities solutions that take a specific input normally. Um these are system events, user messages, messages from other agents and they are built using different LMS. They themselves have instructions and access to tools. They're using these tools to get access to the long-term memory, for example. They can check what they um talked about with a specific user um during the last session, but they have also access to actions. They can change something in our ERP system, in our um um content management system, etc. they can build new records in our databases, right? They can uh run uh SQL queries and in this way they produce some output.

[00:06:35]So basically AI agents um are entities are units are tools that reason act and learn and um there are different way to structure them right so we don't always need an agent although currently it's it's very trendy to to employ agents but um in in the easiest in the simplest cases we absolutely don't um the easiest Best way to implement agents are uh single agents. So um we have an agent that has access to a specific search index. So it can search information uh that it needs to answer queries for example from users and then we have it has also access to another for example LLM as a tool right and that's how it formulates the answer but aentic systems can also be super complex of course so we can have this coordinator agent that has several subordinate agents that uses them to get information to validate information and um then to to get to the result and the result can be an answer but it can be also doing something right and the number of the structures these schemas these diagrams that we can build is uh of course super high super huge. Here you have an example uh of a stock market stock analysis agent. I will show you an example that I built for something very similar um in around 20 minutes. So you have the main agent, the boss, the manager, and then you have um the subordinate, the secondary agents that perform something for the main one. So um if I'm asking about uh a sentiment about a certain stock for example this agent will probably use this subordinate agent sentiment analysis agent. So it will get some news some social media and it will tell me what the public thinks about a specific stock about the specific equity right.

[00:09:01]Um, we could also add even a second layer here of agents.

[00:09:10]It it depends on you how complex you want to do it and how much you want to control how this functions.

[00:09:17]Okay, but let's take something super easy just to give you an example of an agentic system. Let's assume that we want to uh automate our contact center.

[00:09:29]contact center that um manages emails from customers that come to a specific company. Right? So at the beginning, let's say that we want to have a categorization of this email, whether the email focuses on some technical aspects, commercial aspects, whether it's an escalation, maybe it's a general question about how long uh our stores are open, right?

[00:09:59]And then after this categorization we analyze the sentiment whether the customer the user that wrote us an email was very angry or not really. And based on the answer of this first categorizations we forward the query we forward the email to a specific agent here which also have access sometimes to specific tools.

[00:10:22]So just focusing on some of them.

[00:10:25]Um, how do the agents differ? The first one, the categorizer could probably use a very simple, very cheap model because it's it's an easy task to categorize emails in one of four categories.

[00:10:42]Whereas those agents that answer questions should be based on some more advanced models. GPT40 is an older model but but it's still very good for many many cases. We don't need GPT 5.5 for every use case. Right? Then look they also differ in the temperature and in other parameters like max tokens. So how long should the answer be? The role definition is different because they are supposed to do different stuff, different tasks. Like the first one um is a customer contact center expert and uh its role is to categorize the below email as one of three categories.

[00:11:28]Whereas here you are a technical expert specializ in da da da. You address technical questions from customers. They also have access to different tools because they need different tools to do their jobs and then to different context.

[00:11:43]Okay. And what I mentioned, you can order that in plenty of ways or you can have a manager that selects the um collaborators, the employees that he that it wants to use in order to address a a query, whichever you prefer.

[00:12:05]And um when we are talking about implementing agents there are some main considerations.

[00:12:13]One of them is to um imagine a good process.

[00:12:18]So when I meet with my customers normally it's the first step I ask them so how do you imagine the the process that the agent should follow because in this way we can create our um agents define them appropriately and the relationships between them right and the knowledge they have and the tools that they have at at their disposal but during today's presentations I will be focusing mostly on this on security and develop valuation because if you want to create a solution that goes beyond a simple P you have to do that you have to focus on that so this basically tells you whether your solution makes business sense at all um but the other things the nonfunctional requirements linked to security and evaluation um are super important to to secure the quality and the well the security of solution. Okay, so let us spend the next maybe like 10 minutes discussing agents on Azure and how you can do it. But before we focus on Azure, remember that there are many ways to do agents using Microsoft stack. You can use it out of the box. So uh you have Microsoft 365 copilot uh and then you have the low code way agent builder copilot studio and only then um you have the proc code way Microsoft foundry and other ways like putting your uh open source model on a virtual machine and then using the model to create your agents with lang graph or lang chain right and Um here I I included just two screenshots from um simple tools that are available to everybody that can help you u decide which of the stacks I just mentioned is the most suitable one uh for you.

[00:14:30]If you can't find it after this meeting I will be happy to share links.

[00:14:35]Okay. So just to show you how it can work um I mean in this case Microsoft 365 copilot this is one of the agents available there it's called researcher and you can ask you can ask it to to write you something more elaborate than than simple answers and it will do that. Uh, it normally plans its work. It asks you questions. I will speed it up slightly.

[00:15:17]Okay.

[00:15:19]Here you see the stages. So, it plans its work. Um, it can take even a few minutes because um, it can produce even an analysis of 10 of 20 pages, right?

[00:15:32]And this is something that you simply buy. You buy a license, you assign it to your um team and then immediately after a few seconds they they can use it and they can um work with it. Here um the important news is that we've launched co-work which is uh a great new agent that can help you to create presentations. For example, just before this meeting, I actually asked it to create a presentation for me. I'm about to present tomorrow morning uh something uh on the regional level at Microsoft and I wanted to check how it would um manage that and it did it pretty well.

[00:16:17]So I just changed a few things and it's it's a readym made presentation for me. A great thing. And then we have the low code way.

[00:16:33]Let me just skip a bit to show you how it can work. um you basically have these boxes at parts of the process and uh you order them in a specific way in a specific sequence in order to pinpoint what should be done. Um you normally don't have to code in some places you have an opportunity to add uh code snippets to um for your agent to do specific things but it's not necessary to know how to code. So if you have something standard to um automate then um copilot studio may be your friend. If you want for example to build a solution um in the vein of talk to your data or a simple rack then copilot studio may may be a good solution if it's uh something standard. If you want to customize it to a high degree go for the proc code solution.

[00:17:39]Okay, let me finish that.

[00:17:48]Okay, I can't move to the next slide, but that's not the problem. So, let me Oh, I have it here.

[00:17:58]So, moving back to to the overview a bit. Um there are several ways to to to build uh agents using the proc code way.

[00:18:09]Um the one I mentioned just a bit is to use our compute and then to build your agents that that use it using some framework like um Microsoft agent framework, langraph or or similar. Um but the one that we are actually promoting and which offers you a lot of advantages is this one in the middle. So it's using Microsoft Foundry and even in Microsoft Foundry there are several ways to build agents.

[00:18:41]Um there are two foundaries actually the old one called Azure Foundry and the new one Microsoft Foundry. And let me show you uh how to do that first in the old foundry because I know there are still plenty of companies that use this functionality.

[00:19:01]This is a screenshot from it. It still exists. It just it's not developed anymore. Uh we um encourage migration to the new version. So um you are able simply to define the model that the agent should use to define instructions on the system message add specific knowledge and actions meaning tools and uh let me give you a short demo of that parts of it are in Polish but that's the only part in Polish that I have in today's presentation I build that based on uh the called Azure uh Fundry and I wanted to build an app that will answer questions based on my data. This is the data that I will be using as my test set basically. So let me add the file which is an Excel file as you see and I will be speeding it up a bit.

[00:20:01]Yeah. And it actually tried to open it.

[00:20:04]At the beginning it didn't uh work out but then it managed to do that.

[00:20:12]and I can ask questions about that. So it can use code in the background in order to to show me what's in the data to analyze and to to answer questions based on that. Right? It can also for example create diagrams.

[00:20:37]Okay. And uh it can also send emails for example.

[00:21:05]Let me stop this because I'm not sure if uh voice and uh um if you can hear what what is being played.

[00:21:16]So let me speed it up a bit.

[00:21:19]Um and let me show you what was behind that. So um I created the whole app of course uh using code. I created an in using Python but um I created in Fundry. So u I could see all the tools here and all the knowledge for example and functions that I added to it here.

[00:21:46]One of them was send email to Agneska.

[00:21:49]Right. So uh Fundry became the centralized place to create observability and u where you were able to centralize the creation of your um agents and this is something that has been continued in the new u Microsoft Foundry. We currently encourage migration to the new experience to the new Microsoft Foundry. You can find uh very detailed explanations how to do that online.

[00:22:22]The new foundry is a bit different. So keep that in mind. Um but it offers you even more than what you had access to before. So currently when it comes to the um agentic stack, we offer workflows, declarative agents and hosted agents.

[00:22:48]So basically when it comes to Microsoft Foundry you can use whichever framework you would like to use including Microsoft agent framework including a lang graph lchain yama index or any other actually but foundry is much more than that foundry offers you tracing logging monitoring evaluation experimentation and It simplifies doing that by offering you this interface. You can do everything in Fundry by using um APIs.

[00:23:24]We're using SDKs, but you can also just click yourself through it, which for some customers is important, right? And it also offers you, for example, plenty of MCP tools that you can use, not to mention a huge number of models that that you can leverage. um many thousands of models. So just to show you what it looks like declarative agents here is an agent like that.

[00:23:56]It's the simplest way to to create agents in the new foundry. So um you can define it here somehow. This is the message. You can add tools and there are quite a few as you see.

[00:24:21]You can add memory. It's about the long-term memory. So the memory between sessions guard the race of course. So um you can define whether you want this agent to be bound by some rules concerning sexual self harm etc content or not and you can test it and what can you help me with?

[00:24:54]You can also define here what traces you want to uh collect and this works by connecting you to app insights.

[00:25:05]Monitoring and evaluations are also um important parts of it. Here in monitor you have for example information about the tokens but also about the runs and other metrics which you might need.

[00:25:24]Um what is also there are the workflows.

[00:25:29]Using workflows you can um create specific processes reflects probably processes from your everyday life. Um in um also here um when you can uh employ agents to perform parts of the process but also connected with the classic uh code coding u elements like if else conditions and similar right you have some examples here that you could use or you can start from scratch. The good thing about it that you can always see it also as as code in this way you can be for example um do versioning and you that's why that that's not really um a low code solution although it might seem so okay and moving on you can also have so-called hosted agents Um, I created a video on that uh a few weeks ago and if you would like to go deeper into what hosted agents are then you can uh watch this. You you are welcome but just to um tell you the most important part. You normally develop your hosted agent as code in your code editor. Then you create a docker image.

[00:27:01]you push it to the Azure container registry and then you register your um agent in Microsoft Foundry based on a link towards this specific container registry. Why would you do that?

[00:27:21]uh because in this way you can publish the agents very easily to your channels such as teams and Microsoft 365 copilot.

[00:27:32]So um you simply click on this.

[00:27:42]Okay. And this gives you the opportunity to publish the the agents. I created two agents like this.

[00:27:51]um August travel agent and AGAS uh stock agent. I developed them uh in my visual studio code of course and then I followed this process to get them into the foundry which gave me governability observability but uh also the possibility to to publish the the agent here on teams and on Microsoft 365 copilot. I gave both of them access to a lot of MCP servers so that they can for example tell me what the price for specific stock is currently now in this very moment.

[00:28:34]Okay. And uh what this exercise also gives you is the control over the life cycle of your agents. So every agent has a specific status. it is live or um it is uh not activated for example you have access to that um simply by clicking yourself through Foundry but also uh programmatically okay let me skip that um you can use a lot of different frameworks in order to create hosted agents and push them afterwards towards Fundry you can use Microsoft agent framework which is not in public preview anymore for for a few days it has been in G already and you can also use lang graph and it sometimes makes sense even to combine the two I I'm a huge fan of lang graph I I find the way it works and especially how it tackles memory uh quite cool but if you want to use something totally different that's possible too okay so let us spend the last 20 minutes focusing on scalab ability and focusing on is security and governability.

[00:29:53]One lesson that uh I learned while working with enterprise grade customers was about the importance of guard rails and security in general. Um this has changed a lot in the last months. Um before that most customers were um on this stage of experimentation so they didn't care that much about them the agents being secure but currently it's the absolutely most important thing for most of them.

[00:30:27]Um so when creating a solution uh an agentic solution AI solution please keep in mind that it's abs absolutely crucial to analyze what can go wrong at different places right and this includes um prompt injection attacks direct indirect prompt injection attacks but also tools m misuse supply chain risks data poisoning Um and the attacks are getting more and more complex with time. So it's super important to test your uh solution using simple attacks but also those which are more complex like when it comes to prompt injection the crescendo type of attacks and depending on what you are trying to build is this a rack solution or something else. Uh these risks will be different. Of course, you should analyze it for your specific solution.

[00:31:31]Um, what fun gives you is the observability plane. You have to set it up slightly. Just add, for example, the the the metrics that you would like to have and configure the the gut rays that are most important for you. But afterwards, you get this view.

[00:31:54]Um and remember about this new um a agent 365 view. Uh this is the place where um you are supposed to see all your agents and decide who has access to what on the level of the whole organization and not just you know a fragmented view of what some people did. So uh it's supposed to give you the the full control over everything agentic that is happening at your organization.

[00:32:27]When it comes to evaluations and guard rates generally um these are the most prevalent the most popular ones for agents.

[00:32:36]So um you can test intent resolution for example intent classification whether an agent uh correctly identified what the user intent was. You have the tool agent um so basically you can u evaluate single step accuracy so whether the steps that were taken for specific case of a user's interactions with agents were correct or not.

[00:33:07]um and response completeness and similar you can also assess that um these are evaluators which are available for you out of the box. So you don't have to define that. You just connect to your API. You define the model, the LLM that you want the evaluation to use because it's the LLM as a judge approach of course and and then you have your results and additionally you have all the evaluators which are uh also used in the agent context. So uh you can um you you can find out how good the the responses are in terms of fluency, coherence, crowdedness. So whether the responses from your uh agent are based on specific sources that they should be based on.

[00:34:06]There are plenty of evaluators uh in terms of risks and safety of course and this one we've already discussed that um what is also an interesting thing is that we've added Asianic uh ids. So every agent that you create in the foundry has its own uh ID in entra just as people do. And in this way you can see your agents but you can also create some conditional access policies for example right you can audit what the specific agent did so it provides you even more observability.

[00:34:48]Um, one thing that is super important to remember is that you can switch off some of the security features like if you are okay with self harm or violent or sexual content. Um, this type of queries being made to your agent that's completely fine. But there is abuse monitoring which is always there and you cannot switch it off. you can modify it. There is a form online which makes it possible for you uh to um limit it. So normally abuse monitoring is done in an automated way and then afterwards by humans. So if you don't want to have this human review for privacy for example reasons that that's fine. You just fill out the form but the automatic uh abuse monitoring will always take place. So if you are planning to send plenty of very violent content for example to the models which are hosted on Azure to GPT 40 to GPT52 etc. Um be sure to think about that because if you send too many of such prompts and if they are too radical for example questions about some hemical weapons etc you should be careful about getting blocked. So it's better that you talk to us and you ask how to protect yourself. Um so this is this frequent misunderstanding about content filtering and abuse monitoring on the other uh hand.

[00:36:37]Then um another thing that I frequently observe as critical but frequently not given enough attention is about storing locks and storing locks and metrics will save you so much headache. Um, and you should really have information, have logs and have metrics about every step of your process. So, this is a super easy uh schema for a rack architecture of course. So we have our data sources and then after a user asks a question, it's probably a chatbot or voice bot.

[00:37:17]Then we first search for the passages for the chunks that most accurately address the question and then the LLM takes the chunks and answers the user's questions based on these chunks.

[00:37:32]And what I observe a lot is that this part is there. So, um, teams actually store the prompt that was directed to to the LLM and store the LLM's response. No problem. But frequently, we don't store what happened here. And this is critical because we we have to know which information was uh searched for in our AI search or some database where our chunks chunks are.

[00:38:05]It's not trivial at all because normally the user's query doesn't go directly towards the knowledge base. Normally it's paraphrased. We want to search something different than just copy what the user said. Also, uh this part of the process is getting more and more complex. So we currently have agentic retrieval um and we've frequently searched for a content in several knowledge bases for example. So we absolutely need to know what happened here.

[00:38:44]Also here um this is um this is a baseline architecture for uh chatbot.

[00:38:53]Um there are three of them that you will find uh on this website actually. Um and this is a bit more complex one not the basic one. Um I copied it because here you see Azure Cosmos DB Azure Coosmos DB which is frequently used to store uh the history of the conversation of your um of your applications.

[00:39:18]So um look here is the foundry and um in some cases we tend to think that you know Microsoft foundry is the most important thing when it comes to building genai solutions based on Azure.

[00:39:33]Well it might be the case but apart from that we need so many other things we need to think about security. So private endpoints we don't want to communicate uh over um the public internet for example right um we have the app layer we have the search of course we have uh key volt to store our secrets and we also have this Azure cosmos DB to store the whole conversation the the whole history of the conversation and let's not forget this is the typical um heaven and spoke approach so this is a landing zone based chatbot Here we have the hub. So even more of those things that don't have a functional role, they aren't visible from the users point of view, but they are critical when it comes to securing the the security of our uh solution. Right? For example, this private DNS uh for for private link.

[00:40:37]Um what is also important when you are going live is to calculate your demand the the baseline and the spikes. What I mean by the baseline is normally when you launch a solution and if this solution isn't for example an internal automation uh but something that is customerf facing then uh you will have some traffic but you will also have some spikes after for example some uh commercial for your solution runs on the TV or after you um increase the number of users by inviting new people um to to use your solution uh and not just family and friends.

[00:41:22]And in this case, it makes sense to remember that we are past the time when we just could use the standard pay as you go deployments. Currently, there are also so-called reservations.

[00:41:37]And in many cases, it makes sense simply to make a reservation for how much tokens, how much requests you need for the baseline and cover the spikes by pay as you go. Right? The other options here are also interesting in some context but not not precisely in our today's context.

[00:41:57]Um yeah and um in this context it makes sense also to mention API management or API how we call it in many our um reference architectures we have that for example here you have Azure APIM here you have Azure um AI gateway in APIM etc. it makes sense to uh consider using that because in many cases if you want to scale up your solution, you will need to deploy your endpoints especially endpoints in several regions or and using reservations and pay as you go and then you have to create some logic behind that. for example that your app should first be using the PTU the reservations and after they are used up the capacity is used up app up you go uh towards pay as you go right or uh you want to do round robbin between your main endpoints for example you have three PTU uh endpoints and then after they used up go towards pay as you go and APIM makes it possible for you to um implement that easily.

[00:43:19]It has plenty of more interesting um capabilities here.

[00:43:25]It's becoming this centralized thing where you have all your MCP servers for example and access also to to other agent via um A to A. You can transform um every REST app, every REST API into um an MCP server easily using APIM. But let me now not go into that because we don't have that much time. Just remember that APIM may help you in many cases if you are going into production especially.

[00:44:00]Okay. And another thing that I'm frequently asked about is about improving latency.

[00:44:06]That's especially important if we are talking about the agentic word, right?

[00:44:10]Because um well um just three years ago um we normally worked with just one agent with just one LLM. So uh a user asked a question, the question was typically forwarded to an LLM that replied and that that's it. But currently there's normally a whole process. So um there is the categorization of the question, the rewriting of the question at the very beginning and then there's so much more.

[00:44:42]I showed you at the beginning um using the example of the stock analyst agent that you can have five agents just analyzing basic stock information for you and that's just the basic version, right? And every each of these agents basically adds agency latency if if this is how the the process goes after the the the user um asks a question, right?

[00:45:14]Um and one way to address that is by leveraging caching. There are several ways of caching that we normally talk about in the context of LLMs and Genai.

[00:45:25]There's this traditional caching that kicks in when we ask exactly the same question. It's less important in the context of genai prompt caching that kicks in automatically, but you can change the way it works to some degree.

[00:45:41]For example, how long the the cached uh prompts are stored and it's important to look into that because you can save so much money using that.

[00:45:52]So much money and so much time. And then there is semantic caching which kicks in uh when a user asks a similar question to one that was already asks asked and u this is something that you implement yourself.

[00:46:10]Um let me skip that.

[00:46:13]Yeah. So um once again why caching is important because this is an example from lang chain. Um we asked the same question twice and at the beginning uh it cost us 20 uh 25 prompt tokens and over 200 completion tokens. The second question, the same one asks after a few minutes cost us nothing and was much much faster because it was simply returned from the cache, right? Um, by using APIM, you can add radius very easily, the managed version of radius, which which means that you can leverage semantic uh caching.

[00:47:02]Here is an example using this exactly same question twice. And you'll see that the second uh answer, the second reply was delivered, was returned very quickly because it was cached. But the cool thing about semantic caching is that it asks for similar questions. So it analyzes the s the the point of the question what is there and not just the form of it the string doesn't have to be the same just what the user is asking about should be similar for example here the first question was I'm starting a new brand of beer propose some creative names and the other propose some creative names for a new brand of beer and you see that the first question without semantic hashing took over a second to return and the second one formulated slightly differently.

[00:47:59]It just took us 10% of the time. Um you can uh perform semantic caching using several databases also Cosmos DB. here.

[00:48:09]Two questions that I asked it were were and it's the same thing but formulated a bit differently. And look, the first one returned an a reply after 15 seconds and the other not even a half second because semantic caching kicked in.

[00:48:36]That's everything from my side. Uh, thank you so much and if you have any questions I will be happy to address them.

[00:49:06]This is a very interesting question and we could go on about it for um a lot of time to to be honest. I'm also not an expert on that to be frank. So uh I can recommend you maybe some some good good sources to deep dive uh into that. Um yeah.

[00:49:55]Look what I normally do uh when creating an agentic process sorry um is I ask business how the process is performed now because just as agents shouldn't uh disturb one another also the people shouldn't right because this could lead to to conflicts.

[00:50:20]Uh so this is our starting point but this is not the the final process that we want to then implement using agents.

[00:50:29]This is the starting point and then after we already have the diagram who talks to whom and how the role division takes place we normally think okay um is everything efficient? Could we for example get rid of this step because it's a duplication of something which was before or um does it make sense maybe to move one step towards the end of the process because um then we have an agent that have has a similar skill set and can address that. Right? And by doing that you can build something you can ideate something that makes sense and um yeah u it makes sense definitely to talk to people who know the process well. So uh as an IT person most of you pro probably are from IT you aren't in a position to define the process yourself. you have to have specialists that know how it goes and what is important, what the input and output data should be. So, um it's similar to creating every other process really. Um you iterate, you talk to stakeholders that know a lot about it in order to to optimize what you have and it's always based on iterations. So even if it's not perfect at the beginning, if it adds something to your existing processes, if it makes them faster, for example, that that's normally enough for it to uh be deployed in production at least for specific people. And afterwards, after they share your observations with you and they uh share with you what they don't like about how it's implemented, you can then iterate and add new uh tools. You can increase or decrease number of agents etc. So I I think about several stages. Uh to be frank um in a typical process of uh application development uh you should have a product owner for example, right? So you should have somebody who knows uh the process and who can imagine how the product should look like and who talks also with other stakeholders to confirm with them uh what is expected and not not really.

[00:53:32]Um when it comes to autonomy um you cannot generalize a lot. It really depends on the specific application uh and uh whether you need a human in the loop or not. Um human in the loop is costly from the point of view of time and money actually if an agent has to wait for for a human for approval. So u if it's possible to to avoid that and it's normally possible for low stake processes or where it's easy to um to find mistakes right that that's normally the the way to go. Uh but it doesn't mean of course that everything that agents do should be simply accepted.

[00:54:23]when it comes to decisions especially that are important that involve people then then we should also uh we should always conduct audit. So it really depends on the business case and this analysis should be done at the beginning but then um the product owner should also uh control whether the the autonomy is enough but not too much. And if this question was also about the technical aspect, then you definitely should control uh the number of tools that every agent has access to. Uh and uh you should audit what it does afterwards, right? So the tools what it can do with the tools.

[00:55:14]For example, when we are talking about GitHub copilot, we can give it access to our GitHub repos but uh just to create pull requests for example but but nothing more not to delete our reposets etc. And I also mentioned during this presentation that you now have agent ID in your entra which makes it possible to create conditional access and and similar policies reducing um what agents can do if this is needed on your project.

[00:55:57]Thanks a lot. This was the last question. Have a great rest of the day.

[00:56:01]Bye.