How do I troubleshoot latency and optimize Amazon Bedrock Agents performance?

本站添加: 2026-06-06

[00:00:00]Hello, I'm Nandini, a technical account manager here at the AWS office in Northern Virginia. Today I'm going to answer the question, how do I troubleshoot latency and optimize Amazon Bedrock agents performance? Let's get started.

[00:00:18]Amazon Bedrock agents use reasoning capabilities to retrieve information from knowledge bases which can sometimes result in high latency. A key metric to understand here is time to first token or TTFT.

[00:00:34]That is the elapse time from when a request is submitted to when the first output token is returned to the user.

[00:00:42]The factors that affect TTFT and overall response time include the model that you select and its size, the loop iteration and pre and post-processing templates, knowledge based retrieval latency, prompt structure and complexity, the number of input and output tokens, the network connectivity and AWS regional infrastructure to troubleshoot ute high latency. Let's start with the model size.

[00:01:16]If you're using large foundation models and experiencing higher latency, then use lighter models for latency sensitive use cases. Invocation latency increases with the combined count of input and output tokens.

[00:01:34]To check which foundation model you're using, open the AWS management console and navigate to Amazon bedrock console. In the navigation pane on the left side, choose agents and then select your agent.

[00:01:54]Select edit in agent builder and review the model field under agent details.

[00:02:01]Note the currently selected foundation model. To change the foundation model under model, choose the drop-own list and compare available models.

[00:02:12]Smaller and faster models include Claude Haiku, Amazon Nova Micro, Amazon Nova Light. Larger and slower models include Claude Sonet, Nova Pro, Llama 3.145B.

[00:02:29]For latency sensitive workloads, select a smaller model. Smaller models can be two to five times faster.

[00:02:38]Before you commit to a change in production, use a playground to benchmark response times across models.

[00:02:46]To use the playground, in the navigation plane on the left side under test, select playground.

[00:02:54]Then select your model.

[00:02:58]Next, let's check the loop iteration and pre and post-processing templates. In each agent orchestration cycle, the reason and act or react loop involves at least one full model invocation. Complex tasks that require multiple tool calls, knowledge based lookups, or reasoning steps multiply the total latency by the number of iterations. Deactivating unnecessary pre and post-processing templates can reduce the agent response time for simple use cases because each template elimination removes an entire model invocation from the critical path.

[00:03:41]To check the number of orchestration steps on the Amazon Bitro console in the navigation pane on left side, choose agents and then select your agent.

[00:03:53]On the agent details page, the test window appears in a pane on the right.

[00:03:59]If it's closed, choose test at the top of the page to reopen it.

[00:04:06]Choose expand prompt editor to show trace.

[00:04:10]Submit a representative query and expand the trace.

[00:04:15]Count the number of orchestration steps.

[00:04:18]Each steps adds a full roundtrip model.

[00:04:25]To resolve this issue on the agent details page, select edit in agent builder.

[00:04:32]Click on edit in advanced prompts.

[00:04:38]Go to the orchestration tab. Turn on override orchestration template defaults. Select confirm.

[00:04:46]In the prompt template editor, simplify the orchestration prompt to reduce ambiguity that causes the model to loop.

[00:04:59]Remove verbose examples that prompt unnecessary multi-step reasoning.

[00:05:07]Add explicit instructions such as answer in a single response. Do not ask for clarification unless the request is completely ambiguous.

[00:05:19]Choose save and exit and then choose prepare in the test window.

[00:05:30]To turn off unnecessary pre and post-processing templates on the agent details page, click edit in agent builder.

[00:05:41]Click edit in advanced prompts.

[00:05:44]If you don't require input validation, clear activate pre-processing templates.

[00:05:52]Go to post-processing template tab. If your application layer doesn't handle output formatting, clear activate post-processing template, choose save and exit and then prepare on the test window.

[00:06:10]Pre and post-processing templates each add an LLM call. Deactivating these templates can reduce the agent response time for simple use cases. Now I'll show you how to resolve knowledge base retrieval latency. When an agent queries a knowledge base, it first uses an embedding model to convert the query into a vector. This step alone adds latency. The agent then performs a vector similarity search where latency is shaped by your vector store type indexing strategy, chunk size, and how many results are retrieved. Querying multiple knowledge bases multiplies this overhead.

[00:06:59]To check the timing for knowledgebased query, the steps to be followed.

[00:07:05]In the navigation pane on left side, choose agents and then select your agent. On the agent details page, the test window appears in a pane on the right. If it's closed, choose test at the top of the page to reopen it.

[00:07:24]Run a query that initiates knowledgebased retrieval and then select show trace.

[00:07:32]Expand the trace and look for knowledgebased query steps. Note the millisecond timing. To resolve this issue, reduce the number of knowledge bases in agent builder. Under knowledge bases, confirm that only one knowledge base is attached. If there is more than one knowledge base, then remove ones that aren't necessary to reduce retrieval latency. You can also optimize the knowledge base. In the navigation pane of the Amazon Bitro console, choose knowledge bases and then select your knowledge base. Choose edit for the data source and review your chunking strategy.

[00:08:16]For faster retrieval, use fixit size chunking with smaller chunk sizes.

[00:08:23]Chunking strategy cannot be changed on an existing data source. You must create a new data source with the desired chunk size and reync.

[00:08:35]Amazon open search serverless is a recommended option for low latency managed vector search.

[00:08:44]Now I'll show you how to simplify your prompt structure to reduce latency.

[00:08:50]Token count is directly proportional to processing time. The model must process every token in the input and longer prompts increase cost, kota use and time to first token.

[00:09:05]How to check your current token usage?

[00:09:09]Open the Amazon Cloudatch console. In the navigation pane on left side, choose metrics and then choose all metrics.

[00:09:19]Select bedrock.

[00:09:23]Choose by model ID.

[00:09:26]Select the input token count and output token count metrics for your model.

[00:09:34]Go to graph metrics.

[00:09:36]Set the period to 1 hour and statistic to average. Note which models are consuming the most input tokens. These are your optimization targets.

[00:09:50]The next step is to simplify your agent instructions. Open the Amazon Bedrock console. In the navigation pane, choose agents and then select your agent.

[00:10:02]Choose edit in agent builder. Under instructions, review the current instruction text.

[00:10:11]Remove verbose examples, redundant context, and unnecessary preamble. Aim to keep instructions under 500 tokens.

[00:10:22]Choose save and prepare.

[00:10:27]The next step is to trim the orchestration prompt. Similar to how we optimize the orchestration, you can also reduce perceived latency by using response streaming. Instead of waiting for the agent to generate the full response, streaming returns tokens to the user as they are generated. This significantly improves the user experience for longer responses. To enable streaming, use the invoke agent API with response streaming enabled in your application code. Call the invoke agent action. The response is returned as a stream by default. Make sure your client application process the response stream incrementally rather than waiting for the complete response. Note response streaming is available through the API and SDK. It's not a console toggle.

[00:11:27]Now I'll show you how to check and reduce your token counts. If your outputs have high token counts, then users must wait for the full response.

[00:11:40]To check token counts, use the output token count metric in Amazon Cloudatch.

[00:11:48]Navigate to Amazon Cloudatch console. In the navigation pane of the Cloudatch console, choose metrics and then choose all metrics.

[00:11:58]Choose bedrock and choose by model ID.

[00:12:02]Add metrics for both input token count and output token count. Go to graph metrics. Set the period to 1 hour and statistic to sum.

[00:12:14]Review the models that are using the most tokens. To set the number of output tokens, navigate to your Amazon Bedrock console.

[00:12:24]Choose agents on your left navigation pane and select your agent. Click on edit in agent builder. Click on edit in advanced prompts.

[00:12:36]Go to orchestration.

[00:12:40]In the inference configuration section, set maximum length to the minimum number of output tokens that you require. For example, reducing from 4096 output tokens to 512 can decrease the generation time proportionally.

[00:12:59]Click on save and exit and then prepare your agent.

[00:13:03]And finally, let's check the network connectivity.

[00:13:08]If your agent uses action group Lambda functions, Lambda cold starts can add latency, especially on the first invocation or after a period of inactivity.

[00:13:22]To reduce this, consider configuring provision concurrency, minimizing your deployment package size, and increasing the functions memory allocation.

[00:13:34]If your Lambda function runs inside a VPC and you're experiencing slow network connections, the traffic might be routing through the public internet instead of staying within the AWS network. To resolve this, use AWS private link to create a VPC endpoint for Amazon Bedrock. This keeps traffic private and reduces network latency. If you experience latency during periods of high regional demand, you can use cross region inference to distribute inference workloads across multiple regions. Cross region inference uses inference profiles that automatically route requests to regions with available capacity. To use it, replace your agents model ID with a cross region inference profile ARN in agent builder.

[00:14:32]So now you know how to troubleshoot high latency and optimize the performance of Amazon Bitrock agents.

[00:14:42]Thanks for watching and happy cloud computing from all of us here at AWS.