AWS connect documentation change
Summary
Updated Amazon Connect AI Agent Performance Dashboard documentation with expanded metrics, new sections (AI agent performance trend, AI tools usage, AI prompt performance, proactive intents trend, AI prompts by invocation latency/success rate), updated security permissions, and renamed sections for clarity.
Security assessment
The change updates the required security profile permissions for accessing the dashboard from a single permission to multiple possible permissions (AI agent designer and Agent applications permissions). This clarifies access control but doesn't indicate a fix for a specific security vulnerability. The documentation now includes more detailed monitoring metrics (faithfulness score, tool accuracy metrics) which could help identify performance issues but aren't directly security fixes.
Diff
diff --git a/connect/latest/adminguide/ai-agent-performance-dashboard.md b/connect/latest/adminguide/ai-agent-performance-dashboard.md index 9b0449ba9..fd4eb327b 100644 --- a//connect/latest/adminguide/ai-agent-performance-dashboard.md +++ b//connect/latest/adminguide/ai-agent-performance-dashboard.md @@ -7 +7 @@ -Enable access to the dashboardSpecify "Time range" and "Compare to" benchmarkSelf-service AI performance summaryAI agent assistance performance summaryAI agents by version chartAI agents by invocation success rateKnowledge base usageAI prompts by version +Enable access to the dashboardSpecify "Time range" and "Compare to" benchmarkSelf-service AI performance summaryAgent assistance AI performance summaryAI agents performanceAI agents by invocation success rateKnowledge base usageAI agent performance trendAI tools usageAI prompt performanceProactive intents trendAI prompts by invocation latencyAI prompts by invocation success rate @@ -23 +23 @@ The dashboard provides a single place to view aggregated AI Agent performance. U - * AI agent assistance performance summary + * Agent assistance AI performance summary @@ -25 +25 @@ The dashboard provides a single place to view aggregated AI Agent performance. U - * AI agents by version chart + * AI agents performance @@ -31 +31,11 @@ The dashboard provides a single place to view aggregated AI Agent performance. U - * AI prompts by version + * AI agent performance trend + + * AI tools usage + + * AI prompt performance + + * Proactive intents trend + + * AI prompts by invocation latency + + * AI prompts by invocation success rate @@ -42 +52,12 @@ Ensure users are assigned the appropriate security profile permissions: -**Agent applications -****Connect Workspace AI Chat Widget** permission: This permission is needed to access the AI Agent Performance dashboard. +At least one of the following security profile permissions is needed to access the AI Agent Performance dashboard: + + * **AI agent designer- AI agent** view permission + + * **AI agent designer- AI prompt** view permission + + * **AI agent designer- AI guardrails** view permission + + * **Agent applications - Connect assistant** view permission + + + @@ -56 +77 @@ Use the **Compare to** filter to select a time period to compare your current da -This section shows health of your AI-Agent initiated Self-Service interactions. It displays the following key metrics for the selected time period filtered by ‘Self service’ usecase: +This section shows health of your AI-Agent initiated Self-Service interactions. It displays the following key metrics for the selected time period filtered by 'Self service' use case: @@ -74 +95,5 @@ Percentage of self service contacts handled by AI agents that was marked for nee - * **Avg. AI conversation turns** : + * **Goal success rate** : + +The proportion of sessions where AI agents successfully resolved customer issues. Value is between 0-1, where 1 indicates successful resolution across all sessions. + + * **Faithfulness score** : @@ -76 +101 @@ Percentage of self service contacts handled by AI agents that was marked for nee -Average number of conversation turns across AI-enabled contacts. +The proportion of AI agent responses that remain faithful to the conversational context, including messages and tool call results. Value is between 0-1, where 1 indicates perfect contextual fidelity. @@ -85 +110 @@ The following image shows an example **Self-service AI performance summary** cha -## AI agent assistance performance summary +## Agent assistance AI performance summary @@ -87 +112 @@ The following image shows an example **Self-service AI performance summary** cha -This widget shows the health of your agent-assisted interactions where AI provides support to human agents. It displays the following key metrics for the selected time period filtered by ‘Agent Assistance’ usecase. +This widget shows the health of your agent-assisted interactions where AI provides support to human agents. It displays the following key metrics for the selected time period filtered by 'Agent Assistance' use case. @@ -105 +130,25 @@ Percentage of detected proactive intents clicked by human agents. - * **Avg. AI conversation turns** : + * **Faithfulness score** : + +The proportion of AI agent responses that remain faithful to the conversational context, including messages and tool call results. Value is between 0-1, where 1 indicates perfect contextual fidelity. + + + + +The following image shows an example **Agent assistance AI performance summary** chart. + + + +## AI agents performance + +This table provides a drill-down view of AI agent performance from use case to individual agent versions and their key performance metrics. + +The table displays performance metrics at two levels: + + * **AI use case level** : + +Aggregated metrics across all agent versions within a use case (End customer self-service, Agent assistance). + + * **AI agent version level** : + +Individual performance metrics for each agent version. + @@ -107 +155,0 @@ Percentage of detected proactive intents clicked by human agents. -Average number of conversation turns across AI-enabled contacts. @@ -109 +156,0 @@ Average number of conversation turns across AI-enabled contacts. - * **Avg. handle time** : @@ -111 +158 @@ Average number of conversation turns across AI-enabled contacts. -Average handle time for contacts where AI Agents were engaged +You can expand or collapse the use case rows to drill down into specific agent versions. To see how each version contributes to the overall use case performance, view the individual metrics for each agent version row. @@ -112,0 +160,13 @@ Average handle time for contacts where AI Agents were engaged +**Metrics displayed:** + + * **AI agent invocation count** : + +Total number of times the AI agent version was invoked. + + * **AI agent invocation success rate** : + +Percentage of AI agent invocations that executed successfully. + + * **Avg. AI agent conversation turns** : + +Average number of conversation turns the AI agent took to reach an outcome. @@ -113,0 +174 @@ Average handle time for contacts where AI Agents were engaged + * **Helpful response** : @@ -114,0 +176 @@ Average handle time for contacts where AI Agents were engaged +The count of AI suggestions rated as helpful with a thumbs-up. @@ -116 +178 @@ Average handle time for contacts where AI Agents were engaged -The following image shows an example **AI agent assistance performance summary** chart. + * **Unhelpful response** : @@ -118 +180 @@ The following image shows an example **AI agent assistance performance summary** - +The count of AI suggestions rated as unhelpful with a thumbs-down. @@ -120 +182 @@ The following image shows an example **AI agent assistance performance summary** -## AI agents by version chart + * **Tool utilization accuracy** : @@ -122 +184 @@ The following image shows an example **AI agent assistance performance summary** -On the **Evaluation score trend** chart you can view trends at intervals of 15 minutes, daily, weekly or monthly, and perform comparison with prior time period and resource benchmarks. The available intervals depend on time range selections. For example, for a Time Range of weekly, you can view trends at daily and weekly intervals. +The rate of correct tool utilization by the AI agent. Value is between 0-1, where 1 indicates perfect utilization. @@ -124 +186 @@ On the **Evaluation score trend** chart you can view trends at intervals of 15 m -In addition to the page filters, you can also add filters to the chart for the evaluation form and the evaluation source. + * **Tool selection accuracy** : @@ -126 +188 @@ In addition to the page filters, you can also add filters to the chart for the e -The following image shows an example **Evaluation score trend** chart. +The rate of correct tool selections by AI Agents. Value is between 0-1, where 1 indicates optimal selection. @@ -128 +190,10 @@ The following image shows an example **Evaluation score trend** chart. - + * **Tool parameter accuracy** : + +The rate of tool invocations where AI Agents provided the correct parameters. Value is between 0-1, where 1 indicates perfect parameter accuracy. + + + + +The following image shows an example **AI agents performance** table. + + @@ -153 +224,80 @@ The following image shows an example **Knowledge base usage** table. -## AI prompts by version +## AI agent performance trend + +The AI agent performance trend is a time-series chart that displays the AI agent invocation success rate (blue bars), Faithfulness score (red line), and Tool utilization accuracy (green line) over a given time period broken down by intervals (15min, daily, weekly, monthly). You can configure different time range intervals by using the "Interval" button directly in the widget. The intervals that you may select depend on the page-level time range filter. + +For example: + + * If you have a "Today" time range filter at the top of your dashboard, you can only see an interval of 15min for the last 24 hours. + + * If you have a "Day" time range filter at the top of your dashboard, you can see a trailing 8 day interval trend, or a 15min interval trend for the trailing 24 hours. + + * If you have a "Week" time range filter at the top of your dashboard, you can see daily or weekly interval trends. + + * If you have a "Month" time range filter at the top of your dashboard, you can see daily or monthly interval trends. + + + + +The following image shows an example **AI agent performance trend** chart. + + + +## AI tools usage + +This table compares tool implementations (e.g., MCP - Model Context Protocol, Return-to-Control) on invocations, average latency and invocation success rate, and tool utilization accuracy. + +For example, you can select an AI agent type (such as Orchestration or Answer Recommendation) and view how different tools and their versions are performing within that agent type. You can identify which tool versions need optimization by sorting columns using **AI tool invocation count** , **AI tool invocation success rate** , or **Avg. AI tool invocation latency**. + +The table displays performance metrics at three levels: + + * **AI agent type level** : + +Aggregated metrics across all tools and versions within an agent type. + + * **AI tool type level** : + +Aggregated metrics across all versions of a specific tool type. + + * **AI tool** : + +Individual performance metrics for each tool version. + + + + +You can expand or collapse the AI agent type and tool type rows to drill down into specific tool versions. To see how each version contributes to the overall tool type performance, view the individual metrics for each tool version row. + +**Metrics displayed:** + + * **AI tool invocation count** : + +Total number of times the AI tool version was invoked. + + * **Avg. AI tool invocation latency** : + +Average invocation latency in milliseconds for the tool version. + + * **AI tool invocation success rate** :