AWS connect documentation change
Summary
Added new AI-related metrics including AI Agent Response Helpful/Not Helpful, AI Tool Parameter/Selection/Utilization Accuracy, and Completeness/Faithfulness/Goal Success Rate scores for monitoring AI agent performance and accuracy
Security assessment
This change adds documentation for new AI performance monitoring metrics. While these metrics help monitor AI system behavior and could indirectly help identify potential issues with AI responses, there is no evidence in the diff that these changes address a specific security vulnerability, weakness, or incident. The metrics focus on performance, accuracy, and user feedback rather than security controls or vulnerabilities.
Diff
diff --git a/connect/latest/adminguide/metrics-definitions.md b/connect/latest/adminguide/metrics-definitions.md index 5bcecf317..21a51abf9 100644 --- a//connect/latest/adminguide/metrics-definitions.md +++ b//connect/latest/adminguide/metrics-definitions.md @@ -7 +7 @@ -Abandonment rateActive AI AgentsActive slotsAdherenceAdherent timeAfter contact work timeAgent ActivityAgent After contact workAgent API connecting timeAgent answer rateAgent average contact first response wait timeAgent callback connecting timeAgent errorAgent idle timeAgent incoming connecting timeAgent interaction and hold timeAgent interaction timeAgent non-productiveAgent non-productive timeAgent non-responseAgent non-response without customer abandonsAgent contact timeAgents countAgents on contactAgent outbound connecting timeAgent talk time percentAI HandoffsAI Handoff RateAI Agent InvocationsAI Agent Invocation SuccessAI Agent Invocation Success RateAI Response Completion RateAI Involved ContactsAI Prompt Invocation SuccessAI Prompt Invocation Success RateAI Prompt InvocationsAI Tool Invocation SuccessAI Tool Invocation Success RateAI Tool InvocationsAverage AI Agent Conversation TurnsAverage AI Conversation TurnsAverage AI Prompt Invocation LatencyAverage AI Tool Invocation LatencyKnowledge Content ReferencesProactive Intents AnsweredProactive Intents DetectedProactive Intents EngagedProactive Intent Engagement RateProactive Intent Response RateAPI contactsAPI contacts handledAutomatic fails percentAvailabilityAvailableAverage active timeAverage after contact work timeAverage agent API connecting timeAverage agent callback connecting timeAverage agent first response timeAverage agent greeting timeAverage agent incoming connecting timeAverage agent interaction and customer hold timeAverage agent interaction timeAverage agent interruptionsAverage agent interruption timeAverage agent message lengthAverage agent messagesAverage agent outbound connecting timeAverage agent pause timeAverage agent response timeAverage agent talk timeAverage bot conversation timeAverage bot conversation turnsAverage bot messagesAverage case resolution timeAverage contact durationAverage contacts per caseAverage conversation close timeAverage conversation durationAverage customer hold timeAverage customer hold time all contactsAverage customer message lengthAverage customer messagesAverage customer response timeAverage customer talk timeAverage dials per minuteAverage evaluation scoreAverage flow timeAverage handle timeAverage holdsAverage messagesAverage non-talk timeAverage outbound after contact work timeAverage outbound agent interaction timeAverage queue abandon timeAverage queue abandon time - customer first callbackAverage queue answer timeAverage queue answer time - customer first callbackAverage queue answer time (enqueue timestamp)Average resolution timeAverage speed of answer - customer first callback dialedAverage talk timeAverage Test Case Execution DurationAverage wait time after customer connection - customer first callbackAverage wait time after customer connectionAverage weighted evaluation scoreBot conversations completedBot intents completedCallback attemptsCallback attempts - customer first callbackCallback contactsCallback contacts handledCampaign contacts abandoned after XCampaign contacts abandoned after X rateCampaign interactionsCampaign progress rateCampaign send attemptsCampaign send exclusionsCapacityCases createdCases reopenedCases resolvedCases resolved on first contactConsultContact flow timeContact handle timeContact StateContact volumeContact volume - agent first callbackContact volume - customer first callbackContacts abandonedContacts abandoned - customer first callbackContacts abandoned in X secondsContacts agent hung up firstContacts answered in X secondsContacts createdContacts consultedContacts disconnectedContacts handledContacts handled (connected to the agent timestamp)Contacts handled - customer first callbackContacts handled incomingContacts handled outboundContacts hold agent disconnectContacts hold customer disconnectContacts hold disconnectContacts in queueContacts incomingContacts put on holdContacts queuedContacts queued (enqueue timestamp)Contacts removed from queue in X secondsContacts resolved in X secondsContacts transferred inContacts transferred in by agentContacts transferred in from queueContacts transferred outContacts transferred out by agentContacts transferred out externalContacts transferred out internalContacts transferred out queueConversations abandonedCurrent casesCustomer hold timeCustomer talk time percentDelivery attemptsDelivery attempt disposition rateDurationEffective staffingError status timeEstimated Wait TimeEvaluations performedFlows outcomeFlows outcome percentageFlows startedHuman answeredMaximum flow timeMaximum queued timeMax Test Case Execution DurationMinimum flow timeNon-adherent timeNon-talk time percentOccupancyOldestOnline agentsOnline timePercent agent on contact timePercent agent idle timePercent agent non-productive timePercent bot conversations outcomePercent bot intents outcomePosition in QueueRecipients attemptedRecipients interactedRecipients targetedScheduled timeScheduledService level XStaffed agentsStep contacts queuedStep expired %Step joinedTalk time percentTest case execution countTest case failed rateTest case success rate +Abandonment rateActive AI AgentsActive slotsAdherenceAdherent timeAfter contact work timeAgent ActivityAgent After contact workAgent API connecting timeAgent answer rateAgent average contact first response wait timeAgent callback connecting timeAgent errorAgent idle timeAgent incoming connecting timeAgent interaction and hold timeAgent interaction timeAgent non-productiveAgent non-productive timeAgent non-responseAgent non-response without customer abandonsAgent contact timeAgents countAgents on contactAgent outbound connecting timeAgent talk time percentAI HandoffsAI Handoff RateAI Agent InvocationsAI Agent Invocation SuccessAI Agent Invocation Success RateAI Agent Response HelpfulAI Agent Response Not HelpfulAI Response Completion RateAI Involved ContactsAI Prompt Invocation SuccessAI Prompt Invocation Success RateAI Prompt InvocationsAI Tool Invocation SuccessAI Tool Invocation Success RateAI Tool InvocationsAI Tool Parameter AccuracyAI Tool Selection AccuracyAI Tool Utilization AccuracyAverage AI Agent Conversation TurnsAverage AI Conversation TurnsAverage AI Prompt Invocation LatencyAverage AI Tool Invocation LatencyCompleteness ScoreFaithfulness ScoreGoal Success RateKnowledge Content ReferencesProactive Intents AnsweredProactive Intents DetectedProactive Intents EngagedProactive Intent Engagement RateProactive Intent Response RateAPI contactsAPI contacts handledAutomatic fails percentAvailabilityAvailableAverage active timeAverage after contact work timeAverage agent API connecting timeAverage agent callback connecting timeAverage agent first response timeAverage agent greeting timeAverage agent incoming connecting timeAverage agent interaction and customer hold timeAverage agent interaction timeAverage agent interruptionsAverage agent interruption timeAverage agent message lengthAverage agent messagesAverage agent outbound connecting timeAverage agent pause timeAverage agent response timeAverage agent talk timeAverage bot conversation timeAverage bot conversation turnsAverage bot messagesAverage case resolution timeAverage contact durationAverage contacts per caseAverage conversation close timeAverage conversation durationAverage customer hold timeAverage customer hold time all contactsAverage customer message lengthAverage customer messagesAverage customer response timeAverage customer talk timeAverage dials per minuteAverage evaluation scoreAverage flow timeAverage handle timeAverage holdsAverage messagesAverage non-talk timeAverage outbound after contact work timeAverage outbound agent interaction timeAverage queue abandon timeAverage queue abandon time - customer first callbackAverage queue answer timeAverage queue answer time - customer first callbackAverage queue answer time (enqueue timestamp)Average resolution timeAverage speed of answer - customer first callback dialedAverage talk timeAverage Test Case Execution DurationAverage wait time after customer connection - customer first callbackAverage wait time after customer connectionAverage weighted evaluation scoreBot conversations completedBot intents completedCallback attemptsCallback attempts - customer first callbackCallback contactsCallback contacts handledCampaign contacts abandoned after XCampaign contacts abandoned after X rateCampaign interactionsCampaign progress rateCampaign send attemptsCampaign send exclusionsCapacityCases createdCases reopenedCases resolvedCases resolved on first contactConsultContact flow timeContact handle timeContact StateContact volumeContact volume - agent first callbackContact volume - customer first callbackContacts abandonedContacts abandoned - customer first callbackContacts abandoned in X secondsContacts agent hung up firstContacts answered in X secondsContacts createdContacts consultedContacts disconnectedContacts handledContacts handled (connected to the agent timestamp)Contacts handled - customer first callbackContacts handled incomingContacts handled outboundContacts hold agent disconnectContacts hold customer disconnectContacts hold disconnectContacts in queueContacts incomingContacts put on holdContacts queuedContacts queued (enqueue timestamp)Contacts removed from queue in X secondsContacts resolved in X secondsContacts transferred inContacts transferred in by agentContacts transferred in from queueContacts transferred outContacts transferred out by agentContacts transferred out externalContacts transferred out internalContacts transferred out queueConversations abandonedCurrent casesCustomer hold timeCustomer talk time percentDelivery attemptsDelivery attempt disposition rateDurationEffective staffingError status timeEstimated Wait TimeEvaluations performedFlows outcomeFlows outcome percentageFlows startedHuman answeredMaximum flow timeMaximum queued timeMax Test Case Execution DurationMinimum flow timeNon-adherent timeNon-talk time percentOccupancyOldestOnline agentsOnline timePercent agent on contact timePercent agent idle timePercent agent non-productive timePercent bot conversations outcomePercent bot intents outcomePosition in QueueRecipients attemptedRecipients interactedRecipients targetedScheduled timeScheduledService level XStaffed agentsStep contacts queuedStep expired %Step joinedTalk time percentTest case execution countTest case failed rateTest case success rate @@ -1333,0 +1334,92 @@ This metric measures the percentage of [AI Agents](./create-ai-agents.html) invo +## AI Agent Response Helpful + +This metric measures the count of AI suggestions rated as helpful with a thumbs-up. + +**Metric type** : Integer + +**Metric category** : AI Agent + +**How to access using the Amazon Connect API** : + + * [GetMetricDataV2](https://docs.aws.amazon.com/connect/latest/APIReference/API_GetMetricDataV2.html) API metric identifier: `AI_AGENT_RESPONSE_HELPFUL` + + + + +**How to access using the Amazon Connect admin website** : + + * Dashboard: [AI Agent performance dashboard](./ai-agent-performance-dashboard.html), AI Agent Response Helpful + + + + +**Calculation logic** : + + * For each AI Agent record + + * If aiAgentId is NOT present, then skip this record. + + * If helpfulResponseCount is present, then set result = helpfulResponseCount. + + * Else, skip this record. + + * Return final_result = sum of the result values from matching records. + + + + +###### Note + + * This metric is updated every 6 hours. + + * This metric is available as part of Connect Customer AI. + + + + +## AI Agent Response Not Helpful + +This metric measures the count of AI suggestions rated as unhelpful with a thumbs-down. + +**Metric type** : Integer + +**Metric category** : AI Agent + +**How to access using the Amazon Connect API** : + + * [GetMetricDataV2](https://docs.aws.amazon.com/connect/latest/APIReference/API_GetMetricDataV2.html) API metric identifier: `AI_AGENT_RESPONSE_NOT_HELPFUL` + + + + +**How to access using the Amazon Connect admin website** : + + * Dashboard: [AI Agent performance dashboard](./ai-agent-performance-dashboard.html), AI Agent Response Not Helpful + + + + +**Calculation logic** : + + * For each AI Agent record + + * If aiAgentId is NOT present, then skip this record. + + * If unhelpfulResponseCount is present, then set result = unhelpfulResponseCount. + + * Else, skip this record. + + * Return final_result = sum of the result values from matching records. + + + + +###### Note + + * This metric is updated every 6 hours. + + * This metric is available as part of Connect Customer AI. + + + + @@ -1601,0 +1694,138 @@ This metric measures the percentage of [AI Tools](./ai-agent-mcp-tools.html) inv +## AI Tool Parameter Accuracy + +This metric measures the rate of tool invocations where [AI Agents](./create-ai-agents.html) provided the correct parameters. Value is between 0-1, where 1 indicates perfect parameter accuracy. + +**Metric type** : Double + +**Metric category** : AI Tool + +**How to access using the Amazon Connect API** : + + * [GetMetricDataV2](https://docs.aws.amazon.com/connect/latest/APIReference/API_GetMetricDataV2.html) API metric identifier: `AI_TOOL_PARAMETER_ACCURACY` + + + + +**How to access using the Amazon Connect admin website** : + + * Dashboard: [AI Agent performance dashboard](./ai-agent-performance-dashboard.html), AI Tool Parameter Accuracy + + + + +**Calculation logic** : + + * For each AI Tool record + + * If aiToolId is NOT present, then skip this record. + + * If toolParameterAccuracy is present, then set result = toolParameterAccuracy. + + * Else, skip this record. + + * Return final_result = average of the result values from matching records. + + + + +###### Note + + * This metric is updated every 24 hours. + + * This metric is available as part of Connect Customer AI. + + + + +## AI Tool Selection Accuracy + +This metric measures the rate of correct tool selections by [AI Agents](./create-ai-agents.html). Value is between 0-1, where 1 indicates optimal selection. + +**Metric type** : Double + +**Metric category** : AI Tool + +**How to access using the Amazon Connect API** : + + * [GetMetricDataV2](https://docs.aws.amazon.com/connect/latest/APIReference/API_GetMetricDataV2.html) API metric identifier: `AI_TOOL_SELECTION_ACCURACY` + + + + +**How to access using the Amazon Connect admin website** : + + * Dashboard: [AI Agent performance dashboard](./ai-agent-performance-dashboard.html), AI Tool Selection Accuracy + + + + +**Calculation logic** : + + * For each AI Tool record + + * If aiToolId is NOT present, then skip this record. + + * If toolSelectionAccuracy is present, then set result = toolSelectionAccuracy. + + * Else, skip this record. + + * Return final_result = average of the result values from matching records. + + + + +###### Note + + * This metric is updated every 24 hours. + + * This metric is available as part of Connect Customer AI. + + + + +## AI Tool Utilization Accuracy + +This metric measures the rate of correct tool utilization by [AI Agents](./create-ai-agents.html), including proper parameters and selection. Value is between 0-1, where 1 indicates perfect utilization. + +**Metric type** : Double + +**Metric category** : AI Tool