AWS connect documentation change
Summary
Added multiple new AI-related metrics including Active AI Agents, AI Handoffs, AI Agent Invocations, AI Response Completion Rate, and various AI tool/prompt metrics with documentation about their calculation and access methods
Security assessment
The changes add operational metrics for monitoring AI agent performance and usage patterns. While they include success rates and error tracking, there is no evidence of addressing security vulnerabilities or describing security-specific features. The metrics focus on performance monitoring rather than security controls.
Diff
diff --git a/connect/latest/adminguide/metrics-definitions.md b/connect/latest/adminguide/metrics-definitions.md index 9321c3c17..11eff1948 100644 --- a//connect/latest/adminguide/metrics-definitions.md +++ b//connect/latest/adminguide/metrics-definitions.md @@ -5 +5 @@ -Abandonment rateActive slotsAdherenceAdherent timeAfter contact work timeAgent ActivityAgent After contact workAgent API connecting timeAgent answer rateAgent average contact first response wait timeAgent callback connecting timeAgent errorAgent idle timeAgent incoming connecting timeAgent interaction and hold timeAgent interaction timeAgent non-productiveAgent non-productive timeAgent non-responseAgent non-response without customer abandonsAgent contact timeAgents countAgents on contactAgent outbound connecting timeAgent talk time percentAPI contactsAPI contacts handledAutomatic fails percentAvailabilityAvailableAverage active timeAverage after contact work timeAverage agent API connecting timeAverage agent callback connecting timeAverage agent first response timeAverage agent greeting timeAverage agent incoming connecting timeAverage agent interaction and customer hold timeAverage agent interaction timeAverage agent interruptionsAverage agent interruption timeAverage agent message lengthAverage agent messagesAverage agent outbound connecting timeAverage agent pause timeAverage agent response timeAverage agent talk timeAverage bot conversation timeAverage bot conversation turnsAverage bot messagesAverage case resolution timeAverage contact durationAverage contacts per caseAverage conversation close timeAverage conversation durationAverage customer hold timeAverage customer hold time all contactsAverage customer message lengthAverage customer messagesAverage customer response timeAverage customer talk timeAverage dials per minuteAverage evaluation scoreAverage flow timeAverage handle timeAverage holdsAverage messagesAverage non-talk timeAverage outbound after contact work timeAverage outbound agent interaction timeAverage queue abandon timeAverage queue abandon time - customer first callbackAverage queue answer timeAverage queue answer time - customer first callbackAverage queue answer time (enqueue timestamp)Average resolution timeAverage speed of answer - customer first callback dialedAverage talk timeAverage Test Case Execution DurationAverage wait time after customer connection - customer first callbackAverage wait time after customer connectionAverage weighted evaluation scoreBot conversations completedBot intents completedCallback attemptsCallback attempts - customer first callbackCallback contactsCallback contacts handledCampaign contacts abandoned after XCampaign contacts abandoned after X rateCampaign interactionsCampaign progress rateCampaign send attemptsCampaign send exclusionsCapacityCases createdCases reopenedCases resolvedCases resolved on first contactConsultContact flow timeContact handle timeContact StateContact volumeContact volume - agent first callbackContact volume - customer first callbackContacts abandonedContacts abandoned - customer first callbackContacts abandoned in X secondsContacts agent hung up firstContacts answered in X secondsContacts createdContacts consultedContacts disconnectedContacts handledContacts handled (connected to the agent timestamp)Contacts handled - customer first callbackContacts handled incomingContacts handled outboundContacts hold agent disconnectContacts hold customer disconnect Contacts hold disconnectContacts in queueContacts incomingContacts put on holdContacts queuedContacts queued (enqueue timestamp)Contacts removed from queue in X secondsContacts resolved in X secondsContacts transferred inContacts transferred in by agentContacts transferred in from queueContacts transferred outContacts transferred out by agentContacts transferred out externalContacts transferred out internal Contacts transferred out queueConversations abandonedCurrent casesCustomer hold timeCustomer talk time percentDelivery attemptsDelivery attempt disposition rateDurationEffective staffingError status timeEvaluations performedFlows outcomeFlows outcome percentageFlows startedHuman answeredMaximum flow timeMaximum queued timeMax Test Case Execution DurationMinimum flow timeNon-adherent timeNon-talk time percentOccupancyOldestOnline agentsOnline timePercent agent on contact timePercent agent idle timePercent agent non-productive timePercent bot conversations outcomePercent bot intents outcomePosition in queueRecipients attemptedRecipients interactedRecipients targetedScheduled timeScheduledService level XStaffed agentsStep contacts queuedStep expired %Step joinedTalk time percentTest case execution countTest case failed rateTest case success rate +Abandonment rateActive AI AgentsActive slotsAdherenceAdherent timeAfter contact work timeAgent ActivityAgent After contact workAgent API connecting timeAgent answer rateAgent average contact first response wait timeAgent callback connecting timeAgent errorAgent idle timeAgent incoming connecting timeAgent interaction and hold timeAgent interaction timeAgent non-productiveAgent non-productive timeAgent non-responseAgent non-response without customer abandonsAgent contact timeAgents countAgents on contactAgent outbound connecting timeAgent talk time percentAI HandoffsAI Handoff RateAI Agent InvocationsAI Agent Invocation SuccessAI Agent Invocation Success RateAI Response Completion RateAI Involved ContactsAI Prompt Invocation SuccessAI Prompt Invocation Success RateAI Prompt InvocationsAI Tool Invocation SuccessAI Tool Invocation Success RateAI Tool InvocationsAverage AI Agent Conversation TurnsAverage AI Conversation TurnsAverage AI Prompt Invocation LatencyAverage AI Tool Invocation LatencyKnowledge Content ReferencesProactive Intents AnsweredProactive Intents DetectedProactive Intents EngagedProactive Intent Engagement RateProactive Intent Response RateAPI contactsAPI contacts handledAutomatic fails percentAvailabilityAvailableAverage active timeAverage after contact work timeAverage agent API connecting timeAverage agent callback connecting timeAverage agent first response timeAverage agent greeting timeAverage agent incoming connecting timeAverage agent interaction and customer hold timeAverage agent interaction timeAverage agent interruptionsAverage agent interruption timeAverage agent message lengthAverage agent messagesAverage agent outbound connecting timeAverage agent pause timeAverage agent response timeAverage agent talk timeAverage bot conversation timeAverage bot conversation turnsAverage bot messagesAverage case resolution timeAverage contact durationAverage contacts per caseAverage conversation close timeAverage conversation durationAverage customer hold timeAverage customer hold time all contactsAverage customer message lengthAverage customer messagesAverage customer response timeAverage customer talk timeAverage dials per minuteAverage evaluation scoreAverage flow timeAverage handle timeAverage holdsAverage messagesAverage non-talk timeAverage outbound after contact work timeAverage outbound agent interaction timeAverage queue abandon timeAverage queue abandon time - customer first callbackAverage queue answer timeAverage queue answer time - customer first callbackAverage queue answer time (enqueue timestamp)Average resolution timeAverage speed of answer - customer first callback dialedAverage talk timeAverage Test Case Execution DurationAverage wait time after customer connection - customer first callbackAverage wait time after customer connectionAverage weighted evaluation scoreBot conversations completedBot intents completedCallback attemptsCallback attempts - customer first callbackCallback contactsCallback contacts handledCampaign contacts abandoned after XCampaign contacts abandoned after X rateCampaign interactionsCampaign progress rateCampaign send attemptsCampaign send exclusionsCapacityCases createdCases reopenedCases resolvedCases resolved on first contactConsultContact flow timeContact handle timeContact StateContact volumeContact volume - agent first callbackContact volume - customer first callbackContacts abandonedContacts abandoned - customer first callbackContacts abandoned in X secondsContacts agent hung up firstContacts answered in X secondsContacts createdContacts consultedContacts disconnectedContacts handledContacts handled (connected to the agent timestamp)Contacts handled - customer first callbackContacts handled incomingContacts handled outboundContacts hold agent disconnectContacts hold customer disconnect Contacts hold disconnectContacts in queueContacts incomingContacts put on holdContacts queuedContacts queued (enqueue timestamp)Contacts removed from queue in X secondsContacts resolved in X secondsContacts transferred inContacts transferred in by agentContacts transferred in from queueContacts transferred outContacts transferred out by agentContacts transferred out externalContacts transferred out internal Contacts transferred out queueConversations abandonedCurrent casesCustomer hold timeCustomer talk time percentDelivery attemptsDelivery attempt disposition rateDurationEffective staffingError status timeEvaluations performedFlows outcomeFlows outcome percentageFlows startedHuman answeredMaximum flow timeMaximum queued timeMax Test Case Execution DurationMinimum flow timeNon-adherent timeNon-talk time percentOccupancyOldestOnline agentsOnline timePercent agent on contact timePercent agent idle timePercent agent non-productive timePercent bot conversations outcomePercent bot intents outcomePosition in queueRecipients attemptedRecipients interactedRecipients targetedScheduled timeScheduledService level XStaffed agentsStep contacts queuedStep expired %Step joinedTalk time percentTest case execution countTest case failed rateTest case success rate @@ -67,0 +68,35 @@ The abandonment rate helps you identify potential issues with long wait times or +## Active AI Agents + +This metric measures the total number of unique [AI Agents](./create-ai-agents.html), where each AI Agent is identified by its unique combination of `Name` and `Version`. + +**Metric type** : Integer + +**Metric category** : AI Agent + +**How to access using the Amazon Connect API** : + + * [GetMetricDataV2](https://docs.aws.amazon.com/connect/latest/APIReference/API_GetMetricDataV2.html) API metric identifier: `ACTIVE_AI_AGENTS` + + + + +**How to access using the Amazon Connect admin website** : + + * Dashboard: [AI Agent performance dashboard](./ai-agent-performance-dashboard.html), Active AI Agents + + + + +**Calculation logic** : + + * For each AI Agent record + + * If aiAgentId is NOT present, then skip this record. + + * If aiAgentNameVersion is present, then return noncontroversial. + + * Return final_result = approximate unique count of aiAgentNameVersion values from matching records. + + + + @@ -1125,0 +1161,786 @@ For a list of all metrics driven by Contact Lens Conversational analytics, see [ +## AI Handoffs + +This metric measures the total count of contacts handled by [AI Agents](./create-ai-agents.html) that escalated to human agents. This metric is specifically applicable to self-service use cases where AI agents initially handle customer inquiries. + +**Metric type** : Integer + +**Metric category** : AI Session + +**How to access using the Amazon Connect API** : + + * [GetMetricDataV2](https://docs.aws.amazon.com/connect/latest/APIReference/API_GetMetricDataV2.html) API metric identifier: `AI_HANDOFFS` + + + + +**How to access using the Amazon Connect admin website** : + + * Historical metrics reports: Active + + * Dashboard: [AI Agent performance dashboard](./ai-agent-performance-dashboard.html), Active AI Agents + + + + +**Calculation logic** : + + * For each AI Agent record + + * If aiAgentId is NOT present, then skip this record. + + * If aiAgentNameVersion is present, then return noncontroversial. + + * Return final_result = approximate unique count of aiAgentNameVersion values from matching records. + + + + +## AI Handoff Rate + +This metric measures the percentage of contacts handled by [AI Agents](./create-ai-agents.html) that had escalation to human agents or additional support. + +**Metric type** : Percent + +**Metric category** : AI Session + +**How to access using the Amazon Connect API** : + + * [GetMetricDataV2](https://docs.aws.amazon.com/connect/latest/APIReference/API_GetMetricDataV2.html) API metric identifier: `AI_HANDOFF_RATE` + + + + +**How to access using the Amazon Connect admin website:** + + * Dashboard [AI Agent performance dashboard](./ai-agent-performance-dashboard.html), Handoff Rate + + + + +**Calculation logic** : + + * Get total AI handoffs count. + + * Get total AI involved contacts count. + + * Calculate percentage: (AI handoffs / AI involved contacts) * 100.0 + + + + +## AI Agent Invocations + +This metric measures the total count of [AI Agents](./create-ai-agents.html) invocations across all AI-Agents per instance. + +**Metric type** : Integer + +**Metric category** : AI Agent + +**How to access using the Amazon Connect API** : + + * [GetMetricDataV2](https://docs.aws.amazon.com/connect/latest/APIReference/API_GetMetricDataV2.html) API metric identifier: `AI_AGENT_INVOCATIONS` + + + + +**How to access using the Amazon Connect admin website** : + + * Dashboard: [AI Agent performance dashboard](./ai-agent-performance-dashboard.html), AI Agent Invocation Count + + + + +**Calculation logic** : + + * For each AI Agent record + + * If aiAgentId is present, then count this record as 1. + + * Else, skip this record. + + * Return final_result = sum of the counts from matching records. + + + + +## AI Agent Invocation Success + +This metric measures the number of [AI Agents](./create-ai-agents.html) invocations that executed successfully without technical failures such as API errors, timeouts, or system issues. + +**Metric type** : Integer + +**Metric category** : AI Agent + +**How to access using the Amazon Connect API** : + + * [GetMetricDataV2](https://docs.aws.amazon.com/connect/latest/APIReference/API_GetMetricDataV2.html) API metric identifier: `AI_AGENT_INVOCATION_SUCCESS` + + + + +**Calculation logic** : + +For each AI Agent record + + * If aiAgentId is NOT present, then skip this record. + + * If invocationSuccess is present and equals true, then count this record as 1. + + * Else, count this record as 0. + + + + + * Return final_result = sum of the counts from matching records. + + + + +## AI Agent Invocation Success Rate + +This metric measures the percentage of [AI Agents](./create-ai-agents.html) invocations that executed successfully without technical failures. + +**Metric type** : Percent + +**Metric category** : AI Agent + +**How to access using the Amazon Connect API** : + + * [GetMetricDataV2](https://docs.aws.amazon.com/connect/latest/APIReference/API_GetMetricDataV2.html) API metric identifier: `AI_AGENT_INVOCATION_SUCCESS_RATE` + + + + +**How to access using the Amazon Connect admin website** : + + * Dashboard: [AI Agent performance dashboard](./ai-agent-performance-dashboard.html), AI Agent Invocation Success Rate