AWS ai documentation change
Summary
Comprehensive update to Amazon Transcribe documentation covering both batch and streaming modes, expanded feature details, security/privacy controls, and responsible AI practices. Added sections on safety features, PII redaction, toxicity detection, encryption, access controls, and compliance certifications.
Security assessment
The changes extensively document security features including PII redaction, vocabulary filtering, TLS encryption, KMS integration, PrivateLink support, IAM controls, and CloudTrail logging. However, there is no evidence of addressing a specific security vulnerability or incident.
Diff
diff --git a/ai/responsible-ai/transcribe-speech-recognition/overview.md b/ai/responsible-ai/transcribe-speech-recognition/overview.md index f9b557b62..5769669ac 100644 --- a//ai/responsible-ai/transcribe-speech-recognition/overview.md +++ b//ai/responsible-ai/transcribe-speech-recognition/overview.md @@ -5 +5 @@ -[Documentation](/index.html)[AWS AI Service Cards](https://aws.amazon.com/ai/responsible-ai/resources/)[Amazon Transcribe - Batch (English-US)](overview.html) +[Documentation](/index.html)[AWS AI Service Cards](https://aws.amazon.com/ai/responsible-ai/resources/)[Amazon Transcribe](overview.html) @@ -7 +7 @@ -OverviewIntended use cases and limitationsDesign of Transcribe - Batch (English-US)Deployment and performance optimization best practicesFurther informationGlossary +Overview of Amazon TranscribeIntended Use Cases and LimitationsDesign of Amazon TranscribeDeployment and performance optimization best practicesFurther informationGlossary @@ -9 +9 @@ OverviewIntended use cases and limitationsDesign of Transcribe - Batch (English- -# Amazon Transcribe - Batch (English-US) +# Amazon Transcribe @@ -13 +13 @@ OverviewIntended use cases and limitationsDesign of Transcribe - Batch (English- -# Amazon Transcribe - Batch (English-US) +# Amazon Transcribe @@ -17 +17 @@ An AWS AI Service Card explains the use cases for which the service is intended, -This AI Service Card applies to the version of Amazon Transcribe – Batch (English-US) that is current as of November 22, 2023. +This Service Card applies to the releases of Amazon Transcribe that are current as of May 2026. @@ -19 +19 @@ This AI Service Card applies to the version of Amazon Transcribe – Batch (Engl -## Overview +## Overview of Amazon Transcribe @@ -21 +21 @@ This AI Service Card applies to the version of Amazon Transcribe – Batch (Engl -Amazon Transcribe enables AWS customers to add speech-to-text capabilities in their voice-enabled applications. Using Automatic Speech Recognition (ASR) technology, customers can use Amazon Transcribe for a variety of business applications. Features provided by the service include automatic speech recognition, speaker diarization, personally identifiable information (PII) redaction, and language identification; see the [Amazon Transcribe Developer Guide](https://aws.amazon.com/transcribe/features/) for additional detail. This AI Service Card describes one of these features, Transcribe – Batch (English-US), implemented by the [Transcribe::StartTranscriptionJob API](https://docs.aws.amazon.com/transcribe/latest/APIReference/API_StartTranscriptionJob.html). This feature performs ASR in the en-US locale at low (8kHz) or high (16kHz) bandwidth. It operates on recorded speech that is available from a static audio file (batch mode). For ASR in near real-time on streaming media, see the [Transcribe::StartStreamTranscription API](https://docs.aws.amazon.com/transcribe/latest/APIReference/API_streaming_StartStreamTranscription.html). +Amazon Transcribe is a cloud-based automatic speech recognition (ASR) service designed for enterprise use cases. Amazon Transcribe converts spoken audio into text, enabling customers to add speech-to-text capabilities to their voice-enabled applications. Customers can use Amazon Transcribe for tasks such as contact center analytics, voicemail transcription, meeting captioning, closed-captioning for media content, transcribing educational content and search and analytics for audio and video archives. This AI Service Card applies to the use of Amazon Transcribe via the StartTranscriptionJob API (batch mode) and the StartStreamTranscription API (streaming mode). Features covered by this Service Card include automatic speech recognition, language identification, speaker diarization, transcribing digits, custom vocabularies, identifying and redacting personally identifiable information (PII), vocabulary filtering and toxicity detection. Amazon Transcribe is a managed AWS AI service; customers can focus on integrating speech-to-text capabilities into their applications without having to provision or manage any infrastructure. Typically, customers use the AWS Console to develop and test applications, and the API for production workloads at scale. @@ -23 +23 @@ Amazon Transcribe enables AWS customers to add speech-to-text capabilities in th -We assess the quality of Transcribe Speech by measuring how well the words from an ASR transcript match the words spoken in the speech sample, as transcribed by a human listener. When a speaker says “This system can really recognize speech,” we expect the transcript to contain the spoken words, not “This system can wreck a nice beach.” Three types of errors may appear in a transcription: substitutions (like recognize for wreck), insertions (extra words such as "nice"), and deletions (missing words such as "really"). Correctly-transcribed words are called hits. Quality metrics like precision, recall, F1, and word error rate (WER) depend on the number of hits and errors. +An Amazon Transcribe _< audio input, configuration parameters, transcript output>_ triple is said to be "effective" if a skilled human evaluator decides that the resulting transcript: 1/ accurately reflects the words spoken in the audio; 2/ correctly attributes speech to speakers when diarization is enabled; 3/ appropriately handles formatting such as punctuation, capitalization, and digit transcription when enabled; and 4/ is suitable for the customer's intended downstream use. Otherwise, the output may require refinement or may not fully meet all evaluation criteria for the specific use case. A customer's workflow must decide if a transcript is effective using human judgment, whether human judgment is applied on a case-by-case basis (as happens when reviewing individual transcripts) or is applied via the customer's choice of an acceptable score on an automated test. @@ -25 +25 @@ We assess the quality of Transcribe Speech by measuring how well the words from -Multiple factors affect the accuracy of any ASR system. The input audio signal consists of the speech itself, modified by a variety of confounding factors. Individual words and utterances differ from speaker to speaker in the frequency with which they are used, in how they are pronounced, and in the ways they are combined with other words. Words that differ in spelling and meaning may not differ in sound. Speakers may overlap or interrupt one another. Recording devices differ in quality, and position relative to the speaker (e.g., farfield vs nearfield). Recording environments differ in the level of background noise, susceptibility to echo, and the presence of other speakers. Transmission lines vary in the level of noise. Transcribe is designed to distinguish between the audio for different words, and ignore the confounding variations. +The "overall effectiveness" of Amazon Transcribe for a specific use case is based on the percentage of use-case-specific inputs for which the service returns an effective result. Customers should define and measure effectiveness for themselves for the following reasons. First, the customer is best positioned to know which audio inputs will best represent their use case and should therefore be included in an evaluation dataset. Second, different ASR systems may respond differently to the same audio input, requiring tuning of the configuration and/or the evaluation mechanism. We assess the quality of Amazon Transcribe by measuring how well the words from an ASR transcript match the words spoken in the audio, as transcribed by a human listener. When a speaker says _"This system can really recognize speech"_ , we expect the transcript to contain the spoken words, not _"This system can wreck a nice beach"_. Three types of errors may appear in a transcription: substitutions (like "recognize" transcribed as "wreck"), insertions (extra words such as "nice"), and deletions (missing words such as "really"). Correctly-transcribed words are called hits. Quality metrics like precision, recall, F1, and word error rate (WER) depend on the number of hits and errors. @@ -27 +27 @@ Multiple factors affect the accuracy of any ASR system. The input audio signal c -## Intended use cases and limitations +As with all ML solutions, Amazon Transcribe must overcome issues of intrinsic and confounding variation. Intrinsic variation refers to features of the audio input that the service needs to pay attention to, such as the specific words spoken by each individual and their unique pronunciation, intonation, speaking rate, and vocabulary. Confounding variation refers to features of the audio input that the service should ignore, for example, variations in background noise, recording equipment, room acoustics, and transmission quality, which should not affect the transcription of the spoken words. The full set of variations encountered in the input audio include: languages, dialects, accents, speaker age, speaker gender, speaking rate, speaking style (read vs. spontaneous), number of speakers, overlapping speech, background noise, recording device quality and placement (near-field vs. far-field), sampling rate, room reverberation, transmission line noise, and domain-specific vocabulary. Since different Amazon Transcribe features use audio input and configuration parameters differently, customers should experiment as necessary to understand how best to adjust their configuration to achieve an effective result. @@ -29 +29 @@ Multiple factors affect the accuracy of any ASR system. The input audio signal c -Transcribe – Batch (English-US) is intended for use on audio samples that contain naturally occurring human speech. It is not designed for mechanically or digitally transformed speech, or synthetic speech. It is intended to transcribe US English words; see [Supported Languages](https://docs.aws.amazon.com/transcribe/latest/dg/supported-languages.html) for additional language locales. Transcribe supports a large general-purpose vocabulary; customers can add [custom vocabularies](https://docs.aws.amazon.com/transcribe/latest/dg/custom-vocabulary.html) and [custom language models](https://docs.aws.amazon.com/transcribe/latest/dg/custom-language-models.html) for coverage of words and phrases from specialized domains. Transcribe supports [speaker partitioning](https://docs.aws.amazon.com/transcribe/latest/dg/diarization.html), also known as diarization. Up to 10 unique speakers can be identified by enabling speaker partitioning in the API call. +## Intended Use Cases and Limitations @@ -31 +31 @@ Transcribe – Batch (English-US) is intended for use on audio samples that cont -Transcribe Batch (English-US) has many possible applications, such as contact center analytics (sentiment/categorization/talk speed), transcribing voicemails, meeting captioning, captioning for media content (audio or video), and search/analytics/keyword analysis for media, including cataloging or indexing media archives. These applications vary in their design by 1/ the number of speakers, 2/ the number of speakers per channel (i.e., per recording device such as a laptop or mobile phone), 3/ the style of speech employed by speakers, 4/ recording conditions (such as location and equipment), and other factors. For example, a contact center transcription application might expect two speakers; one speaker per channel; near-field recording (with the speaker’s mouth close to the microphone); and high background noise from both the caller’s home environment and the contact center operator’s work environment. A second example is an application for doing closed-captioning of instructional videos, an entry point to media analytics, indexing, and search. This application would expect multiple speakers; one audio channel shared across all speakers; scripted speech with fewer filler words, pauses, and disfluencies, but more domain-specific jargon; and lower levels of background noise and other audio occlusions. +**Core Capabilities:** Amazon Transcribe serves a wide range of potential application domains and offers the following core capabilities: @@ -33 +33,64 @@ Transcribe Batch (English-US) has many possible applications, such as contact ce -## Design of Transcribe - Batch (English-US) + * **Batch and Streaming Transcription.** Amazon Transcribe offers two transcription modes. Batch transcription processes pre-recorded audio files stored in Amazon Simple Storage Service and returns a complete transcript after processing; it benefits from full audio context, making it suitable for call recordings, media archives, and voicemails. Streaming transcription processes live audio in near real-time, returning partial and final results as audio is received; it must produce results with limited future context, so partial results may change as more audio arrives. Streaming is suitable for live captioning, real-time call monitoring, and voice chat. Accuracy may differ between modes for the same audio. + + * **Language Identification.** Automatically detects the language spoken in an audio input. When [language identification](https://docs.aws.amazon.com/transcribe/latest/dg/lang-id.html) is enabled, the service analyzes the acoustic and phonetic characteristics of the speech to determine which of the candidate languages is most likely being spoken. Customers can optionally specify a set of candidate language codes, and the service returns the identified language along with a confidence score. Language identification can be used in both batch and streaming modes. In batch mode, the service identifies the dominant language of the entire audio file. In streaming mode, the service can identify the language at the start of the stream. Language identification is most accurate when the set of candidate languages is small and when the candidate languages are phonetically distinct from one another. Accuracy may be reduced when candidate languages are closely related (for example, distinguishing between regional variants of the same language) or when speakers switch languages mid-utterance. + + * **Speaker Diarization.** Differentiates speakers in an audio input based on their voice characteristics, annotating the transcript to indicate which speaker produced each segment of speech. When [speaker diarization](https://docs.aws.amazon.com/transcribe/latest/dg/diarization.html) is enabled, the service analyzes the audio to identify distinct speakers and labels each transcribed segment with a speaker identifier (Speaker 0, Speaker 1, Speaker 2, and so on). Speaker labels are assigned based on voice characteristics within a single audio file or streaming session; labels are randomly assigned and are not consistent across separate audio files or sessions. Customers can optionally specify the expected number of speakers to improve diarization accuracy. Speaker diarization supports up to 30 unique speakers per audio input. The feature performs best when speakers have distinct voice characteristics and when there is minimal overlapping speech. Performance may degrade when speakers have very similar voice characteristics, when there are many speakers in a single recording, or when speakers frequently interrupt one another. Speaker diarization is available in both batch and streaming modes. Speaker diarization does not perform speaker identification or recognition. It cannot be used to identify who a speaker is, to match a speaker against a known voice profile, or to track the same speaker across separate audio files or sessions. + + * **Transcribing Digits.** Converts spoken numbers into their written digit form in the transcription output. For example, when a speaker says "my phone number is five five five one two three four," the service can output "my phone number is 5551234" rather than the spoken-word form. This feature is part of Amazon Transcribe's inverse text normalization (ITN) capability, which also handles formatting of other spoken elements such as dates, times, currencies, and addresses into their conventional written forms. Digit transcription is enabled by default for supported languages and requires no additional configuration. The accuracy of digit transcription depends on factors such as the clarity of the spoken numbers, the presence of background noise, the context in which numbers are spoken (for example, isolated number sequences vs. numbers embedded in conversational speech), and expected adherence to locale variations. Digit transcription is available for a [subset of supported languages](https://docs.aws.amazon.com/transcribe/latest/dg/supported-languages.html). + + * **Custom Vocabularies.** Allows customers to provide a list of domain-specific terms, brand names, proper nouns, acronyms, or other words that the service should recognize with higher accuracy. When a [custom vocabulary](https://docs.aws.amazon.com/transcribe/latest/dg/custom-vocabulary.html) list is provided, the service biases its recognition toward the specified terms, making it more likely to correctly transcribe those words when they occur in the audio. This is particularly useful for words that are uncommon in general speech but frequent in the customer's domain; for example, proprietary product names, medical terminology, legal terms, or financial instrument identifiers. Custom vocabulary is available across all languages, for both Batch and Streaming modes. + + * **PII Identification and Redaction.** [Redacts personally identifiable information](https://docs.aws.amazon.com/transcribe/latest/dg/pii-redaction.html) (PII) like names, phone numbers, credit card numbers, Social Security numbers and other PII types from the transcript. When PII identification is enabled, the service identifies PII entities in the transcript under a placeholder tag. In batch mode, customers can choose to receive both the original (unredacted) and redacted transcripts, or only the redacted transcript. In streaming mode, redaction is completed only at the end of the audio segment. The types of PII that Amazon Transcribe can identify and redact vary between batch and streaming transcriptions, and vary by language. PII identification and redaction uses pattern matching and contextual analysis to detect PII entities; it is not guaranteed to identify all PII instances in all contexts. Customers should not rely solely on automated PII redaction for compliance with data protection regulations without additional review processes, particularly for regulated workloads. The accuracy of PII identification may vary depending on factors such as audio quality, speaker clarity, and the context in which PII is spoken. + + + + +The components differ in the parameters (for example, API arguments) required to invoke them. For more information about these specifications, see the [Amazon Transcribe Documentation](https://docs.aws.amazon.com/transcribe/latest/dg/what-is.html). + +**Supported Languages.** Amazon Transcribe supports [over 100 languages and locales](https://docs.aws.amazon.com/transcribe/latest/dg/supported-languages.html), with varying levels of feature support and accuracy. Transcription accuracy is highest for English (particularly US English), which benefits from the largest and most diverse training data, the broadest range of evaluation datasets, and the most extensive demographic fairness testing. Accuracy is strong for other high-resource languages but may not match English performance across all conditions. For lower-resource languages, accuracy may be noticeably lower, and performance may be less consistent across acoustic conditions. Feature availability also varies by language: digit transcription, custom vocabularies, and PII redaction are available for a subset of supported languages, with the broadest feature set available for English. Customers should test Amazon Transcribe on their own audio content for each language they intend to use, and should set accuracy expectations based on their own evaluation rather than extrapolating from English performance. + +**Supported Input Formats.** Amazon Transcribe operates on audio input only. Batch transcription is supported in the following supported formats: WAV, MP3, MP4, M4A, FLAC, OGG, AMR, WebM. Streaming transcription is supported in the following supported formats: PCM, OGG, FLAC, G711_ALAW, G711_ULAW, G729. Sampling frequencies from 8kHz to 48kHz are supported for both modes. The maximum supported audio file size is 2GB, and the maximum supported audio length is 4 hours. + +**Appropriateness for Use.** Because its output is probabilistic, Amazon Transcribe may produce inaccurate transcriptions. Customers should evaluate outputs for accuracy and appropriateness for their use case, especially if they will be directly surfaced to end users or used for consequential decisions. Additionally, if Amazon Transcribe is used in customer workflows that produce consequential decisions – such as decisions that may impact an individual's rights, access to services, financial standing, or employment – customers must evaluate the potential risks of their use case and implement appropriate human oversight, testing, and other use case-specific safeguards to mitigate such risks. Customers should treat Amazon Transcribe outputs as an assistive tool and not as a sole basis for consequential determinations. For more information, see the [AWS Responsible AI Policy](https://aws.amazon.com/ai/responsible-ai/policy/). + +**Safety Features.** Amazon Transcribe converts speech to text verbatim, including any inappropriate content spoken in the audio. The service is not a generative AI system, does not produce content beyond its transcription of the provided audio, does not use text prompts, and is therefore not susceptible to prompt injection attacks. To help customers manage sensitive content in transcriptions, Amazon Transcribe offers [vocabulary filtering](https://docs.aws.amazon.com/transcribe/latest/dg/vocabulary-filtering.html), which enables customers to mask or remove words that are sensitive or unsuitable for their audience from transcription results based on a customer-defined word list. Amazon Transcribe also enforces a default vocabulary filter for certain sensitive words. Additionally, customers can use the PII redaction feature to automatically identify and remove personally identifiable information from transcripts and the [Amazon Transcribe Toxicity Detection](https://docs.aws.amazon.com/transcribe/latest/dg/toxicity.html) feature for identifying and classifying toxic voice content. + +**Considerations When Choosing Use Cases** + +When assessing an ASR service for a particular use case, we encourage customers to specifically define the use case, i.e., by considering at least the following factors: the **business problem** being solved; the **stakeholders** in the business problem and deployment process; the **workflow** that solves the business problem, with the model and oversight as components; key system **inputs and outputs** ; the expected intrinsic and confounding **variation** ; and the types of **errors** possible and the relative impact of each. + +Consider the following use case of utilizing Amazon Transcribe to support a contact center analytics workflow that helps a customer service operations team analyze agent-customer phone calls. The **business goal** is to generate accurate transcripts of recorded customer calls for downstream analysis, enabling the operations team to derive insights about conversation quality, customer sentiment, and agent performance. The **stakeholders** include the contact center operations manager, who wants to improve agent productivity and customer satisfaction through data-driven insights, and the contact center agents, whose interactions are transcribed and analyzed. The **workflow** is: 1/ the contact center records customer calls, with one speaker per audio channel (agent on one channel, customer on the other); 2/ recorded audio files are uploaded to Amazon Simple Storage Service; 3/ the operations team submits the audio to Amazon Transcribe via the StartTranscriptionJob API, specifying the language, enabling speaker diarization, and optionally enabling PII redaction; 4/ Amazon Transcribe processes the audio and returns a transcript with speaker labels and timestamps; 5/ the operations team uses the transcribed text with downstream analytics tools to generate insights, which supervisors review to improve processes and training. **Input audio** contains information regarding customer inquiries, complaints, account details, agent responses, hold times, and conversational dynamics. **Output transcripts** contain text representations of the spoken words, speaker attribution, word-level timestamps, confidence scores, and optionally redacted PII. Input **variations** include: 1/ different accents, dialects, and speaking rates of callers and agents; 2/ background noise from the caller's environment (home, car, public space) and the contact center floor; 3/ telephony audio quality at 8kHz sampling rate with potential line noise; 4/ domain-specific terminology related to the customer's industry (such as financial product names, insurance policy terms, or technical support jargon); 5/ overlapping speech when the agent and caller talk simultaneously; 6/ varying call durations and conversation styles (scripted vs. unscripted). The **error types** , ranked in order of estimated negative impact on stakeholders, include: 1/ substitution errors on critical terms such as product names, account numbers, or action items, which could lead to incorrect downstream analytics or decisions (medium-high impact); 2/ deletion errors where important spoken content is missing from the transcript, causing incomplete analysis (medium impact); 3/ insertion errors where words not spoken appear in the transcript, potentially introducing noise into analytics (low-medium impact); 4/ incorrect speaker attribution, where speech is assigned to the wrong speaker, affecting agent performance analysis (medium impact); 5/ PII not correctly identified and redacted, creating a compliance risk (medium-high impact); 6/ PII incorrectly identified and redacted from non-PII content, causing information loss (low-medium impact). With this in mind, we would expect the operations team to test a sample of call recordings through the API, compare the resulting transcripts against human-generated reference transcripts, and review the output for their specific use case. + +Amazon Transcribe processes audio that may contain personal information in the form of human speech. Customers must ensure compliance with all laws and regulations applicable to their use of Amazon Transcribe. This includes understanding obligations under privacy, data protection, and communication laws – including consent, eavesdropping, wiretapping, and recording laws – that apply in their jurisdiction and to the individuals whose speech is being processed. Customers should collect and process only audio that is within the reasonable expectations of the individuals whose speech is being recorded and transcribed. This includes ensuring that all necessary and appropriate consents are obtained from speakers before their audio is submitted to the service, both for the recording of their speech and for the processing of that recording by a cloud-based service. + +Many applications are designed to capture audio from a specific set of speakers (for example, the two parties on a phone call, or participants in a meeting who have consented to recording). However, microphones may also capture speech from individuals who are not the intended users of the application. To avoid unintentionally processing the speech of non-consenting individuals, customers should consider the following: + + * **Microphone placement and scope:** Customers cannot always control who may speak near the recording device. Workflows should be designed to minimize the capture of speech from unintended parties, particularly in public or open environments. + + * **Reasonable expectations of speakers:** Amazon Transcribe should be used only in scenarios where the collection and processing of speech data is within the reasonable expectations of the speakers. + + * **Disclosure to affected individuals:** Where appropriate, customers should inform individuals that their speech is being recorded and processed using automated speech recognition technology. + + + + +Organizations operating in regulated industries or sensitive domains should evaluate any specific legal, regulatory, or policy obligations when using Amazon Transcribe, as it may not be appropriate for every industry or scenario without additional safeguards. Customers who are required to conduct data protection impact assessments or AI risk assessments should consider the information in this Service Card, the data processing descriptions in the Privacy section, and may request additional detail through their AWS account team. + +**Unsupported Use Cases.** Amazon Transcribe is designed and optimized for naturally occurring human speech. The following uses are out of scope and may produce unreliable results: + + * **Synthetic speech:** Amazon Transcribe is not designed for audio generated by text-to-speech (TTS) systems, voice synthesis engines, or other speech generation technology. While the service may produce output for synthetic speech inputs, accuracy is not guaranteed and has not been evaluated for this use case. + + * **Mechanically or digitally transformed speech:** Audio that has been altered through pitch shifting, speed modification, voice disguising, vocoding, or other signal transformations is out of scope. Such transformations change the acoustic characteristics that the model relies on for accurate recognition. + + * **Non-speech audio:** Amazon Transcribe is designed to transcribe human speech. It is not designed to interpret non-speech sounds such as music, tones, alarms, or animal sounds. The service may produce insertion errors (spurious words) when processing audio that contains extended non-speech segments. + + * **Speaker identification or biometric recognition:** As noted under Speaker Diarization, the service labels speakers generically (Speaker 0, Speaker 1, etc.) and cannot identify who a speaker is, match a speaker against a known voice profile, or track speakers across separate audio files or sessions. + + * **Covert surveillance:** Amazon Transcribe may not be used for covert audio surveillance or monitoring of individuals without their knowledge and consent, or in any manner that violates applicable laws and regulations. + + + + +Amazon Transcribe is not intended to support any prohibited practices under applicable AI legislation or any other relevant law. Amazon Transcribe can be integrated into an array of systems such as contact center platforms, content management systems, meeting productivity tools, accessibility solutions, compliance monitoring systems, and media analytics workflows. All Amazon Transcribe use cases must comply with the [AWS Acceptable Use Policy](https://aws.amazon.com/aup/). + +## Design of Amazon Transcribe @@ -38 +101,27 @@ Transcribe Batch (English-US) has many possible applications, such as contact ce -Transcribe is built using ML and ASR technology. It works as follows: (1) Identify relevant acoustic features of the audio input. (2) Generate a set of candidate word-level strings, based on these features. (3) Apply language modeling to rank the candidates and return the top-ranked transcription. See [Amazon Transcribe Developer Guide](https://docs.aws.amazon.com/transcribe/latest/dg/what-is.html) for details of the API calls. +Amazon Transcribe performs automatic speech recognition using machine learning, specifically, a neural encoder model combined with a language model. At a high level, the core service works as follows: 1/ Amazon Transcribe receives audio input (along with configuration parameters such as language code and feature selections) via the API or Console; 2/ the service extracts acoustic features from the audio input; 3/ the acoustic features are processed by a neural encoder that generates probabilities over a set of text tokens at each time frame; 4/ a language model is applied to score and rank candidate transcriptions based on these probabilities; 5/ a beam-search decoding algorithm selects the highest-ranking transcription; 6/ additional feature-specific processing is applied as requested (described below); 7/ the final output, including the transcript, word-level timestamps, confidence scores, and any requested feature outputs, is returned to the customer. + +**Language identification** uses a downstream model that consumes high-level representations from the acoustic encoder to produce language-level output probabilities. The model is trained on multilingual data with balanced sampling across languages to ensure all languages are represented during training. **Speaker diarization** analyzes the audio to differentiate and cluster speakers based on unique voice characteristics, used solely to annotate the transcript with speaker labels and discarded after processing. Speaker labels are assigned within a single audio file or session and are not linked to any speaker identity. **Custom vocabularies** are implemented through a contextual adapter module that biases the encoder's output probabilities toward recognizing customer-specified terms. When a customer provides a custom vocabulary list, the terms are encoded into embedding vectors, and these vectors are used to boost the probability of the corresponding terms during beam-search decoding. **PII identification and redaction** is applied as a post-processing step on the transcription output. The service uses pattern matching and contextual analysis to identify PII entities in the transcript text and replaces them with placeholder tags when redaction is enabled. **Word-level timing** is generated by an alignment model trained on top of the ASR encoder, which produces frame-level token probabilities that are used to estimate the start and end time of each word in the transcript through forced alignment. **Confidence scores** are generated for each word based on the model's output probabilities and are calibrated during each model update cycle to ensure consistency across model versions. + +Amazon Transcribe models are trained and tested on licensed and proprietary datasets, off-the-shelf purchased datasets, and open-source datasets in the public domain. The training data covers a diverse range of acoustic conditions, speaker demographics, languages, and use cases. Training data from licensed and purchased datasets was collected from speakers who consented for their voice recordings to be used for commercial purposes. Training data from open-source datasets was released under a commercial-usage license. Amazon Transcribe may use customer content to develop and improve the service. Customers can opt out of having their content used for service improvement via AWS Organizations or other opt-out mechanisms we may provide. When customers opt out, their content is not used for service improvement. Regardless of opt-out status, customer content is never shared between customers. Amazon Transcribe is available pursuant to the [AWS Customer Agreement](https://aws.amazon.com/agreement/) or other relevant agreements with AWS. + +**Controllability** + + +We say that Amazon Transcribe exhibits a particular "behavior" when it generates the same kind of output for the same kinds of audio inputs and configuration parameters (for example, language code, sample rate, speaker diarization settings, custom vocabulary, and PII redaction settings). For a given model architecture, the control levers that we have over the behaviors are primarily a/ the training data corpus (which we curate from a variety of licensed, proprietary, and publicly available sources), b/ the language models and vocabulary that determine how the service ranks candidate transcriptions, c/ the safety filters (such as the default vocabulary filter) applied to post-process outputs, and d/ customer-facing configuration options that allow customers to tailor the service behavior to their use case. Customers can further steer the behavior of Amazon Transcribe to their specific needs through several mechanisms: + + * **Custom vocabularies** bias the model toward recognizing specific terms that are important to the customer's domain, improving accuracy for those terms without requiring model retraining. + + * **PII redaction** allows customers to control what types of sensitive information are identified and removed from outputs, with configurable entity type selection. + + * **Vocabulary filtering** allows customers to control what words are masked or removed from outputs, based on a customer-defined word list. + + * **Speaker diarization settings** allow customers to specify the expected number of speakers, improving speaker attribution accuracy when the number of speakers is known in advance. + + * **Language identification settings** allow customers to specify a set of candidate languages, narrowing the identification task and improving accuracy. + + * **Confidence score thresholds** can be implemented by customers in their downstream workflows to flag or escalate low-confidence segments for human review. + + + + +These mechanisms serve as compensating controls that customers can use to address known model limitations for their specific use case. The effectiveness of these controls should be evaluated by the customer on their own content. @@ -43 +132 @@ Transcribe is built using ML and ASR technology. It works as follows: (1) Identi -Individual and confounding variation will differ between customer applications. This means that performance will also differ between applications, even if they support the same use case. Consider two transcription applications A and B. Application A enables video captioning for a TV talk show, and has multiple voices per recording channel, high-quality boom microphones, and negligible background noise. Application B helps contact centers record customer calls, and has customers speaking close to their mic, one voice per recording channel, and unscripted customer dialog. Because A and B have differing kinds of inputs, they will likely have differing error rates, even assuming that each application is deployed perfectly using Transcribe. +Intrinsic and confounding variation differ between customer applications. This means that performance will also differ between applications, even if they support the same use case. Consider two applications A and B. Application A enables live captioning for a television news broadcast, with professional presenters speaking clearly into high-quality microphones, scripted speech with domain-specific terminology, and minimal background noise. Application B transcribes customer service calls at a contact center, with callers speaking on mobile phones in noisy environments, unscripted conversational speech with filler words and interruptions, and telephony audio at 8kHz sampling rate. Because A and B have differing kinds of inputs, they will likely have differing error rates, even assuming that each application is deployed perfectly using Amazon Transcribe. In addition, streaming transcription provides real-time results but may have different accuracy characteristics compared to batch transcription, which can process the entire audio file before returning results. Furthermore, performance differs across languages: English (particularly US English) generally delivers the highest accuracy due to the largest and most diverse training data, while lower-resource languages may exhibit higher error rates and greater sensitivity to adverse acoustic conditions. As a result, the overall utility of Amazon Transcribe will depend both on the service and on the workflows it enables. Performance results depend on a variety of factors including Amazon Transcribe itself, the customer workflow, and the evaluation dataset; we recommend that customers test Amazon Transcribe using their own content. @@ -48 +137 @@ Individual and confounding variation will differ between customer applications. -We use multiple datasets to evaluate performance. No single evaluation dataset provides an absolute picture of performance. That’s because evaluation datasets vary based on their demographic makeup (the number and type of defined groups), the amount of confounding variation (quality of content, fit for purpose), the types and quality of labels available, and other factors. We measure Transcribe performance by testing it on evaluation datasets containing audio recordings from a variety of speakers who are representative of the population of end users, where each recording is labeled with ground-truth transcriptions and demographic attributes of the speaker. We represent overall performance on a dataset by several metrics, included word error rate and F1, a percentage that evenly balances the percentage of predicted words that are correct (precision) against the percentage of correct words that are included in the prediction (recall). Groups in a dataset can be defined by demographic attributes (such as gender, age and ancestry), confounding variables (such as recording equipment varieties, the distance of each speaker from recording equipment, post-processing and background noises), or a mix of the two. Different evaluation datasets vary across these and other factors. Because of this, all metrics – both overall and for groups – vary from dataset to dataset. Taking this variation into account, our development process examines Transcribe performance using multiple evaluation datasets, takes steps to increase accuracy for groups on which Transcribe performed least well, works to improve the suite of evaluation datasets, and then iterates. +We use multiple datasets, automated evaluations and human inspection to evaluate the performance of Amazon Transcribe models. No single evaluation dataset suffices to completely capture performance. This is because evaluation datasets vary based on use case, intrinsic and confounding variation, the quality of ground truth available, and other factors. Our development testing involves automated testing against publicly available and proprietary datasets, benchmarking against proxies for anticipated customer use cases, evaluation across multiple languages and acoustic conditions, and iterative refinement based on results. Our development process examines Amazon Transcribe's performance using these tests, takes steps to improve the model and the suite of evaluation datasets, and then iterates. In this Service Card, we provide examples of test results to illustrate our methodology. @@ -50 +139 @@ We use multiple datasets to evaluate performance. No single evaluation dataset p -**Fairness and bias** +**Automated Evaluations:** Automated testing provides consistent comparisons between candidate models by applying standardized metrics across evaluation datasets. The primary metric for transcription accuracy is word error rate (WER), which measures the aggregate of insertion, deletion, and substitution errors divided by the total number of words in the reference transcript. We also measure F1 scores for evaluating the accuracy of specific output elements – such as custom vocabulary terms, named entities, digit transcription, punctuation, and capitalization – which balance the percentage of predicted items that are correct (precision) against the percentage of correct items that are included in the prediction (recall). For features such as speaker diarization, custom vocabulary recognition, punctuation, and capitalization, we use feature-specific metrics including diarization accuracy, custom vocabulary F1 scores, and punctuation and casing F1 scores. Models are evaluated on datasets totaling thousands of hours of audio data, covering multiple languages, acoustic conditions (including telephony at 8kHz and broadband at 16kHz), and speaker demographics. Evaluation datasets include internally commissioned datasets representative of customer use cases, purchased off-the-shelf datasets, and publicly available benchmarking datasets used by the scientific community. All evaluation datasets are versioned, archived, and subject to legal review. @@ -51,0 +141 @@ We use multiple datasets to evaluate performance. No single evaluation dataset p +**Model Quality Monitoring:** Amazon Transcribe employs automated monitoring to track service behavior over time. Canary systems periodically evaluate the service by measuring specific metrics (including WER, latency, and processing speed) against pre-determined reference datasets. These systems automatically trigger alerts when service metrics demonstrate any significant drift or anomalous behavior. Mechanisms are in place to roll back the service to a previously known stable state if needed. @@ -53 +143 @@ We use multiple datasets to evaluate performance. No single evaluation dataset p -Our goal is for Transcribe – Batch (English-US) to work well for speakers of US English across the variety of pronunciations, intonations, vocabularies, and grammatical features that these speakers may use. We consider speaker communities defined by regions, such as the Midwest or New York City, and communities defined by multiple dimensions of identity, including ancestry, age, and gender. To achieve this, we use the iterative development process described above. As part of this process, we build datasets to capture a diverse range of human speakers under a wide range of confounding factors. We routinely test on datasets for which we have reliable demographic labels. We find that Transcribe performs well across demographic attributes. As an example, on one dataset of natural speech with 65 demographic groups, defined by age, ancestry, gender and regional dialect (such as Female+European, Male+Under 45), we find that F1 word recognition accuracy is 92% or higher for every group of speakers. For transcriptions with speaker partitioning (diarization) enabled, on the same dataset we find that diarization accuracy is 98% or higher for every group of speakers. Because results depend on Transcribe, the customer workflow and the evaluation dataset, we recommend that customers additionally test Transcribe on their own content. +**Input Validation:** Amazon Transcribe validates the format of the input audio (for example, encoding, sample rate, and channel configuration) before processing it. Non-adherence to prescribed format specifications will result in an error response. All data used in model development is validated and checked for quality and consistency before use in training or evaluation. @@ -55 +145,6 @@ Our goal is for Transcribe – Batch (English-US) to work well for speakers of U -**Explainability** +**Safety** + + +Safety is a shared responsibility between AWS and our customers. Our goal for safety is to mitigate key risks of concern to our enterprise customers, and to society more broadly. Amazon Transcribe is an ASR service that converts speech to text verbatim. It is not a generative AI service and does not generate any content that is not spoken in the input audio. This fundamental design characteristic means that the service does not produce hallucinated, fabricated, or novel content. Amazon Transcribe does not use text prompts and is not susceptible to prompt injection attacks. + +In a case where a user speaks inappropriate content, the service will transcribe that content verbatim. To help customers manage such scenarios, Amazon Transcribe provides the following safety-relevant features: @@ -56,0 +152 @@ Our goal is for Transcribe – Batch (English-US) to work well for speakers of U +**Vocabulary Filtering:** Customers can define a list of words (e.g., offensive, profane, or otherwise inappropriate words) to mask or remove from transcription results. The default vocabulary filter removes certain sensitive words; customers can supplement this with their own word lists. Note that the default vocabulary filter may not include words from all supported languages; customers should review and supplement the filter for their specific language and use case. @@ -58 +154,16 @@ Our goal is for Transcribe – Batch (English-US) to work well for speakers of U -When Amazon Transcribe transcribes audio, it creates different versions of the same transcript and assigns a confidence score to each version. If customers enable alternative transcriptions, Amazon Transcribe returns alternative versions of the transcript that have lower confidence levels. Customers can explore [alternative transcriptions](https://docs.aws.amazon.com/transcribe/latest/dg/alternatives.html) to gain more insight into candidate words and phrases that were generated for each audio input. +**PII Redaction:** Customers can automatically identify and redact personally identifiable information from transcripts, including names, addresses, credit card numbers, Social Security numbers, and other PII types. However, PII identification is not guaranteed to detect all PII instances in all contexts, and false positives (non-PII content incorrectly identified as PII) may also occur. Customers operating in regulated environments should not rely solely on automated PII redaction without implementing additional review processes appropriate to their compliance requirements. + +**Toxicity Detection:** Amazon Transcribe Toxicity Detection (available as a separate feature) leverages both audio and text-based cues to identify and classify toxic voice content across seven categories including sexual harassment, hate speech, threats, abuse, profanity, insults, and graphic content. Toxicity Detection is not guaranteed to detect all instances of toxic language in all contexts, and false positives (non-toxic content incorrectly identified as toxic) may also occur. Customers operating in regulated environments should not rely solely on automated toxicity detection without implementing additional review processes appropriate to their compliance requirements. Please see the [Service Card for Toxicity Detection](https://docs.aws.amazon.com/ai/responsible-ai/transcribe-toxicity-detection/overview.html) for additional useful information. + +Customers are responsible for end-to-end testing of their applications on datasets representative of their use cases and any additional safety mitigations, and deciding if test results meet their specific expectations of safety, fairness, and other properties, as well as overall effectiveness. + +**Fairness** + + +Amazon Transcribe is designed to work well for speakers across the variety of pronunciations, intonations, vocabularies, and grammatical features that speakers of each supported language may use. We consider speaker communities defined by regions – for example, speakers of different English varieties such as US, British, Australian, Indian, and South African English – and communities defined by multiple dimensions of identity, including accent or ancestry, age, and gender. To measure accuracy across these dimensions, we perform extensive evaluations covering both demographic variation and confounding factors such as acoustic conditions, pronunciation, and speaking rate. For controlled comparison across demographic groups in English, we use evaluation datasets designed to minimize confounding variation, enabling fair comparisons across dimensions such as accent/ancestry, age, and gender. + +We find that Amazon Transcribe performs reasonably well across demographic attributes. As an example, on one dataset of read speech with 23 demographic groups, defined by ancestry, gender, and regional dialect (such as Liberian English, New England English), we find the median WER to be 2.4%, and statistically significant differences (p < 0.001) in accuracy between native (WER of 0.9%) and non-native English speakers (WER of 4.8%). Along the gender dimension, the median error rate for females is slightly lower (less than 1 error difference) than that of males, and the error distributions indicate these differences are statistically significant (p < 0.001). + +For non-English languages, transcription accuracy varies across languages and locales. This variation is expected and reflects differences in the volume and diversity of training data available for each language, the linguistic characteristics of each language, and the representativeness of available evaluation datasets. In general, languages with larger training datasets and more diverse acoustic coverage perform better than languages with limited training data. We apply quality thresholds as launch criteria for all supported languages, and we continuously work to improve accuracy across all languages by expanding training data diversity and volume. For most non-English evaluation datasets, fine-grained demographic information (such as speaker gender, age, and accent) is not available, which limits the measurement of within-language demographic fairness for those languages. We are investing in expanding the demographic coverage of datasets for non-English languages. Customers should evaluate Amazon Transcribe on their own content for each language and should not assume that performance observed for one language will generalize to another. + +Because results depend on Amazon Transcribe, the customer workflow and the evaluation dataset, we recommend that customers additionally test Amazon Transcribe on their own content. @@ -63 +174,31 @@ When Amazon Transcribe transcribes audio, it creates different versions of the s -We maximize robustness with a number of techniques, including using large training datasets that capture many kinds of variation across many individuals. Ideal audio inputs to Transcribe ASR contain audio with high recording quality, low background noise, and low room reverberation. However, Transcribe is trained to be resilient even when inputs vary from ideal conditions and can perform well in noisy and multi-speaker settings. +We maximize robustness with a number of techniques, including using large training datasets that capture many kinds of variation across many speakers, and simulating a variety of acoustic conditions during training (such as different noise types, reverberation levels, and telephony codecs). Ideal audio inputs to Amazon Transcribe contain audio with high recording quality, low background noise, and low room reverberation. However, Amazon Transcribe is trained to be resilient even when inputs vary from ideal conditions and can perform well in noisy and multi-speaker settings. We measure model robustness by evaluating performance across a range of acoustic conditions, including clean speech, moderate and high background noise, far-field recording, overlapping speech, and telephony audio. Amazon Transcribe models degrade gracefully under adverse conditions rather than producing catastrophic errors. Additionally, given the architecture of the model (a neural encoder combined with a language model operating on continuous-valued audio inputs) the risk of adversarial attacks such as training data extraction or membership inference is significantly lower compared to text-based generative models. We are not aware of published work demonstrating such attacks on encoder-only ASR architectures of this type. + +**Explainability** + + +When Amazon Transcribe transcribes audio, it assigns a confidence score to each word in the transcript, indicating the service's confidence that the word was correctly recognized. Confidence scores are calibrated during each model update cycle to ensure consistency across model versions, which supports the stability of downstream workflows that rely on specific confidence thresholds. Customers can use these confidence scores to identify portions of the transcript that may require additional review, to implement quality thresholds appropriate for their use case, or to trigger human review for low-confidence segments. Additionally, Amazon Transcribe provides word-level timing information, indicating the start and end time of each recognized word in the audio. This "acoustic grounding" can help customers identify challenging portions of the audio where the service may have made errors, and facilitates alignment of transcripts with the source audio for verification purposes. If customers enable alternative transcriptions (available in batch mode), Amazon Transcribe returns alternative versions of the transcript that have lower confidence levels. Customers can explore alternative transcriptions to gain more insight into candidate words and phrases that were generated for each audio input. + +**Privacy** + + +Amazon Transcribe processes only audio input data. Audio inputs are never included in the output returned by the service. Inputs and outputs are never shared between customers. Amazon Transcribe may use customer content to develop and improve the service. Customers can opt out of having their content used for service improvement via AWS Organizations or other opt-out mechanisms we may provide. When customers opt out, their content is not used for service development or improvement. Regardless of opt-out status, inputs and outputs are never shared between customers. For more information, see Section 50.3 of the [AWS Service Terms](https://aws.amazon.com/service-terms/) and the [AWS Data Privacy FAQs](https://aws.amazon.com/compliance/data-privacy-faq/). For service-specific privacy and security information, see [Amazon Transcribe FAQs](https://aws.amazon.com/transcribe/faqs/) and [Amazon Transcribe Security](https://docs.aws.amazon.com/transcribe/latest/dg/security.html). + +**How Amazon Transcribe processes data.** Amazon Transcribe processes audio input directly within the AWS infrastructure in the AWS Region where the customer is using the service. The processing flow differs by mode: + + * **Batch transcription:** Customers specify their own Amazon Simple Storage Service (Amazon S3) storage location for both the input audio files and the output transcript files. Amazon Transcribe reads the audio from the customer-specified location, processes it, and writes the transcript output back to the customer-specified location. The customer controls the storage and retention of both input and output data. + + * **Streaming transcription:** Audio is streamed to the service in real-time. The audio is processed on AWS server memory, and transcription results are returned to the customer in real-time within the same connection. The service does not separately store audio input or transcription output data at rest beyond the duration of the streaming session. + + * **Speaker diarization:** When customers enable speaker diarization, the speech recognition engine analyzes voice characteristics in the audio to differentiate between speakers. These voice characteristic signals are used solely for the purpose of annotating the transcript with speaker labels and are not retained after processing is complete. Speaker diarization does not support speaker identity recognition and cannot track speakers across separate audio files. + + * **Language identification:** When language identification is enabled, the service analyzes acoustic characteristics of the audio to determine the spoken language. This analysis is performed as part of the transcription process and no additional data is stored.