AWS bedrock: Pegasus 1.5 model params: prompt_v2, responseFormat types, time ranges
Summary
Documents new TwelveLabs Pegasus 1.5 request/response parameters: prompt_v2 with image media_sources referenced by <@name> placeholders, responseFormat.type (json_schema / segment_definitions) with timestamp schema type, start_time/end_time clip selection, segment_time_format, enum-restricted segment fields, same-Region S3 bucket requirement for mediaSource.s3Location.uri, and the optional _definition_id in segment responses, plus new request/response examples.
Security assessment
The change is purely API/parameter reference documentation for a Bedrock model: it describes new prompt_v2 image inputs, responseFormat type values, clip time ranges, output formatting, and validation behaviors. The Region-matching constraint for the S3 source is a functional/config requirement (data residency-adjacent) rather than a stated fix for a vulnerability, and no CVE, advisory, authz, credential, or encryption control is mentioned, so it is not security documentation.
Evidence
+`mediaSource.s3Location.uri` | string | No | S3 URI where the video can be downloaded from. Max: up to 2 hours long (less than 2 GB file size). The bucket must be in the same AWS Region that you send the request to. This applies to the geo and global inference profiles as well: the profile determines where the request can be served, but it does not widen where the bucket can live. A bucket in any other Region fails with a `ValidationException`.
Diff
diff --git a/bedrock/latest/userguide/model-parameters-pegasus-1-5.md b/bedrock/latest/userguide/model-parameters-pegasus-1-5.md index ad7116a..55f6450 100644 --- a//bedrock/latest/userguide/model-parameters-pegasus-1-5.md +++ b//bedrock/latest/userguide/model-parameters-pegasus-1-5.md @@ -40,2 +40,8 @@ TwelveLabs Pegasus 1.5 request parameters Field | Type | Required | Description -`analysis_mode` | string | No | The analysis to perform. Valid values: `general` (default) returns free-form text analysis driven by `inputPrompt`; `time_based_metadata` returns structured, timestamped segments defined by `segment_definitions`. -`inputPrompt` | string | Conditional | Prompt to analyze the video. Required when `analysis_mode` is `general`. +`analysis_mode` | string | No | The analysis to perform. Valid values: `general` (default) returns free-form text analysis driven by `inputPrompt` or `prompt_v2`; `time_based_metadata` returns structured, timestamped segments defined by `segment_definitions`. +`inputPrompt` | string | Conditional | Prompt to analyze the video. Required when `analysis_mode` is `general`, unless you supply `prompt_v2` instead. Must be omitted when `analysis_mode` is `time_based_metadata`; including it returns a `ValidationException`. In that mode the `segment_definitions` entries describe what to extract instead. +`prompt_v2` | object | Conditional | An alternative to `inputPrompt` that can reference images from the prompt text. Use either `inputPrompt` or `prompt_v2`, not both; supplying both returns a `ValidationException`. Like `inputPrompt`, it must be omitted when `analysis_mode` is `time_based_metadata`. A video in `mediaSource` is still required: a request that supplies `prompt_v2` and no video fails. +`prompt_v2.input_text` | string | Conditional | The prompt text. Required when you use `prompt_v2`. To refer to an image, include a placeholder of the form `<@`name`>`, where `name` matches the `name` of an entry in `media_sources`. A placeholder with no matching entry returns a `ValidationException`. +`prompt_v2.media_sources` | array | No | Images that `input_text` can refer to by placeholder. Each entry takes a `name`, a `media_type`, and a `base64String`. These images supplement the video; they don't replace it. +`prompt_v2.media_sources[].name` | string | Conditional | The name used to refer to this image from `input_text`, as `<@`name`>`. Required for each entry. +`prompt_v2.media_sources[].media_type` | string | Conditional | The kind of media in this entry. The only valid value is `image`. Required for each entry. +`prompt_v2.media_sources[].base64String` | string | Conditional | Base64 encoded byte string for the image. Required for each entry. @@ -44 +50 @@ TwelveLabs Pegasus 1.5 request parameters Field | Type | Required | Description -`mediaSource.s3Location.uri` | string | No | S3 URI where the video can be downloaded from. Max: up to 2 hours long (less than 2 GB file size). +`mediaSource.s3Location.uri` | string | No | S3 URI where the video can be downloaded from. Max: up to 2 hours long (less than 2 GB file size). The bucket must be in the same AWS Region that you send the request to. This applies to the geo and global inference profiles as well: the profile determines where the request can be served, but it does not widen where the bucket can live. A bucket in any other Region fails with a `ValidationException`. @@ -48,2 +54,10 @@ TwelveLabs Pegasus 1.5 request parameters Field | Type | Required | Description -`responseFormat` | object | No | Specifies a structured-output JSON Schema. Set the schema object directly under `responseFormat.jsonSchema`. Applies to `general` analysis mode. -`segment_definitions` | array | Conditional | Required when `analysis_mode` is `time_based_metadata`. Each entry defines a segment type to extract, with an `id`, a `description`, and a list of `fields` (each with `name`, `type`, and `description`) to populate per segment. +`responseFormat` | object | No | Specifies a structured output format. When you include `responseFormat`, you must also set `responseFormat.type`; omitting it returns a `ValidationException`. Note that this `type` is a sibling of `jsonSchema` and is not the same as the `type` inside the schema, which describes the schema's own root type. Valid values, each tied to an analysis mode: + + * `json_schema` — required when `analysis_mode` is `general`. Supply the schema under `responseFormat.jsonSchema`. In addition to the standard JSON Schema types, a property can use `"type": "timestamp"` with a `format` of `seconds` or `hh:mm:ss.fff` to have the model return a video timestamp. + * `segment_definitions` — required when `analysis_mode` is `time_based_metadata`. Supply the definitions under `responseFormat.segment_definitions`, as an alternative to the top-level `segment_definitions` field. + +Using a value that doesn't match the analysis mode returns a `ValidationException`. You can omit `responseFormat` entirely, in which case `general` mode returns free-form text and `time_based_metadata` mode uses the top-level `segment_definitions`. +`start_time` | double | No | Analyze only the portion of the video starting at this offset, in seconds. Use with `end_time` to analyze a clip rather than the whole video. Applies to both analysis modes; in `time_based_metadata` mode only segments inside the clip are returned. +`end_time` | double | No | Analyze only the portion of the video ending at this offset, in seconds. Must be greater than `start_time`; otherwise the request returns a `ValidationException`. +`segment_time_format` | string | No | Controls how segment boundaries are formatted in the response when `analysis_mode` is `time_based_metadata`. Use `seconds` to return `start_sec` and `end_sec` as numbers, or `hh:mm:ss.fff` to return them as timestamp strings. +`segment_definitions` | array | Conditional | Required when `analysis_mode` is `time_based_metadata`. Each entry defines a segment type to extract, with an `id`, a `description`, and a list of `fields` (each with `name`, `type`, and `description`) to populate per segment. A field can also carry an `enum` array, which restricts the model's output for that field to the listed values. @@ -59 +73 @@ TwelveLabs Pegasus 1.5 response fields Field | Type | Description -`segments` | array | Present when `analysis_mode` is `time_based_metadata`. The list of extracted segments, each carrying the fields declared in the matching `segment_definitions` entry along with its time range. +`segments` | array | Present when `analysis_mode` is `time_based_metadata`. The list of extracted segments, each carrying the fields declared in the matching `segment_definitions` entry along with its `start_sec` and `end_sec` time range. When you supply two or more `segment_definitions` entries, each segment also carries a `_definition_id` field holding the `id` of the definition that produced it. With a single definition, no identifier field is returned. @@ -118,0 +133 @@ The following examples show request formats for the TwelveLabs Pegasus 1.5 model + "type": "json_schema", @@ -130,0 +146,23 @@ The following examples show request formats for the TwelveLabs Pegasus 1.5 model +**Referring to an image from the prompt:** + + + { + "prompt_v2": { + "input_text": "Does the logo in <@logo> appear anywhere in this video?", + "media_sources": [ + { + "name": "logo", + "media_type": "image", + "base64String": "base64-encoded image" + } + ] + }, + "mediaSource": { + "s3Location": { + "uri": "s3://path-to-video-object-in-s3", + "bucketOwner": "bucket-owner-account-id" + } + }, + "maxOutputTokens": 4096 + } + @@ -177,0 +216,2 @@ The following examples show response formats from the TwelveLabs Pegasus 1.5 mod +The following response corresponds to the single `segment_definitions` entry in the preceding request. Each segment carries the fields declared in that definition along with its time range, and no identifier field is present. + @@ -182 +221,0 @@ The following examples show response formats from the TwelveLabs Pegasus 1.5 mod - "id": "scenes", @@ -188 +226,0 @@ The following examples show response formats from the TwelveLabs Pegasus 1.5 mod - "id": "scenes", @@ -198,0 +237,41 @@ The following examples show response formats from the TwelveLabs Pegasus 1.5 mod +**Response with time-based metadata and multiple definitions:** + +When the request supplies two or more `segment_definitions` entries, each segment additionally carries a `_definition_id` field holding the `id` of the definition that produced it. Use that field to correlate a segment back to its definition. The following response is for a request with the definitions `scenes` and `colors`. + + + { + "segments": [ + { + "start_sec": 0, + "end_sec": 12, + "summary": "Opening shot establishing the location", + "_definition_id": "scenes" + }, + { + "start_sec": 0, + "end_sec": 12, + "palette": "Muted blues and greys", + "_definition_id": "colors" + }, + { + "start_sec": 12, + "end_sec": 30, + "summary": "Main subject enters and the action begins", + "_definition_id": "scenes" + }, + { + "start_sec": 12, + "end_sec": 30, + "palette": "Warm reds and oranges", + "_definition_id": "colors" + } + ], + "definition_errors": [], + "stopReason": "stop", + "finish_reason": "stop" + } + +###### Note + +The segment identifier is named `_definition_id`, not `id`. The `id` field appears only on the request side, in each `segment_definitions` entry. The response carries `_definition_id` only when the request supplies two or more definitions, so a parser should treat it as optional. This behavior is the same for `InvokeModel` and `InvokeModelWithResponseStream`; in the streaming case the `segments` array arrives in the final chunk. +