AWS nova documentation change
Summary
Added documentation for few-shot exemplars, document content extraction to Markdown, and vision inference parameters. Updated examples and templates for multi-image handling.
Security assessment
The changes focus on improving model performance through prompt engineering techniques and output formatting. While base64 encoding is mentioned for image handling, this is standard data representation rather than a security measure. No vulnerabilities, security controls, or security-related features are discussed.
Diff
diff --git a/nova/latest/userguide/prompting-vision-prompting.md b/nova/latest/userguide/prompting-vision-prompting.md index b80a00d9e..35866716f 100644 --- a//nova/latest/userguide/prompting-vision-prompting.md +++ b//nova/latest/userguide/prompting-vision-prompting.md @@ -5 +5 @@ -Placement mattersMultiple media filesImproved instruction followingBounding box detectionRicher outputs or style +Placement mattersMultiple media filesImproved instruction followingFew shot exemplarsBounding box detectionRicher outputs or styleExtract document contents into MarkdownInference parameter settings for vision understanding @@ -17 +17,3 @@ The following vision prompting techniques will help you create better prompts fo - * Improved instruction following for video understanding + * Use user instructions for improved instruction following for vision understanding tasks + + * Few shot exemplars @@ -22,0 +25,4 @@ The following vision prompting techniques will help you create better prompts fo + * Extract document contents into Markdown + + * Inference parameter settings for vision understanding + @@ -63,7 +68 @@ The following template can be used to place multiple media files: - "content": [ - { - "image 1": "..." - }, - { - "image 2": "..." - }, + messages = [ @@ -71 +70,14 @@ The following template can be used to place multiple media files: - "text": "Describe what you see in the second image." + "role": "user", + "content": [ + {"text":"Image 1:"}, + {"image": {"format": "jpeg", "source": {"bytes": img_1_base64}}}, + {"text":"Image 2:"}, + {"image": {"format": "jpeg", "source": {"bytes": img_2_base64}}}, + {"text":"Image 3:"}, + {"image": {"format": "jpeg", "source": {"bytes": img_3_base64}}}, + {"text":"Image 4:"}, + {"image": {"format": "jpeg", "source": {"bytes": img_4_base64}}}, + {"text":"Image 5:"}, + {"image": {"format": "jpeg", "source": {"bytes": img_5_base64}}}, + {"text":user_prompt}, + ], @@ -78,2 +89,2 @@ Unoptimized Prompt | Optimized Prompt -Describe what you see in the second image. [Image1.png] [image2.png] | [Image1.png] [Image2.png] Describe what you see in the second image. -Is the second image described in the included document? [Image1.png] [image2.png] [Document1.pdf] | [Image1.png] [Image2.png] [Document1.pdf] Is the second image described in the included document? +Describe what you see in the second image. [Image1.png] [Image2.png] | [Image1.png] [Image2.png] Describe what you see in the second image. +Is the second image described in the included document? [Image1.png] [Image2.png] [Document1.pdf] | [Image1.png] [Image2.png] [Document1.pdf] Is the second image described in the included document? @@ -83 +94 @@ Due to the long context tokens of the media file types, the system prompt indica -## Improved instruction following for video understanding +## Use user instructions for improved instruction following for vision understanding tasks @@ -105,0 +117,18 @@ The following template can be used to for improved instructions: +Just like for text, we recommended applying chain-of-thought for images and videos to gain improved performances. We also recommended that you place the chain-of-thought directives in the system prompt, while keeping other instructions in the user prompt. + +###### Important + +The Amazon Nova Premier model is a higher intelligence model in the Amazon Nova family, able to handle more complex tasks. If your tasks require advanced chain-of-thought thinking, we recommend that you utilize the prompt template provided in [Give Amazon Nova time to think (chain-of-thought)](https://docs.aws.amazon.com/nova/latest/userguide/prompting-chain-of-thought.html). This approach can help enhance the model's analytical and problem-solving abilities. + +## Few shot exemplars + +Just like for text models, we recommend that you provide examples of images for improved image understanding performance (videos exemplars cannot be provided, due to the single-video-per-inference limitation). We recommended that you place the examples in the user prompt, after the media file, as opposed to providing it in the system prompt. + +| 0-Shot | 2-Shot +---|---|--- +User | | [Image 1] +Assistant | | The image 1 description +User | | [Image 2] +Assistant | | The image 2 description +User | [Image 3] Explain what is happening in the image | [Image 3] Explain what is happening in the image + @@ -127,0 +157,37 @@ Further modifications to the responses can be achieved with [one-shot and few-sh +## Extract document contents into Markdown + +Amazon Nova Premier demonstrates enhanced capabilities to understand charts embedded within documents and the ability to read and comprehend content from complex domains such as scientific papers. Additionally, Amazon Nova Premier shows improved performance when extracting document contents and can output this information into Markdown Table and Latex formats. + +The following example provides a table in an image, along with a prompt for Amazon Nova Premier convert the contents of the image to a Markdown table. After the Markdown (or Latex Representation) is created, you can use tools to convert the content into JSON or other structured output. + + + Make a table representation in Markdown of the image provided. + + + +Here is the output provided by the model: + + + | No. | DATE | DESCRIPTION | AMOUNT | + |-----|------------|-------------|--------| + | 1 | 2021-04-19 | Guest room | $180 | + | 2 | 2021-04-19 | Breakfast | $30 | + | 3 | 2021-04-20 | Guest room | $180 | + | 4 | 2021-04-20 | Laundry | $15 | + | 5 | 2021-04-20 | Dinner | $70 | + | 6 | 2021-04-20 | Breakfast | $30 | + | 7 | 2021-04-21 | Guest room | $180 | + | 8 | 2021-04-21 | Breakfast | $30 | + | 9 | 2021-04-21 | Dinner | $70 | + | 10 | 2021-04-21 | Laundry | $15 | + | | | Subtotal | $800 | + | | | Discounts | $24 | + | | | Taxes | $68.9 | + | | | Total | $844.9 | + +This output uses custom table notation where `||` is used as the column separator and `&&` is used as the row separator. + +## Inference parameter settings for vision understanding + +For vision understanding use cases, we recommended that you start with the inference parameters `temperature` set to `0` and `topK` set to `1`. After observing the model's output, you can then adjust the inference parameters based on the use case. These values usually depend on the task and variance required, increase the temperature setting to induce more variations in the answers. + @@ -136 +202 @@ Vision understanding -Generating creative content +General prompting tips