AWS nova documentation change
Summary
Added guidance on when to use supervised fine-tuning vs RAG, fixed typos, improved formatting, and added comprehensive section on designing effective training examples with data transformation patterns
Security assessment
The changes are primarily documentation improvements and guidance on data preparation for fine-tuning. While one example mentions multi-factor authentication reset procedures, this is just an example of procedural documentation transformation, not documentation of security features or addressing security vulnerabilities. No evidence of addressing specific security issues or vulnerabilities.
Diff
diff --git a/nova/latest/nova2-userguide/fine-tune-prepare-data-understanding.md b/nova/latest/nova2-userguide/fine-tune-prepare-data-understanding.md index 8556c20f9..c5ff669dd 100644 --- a//nova/latest/nova2-userguide/fine-tune-prepare-data-understanding.md +++ b//nova/latest/nova2-userguide/fine-tune-prepare-data-understanding.md @@ -7 +7 @@ -Example dataset formatsDataset constraints +Designing effective training examplesExample dataset formatsDataset constraints @@ -10,0 +11,4 @@ Example dataset formatsDataset constraints +###### Important + +Before you begin preparing your dataset, make sure supervised fine-tuning (SFT) is the right approach for your use case. SFT teaches the model new _behaviors_ , response formats, and reasoning patterns. It does not teach the model new factual knowledge. If your primary goal is to introduce domain-specific facts, terminology, or knowledge the model hasn't seen, consider retrieval-augmented generation (RAG) to supply that context at inference time. For guidance on choosing between SFT, reinforcement fine-tuning (RFT), and RAG, see [](./.html#custom-fine-tune-models-sagemaker-tj). + @@ -17 +21 @@ The following are guidelines and requirements for preparing data for fine-tuning - 3. Traning and validation datasets must be JSONL files, where each line is a JSON object corresponding to a record. These file names can consist of only alphanumeric characters, underscores, hyphens, slashes and dots. + 3. Training and validation datasets must be JSONL files, where each line is a JSON object corresponding to a record. These file names can consist of only alphanumeric characters, underscores, hyphens, slashes, and dots. @@ -33 +37 @@ The following are guidelines and requirements for preparing data for fine-tuning - 8. The first turn in `messages` should always start with `"role": "user"`. The last turn is the bot's response, denoted by "role": "assistant". + 8. The first turn in `messages` should always start with `"role": "user"`. The last turn is the bot's response, denoted by `"role": "assistant"`. @@ -41 +45 @@ The following are guidelines and requirements for preparing data for fine-tuning - 12. The terms `User:`, `Bot:`, `Assistant:`, `System:`, `<image>`, `<video>` and `[EOS]` are reserved keywords. If a user prompt or system prompt starts with any of these keywords, or have these keywords anywhere in their prompts, your training job will fail due to data issues. If you need to use these keywords for your use case, you must substitute it for a different keyword with a similar meaning so that your training can proceed. + 12. The terms `User:`, `Bot:`, `Assistant:`, `System:`, `<image>`, `<video>`, and `[EOS]` are reserved keywords. If a user prompt or system prompt starts with any of these keywords, or has these keywords anywhere in the prompt, your training job will fail due to data issues. If you need to use these keywords for your use case, you must substitute them for different keywords with similar meanings so that your training can proceed. @@ -49,0 +54,126 @@ To validate your dataset before submitting a fine-tuning job, you can use the [d +## Designing effective training examples + +Your training data should demonstrate the _behavior_ you want the model to exhibit. SFT teaches the model _how_ to respond, not _what_ to know. If you find yourself creating training examples primarily to inject factual knowledge (for example, "What does error code E-45 mean?" with the answer "E-45 indicates a sensor timeout"), consider whether RAG or prompt engineering could achieve the same result without fine-tuning. + +When converting source data into the conversational format, follow these principles: + + 1. **Start from real user queries.** Write user turns that mirror how end users will actually prompt the model in production. Avoid artificial or overly simplified questions that don't reflect real usage patterns. + + 2. **Write gold-standard assistant responses.** Each assistant turn should be the ideal response you want the model to produce — correct, well-formatted, and consistent in tone and structure across all examples. + + 3. **Use the system prompt strategically.** Place persistent instructions such as output format, persona, and domain context in the system turn. Use the same system prompt during both training and inference to achieve the best results. + + 4. **Keep examples focused on one skill per sample.** If you need the model to classify, answer questions, and summarize, create separate example sets for each skill rather than combining multiple skills in a single sample. + + 5. **Prioritize quality over quantity.** A few hundred high-quality, consistent examples typically outperform thousands of noisy or contradictory ones. Deduplicate and review your data before training. + + + + +### Common data transformation patterns + +The following patterns illustrate how to convert common enterprise data formats into effective SFT training examples. + +###### Pattern 1: Structured records to classification or lookup + +When your source data is a table or database of records with fields such as `issue`, `cause`, and `resolution`, frame each record as a user question with the relevant input fields and the assistant response as the target output fields. + + + // train.jsonl + { + "schemaVersion": "bedrock-conversation-2024", + "system": [ + { + "text": "You are a technical support assistant. Given an issue description, identify the most likely root cause and recommended resolution." + } + ], + "messages": [ + { + "role": "user", + "content": [ + { + "text": "A user reports: 'Application crashes on startup after the latest update.' What is the likely cause and resolution?" + } + ] + }, + { + "role": "assistant", + "content": [ + { + "text": "Cause: Incompatible configuration file from the previous version.\nResolution: Delete the local configuration cache and restart the application to regenerate default settings." + } + ] + } + ] + } + +###### Pattern 2: Hierarchical or taxonomy data to structured Q&A + +When your source data is a code hierarchy or taxonomy such as product category trees or classification systems, generate questions that ask the model to navigate the hierarchy. Include the classification context in the system prompt or user turn. + + + // train.jsonl + { + "schemaVersion": "bedrock-conversation-2024", + "system": [ + { + "text": "You are a product catalog assistant. Answer questions about product categories and classifications accurately." + } + ], + "messages": [ + { + "role": "user", + "content": [ + { + "text": "What top-level category does product code 'EL-4520' belong to, and what subcategories does it include?" + } + ] + }, + { + "role": "assistant", + "content": [ + { + "text": "Product code EL-4520 belongs to the Electronics category. It includes the subcategories: Audio Equipment, Portable Speakers, and Wireless Accessories." + } + ] + } + ] + } + +###### Pattern 3: Procedural documents to instructional dialog + +When your source data consists of standard operating procedures, runbooks, or step-by-step guides, convert procedures into task-oriented questions. The user asks how to perform a task and the assistant provides the steps in a consistent format. + + + // train.jsonl + { + "schemaVersion": "bedrock-conversation-2024", + "system": [ + { + "text": "You are an IT operations assistant. Provide step-by-step instructions based on approved runbooks and standard operating procedures." + } + ], + "messages": [ + { + "role": "user", + "content": [ + { + "text": "How do I reset a user's multi-factor authentication when they have lost access to their device?" + } + ] + }, + { + "role": "assistant", + "content": [ + { + "text": "Follow these steps to reset a user's multi-factor authentication:\n1. Verify the user's identity using the approved verification procedure.\n2. Disable the existing MFA configuration in the identity management console.\n3. Generate a new temporary access code and send it to the user's verified email.\n4. Instruct the user to log in with the temporary code and re-enroll their new device.\n5. Confirm the new MFA device is working by requesting a test authentication.\n6. Document the reset in the support ticket." + } + ] + } + ] + } + +###### Note + +In Patterns 1 and 2, if the factual content (such as codes, categories, or resolutions) is not already in the model's pre-training data, consider using RAG to supply this information at inference time rather than relying solely on SFT to memorize it. SFT is most effective for teaching the model the response format and reasoning pattern, while RAG handles the factual grounding. +