AWS sagemaker-unified-studio documentation change
Summary
Restructured documentation to add a new 'Create a knowledge base with a web crawler' section with detailed steps, reorganized existing content under 'Web crawler behavior', updated terminology consistency (e.g., 'Web Crawler' to 'web crawler'), and added explicit instructions for crawler configuration.
Security assessment
Changes focus on usability improvements and procedural guidance without addressing vulnerabilities. Security mentions (robots.txt compliance and Acceptable Use Policy) are maintained from previous version but not expanded. No evidence of patching vulnerabilities or incident response.
Diff
diff --git a/sagemaker-unified-studio/latest/userguide/data-source-document-web-crawler.md b/sagemaker-unified-studio/latest/userguide/data-source-document-web-crawler.md index dc5f652f9..612c0eb86 100644 --- a//sagemaker-unified-studio/latest/userguide/data-source-document-web-crawler.md +++ b//sagemaker-unified-studio/latest/userguide/data-source-document-web-crawler.md @@ -5 +5 @@ -Source URLsWebsite domain range for crawling URLsUse a URL regex filter to include or exclude URLsThrottle crawling speedIncremental syncing +Web crawler behaviorCreate a knowledge base with a web crawler @@ -7 +7 @@ Source URLsWebsite domain range for crawling URLsUse a URL regex filter to inclu -# Web crawler data source +# Use a web crawler as a data source @@ -11 +11 @@ The Amazon Bedrock in SageMaker Unified Studio provided Web Crawler connects to -The Web Crawler connects to and crawls HTML pages starting from the seed URL, traversing all child links under the same top primary domain and path. If any of the HTML pages reference supported documents, the Web Crawler will fetch these documents, regardless of if they are within the same top primary domain. +The web brawler connects to and crawls HTML pages starting from the seed URL, traversing all child links under the same top primary domain and path. If any of the HTML pages reference supported documents, the Web Crawler will fetch these documents, regardless of if they are within the same top primary domain. @@ -13 +13 @@ The Web Crawler connects to and crawls HTML pages starting from the seed URL, tr -The following is supported for you to: +The web crawler lets you: @@ -17 +17 @@ The following is supported for you to: - * Respect standard robots.txt directives like 'Allow' and 'Disallow' + * Respect standard `robots.txt` directives like 'Allow' and 'Disallow' @@ -27,0 +28,11 @@ There are limits to how many web page content items and MB per content item that +###### Topics + + * Web crawler behavior + + * Create a knowledge base with a web crawler + + + + +## Web crawler behavior + @@ -30 +41 @@ You can modify the crawling behavior by changing the following configuration cha -## Source URLs +### Source URLs @@ -36 +47 @@ You specify the source URLs that you want the Knowledge Base to crawl. Before yo - * Check the path to robots.txt corresponding to your source URLs doesn't block the URLs from being crawled. The Web Crawler adheres to the standards of robots.txt: `disallow` by default if robots.txt is not found for the website. The Web Crawler respects robots.txt in accordance with the [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html). + * Check the path to robots.txt corresponding to your source URLs doesn't block the URLs from being crawled. The web crawler adheres to the standards of robots.txt: `disallow` by default if robots.txt is not found for the website. The web crawler respects robots.txt in accordance with the [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html). @@ -45 +56 @@ You specify the source URLs that you want the Knowledge Base to crawl. Before yo -When selecting websites to crawl, you must adhere to the [Amazon Acceptable Use Policy](https://aws.amazon.com/aup/) and all other Amazon terms. Remember that you must only use the Web Crawler to index your own web pages, or web pages that you have authorization to crawl. +When selecting websites to crawl, you must adhere to the [Amazon Acceptable Use Policy](https://aws.amazon.com/aup/) and all other Amazon terms. Remember that you must only use the web crawler to index your own web pages, or web pages that you have authorization to crawl. @@ -51 +62 @@ Make sure you are not crawling potentially excessive web pages. We recommend tha -## Website domain range for crawling URLs +### Website domain range for crawling URLs @@ -72 +83 @@ Make sure you are not crawling potentially excessive web pages. It's not recomme -## Use a URL regex filter to include or exclude URLs +### Use a URL regex filter to include or exclude URLs @@ -82 +93 @@ An example of a regular expression filter pattern to exclude URLs that end with -## Throttle crawling speed +### Throttle crawling speed @@ -86 +97 @@ You can set the number of URLs that Amazon Bedrock in SageMaker Unified Studio c -## Incremental syncing +### Incremental syncing @@ -88 +99 @@ You can set the number of URLs that Amazon Bedrock in SageMaker Unified Studio c -Each time the the Web Crawler runs, it retrieves content for all URLs that are reachable from the source URLs and which match the scope and filters. For incremental syncs after the first sync of all content, Amazon Bedrock will update your knowledge base with new and modified content, and will remove old content that is no longer present. Occasionally, the crawler may not be able to tell if content was removed from the website; and in this case it will err on the side of preserving old content in your knowledge base. +Each time the the web crawler runs, it retrieves content for all URLs that are reachable from the source URLs and which match the scope and filters. For incremental syncs after the first sync of all content, Amazon Bedrock will update your knowledge base with new and modified content, and will remove old content that is no longer present. Occasionally, the crawler may not be able to tell if content was removed from the website; and in this case it will err on the side of preserving old content in your knowledge base. @@ -91,0 +103,71 @@ To sync your data source with your knowledge base, see [Synchronize an Amazon Be +## Create a knowledge base with a web crawler + +###### To create a Knowledge Base with a web crawler + + 1. Navigate to the Amazon SageMaker Unified Studio landing page by using the URL from your administrator. + + 2. Access Amazon SageMaker Unified Studio using your IAM or single sign-on (SSO) credentials. For more information, see [Access Amazon SageMaker Unified Studio](./getting-started-access-the-portal.html). + + 3. Choose the **Build** menu at the top of the page. + + 4. In the **MACHINE LEARNING & GENERATIVE AI** section, choose **My apps**. + + 5. In the **Select or create a new project to continue** dialog box, select the project that you want to use. + + 6. In the left pane, choose **Asset gallery**. + + 7. Choose **My components**. + + 8. In the **Components** section, choose **Create component** and then **Knowledge Base**. The **Create Knowledge Base** pane is shown. + + 9. For **Name** , enter a name for the Knowledge Base. + + 10. For **Description** , enter a description for the Knowledge Base. + + 11. In **Select data source type** , do one of the following: + + * Use a document as a data source by doing the following: + + 1. Select **Local file**. + + 2. Choose **Click to upload** and upload the document that you want the Knowledge Base to use. Alternatively, add your source documents by dragging and dropping the document from your computer. + +For more information, see [Use a Local file as a data source](./data-source-document.html). + + * Use a web crawler as a data source by doing the following: + + 1. Select **Web crawler**. + + 2. Provide the **Source URLs** of the URLs you want to crawl. You can add up to 9 additional URLs by selecting **Add Source URLs**. By providing a source URL, you are confirming that you are authorized to crawl its domain. + + 3. (Optional) Choose **Edit advanced web crawler configs** to make the following optional configuration changes: + + * **Website domain range**. Set the domain that you want the Knowledge Base to crawl. For more information, see Website domain range for crawling URLs. + + * **Maximum throttling of crawling speed**. Set the speed at which the Knowledge Base crawls through the source URLs. For more information, see Throttle crawling speed. + + * **URL regex filter**. Set regex filters for including (**Include patterns**) or excluding **Exclude patterns** URLS from the web crawl. For more information, see Use a URL regex filter to include or exclude URLs. + + * Choose **Back** to leave the web crawler configuration pane. + + 12. For **parsing** Choose either **default** parsing or choose **parsing with foundation model**. + + 13. If you choose **parsing with foundation model** , do the following: + + 1. For **Choose a foundation model for parsing** select your preferred foundation model. You can only choose models that your administrator has enabled for parsing. If you don't see a suitable model, contact your administrator. + + 2. (Optional) Overwrite the **Instructions for the parser** to suit your specific needs. + + 14. (Optional) For **Embeddings model** , choose a model for converting your data into vector embeddings, or use the default model. + + 15. Choose **Create** to create the Knowledge Base. + + 16. Use the Knowledge Base in an app, by doing one of the following: + + * If your app is a chat agent app, do [Add an Amazon Bedrock Knowledge Base component to a chat agent app](./add-kb-component-chat-app.html). + + * If your app is a flow app, do [Add a Knowledge Base component to a flow app](./add-kb-component-prompt-flow-app.html). + + + + @@ -98 +180 @@ To use the Amazon Web Services Documentation, Javascript must be enabled. Please -Document data source +Use a Local file as a data source @@ -100 +182 @@ Document data source -Content chunking and parsing +Use project data as a data source