AWS quick documentation change
Summary
Structural and editorial improvements including section renames ('What you can do' to 'Web Crawler capabilities'), prerequisite clarifications, authentication method simplifications, and VPC-related additions (configuration guidance, limitations)
Security assessment
Adds documentation about security-relevant configurations (e.g., SAML SSO authentication details, public CA requirements for VPC crawling) but does not address specific vulnerabilities. Security documentation enhancements include SSL/TLS guidance and authentication method requirements.
Diff
diff --git a/quick/latest/userguide/web-crawler-integration.md b/quick/latest/userguide/web-crawler-integration.md index cc0885f42..01c6ff8f0 100644 --- a//quick/latest/userguide/web-crawler-integration.md +++ b//quick/latest/userguide/web-crawler-integration.md @@ -5 +5 @@ -What you can doBefore you beginPrepare website access and authenticationSet up Web Crawler integrationConfigure crawlingManage knowledge basesTroubleshooting +Web Crawler capabilitiesPrerequisitesPrepare website access and authenticationSet up Web Crawler integrationConfigure crawlingManage knowledge basesTroubleshooting @@ -9 +9 @@ What you can doBefore you beginPrepare website access and authenticationSet up W -With Web Crawler integration in Amazon Quick, you can create knowledge bases from website content by crawling and indexing web pages. This integration supports data ingestion capabilities with different authentication options based on your user tier. +With Web Crawler integration in Amazon Quick, you can create knowledge bases from website content by crawling and indexing web pages. This integration supports data ingestion capabilities with different authentication options. @@ -11 +11 @@ With Web Crawler integration in Amazon Quick, you can create knowledge bases fro -## What you can do +## Web Crawler capabilities @@ -13 +13,3 @@ With Web Crawler integration in Amazon Quick, you can create knowledge bases fro -Web Crawler users can ask questions about content stored on websites and web pages. For example, users can inquire about documentation sites, knowledge bases, or search for specific information across multiple web pages. The integration enables users to quickly access and understand information from web content, regardless of location or type, while providing contextual details such as publication dates, modification history, and page ownership—all contributing to more efficient information discovery and better-informed decision making. +Web Crawler users can ask questions about content stored on websites and web pages. For example, users can search documentation sites, knowledge bases, or specific information across multiple web pages. + +The integration helps users access and understand web content regardless of location or type. It provides contextual details such as publication dates, modification history, and page ownership for more efficient information discovery. @@ -19 +21 @@ Web Crawler integration supports data ingestion only. It doesn't provide action -## Before you begin +## Prerequisites @@ -25 +27 @@ Before you set up Web Crawler integration, make sure you have the following: - * Amazon Quick Enterprise subscription + * An Amazon Quick Enterprise subscription. @@ -27 +29 @@ Before you set up Web Crawler integration, make sure you have the following: - * The website you want to crawl needs to be public and can't be behind a firewall or require special browser plugins to connect. + * A website that is not behind a firewall and does not require special browser plugins to connect. @@ -34 +36 @@ Before you set up Web Crawler integration, make sure you have the following: -Before setting up the integration in Amazon Quick, prepare your website access credentials. Web Crawler integration supports different authentication methods based on your user role: +Before setting up the integration in Amazon Quick, prepare your website access credentials. Web Crawler integration supports different authentication methods: @@ -39 +41 @@ Before setting up the integration in Amazon Quick, prepare your website access c -Available for all users. Use for crawling public websites that don't require authentication. +Use for crawling websites that don't require authentication. @@ -44 +46 @@ Available for all users. Use for crawling public websites that don't require aut -Standard HTTP Basic Authentication for secured websites. HTTP Basic Authentication is a simple way to protect web resources by requiring a username and password. When you visit a protected site using Basic Authentication, your browser will show a pop-up dialog box asking for your credentials. +Standard HTTP Basic Authentication for secured websites. When you visit a protected site, your browser displays a dialog box that asks for your credentials. @@ -48 +50 @@ Standard HTTP Basic Authentication for secured websites. HTTP Basic Authenticati - * Login page URL - The URL of the login page + * **Login page URL** \- The URL of the login page @@ -60 +62 @@ Standard HTTP Basic Authentication for secured websites. HTTP Basic Authenticati -For websites that use HTML form-based login pages. +For websites that use HTML form-based login pages. You specify XPath expressions to identify the form fields on the login page. @@ -62 +64 @@ For websites that use HTML form-based login pages. -The formis set up to for you to sepecify XPath. XPath (XML Path Language) is a query language used to navigate through elements and attributes in an HTML or XML document. To identify an XPath for a web page element, a user can utilize their browser's developer tools, typically accessed by right-clicking on the desired element and selecting "Inspect" or pressing F12. Once the element is highlighted in the developer tools, the user can right-click on the corresponding HTML code, select "Copy," and then choose "Copy XPath" from the submenu. This generates a unique path that identifies the element's exact location in the document structure. The resulting XPath might look something like //input[@id='username'] or //button[@type='submit'], where the double forward slashes (//) indicate the path can start anywhere in the document, and the square brackets contain attributes that help identify the specific element. +XPath (XML Path Language) is a query language for navigating elements in an HTML or XML document. To find an XPath for a web page element, right-click the element in your browser and choose **Inspect**. In the developer tools, right-click the highlighted HTML code, choose **Copy** , and then choose **Copy XPath**. @@ -66 +68 @@ The formis set up to for you to sepecify XPath. XPath (XML Path Language) is a q - * **Login page URL** \- URL of the login form (e.g., `https://example.com/login`) + * **Login page URL** \- URL of the login form (for example, `https://example.com/login`) @@ -72 +74 @@ The formis set up to for you to sepecify XPath. XPath (XML Path Language) is a q - * **Username field XPath** \- XPath to username input field (e.g., `//input[@id='username']`) + * **Username field XPath** \- XPath to username input field (for example, `//input[@id='username']`) @@ -74 +76 @@ The formis set up to for you to sepecify XPath. XPath (XML Path Language) is a q - * **Username button XPath** (Optional) - XPath to username button field (e.g., `//input[@id='username_button']`) + * **Username button XPath** (Optional) - XPath to username button field (for example, `//input[@id='username_button']`) @@ -76 +78 @@ The formis set up to for you to sepecify XPath. XPath (XML Path Language) is a q - * **Password field XPath** \- XPath to password input field (e.g., `//input[@id='password']`) + * **Password field XPath** \- XPath to password input field (for example, `//input[@id='password']`) @@ -78 +80 @@ The formis set up to for you to sepecify XPath. XPath (XML Path Language) is a q - * **Password button XPath** \- XPath to password button (e.g., `//button[@type='password']`) + * **Password button XPath** \- XPath to password button (for example, `//button[@type='password']`) @@ -86 +88 @@ The formis set up to for you to sepecify XPath. XPath (XML Path Language) is a q -For websites that use SAML-based single sign-on authentication. +For websites that use SAML-based single sign-on (SSO) authentication. @@ -88 +90 @@ For websites that use SAML-based single sign-on authentication. -SAML (Security Assertion Markup Language) authentication is a federated identity standard that enables single sign-on (SSO) by allowing users to authenticate through a centralized identity provider rather than entering credentials directly into each application. Unlike traditional form authentication where users type their username and password into fields on the application's login page, SAML redirects users to their organization's identity provider (like Microsoft Azure AD or Okta) to authenticate, then passes a secure token back to the application to grant access. This approach provides a seamless user experience across multiple applications, centralized user management for IT administrators, and enhanced security through features like multi-factor authentication, while form authentication requires separate credential management for each individual application +SAML (Security Assertion Markup Language) authentication is a federated identity standard that enables SSO. Users authenticate through a centralized identity provider (such as Microsoft Azure AD or Okta) instead of entering credentials directly into each application. The identity provider passes a secure token back to the application to grant access. @@ -98 +100 @@ SAML (Security Assertion Markup Language) authentication is a federated identity - * **Username field XPath** \- XPath to username input field (e.g., `//input[@id='username']`) + * **Username field XPath** \- XPath to username input field (for example, `//input[@id='username']`) @@ -100 +102 @@ SAML (Security Assertion Markup Language) authentication is a federated identity - * **Username button XPath** (Optional) - XPath to username button field (e.g., `//input[@id='username_button']`) + * **Username button XPath** (Optional) - XPath to username button field (for example, `//input[@id='username_button']`) @@ -102 +104 @@ SAML (Security Assertion Markup Language) authentication is a federated identity - * **Password field XPath** \- XPath to password input field (e.g., `//input[@id='password']`) + * **Password field XPath** \- XPath to password input field (for example, `//input[@id='password']`) @@ -104 +106 @@ SAML (Security Assertion Markup Language) authentication is a federated identity - * **Password button XPath** \- XPath to password button (e.g., `//button[@type='password']`) + * **Password button XPath** \- XPath to password button (for example, `//button[@type='password']`) @@ -142 +144,5 @@ After preparing your website access requirements, create the Web Crawler integra - 1. Select the authentication type for your web crawler integration. + 1. Choose the authentication type for your web crawler integration. + + 2. Enter the required details based on your chosen authentication method. + + 3. (Optional) Choose a VPC connection to crawl sites hosted in your private network. The VPC connection must be configured in admin settings before you can choose it here. For more information, see [Setting up a VPC to use with Amazon Quick](./vpc-setup-for-quicksight.html). @@ -144 +150,3 @@ After preparing your website access requirements, create the Web Crawler integra - 2. Fill in the required details based on your selected authentication method. +###### Note + +You can't change the VPC connection after the integration is created. To use a different VPC connection, create a new integration. @@ -146 +154 @@ After preparing your website access requirements, create the Web Crawler integra - 3. Select **Create and continue**. + 4. Choose **Create and continue**. @@ -148 +156 @@ After preparing your website access requirements, create the Web Crawler integra - 4. Fill in the Name and description for your knowledge base. + 5. Enter the name and description for your knowledge base. @@ -150 +158 @@ After preparing your website access requirements, create the Web Crawler integra - 5. Add the content URLs you want to crawl. + 6. Add the content URLs that you want to crawl. @@ -152 +160 @@ After preparing your website access requirements, create the Web Crawler integra - 6. Select **Create**. + 7. Choose **Create**. @@ -157 +165 @@ After preparing your website access requirements, create the Web Crawler integra -After clicking create, the data sync is started automatically. +After you choose **Create** , the data sync starts automatically. @@ -216 +224 @@ To view these settings, you must first set up a knowledge base and then examine - * The amount of time the web crawler will wait for each page after the page reaches "page ready" state. This is useful for pages that have dynamic javascript load characteristics where the page has content blocks that loads after the main template is loaded.Increase wait time if you have visually rich content or anticipate high load times. + * The time (in seconds) that the web crawler waits for each page after the page reaches a ready state. Increase this value for pages with dynamic JavaScript content that loads after the main template. @@ -231 +239 @@ You can modify your existing Web Crawler knowledge bases: - 2. Select your Web Crawler knowledge base from the list. + 2. Choose your Web Crawler knowledge base from the list. @@ -244 +252 @@ You can create multiple knowledge bases from the same Web Crawler integration: - 1. In the Amazon Quick console, choose **Integrations** , and then select the **Data** tab. + 1. In the Amazon Quick console, choose **Integrations** , and then choose the **Data** tab. @@ -261 +269 @@ Control whether the system processes files and attachments linked from web pages - * Enable file attachment crawling - Select this option to crawl and index files and attachments found on web pages, such as PDFs, documents, and media files. + * **Enable file attachment crawling** – Choose this option to crawl and index files and attachments found on web pages, such as PDFs, documents, and media files. @@ -270 +278 @@ Your Web Crawler integration follows these crawling practices: - * **Incremental sync model:** First sync performs full crawl, subsequent syncs capture changes only + * **Incremental sync model:** First sync performs full crawl. Subsequent syncs capture changes only. @@ -272 +280 @@ Your Web Crawler integration follows these crawling practices: - * **Automatic retry:** Built-in retry logic for failed requests + * **Automatic retry:** Built-in retry logic for failed requests. @@ -274 +282 @@ Your Web Crawler integration follows these crawling practices: - * **Duplicate handling:** Automatic detection and handling of URLs + * **Duplicate handling:** Automatic detection and deduplication of URLs. @@ -276 +284 @@ Your Web Crawler integration follows these crawling practices: - * **Crawler identification:** Identifies itself with user-agent string "aws-quick-on-behalf-of-<UUID>" in request headers + * **Crawler identification:** Identifies itself with user-agent string "aws-quick-on-behalf-of-<UUID>" in request headers. @@ -291 +299 @@ Web Crawler respects the robots.txt protocol and honors user-agent and allow/dis - * **Fallback behavior:** If Web Crawler cannot fetch robots.txt due to blocking, parsing errors, or timeouts, it will behave as if robots.txt does not exist and will crawl the site + * **Fallback behavior:** If Web Crawler can't fetch robots.txt due to blocking, parsing errors, or timeouts, it behaves as if robots.txt doesn't exist. In this case, the crawler proceeds to crawl the site. @@ -303 +311 @@ Web Crawler recognizes these robots.txt fields (field names are case-insensitive -Identifies which crawler the rules apply to +Identifies which crawler the rules apply to. @@ -308 +316 @@ Identifies which crawler the rules apply to -A URL path that may be crawled +A URL path that may be crawled. @@ -313 +321 @@ A URL path that may be crawled -A URL path that may not be crawled +A URL path that may not be crawled. @@ -318 +326 @@ A URL path that may not be crawled -The complete URL of a sitemap +The complete URL of a sitemap. @@ -323 +331 @@ The complete URL of a sitemap -Specified amount of time (in seconds) to wait between requests to your website +The time (in seconds) to wait between requests to your website. @@ -334 +342 @@ Web Crawler supports page-level robots meta tags that you can use to control how -Do not index the page. If you don't specify this rule, the page may be indexed and eligible to appear in experiences +Do not index the page. If you don't specify this rule, the page may be indexed and eligible to appear in experiences. @@ -339 +347 @@ Do not index the page. If you don't specify this rule, the page may be indexed a -Do not follow the links on this page. If you don't specify this rule, Web Crawler may use the links on the page to discover those linked pages +Do not follow the links on this page. If you don't specify this rule, Web Crawler may use the links on the page to discover those linked pages. @@ -345 +353 @@ You can combine multiple values using a comma (for example, "noindex, nofollow") -To detect meta tags, Web Crawler needs to access your page, so do not block your page with robots.txt which will prevent it from being recrawled. +To detect meta tags, Web Crawler must access your page. Don't block your page with robots.txt, because this prevents the page from being recrawled. @@ -368 +376 @@ Use this section to resolve common issues with Web Crawler integration. - 1. Verify the site is reachable from the AWS region the Amazon Quick instance is setu + 1. Verify the site is reachable from the AWS Region where the Amazon Quick instance is set up. @@ -370 +378 @@ Use this section to resolve common issues with Web Crawler integration. - 2. Verify credentials accuracy and ensure they haven't expired + 2. Verify that your credentials are correct and haven't expired. @@ -372 +380 @@ Use this section to resolve common issues with Web Crawler integration. - 3. Check authentication endpoint availability and accessibility + 3. Check authentication endpoint availability and accessibility. @@ -374 +382 @@ Use this section to resolve common issues with Web Crawler integration. - 4. Validate XPath configurations by testing them in browser developer tools + 4. Validate XPath configurations by testing them in browser developer tools. @@ -376 +384 @@ Use this section to resolve common issues with Web Crawler integration. - 5. Review browser network logs to understand the authentication flow + 5. Review browser network logs to understand the authentication flow. @@ -378 +386 @@ Use this section to resolve common issues with Web Crawler integration. - 6. Ensure login page URL is correct and accessible + 6. Ensure the login page URL is correct and accessible. @@ -380 +388 @@ Use this section to resolve common issues with Web Crawler integration. - 7. Test authentication manually using the same credentials + 7. Test authentication manually using the same credentials. @@ -400 +408 @@ Use this section to resolve common issues with Web Crawler integration. - 1. Verify network connectivity to target websites + 1. Verify network connectivity to target websites. @@ -404 +412,10 @@ Use this section to resolve common issues with Web Crawler integration. - * Check DNS resolution for target domains + * Check DNS resolution for target domains. + + * Verify SSL/TLS configuration and certificates. + + * Test access from different networks if possible. + + + + +### DNS resolution @@ -406 +423 @@ Use this section to resolve common issues with Web Crawler integration. - * Verify SSL/TLS configuration and certificates +The Web Crawler uses DNS to resolve website hostnames (for example, `www.example.com`) to IP addresses. By default, it uses public DNS resolution. @@ -408 +425 @@ Use this section to resolve common issues with Web Crawler integration. - * Test access from different networks if possible +When crawling sites inside a VPC, you may need to configure a private DNS server so the crawler can resolve hostnames for internal sites. Choose one of the following options based on your VPC configuration: @@ -409,0 +427 @@ Use this section to resolve common issues with Web Crawler integration. + 1. **Use the VPC-provided DNS server** — If your VPC has both **DNS hostnames** and **DNS resolution** enabled, you can use the default VPC DNS resolver (typically 10.0.0.2, or more generally the VPC CIDR base+2). For more information, see [VPC](./vpc-amazon-virtual-private-cloud.html).