AWS opensearch-service: Dashboards troubleshooting guidance for allow_explicit_index=false
Summary
Expands the OpenSearch Dashboards troubleshooting page: renames the allow_explicit_index section, adds three new symptom descriptions (startup hang, HTTP 400 errors from api/saved_objects/_bulk_get, missing saved objects), a describe-domain-config CLI check, a more detailed root-cause explanation of why the bulk/mget/msearch restriction breaks Dashboards and blocks upgrades, a note that advanced options cannot be changed during an in-progress upgrade/blue-green deployment, a warning not to delete leftover Dashboards indexes yourself, and tighter guidance on when (if ever) setting rest.action.multi.allow_explicit_index=false is justified, pointing to a per-user access-control alternative.
Security assessment
The change documents the operational and security tradeoff of the rest.action.multi.allow_explicit_index advanced option, which exists to restrict subresource/index access via resource-based policies; the added text warns that disabling it is only warranted when intending such access restriction and no Dashboards is needed, links to Advanced options and API considerations for a per-user alternative, and warns against manually deleting Dashboards indexes that hold saved objects. No vulnerability, CVE, or incident is referenced.
Evidence
+ * You intend to restrict index access through resource-based policies.
Diff
diff --git a/opensearch-service/latest/developerguide/dashboards-troubleshooting.md b/opensearch-service/latest/developerguide/dashboards-troubleshooting.md index edbbd1075..c1f79e5ba 100644 --- a//opensearch-service/latest/developerguide/dashboards-troubleshooting.md +++ b//opensearch-service/latest/developerguide/dashboards-troubleshooting.md @@ -7 +7 @@ -Dashboards is stuck at "server is not ready yet" (HTTP 503 not ready error)Dashboards fails to load with an allow_explicit_index messageDashboards runs out of memoryRequests fail because of payload sizeDashboards is unavailable during a version upgradeAn engine version upgrade fails the pre-upgrade check with an incompatible indexDashboards doesn't load properly, shows a blank page, or generates blank reportsUnsupported configurationsRelated cluster and access issues +Dashboards is stuck at "server is not ready yet" (HTTP 503 not ready error)Dashboards fails to load or start because explicit indexes are disabledDashboards runs out of memoryRequests fail because of payload sizeDashboards is unavailable during a version upgradeAn engine version upgrade fails the pre-upgrade check with an incompatible indexDashboards doesn't load properly, shows a blank page, or generates blank reportsUnsupported configurationsRelated cluster and access issues @@ -47,0 +48,2 @@ Dashboards reports "server is not ready yet" until all of its core services fini + * The `rest.action.multi.allow_explicit_index` advanced cluster option is set to `false`, which blocks the bulk requests that the migration uses. Unlike the other causes in this list, you can correct this one yourself. For more information, see Dashboards fails to load or start because explicit indexes are disabled. + @@ -68 +70,5 @@ How to mitigate - 3. If the message persists while cluster health is green, gather some read-only diagnostics that help AWS Support resolve the issue faster, then contact [AWS Support](https://aws.amazon.com/premiumsupport/) to repair the OpenSearch Dashboards index. Include the output of the following commands in your case: + 3. Check whether the `rest.action.multi.allow_explicit_index` advanced cluster option is set to `false`. If it is, that alone prevents Dashboards from starting. Set it back to `true` as described in Dashboards fails to load or start because explicit indexes are disabled, then skip the remaining steps. + +You can usually make this change yourself without a Support case. The exception is a domain that is stuck part-way through a version upgrade. You can't change advanced options while an upgrade is in progress, so the request is rejected. In that case, contact [AWS Support](https://aws.amazon.com/premiumsupport/) to clear the upgrade first. + + 4. If the message persists while cluster health is green, gather some read-only diagnostics that help AWS Support resolve the issue faster, then contact [AWS Support](https://aws.amazon.com/premiumsupport/) to repair the OpenSearch Dashboards index. Include the output of the following commands in your case: @@ -75 +81 @@ Repairing the blocked migration is the fix for an already-stuck domain. Installi - 4. To prevent recurrence, keep the domain on the latest service software update; current releases fix the common causes of migration failure. Install the update after the domain returns to **Active**. + 5. To prevent recurrence, keep the domain on the latest service software update; current releases fix the common causes of migration failure. Install the update after the domain returns to **Active**. @@ -77 +83 @@ Repairing the blocked migration is the fix for an already-stuck domain. Installi - 5. Take a manual snapshot before every version upgrade so that you can restore saved objects if a migration fails. For more information about taking snapshots, see [Creating index snapshots in Amazon OpenSearch Service](./managedomains-snapshots.html). + 6. Take a manual snapshot before every version upgrade so that you can restore saved objects if a migration fails. For more information about taking snapshots, see [Creating index snapshots in Amazon OpenSearch Service](./managedomains-snapshots.html). @@ -87 +93 @@ Keep your domain on a current service software version and take a snapshot befor -## Dashboards fails to load with an allow_explicit_index message +## Dashboards fails to load or start because explicit indexes are disabled @@ -92 +98,3 @@ Symptom -Dashboards fails to load and displays a message similar to the following: +Dashboards is unavailable or unusable, in one of three ways. + +On older engine versions, Dashboards loads and displays a message that names the option: @@ -96,0 +105,11 @@ Dashboards fails to load and displays a message similar to the following: +On current versions there is no such message. If a saved-object migration is pending, Dashboards reports `OpenSearch Dashboards server is not ready yet` and never finishes starting. This looks the same as any other blocked migration (see Dashboards is stuck at "server is not ready yet" (HTTP 503 not ready error)). + +If no migration is pending, Dashboards starts and reports that it is healthy, but it can't read its saved objects. The page loads, and index patterns, saved searches, visualizations, and dashboards are missing or fail to load. Your browser's developer console shows HTTP 400 responses from requests to `api/saved_objects/_bulk_get`. Your saved objects have not been lost. Dashboards can't read them while this option is `false`. + +Because none of these messages names the option, check its value directly before you assume another cause. You can see the current value in the console under **Advanced cluster settings** , or with the AWS CLI: + + + aws opensearch describe-domain-config \ + --domain-name my-domain \ + --query "DomainConfig.AdvancedOptions" + @@ -100 +119,10 @@ Root cause -The `rest.action.multi.allow_explicit_index` advanced cluster option is set to `false`. Set this option to `true` so that Dashboards can perform its bulk, mget, and msearch operations. +The `rest.action.multi.allow_explicit_index` advanced cluster option is set to `false`. Dashboards needs the bulk, mget, and msearch APIs with index names in the request body. It needs them both to load data and to migrate its own saved objects, so this option prevents it from working: + + * Dashboards can't read its own saved objects, because it retrieves them with mget and msearch. This happens as soon as the option is set, whether or not a migration is pending, and it happens even when Dashboards reports that it is healthy. + + * Dashboards can fail to start entirely. Whenever the saved-object format changes (for example, after an engine version upgrade), Dashboards migrates the OpenSearch Dashboards index into a new index on startup. This migration uses the bulk API with the index named in the request body. It fails while this option is `false`, and Dashboards doesn't recover on its own. + + * An engine version upgrade can stop before it finishes, because the upgrade verifies that Dashboards is healthy before it completes. + + + @@ -113,0 +142,6 @@ Changing an advanced option triggers a blue/green deployment, so the change take +###### Note + +You can't change advanced options while a configuration change or version upgrade is in progress. If an upgrade has already stalled because Dashboards can't start, this request is rejected. You can't apply the fix yourself in this case—contact [AWS Support](https://aws.amazon.com/premiumsupport/). To avoid this, set the option to `true` before you start an upgrade. + +If the domain was already upgraded while this option was `false`, Dashboards might have left behind one or more unused OpenSearch Dashboards indexes from the failed migration attempts. Setting the option back to `true` doesn't always clear them. If Dashboards still reports `OpenSearch Dashboards server is not ready yet` after the change is applied, contact [AWS Support](https://aws.amazon.com/premiumsupport/) to repair the OpenSearch Dashboards index. Don't delete it yourself—it holds your index patterns, visualizations, and dashboards. + @@ -117 +151,10 @@ Suggested action -Don't set `rest.action.multi.allow_explicit_index` to `false` unless you intend to restrict index access through resource-based policies. Leaving it at the default (`true`) keeps Dashboards operational. +Don't set `rest.action.multi.allow_explicit_index` to `false` unless both of the following are true: + + * You intend to restrict index access through resource-based policies. + + * You don't need Dashboards on the domain. + + + + +Leaving it at the default (`true`) keeps Dashboards operational. If you do need to restrict subresource access this way, see [Advanced options and API considerations](./ac.html#ac-advanced) for the tradeoff and for a per-user alternative that keeps Dashboards working. Before any engine version upgrade, confirm this option is `true`. An upgrade started while it is `false` can leave both Dashboards and the upgrade stuck. @@ -279,0 +323,2 @@ Avoid these configurations, which are common sources of Dashboards problems: + * **Dashboards isn't supported on domains where`rest.action.multi.allow_explicit_index` is set to `false`.** Dashboards requires the bulk, mget, and msearch APIs with index names in the request body, so this option is not compatible with running Dashboards. Setting the option back to `true` is the only way to restore Dashboards. No configuration keeps both. AWS Support can help you repair the OpenSearch Dashboards index after you re-enable the option, but can't make Dashboards work while it stays disabled. Decide per domain which one you need. For more information, see Dashboards fails to load or start because explicit indexes are disabled. +