AWS opensearch-service documentation change
Summary
Updated documentation to add JSONL as a supported data format alongside Parquet for OpenSearch Serverless auto-optimize feature, including format specifications, examples, and bidirectional conversion scripts.
Security assessment
The changes are feature enhancements adding JSONL format support. No security vulnerabilities, incidents, or weaknesses are mentioned. The permission requirements remain unchanged (same S3 permissions). The update clarifies that folders must contain files of a single format, which is a data integrity consideration but not a security fix.
Diff
diff --git a/opensearch-service/latest/developerguide/serverless-auto-optimize.md b/opensearch-service/latest/developerguide/serverless-auto-optimize.md index 1df809caf..1233382be 100644 --- a//opensearch-service/latest/developerguide/serverless-auto-optimize.md +++ b//opensearch-service/latest/developerguide/serverless-auto-optimize.md @@ -5 +5 @@ -BenefitsHow it worksPrerequisitesUse cases for auto-optimizeLimitationsBilling and costsConvert JSONL to ParquetRelated features +BenefitsHow it worksPrerequisitesUse cases for auto-optimizeLimitationsBilling and costsSupported data formatsRelated features @@ -34 +34 @@ Auto-optimize operates through a job-based architecture that analyzes your vecto - * Users share their datasets as OpenSearch JSON documents in parquet format in an Amazon S3 bucket. + * Users share their datasets in Parquet or JSONL format in an Amazon S3 bucket. @@ -47 +47 @@ Auto-optimize operates through a job-based architecture that analyzes your vecto - * **Dataset format and permissions** \- You must have dataset available as a set (one or many) parquet files in an Amazon S3 bucket folder. For instance, if the dataset is split into two files called `s3://dataset-bucket-us-east-1/dataset_folder/first_half.parquet` and `s3://dataset-bucket-us-east-1/dataset_folder/second_half.parquet`, then you should place `s3://dataset-bucket-us-east-1/dataset_folder/` here. This dataset will be used to generate the recommendations. Ensure that your federated role has the following Amazon S3 permissions on that resource: `"s3:Get*", "s3:List*", "s3:Describe*"`. + * **Dataset format and permissions** \- You must have your dataset available as one or more Parquet or JSONL files in an Amazon S3 bucket folder. For example: @@ -49 +49 @@ Auto-optimize operates through a job-based architecture that analyzes your vecto - * **Specify correct dataset metadata** \- The provided parquet dataset must contain rows of float values. The name of each column and dimensionality of each vector must match the options provided in the console. For example, if the dataset contains vectors that are named `train_data` which are each `768` dimension, these values must match the auto-optimize console. + * Parquet: `s3://dataset-bucket-us-east-1/dataset_folder/first_half.parquet` and `s3://dataset-bucket-us-east-1/dataset_folder/second_half.parquet` @@ -51 +51,7 @@ Auto-optimize operates through a job-based architecture that analyzes your vecto - * **(If using vector ingestion feature)** \- If you plan to utilize the ingestion feature (taking auto-optimize recommendations to automatically create index and ingest data), you must configure your OpenSearch cluster to give auto-optimize permission to ingest your parquet dataset into the OpenSearch cluster. For OpenSearch domains with a domain access policy, grant the newly created role access through that policy. For OpenSearch domains with fine-grained access control, add the pipeline role as a backend role. For OpenSearch Serverless collections, add the pipeline role to the data access policy. + * JSONL: `s3://dataset-bucket-us-east-1/dataset_folder/data.jsonl` + +Provide the enclosing folder URI (for example, `s3://dataset-bucket-us-east-1/dataset_folder/`). The folder must contain files of a single format — do not mix Parquet and JSONL files in the same folder. This dataset will be used to generate the recommendations. Ensure that your federated role has the following Amazon S3 permissions on that resource: `"s3:Get*", "s3:List*", "s3:Describe*"`. + + * **Specify correct dataset metadata** \- The provided dataset must contain rows of float values. The name of each column and dimensionality of each vector must match the options provided in the console. For example, if the dataset contains vectors that are named `train_data` which are each `768` dimension, these values must match the auto-optimize console. + + * **(If using vector ingestion feature)** \- If you plan to utilize the ingestion feature (taking auto-optimize recommendations to automatically create index and ingest data), you must configure your OpenSearch cluster to give auto-optimize permission to ingest your dataset into the OpenSearch cluster. For OpenSearch domains with a domain access policy, grant the newly created role access through that policy. For OpenSearch domains with fine-grained access control, add the pipeline role as a backend role. For OpenSearch Serverless collections, add the pipeline role to the data access policy. @@ -140 +146 @@ Nmslib | No | No - * Supported formats: parquet + * Supported formats: Parquet, JSONL @@ -157 +163,27 @@ For pricing information, see [Amazon OpenSearch Service Pricing](https://aws.ama -## Convert JSONL to Parquet +## Supported data formats + +Auto-optimize supports the following data formats for vector datasets stored in Amazon S3: + +### Parquet format + +Parquet is a columnar storage format optimized for analytical workloads. Each Parquet file should contain a column of float arrays representing your vector data. + +Example Parquet file structure (viewed as a table): + + + | id | train_data | + |-----|--------------------------------| + | 1 | [0.12, 0.45, 0.78, ..., 0.33] | + | 2 | [0.56, 0.89, 0.12, ..., 0.67] | + | 3 | [0.34, 0.67, 0.90, ..., 0.11] | + +### JSONL format + +JSONL (JSON Lines) is a text format where each line is a valid JSON object. Each line should contain a field with a float array representing your vector data. + +Example JSONL file: + + + {"id": 1, "train_data": [0.12, 0.45, 0.78, 0.33]} + {"id": 2, "train_data": [0.56, 0.89, 0.12, 0.67]} + {"id": 3, "train_data": [0.34, 0.67, 0.90, 0.11]} @@ -159 +191,5 @@ For pricing information, see [Amazon OpenSearch Service Pricing](https://aws.ama -If your data is in JSONL format, you can use the following Python script to convert it to Parquet format for use with auto-optimize: +### Converting between formats + +If your data is in a different format, you can use the following Python scripts to convert it. + +###### Convert JSON or JSONL to Parquet @@ -212 +248 @@ If your data is in JSONL format, you can use the following Python script to conv - print(f"✔ Wrote {len(records)} rows to {parquet_path}") + print(f"Wrote {len(records)} rows to {parquet_path}") @@ -215,4 +251,4 @@ If your data is in JSONL format, you can use the following Python script to conv - def print_parquet_rows(parquet_path: str, limit: int = 5): - """Print first N rows from the parquet file.""" - table = pq.read_table(parquet_path) - df = table.to_pandas() + if __name__ == "__main__": + INPUT_JSON = "vectors.jsonl" + OUTPUT_PARQUET = "vectors.parquet" + json_to_parquet(INPUT_JSON, OUTPUT_PARQUET) @@ -220,3 +256 @@ If your data is in JSONL format, you can use the following Python script to conv - print(f"\n=== Showing first {min(limit, len(df))} rows from {parquet_path} ===") - print(df.head(limit).to_string()) - print("===========================================================\n") +###### Convert Parquet to JSONL @@ -225,3 +259,3 @@ If your data is in JSONL format, you can use the following Python script to conv - if __name__ == "__main__": - INPUT_JSON = "movies_10k.json" - OUTPUT_PARQUET = "movies.parquet" + #!/usr/bin/env python3 + import json + import pyarrow.parquet as pq @@ -229,2 +262,0 @@ If your data is in JSONL format, you can use the following Python script to conv - # Convert JSON → Parquet - json_to_parquet(INPUT_JSON, OUTPUT_PARQUET) @@ -232,2 +264,14 @@ If your data is in JSONL format, you can use the following Python script to conv - # Print some rows from Parquet - print_parquet_rows(OUTPUT_PARQUET, limit=3) + def parquet_to_jsonl(parquet_path: str, jsonl_path: str): + """Convert a Parquet file to JSONL format.""" + table = pq.read_table(parquet_path) + rows = table.to_pylist() + with open(jsonl_path, "w") as f: + for row in rows: + f.write(json.dumps(row) + "\n") + print(f"Wrote {len(rows)} rows to {jsonl_path}") + + + if __name__ == "__main__": + INPUT_PARQUET = "vectors.parquet" + OUTPUT_JSONL = "vectors.jsonl" + parquet_to_jsonl(INPUT_PARQUET, OUTPUT_JSONL)