AWS prescriptive-guidance documentation change
Summary
Updated documentation structure with detailed naming conventions for data lake layers (Landing zone, Raw, Stage, Analytics), added partition strategy guidance, and expanded examples
Security assessment
Changes focus on organizational naming conventions and partition strategies without addressing vulnerabilities or security controls. While encryption/access requirements are mentioned in general terms, no new security features or mitigations are introduced.
Diff
diff --git a/prescriptive-guidance/latest/defining-bucket-names-data-lakes/naming-structure-data-layers.md b/prescriptive-guidance/latest/defining-bucket-names-data-lakes/naming-structure-data-layers.md index 3d526f52d..8cf56237f 100644 --- a//prescriptive-guidance/latest/defining-bucket-names-data-lakes/naming-structure-data-layers.md +++ b//prescriptive-guidance/latest/defining-bucket-names-data-lakes/naming-structure-data-layers.md @@ -3 +3 @@ -[Documentation](/index.html)[AWS Prescriptive Guidance](https://aws.amazon.com/prescriptive-guidance/)[Defining S3 bucket and path names for data lake layers on the AWS Cloud](welcome.html) +[Documentation](/index.html)[AWS Prescriptive Guidance](https://aws.amazon.com/prescriptive-guidance/)[Defining Amazon S3 bucket and path names for data lake layers](welcome.html) @@ -5 +5 @@ -# Naming S3 buckets in your data layers +Landing zoneRaw layerStage layerAnalytics layer @@ -7 +7 @@ -The following sections provide naming structures for Amazon Simple Storage Service (Amazon S3) buckets in your data lake layers. However, you can customize the S3 bucket and path names according to your organization's requirements. We recommend that you create separate S3 buckets for each individual layer because archiving, versioning, access, and encryption requirements can vary for each layer. +# Naming Amazon S3 buckets in your data layers @@ -9 +9,3 @@ The following sections provide naming structures for Amazon Simple Storage Servi -The following diagram shows the recommended naming structure for S3 buckets in the three recommended data lake layers, including separating multiple business units, file formats, and partitions. You can adapt data partitions according to your organization's requirements, but you should use lowercase and key-value pairs (For example, `year=yyyy`, not `yyyy`) so that you can update the catalog with the `MSCK REPAIR TABLE` command. +The following sections provide naming structures for Amazon Simple Storage Service (Amazon S3) buckets in your data lake layers. However, you can customize the Amazon S3 bucket and path names according to your organization's requirements. We recommend that you create separate buckets for each individual layer because archiving, versioning, access, and encryption requirements can vary for each layer. + +The following diagram shows the recommended naming structure for Amazon S3 buckets in the recommended data lake layers. The naming structure separates multiple business units, file formats, and partitions. @@ -15 +17,173 @@ The following diagram shows the recommended naming structure for S3 buckets in t -S3 buckets must follow the naming guidelines from [Bucket naming rules](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html) in the Amazon S3 documentation. +Amazon S3 buckets must follow the naming guidelines from [Bucket naming rules](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html) in the Amazon S3 documentation. + +You can adapt data partitions according to your organization's requirements. However, you should use lowercase and key-value pairs (for example, `year=yyyy` instead of `yyyy`) so that you can update the catalog with the `MSCK REPAIR TABLE` command. + +Defining a partition strategy depends on the nature of your data and, most importantly, the nature of your user queries. We recommend that you analyze consumption and data processing patterns to find the most suitable strategy for your organization. In general, it makes sense to provide higher hierarchy levels, such as `year=yyyy`, `month=mm`, and `day=dd`, on the raw data layer and lower hierarchy levels on consumption data layers, such as the stage layer and analytics layer. This is because raw data layers usually do not have the complex consumption patterns of data processing pipelines. + +## Landing zone Amazon S3 bucket + +You require an Amazon S3 bucket for your landing zone if sensitive datasets contain elements that must be masked before data is moved to the raw bucket. + +The following table provides the naming structure, a description of the naming structure, and a name example for the Amazon S3 bucket in your landing zone layer. + +Naming format | Example +---|--- +`s3://companyname-landingzoneawsregion-awsaccount|uniqidenv/source/source_region/table/year=yyyy/month=mm/day=dd/table_<yearmonthday>.avro|csv` + + * `companyname` – The organization's name (optional) + + + + * `awsregion` – The AWS Region, such as `us-east-1` or `sa-east-1` + + + + * `awsaccount|uniqid` – The unique identifier or AWS account ID + + + + * `env` – The deployment environment, such as `dev`, `test`, or `prod` + + + + * `source` – The source or content, such as MySQL database, ecommerce, or SAP + + + + * `source_region` – Global business region, such as `us` or `asia` + + + + * `table` – `tb_customer`, `tb_transactions`, or `tb_products` + +| `s3://anycompany-landingzoneuseast1-12345-dev/socialmedia/us/tb_products/year=2021/month=03/day=01/products_20210301.csv` + +## Raw layer Amazon S3 bucket + +The raw data layer contains ingested data that has not been transformed and is in its original file format, such as JSON or CSV. This data is typically organized by data source and the date that it was ingested into the raw data layer's Amazon S3 bucket. + +The following table provides the naming structure, a description of the naming structure, and a name example for the Amazon S3 bucket in your raw data layer. + +Naming format | Example +---|--- +`s3://companyname-raw-awsregion-awsaccount|uniqid-env/source/source_region/table/year=yyyy/month=mm/day=dd/table_<yearmonthday>.avro|csv` + + * `companyname` – The organization's name (optional) + + + + * `awsregion` – The AWS Region, such as `us-east-1` or `sa-east-1` + + + + * `awsaccount|uniqid` – The unique identifier or AWS account ID + + + + * `env` – The deployment environment, such as `dev`, `test`, or `prod` + + + + * `source` – The source or content, such as MySQL database, ecommerce, or SAP + + + + * `source_region` – Global business region, such as `us` or `asia` + + + + * `table` – `tb_customer`, `tb_transactions`, or `tb_products` + +| `s3://anycompany-raw-useast1-12345-dev/socialmedia/us/tb_products/year=2021/month=03/day=01/products_20210301.csv` + +## Stage layer Amazon S3 bucket + +Data in the stage layer is read and transformed from the raw layer (for example, by using an AWS Glue or Amazon EMR job). This process validates the data (for example, by checking data types and headers) and then stores it in a consumption-ready file format, such as Apache Parquet. The metadata is stored in a table in the [AWS Glue Data Catalog](https://docs.aws.amazon.com/glue/latest/dg/components-overview.html). + +The following table provides the naming structure, a description of the naming structure, and a name example for the Amazon S3 bucket in your stage data layer. + +Naming format | Example +---|--- +`s3://companyname-stageawsregion-awsaccount|uniqidenv/source/source_region/ business_unit/table/<partitions>/table_<table_name>_<yearmonthday>.snap` + + * `companyname` – The organization's name (optional) + + + + * `awsregion` – The AWS Region, such as `us-east-1` or `sa-east-1` + + + + * `awsaccount|uniqid` – The unique identifier or AWS account ID + + + + * `env` – The deployment environment, such as `dev`, `test`, or `prod` + + + + * `source` – The source or content, such as MySQL database, ecommerce, or SAP + + + + * `source_region` – Global business region, such as `us` or `asia` + + + + * `business_unit` – The business unit that the data is processed for + + + + * `table` – `tb_customer`, `tb_transactions`, or `tb_products` + + + + * `partitions` – Partitions that provide the best performance for the consumer, allowing the query engine to avoid full data scans + +| `s3://anycompany-stagesaeast1-12345-dev/sap/br/customers/validated/dt=2021-03-01/table_customers_20210301.snappy.parquet py.parquet` + +## Analytics layer Amazon S3 bucket + +The analytics layer is similar to the stage layer because the data is in a processed file format, but the data is then aggregated according to your organization's requirements. + +The following table provides the naming structure, a description of the naming structure, and a name example for the Amazon S3 bucket in your analytics data layer. + +Naming format | Example +---|--- +`s3://companyname-analytics-awsregion-awsaccount|uniqid-env/source_region/business_unit/tb_<region>_<table_name>_<file_format>/<partition_0>/<partition_1>/.../<partition_n>/xxxxx.<compression>.<file_format>` + + * `companyname` – The organization's name (optional) + + + + * `awsregion` – The AWS Region, such as `us-east-1` or `sa-east-1` + + + + * `awsaccount|uniqid` – The unique identifier or AWS account ID + + + + * `env` – The deployment environment, such as `dev`, `test`, or `prod` + + + + * `source` – The source or content, such as MySQL database, ecommerce, or SAP + + + + * `source_region` – Global business region, such as `us` or `asia` + + + + * `business_unit` – The business unit that the data is processed for + + + + * `table` – `tb_customer`, `tb_transactions`, or `tb_products` + + + + * `partitions` – Partitions that provide the best performance for the consumer, allowing the query engine to avoid full data scans + +| `s3://anycompany-analytics-useast1-12345-dev/us/sales/tb_us_customers_parquet/<partitions>/part-000001-20218c886790.c000.snappy.parquet` @@ -25 +199 @@ Recommended data layers -Landing zone S3 bucket +Mapping buckets to IAM policies