AWS wellarchitected documentation change
Summary
Restructured document with updated best practice numbering, added implementation guidance, risk levels, and resource sections. Revised content focuses on cost optimization strategies for IoT data handling/storage.
Security assessment
Changes focus on cost optimization through data aggregation, storage tiering, and message size reduction. No explicit security vulnerabilities or security feature additions are mentioned. References to encryption/identity controls are existing capabilities rather than new security documentation.
Diff
diff --git a/wellarchitected/latest/iot-lens/cost-effective-resources.md b/wellarchitected/latest/iot-lens/cost-effective-resources.md index 12028ece8..18834c775 100644 --- a//wellarchitected/latest/iot-lens/cost-effective-resources.md +++ b//wellarchitected/latest/iot-lens/cost-effective-resources.md @@ -3 +3,3 @@ -[Documentation](/index.html)[AWS Well-Architected](https://aws.amazon.com/architecture/well-architected/)[AWS Well-Architected Framework](abstract-and-introduction.html) +[Documentation](/index.html)[AWS Well-Architected](https://aws.amazon.com/architecture/well-architected/)[AWS Well-Architected Framework](iot-lens.html) + +IOTCOST01-BP01 Use a data lake for raw telemetry dataIOTCOST01-BP02 Provide a self-service interface for end users to search, extract, manage, and update IoT dataIOTCOST01-BP03 Track and manage the utilization of data sourcesIOTCOST01-BP04 Aggregate data at the edge where possibleIOTCOST02-BP01 Use lifecycle policies to archive your dataIOTCOST02-BP02 Evaluate storage characteristics for your use case and align with the right servicesIOTCOST02-BP03 Store raw archival data on cost effective servicesIOTCOST03-BP01 Select services to optimize costIOTCOST03-BP02 Implement and configure telemetry to reduce data transfer costsIOTCOST03-BP03 Use shadow only for slow changing dataIOTCOST03-BP04 Group and tag IoT devices and messages for cost allocationIOTCOST03-BP05 Implement and configure device messaging to reduce data transfer costs @@ -7 +9 @@ -Given the scale of devices and data that can be generated by an IoT application, using the appropriate AWS services for your system is key to cost savings. In addition to the overall cost for your IoT solution, IoT architects often look at connectivity through the lens of bill of materials (BOM) costs. For BOM calculations, you must predict and monitor what the long-term costs will be for managing the connectivity to your IoT application throughout the lifetime of that device. Leveraging AWS services will help you calculate initial BOM costs, make use of cost-effective services that are event driven, and update your architecture to continue to lower your overall lifetime cost for connectivity. +Given the scale of devices and data that can be generated by an IoT application, using the appropriate AWS services for your system is key to cost savings. In addition to the overall cost for your IoT solution, IoT architects often look at connectivity through the lens of bill of materials (BOM) costs. For BOM calculations, you must predict and monitor what the long-term costs will be for managing the connectivity to your IoT application throughout the lifetime of that device. AWS services can help you calculate initial BOM costs, make use of cost-effective services that are event driven, and update your architecture to continue to lower your overall lifetime cost for connectivity. @@ -9 +11 @@ Given the scale of devices and data that can be generated by an IoT application, -The most straightforward way to increase the cost-effectiveness of your resources is to group IoT events into batches and process data collectively. By processing events in groups, you are able to lower the overall compute time for each individual message. Aggregation can help you save on compute resources and enable solutions when data is compressed and archived before being persisted. This strategy decreases the overall storage footprint without losing data or compromising the query ability of the data. +The recommended approach to increase the cost-effectiveness of your resources is to group IoT events into batches and process data collectively. By processing events in groups, you are able to lower the overall compute time for each individual message. Aggregation can help you save on compute resources and enable solutions when data is compressed and archived before being persisted. This strategy decreases the overall storage footprint without losing data or compromising the query ability of the data. @@ -11 +13 @@ The most straightforward way to increase the cost-effectiveness of your resource -IOTCOST 01. How do you choose cost-efficient tools for data aggregation of your IoT workloads? +IOTCOST01: How do you choose cost-efficient tools for data aggregation of your IoT workloads? @@ -14,3 +16 @@ IOTCOST 01. How do you choose cost-efficient tools for data aggregation of your -AWS IoT is best suited for streaming data for either immediate consumption or historical analyses. There are several ways to batch data from AWS IoT Core to other AWS services and the differentiating factor is driven by batching raw data (as is) or enriching the data and then batching it. Enriching, transforming, and filtering IoT telemetry data during (or immediately after) ingestion is best performed by creating an AWS IoT rule that sends the data to Kinesis Data Streams, Firehose, AWS IoT Analytics, or Amazon Simple Queue Service (Amazon SQS). These services allow you to process multiple data events at once. - -When dealing with raw device data from this batch pipeline, you can use AWS IoT Analytics and Amazon Data Firehose to transfer data to S3 buckets and Amazon Redshift. To lower storage costs in Amazon S3, an application can use lifecycle policies that archive data to lower cost storage, such as Amazon S3 Glacier. +AWS IoT is best suited for streaming data for either immediate consumption or historical analyses. There are several ways to batch data from AWS IoT Core to other AWS services and the differentiating factor is driven by batching raw data (as is) or enriching the data and then batching it. Enriching, transforming, and filtering IoT telemetry data during (or immediately after) ingestion is best performed by creating an AWS IoT rule that sends the data to other AWS services such as Kinesis Data Streams, Firehose or Amazon SQS. These services allow you to process multiple data events at once. @@ -18 +18 @@ When dealing with raw device data from this batch pipeline, you can use AWS IoT -Raw data from devices can also be processed at the edge using AWS IoT Greengrass thus eliminating the need to send all the data to the cloud for storage and processing. This can result in lower network cost and lower cost in cloud services. You can dynamically change or update that logic, as well as frequency of transmission using AWS IoT Greengrass since it's not hardcoded and can be adjusted as needed by the use case. This gives you added flexibility for cost optimization. +When dealing with raw device data from this batch pipeline, you can use Amazon Data Firehose to transfer data to S3 buckets and Amazon Redshift. To lower storage costs in Amazon S3, an application can use lifecycle policies that archive data to lower cost storage, such as Amazon S3 Glacier. @@ -20 +20 @@ Raw data from devices can also be processed at the edge using AWS IoT Greengrass -In addition, observe the following general practice recommendations: +Raw data from devices can also be processed at the edge using AWS IoT Greengrass thus alleviating the need to send all the data to the cloud for storage and processing. This can result in lower network cost and lower cost in cloud services. Customers can dynamically change or update that logic, as well as frequency of transmission using AWS IoT Greengrass since it's not hardcoded and can be adjusted as needed by the use case. This gives customers added flexibility for cost optimization. @@ -24 +24,3 @@ Methods and tools for how data is acquired, validated, categorized, and stored i -**Best practice IOTCOST_1.1** – **Use a data lake for raw telemetry data** +## IOTCOST01-BP01 Use a data lake for raw telemetry data + +A _data lake_ brings different data sources together and provides a common management framework for browsing, viewing, and extracting the sources. An effective data lake enables IoT cost management by storing data in the right format for the right use case. With a data lake, storage and interaction characteristics can be aligned to a specific dataset format and required interfaces. @@ -26 +28 @@ Methods and tools for how data is acquired, validated, categorized, and stored i -A data lake brings different data sources together and provides a common management framework for browsing, viewing, and extracting the sources. An effective data lake enables IoT cost management by storing data in the right format for the right use case. With a data lake, storage and interaction characteristics can be aligned to a specific dataset format and required interfaces. +**Level of risk exposed if this best practice is not established:** Medium @@ -28 +30 @@ A data lake brings different data sources together and provides a common managem -**Recommendation IOTCOST_1.1.1** – _Categorize telemetry types and map to storage capabilities_ +**Prescriptive guidance** @@ -32 +34,5 @@ A data lake brings different data sources together and provides a common managem - * Map each stream into the appropriate storage capability. For example, a stream that sends an MQTT message with a JSON payload every second would be an ideal candidate for being batched, compressed then stored in Amazon S3. + * Map each stream into the appropriate storage capability. + + * For example, a stream that sends an MQTT message with a JSON payload every second would be an ideal candidate for being batched, compressed then stored in Amazon S3. + + * For high velocity data streaming, utilize IoT Basic Ingest and AWS IoT rules to route data to the appropriate storage service such as Amazon Timestream or Kinesis Data Streams. @@ -34 +40,4 @@ A data lake brings different data sources together and provides a common managem - * For more information: + + + +### Resources @@ -39,0 +49,2 @@ A data lake brings different data sources together and provides a common managem + * [AWS IoT rule actions](https://docs.aws.amazon.com/iot/latest/developerguide/iot-rules.html) + @@ -43 +54 @@ A data lake brings different data sources together and provides a common managem -**Best practice IOTCOST_1.2 – Provide a self-service interface for end users to search, extract, manage, and update IoT data** +## IOTCOST01-BP02 Provide a self-service interface for end users to search, extract, manage, and update IoT data @@ -45 +56 @@ A data lake brings different data sources together and provides a common managem -With inexpensive cloud computing resources, pay-as-you-go pricing, and strong identity and encryption controls, your organization should allow groups to define and share data models in the format that makes the most sense for them. Self-service interfaces will encourage experimentation and speed up change by removing barriers for teams to gain access to the data they need to make decisions. +With flexible cloud computing resources, pay-as-you-go pricing, and strong identity and encryption controls, your organization should allow groups to define and share data models in the format that makes the most sense for them. Self-service interfaces encourage experimentation and speed up change by removing barriers for teams to gain access to the data they need to make decisions. @@ -47 +58 @@ With inexpensive cloud computing resources, pay-as-you-go pricing, and strong id -**Recommendation IOTCOST_1.2.1** – _Use an architecture that allows various end users to easily find, obtain, enhance, and share data_ +**Level of risk exposed if this best practice is not established:** Low @@ -49 +60 @@ With inexpensive cloud computing resources, pay-as-you-go pricing, and strong id -**Recommendation IOTCOST_1.2.2** – _Use a subscriber model, which allows teams to subscribe to data feeds and receive notification of updates, to reduce the need for active polling and re-synching with data sources_ +**Prescriptive guidance** @@ -51 +62,8 @@ With inexpensive cloud computing resources, pay-as-you-go pricing, and strong id -For more information: + * Use an architecture that allows various end users to easily find, obtain, enhance, and share data + + * Use a subscriber model, which allows teams to subscribe to data feeds and receive notification of updates, to reduce the need for active polling and re-synching with data sources + + + + +### Resources @@ -62 +80 @@ For more information: -**Best practice IOTCOST_1.3 – Track and manage the utilization of data sources** +## IOTCOST01-BP03 Track and manage the utilization of data sources @@ -64 +82 @@ For more information: -Applications and users evolve over time, and IoT solutions can generate large volumes of data quickly. As your application matures, it’s important for cost management of your IoT workload to track that data collected is still being used. Consistent tracking and review of data utilization provides an objective basis for cost optimization decisions. +Applications and users evolve over time, and IoT solutions can generate large volumes of data quickly. As your application matures, it's important for cost management of your IoT workload to track that data collected is still being used. Consistent tracking and review of data utilization provides an objective basis for cost optimization decisions. @@ -66 +84,3 @@ Applications and users evolve over time, and IoT solutions can generate large vo -**Recommendation IOTCOST_1.3.1** – _Track and manage the utilization of data sources to identify hot and cold spots to assess value of data_ +**Level of risk exposed if this best practice is not established:** Medium + +**Prescriptive guidance** @@ -70 +90,5 @@ Applications and users evolve over time, and IoT solutions can generate large vo - * Use automated guidance tools, such as [AWS Cost Explorer](https://aws.amazon.com/aws-cost-management/aws-cost-explorer/) and [AWS Trusted Advisor](https://aws.amazon.com/premiumsupport/technology/trusted-advisor/), to identify under-utilized or resizable components of your workload. AWS Cost Explorer has a forecast feature that predicts how much you will use AWS services over the forecast time period you selected. + * Use automated guidance tools, such as [AWS Cost Explorer](https://aws.amazon.com/aws-cost-management/aws-cost-explorer/) and [AWS Trusted Advisor](https://aws.amazon.com/premiumsupport/technology/trusted-advisor/), to identify under-utilized or resizable components of your workload. AWS Cost explorer has a forecast feature that predicts how much you will use AWS services over the forecast time period you selected. + + * Use [AWS Budgets](https://aws.amazon.com/aws-cost-management/aws-budgets/) and [Cost Anomaly detection](https://aws.amazon.com/aws-cost-management/aws-cost-anomaly-detection/) to help prevent surprise bills. + + @@ -72 +95,0 @@ Applications and users evolve over time, and IoT solutions can generate large vo - * Use [AWS Budgets](https://aws.amazon.com/aws-cost-management/aws-budgets/) and [AWS Cost Anomaly Detection](https://aws.amazon.com/aws-cost-management/aws-cost-anomaly-detection/) to prevent surprise bills. @@ -74 +97 @@ Applications and users evolve over time, and IoT solutions can generate large vo - * For more information: +### Resources @@ -87 +110 @@ Applications and users evolve over time, and IoT solutions can generate large vo -**Best practice IOTCOST_1.4** – **Aggregate data at the edge where possible** +## IOTCOST01-BP04 Aggregate data at the edge where possible @@ -91,6 +114 @@ Data aggregation is an architectural decision that can have impacts on data fide -**Recommendation IOTCOST_1.4.1** – _Examine device telemetry for opportunities to batch and aggregate data_ - - * A common mechanism includes combining multiple status updates to a final status, or combining a series of measurements generated by the device into a single message. - - * For example, 10 KB of device telemetry data might be packaged as one 10-KB message, two 5-KB messages, or 10 1-KB messages. Each packaging format has implications outside of cost such as network traffic (10 1-KB messages will each add their own header messaging as opposed to a single 10-KB message with one header) and the impact of a lost or delayed message. Optimizing message size should consider how a lost message impacts the functional or non-functional characteristics of the system. - +**Level of risk exposed if this best practice is not established:** Medium @@ -97,0 +116 @@ Data aggregation is an architectural decision that can have impacts on data fide +**Prescriptive guidance** @@ -98,0 +118 @@ Data aggregation is an architectural decision that can have impacts on data fide + * A common mechanism includes combining multiple status updates to a final status, or combining a series of measurements generated by the device into a single message. @@ -100 +120 @@ Data aggregation is an architectural decision that can have impacts on data fide -**Recommendation IOTCOST_1.4.2** – _Use[ cost calculators](https://calculator.aws/#/) to model different approaches for message size and count_ + * For example, 10000 of device telemetry data might be packaged as one 10000 message, two 5000 messages, or ten separate 1000 messages. Each packaging format has implications outside of cost such as network traffic (ten 1000 messages will each add their own header messaging as opposed to a single 10000 message with one header) and the impact of a lost or delayed message. Optimizing message size should consider how a lost message impacts the functional or non-functional characteristics of the system. @@ -102 +122 @@ Data aggregation is an architectural decision that can have impacts on data fide - * The [AWS Pricing Calculator](https://calculator.aws/#/) can estimate IoT costs for specific message sizes, traffic, and operations. + * Use [cost calculators](https://calculator.aws/#/) to model different approaches for message size and count @@ -107 +127 @@ Data aggregation is an architectural decision that can have impacts on data fide -IOTCOST 02. How do you optimize the cost of raw telemetry data? +IOTCOST02: How do you optimize cost of raw telemetry data? @@ -112 +132 @@ Raw telemetry is an original source for analytics but can also be a major compon -**Best practice IOTCOST_2.1 – Use lifecycle policies to archive your data** +## IOTCOST02-BP01 Use lifecycle policies to archive your data @@ -114 +134 @@ Raw telemetry is an original source for analytics but can also be a major compon -When selecting an automated lifecycle policy for data, there are tradeoffs to consider. For example, do you want to optimize for speed to market or cost? In some cases, it's best to optimize for speed—going to market quickly, shipping new features, or meeting a deadline—rather than investing in upfront cost optimization. Use your organization’s data classification strategies to define a lifecycle policy to take raw telemetry measurements through various services. Setting milestones by time sets expectations and encourages aggregation and production of data over mere collection. +When selecting an automated lifecycle policy for data, there are tradeoffs to consider. For example, do you want to optimize for speed to market or cost? In some cases, it's best to optimize for speed rather than investing in upfront cost optimization. Use your organization's data classification strategies to define a lifecycle policy to take raw telemetry measurements through various services. Setting milestones by time sets expectations and encourages aggregation and production of data over mere collection. @@ -116 +136 @@ When selecting an automated lifecycle policy for data, there are tradeoffs to co -**Recommendation IOTCOST_2.1.1** – _Evaluate your organization’s data retention and handling requirements and configure your AWS services to support them_ +**Level of risk exposed if this best practice is not established:** Medium @@ -118 +138,3 @@ When selecting an automated lifecycle policy for data, there are tradeoffs to co - * Check your organization’s data management policy for requirements on retention, deletion, and encryption and align your retention policies and tools with those guidelines. +**Prescriptive guidance** + + * Check your organization's data management policy for requirements on retention, deletion, and encryption, and align your retention policies and tools with those guidelines. @@ -125 +147,3 @@ When selecting an automated lifecycle policy for data, there are tradeoffs to co -**Best practice IOTCOST_2.2 – Evaluate storage characteristics for your use case and align with the right services** +## IOTCOST02-BP02 Evaluate storage characteristics for your use case and align with the right services + +Not all data needs to be stored in the same way, and data storage needs change through a data item's lifecycle. A growing fleet of devices can exponentially scale its messaging rate and device operation traffic. This scaling of message volumes can also mean an increase in storage costs. @@ -127 +151 @@ When selecting an automated lifecycle policy for data, there are tradeoffs to co -Not all data needs to be stored in the same way, and data storage needs change through a data item’s lifecycle. A growing fleet of devices can exponentially scale its messaging rate and device operation traffic. This scaling of message volumes can also mean an increase in storage costs. +**Level of risk exposed if this best practice is not established:** Low @@ -129 +153 @@ Not all data needs to be stored in the same way, and data storage needs change t -**Recommendation IOTCOST_2.2.1** – _Evaluate velocity and the volume of data coming from IoT devices when selecting storage services_ +**Prescriptive guidance** @@ -131 +155 @@ Not all data needs to be stored in the same way, and data storage needs change t - * For data at high scale of devices, time, or other characteristics—Consider a data warehouse such as Amazon Redshift or Amazon S3 with Amazon Athena. The data partitioning and tiering features of AWS storage services can help reduce storage costs. + * For data at high scale of devices, time, or other characteristics, consider a data warehouse such as Amazon Redshift or Amazon S3 with Amazon Athena. The data partitioning and tiering features of AWS storage services can help reduce storage costs. @@ -133 +157 @@ Not all data needs to be stored in the same way, and data storage needs change t - * For data at lower scale of time, devices, or other characteristics—Consider Amazon DynamoDB, Amazon OpenSearch Service (OpenSearch Service), or Amazon Aurora for short-term historical data. Use your data lifecycle policies to optimize what is kept in the short-term storage. + * For data at lower scale of time, devices, or other characteristics, consider Amazon DynamoDB, Amazon OpenSearch Service (OpenSearch Service), or Aurora for short-term historical data. Use your data lifecycle policies to optimize what is kept in the short-term storage. @@ -138 +162 @@ Not all data needs to be stored in the same way, and data storage needs change t -**Best practice IOTCOST_2.3 – Store raw archival data on cost effective services** +## IOTCOST02-BP03 Store raw archival data on cost effective services @@ -142 +166 @@ Using the right storage solution for a specific data type will align costs with -**Recommendation IOTCOST_2.3.1** – _Use an object store for archival storage_ +**Level of risk exposed if this best practice is not established:** Medium @@ -144 +168,3 @@ Using the right storage solution for a specific data type will align costs with - * Use an object store, such as Amazon S3, for raw archival storage. Object stores are immutable and often more efficient and cost-effective than block storage, especially for data which doesn’t require editing. +**Prescriptive guidance** + + * Use an object store, such as Amazon S3, for raw archival storage. Object stores are immutable and often more efficient and cost-effective than block storage, especially for data which doesn't require editing. @@ -151 +177 @@ Using the right storage solution for a specific data type will align costs with -IOTCOST 03. How do you optimize the cost of interactions between devices and your IoT cloud solution? +IOTCOST03: How do you optimize cost of interactions between devices and your IoT cloud solution? @@ -154 +180 @@ IOTCOST 03. How do you optimize the cost of interactions between devices and you -Interactions to and from devices can be a significant driver of your workload’s overall cost. Understanding and optimizing interactions between devices and cloud solution can be a significant factor of cost management. +Interactions to and from devices can be a significant driver of your workload's overall cost. Understanding and optimizing interactions between devices and cloud solution can be a significant factor of cost management. @@ -156 +182 @@ Interactions to and from devices can be a significant driver of your workload’ -**Best practice IOTCOST_3.1 – Select services to optimize cost** +## IOTCOST03-BP01 Select services to optimize cost @@ -160 +186,3 @@ Understand how services use and charge for messaging, as well as operating modes -**Recommendation IOTCOST_3.1.1** – _Select services to optimize cost_ +**Level of risk exposed if this best practice is not established:** Medium + +**Prescriptive guidance** @@ -164 +192 @@ Understand how services use and charge for messaging, as well as operating modes - * With AWS IoT Core Rules Engine Basic Ingest, you can publish directly to a rule without messaging charges. + * With AWS IoT Core Basic Ingest, you can publish directly to a rule without messaging charges. @@ -168 +196 @@ Understand how services use and charge for messaging, as well as operating modes - * For your device’s shadow, minimize the frequency of reads and writes to reduce the total metered operation and your operating costs. + * For your device's shadow, minimize the frequency of reads and writes to reduce the total metered operation and your operating costs. @@ -170 +197,0 @@ Understand how services use and charge for messaging, as well as operating modes - * For more information: @@ -172 +198,0 @@ Understand how services use and charge for messaging, as well as operating modes - * [AWS IoT Rules Engine Basic Ingest](https://docs.aws.amazon.com/iot/latest/developerguide/iot-basic-ingest.html) @@ -174 +199,0 @@ Understand how services use and charge for messaging, as well as operating modes - * [AWS IoT Pricing](https://calculator.aws/) @@ -175,0 +201 @@ Understand how services use and charge for messaging, as well as operating modes +### Resources @@ -176,0 +203 @@ Understand how services use and charge for messaging, as well as operating modes + * [AWS IoT Rules Engine Basic Ingest](https://docs.aws.amazon.com/iot/latest/developerguide/iot-basic-ingest.html) @@ -177,0 +205 @@ Understand how services use and charge for messaging, as well as operating modes + * [AWS IoT Pricing](https://calculator.aws/) @@ -179 +206,0 @@ Understand how services use and charge for messaging, as well as operating modes