AWS wellarchitected documentation change
Summary
Updated IoT failure management documentation with new best practices (IOTREL10-BP01 to IOTREL14-BP03) covering cloud service capabilities, device reconnection logic, multi-region support, edge data storage, and disaster recovery planning.
Security assessment
The changes introduce security-related guidance such as AWS IoT Secure Tunneling for debugging (IOTREL10-BP01-02) and device certificate management (IOTREL14-BP02-02), but there is no explicit mention of addressing a specific security vulnerability. The updates primarily focus on reliability and fault tolerance.
Diff
diff --git a/wellarchitected/latest/iot-lens/failure-management.md b/wellarchitected/latest/iot-lens/failure-management.md index 2e0821908..58ec174fb 100644 --- a//wellarchitected/latest/iot-lens/failure-management.md +++ b//wellarchitected/latest/iot-lens/failure-management.md @@ -3 +3,3 @@ -[Documentation](/index.html)[AWS Well-Architected](https://aws.amazon.com/architecture/well-architected/)[AWS Well-Architected Framework](abstract-and-introduction.html) +[Documentation](/index.html)[AWS Well-Architected](https://aws.amazon.com/architecture/well-architected/)[AWS Well-Architected Framework](iot-lens.html) + +IOTREL10-BP01 Use cloud service capabilities to handle component failuresIOTREL11-BP01 Implement device logic to automatically reconnect to the cloudIOTREL11-BP02 Design devices to use multiple methods of communicationIOTREL11-BP03 Automate alerting for devices that are unable to reconnectIOTREL12-BP01 Provide adequate device storage for offline operationsIOTREL12-BP02 Synchronize device states upon connection to the cloudIOTREL13-BP01 Configure cloud services to reliably handle message processing IOTREL13-BP02 Send logs directly to the cloudIOTREL13-BP03 Design devices to allow for remote configuration of message publication frequency IOTREL14-BP01 Design server software to initiate communication only with devices that are onlineIOTREL14-BP02 Implement multi-Region support for IoT applications and devicesIOTREL14-BP03 Use edge devices to store and analyze data @@ -7 +9 @@ -IOTREL 08. How do you implement cloud-side mechanisms to control and modify the message frequency to the device? +IOTREL10: How do you implement your IoT workload to withstand component and system faults? @@ -10 +12 @@ IOTREL 08. How do you implement cloud-side mechanisms to control and modify the -Because IoT is an event-driven workload, your application code must be resilient to handling known and unknown errors that can occur as events are permeated through your application. A well-architected IoT application has the ability to log and retry errors in data processing. An IoT application will archive all data in its raw format. By archiving all data, valid and invalid, an architecture can more accurately restore data to a given point in time. +Understanding and predicting the fault scenarios in the system helps you to architect for failure conditions and use service features to handle them. Therefore, the handling of such predicted system faults and recovering from them should be architected into the system. @@ -12 +14 @@ Because IoT is an event-driven workload, your application code must be resilient -With the IoT rules engine, an application can enable an IoT error action. If a problem occurs when invoking an action, the rules engine will invoke the error action. This allows you to capture, monitor, alert, and eventually retry messages that could not be delivered to their primary IoT action. We recommend that an IoT error action is configured with a different AWS service from the primary action. Use durable storage for error actions such as Amazon SQS or Amazon Kinesis +## IOTREL10-BP01 Use cloud service capabilities to handle component failures @@ -14 +16 @@ With the IoT rules engine, an application can enable an IoT error action. If a p -Beginning with the rules engine, your application logic should initially process messages from a queue and validate that the schema of that message is correct. Your application logic should catch and log any known errors and optionally move those messages to their own dead letter queue (DLQ) for further analysis. Have a catch-all IoT rule that uses Amazon Data Firehose and AWS IoT Analytics channels to transfer all raw and unformatted messages into long-term storage in Amazon S3, AWS IoT Analytics data stores, and Amazon Redshift for data warehousing. +An IoT design consists of device software, connectivity and control services, and analytics services. Test the entire IoT landscape for resiliency, starting with device firmware, data flow, the cloud services used, and error handling. Vendors have services integrated with each other to provide a simplified integration and fault handling. @@ -16 +18 @@ Beginning with the rules engine, your application logic should initially process -IoT implementations must allow for multiple types of failure at the device level. Failures can be due to hardware, software, connectivity, or unexpected adverse conditions. One way to plan for thing failure is to deploy devices in pairs, if possible, or to deploy dual sensors across a fleet of devices deployed over the same coverage area (meshing). +**Level of risk exposed if this best practice is not established:** High @@ -18 +20 @@ IoT implementations must allow for multiple types of failure at the device level -Regardless of the underlying cause for device failures, if the device can communicate to your cloud application, it should send diagnostic information about the hardware failure to AWS IoT Core using a diagnostics topic. If the device loses connectivity because of the hardware failure, use Fleet Indexing with connectivity status to track the change in connectivity status. If the device is offline for extended periods of time, trigger an alert that the device may require remediation. +**Prescriptive guidance IOTREL10-BP01-01** _Understand and apply the standard libraries available to manage your device firmware or software._ @@ -20 +22 @@ Regardless of the underlying cause for device failures, if the device can commun -Devices can be restricted in message processing capacity and messages from the cloud might need to be throttled. The cloud-side message delivery rate might need to be architected based on the type of devices that are connected to control the frequency of message delivery to the device. + * Devices can be built on [FreeRTOS](https://aws.amazon.com/freertos/) which provides connectivity, messaging, power management and device management libraries that are tested for reliability and designed for ease of use. @@ -22,2 +24 @@ Devices can be restricted in message processing capacity and messages from the c -IOTREL 09. How do you plan for disaster recovery (DR) in your IoT workloads? ---- + * AWS provides IoT device SDKs and Mobile SDKs, comprised of open-source libraries, developer guides, sample apps, and porting guides to help you build IoT solutions with AWS IoT and your choice of hardware systems. @@ -25 +25,0 @@ IOTREL 09. How do you plan for disaster recovery (DR) in your IoT workloads? -When companies run their core production operations and cybersecurity functions in the cloud, it is important to design resilience at the edge and cloud in IoT systems. IoT implementations must allow for loss of internet connectivity, local data storage and processing. @@ -27 +26,0 @@ When companies run their core production operations and cybersecurity functions -**Best practice IOTREL_9.1 – Design server software to initiate communication only with devices that are online** @@ -29 +27,0 @@ When companies run their core production operations and cybersecurity functions -Communication should be server initiated with devices that are online rather than client-server requests. This enables you to design client software to accept commands from the server. @@ -31 +29 @@ Communication should be server initiated with devices that are online rather tha -**Recommendation IOTREL_9.1.1** – _Design client software to accept commands from the server_ +**Prescriptive guidance IOTREL10-BP01-02** _Use log levels appropriate to the lifecycle stage of your workload._ @@ -33 +31 @@ Communication should be server initiated with devices that are online rather tha - * FreeRTOS provides pub/sub and shadow library to connected devices. + * AWS IoT logs can be set up per region and per account with the logging level set to DEBUG during product development phase to provide insights on data flow and resources used. This data can be used to improve the IoT system security and performance. @@ -35 +33 @@ Communication should be server initiated with devices that are online rather tha - * AWS IoT Core provides device shadow capability to persist device states. + * [AWS IoT Secure Tunneling](https://aws.amazon.com/blogs/iot/securing-amazon-freertos-devices-at-scale-with-infineon-optiga-trust-x/) can be used to test and debug devices that are behind a restrictive firewall in the field. @@ -37 +34,0 @@ Communication should be server initiated with devices that are online rather tha - * AWS IoT Device Registry contains a list of devices connected to AWS IoT Core. AWS IoT Device Registry lets you manage devices by grouping them. @@ -40,0 +38,2 @@ Communication should be server initiated with devices that are online rather tha +IOTREL11: How do you verify that your IoT device operates with intermittent connectivity to the cloud? +--- @@ -42 +41 @@ Communication should be server initiated with devices that are online rather tha -**Best practice IOTREL_9.2 – Implement multi-region support for IoT applications and devices** +IoT solution reliability must also encompass the device itself. Devices may be deployed in remote locations and deal with intermittent connectivity, or loss in connectivity, due to a variety of external factors that are out of your IoT application's control. @@ -44 +43 @@ Communication should be server initiated with devices that are online rather tha -Cloud service providers have the same service in multiple regions. This architecture enables you to divert device data to a regional endpoint that is in not down. Data consumers should be enabled in all regions that consume the diverted device data. +For example, if an ISP is interrupted for several hours, how will the device behave and respond to these long periods of potential network outage? Implement a minimum set of embedded operations on the device to make it more resilient to the nuances of managing connectivity and communication to AWS IoT Core. @@ -46 +45 @@ Cloud service providers have the same service in multiple regions. This architec -**Recommendation IOTREL_9.2.1** – _Architect device software to reach multiple regions in case one is not available_ +## IOTREL11-BP01 Implement device logic to automatically reconnect to the cloud @@ -48 +47 @@ Cloud service providers have the same service in multiple regions. This architec - * AWS IoT is available in multiple Regions with different endpoints. If an endpoint is not available, divert device traffic to a different endpoint. +Your IoT device will likely become disconnected due to networking issues, power loss, or other unforeseen situations. This might be true of a single device, or for your entire fleet of devices. Whether a single device or the entire fleet becomes disconnected, the following best practices will make sure that the entire fleet is able to automatically reconnect. @@ -50 +49 @@ Cloud service providers have the same service in multiple regions. This architec - * AWS IoT configurable endpoints can be used with Amazon Route 53 to divert IoT traffic to a new regional endpoint. +**Level of risk exposed if this best practice is not established:** Medium @@ -52 +51 @@ Cloud service providers have the same service in multiple regions. This architec - * [AWS IoT Configurable Endpoints](https://docs.aws.amazon.com/iot/latest/developerguide/iot-custom-endpoints-configurable.html) +Prescriptive guidance IOTREL11-BP01-01 _Use an exponential backoff with jitter and retry logic to connect remote devices to the cloud._ @@ -53,0 +53 @@ Cloud service providers have the same service in multiple regions. This architec +Consider implementing a retry mechanism for IoT device software. The retry mechanism should have exponential backoff with a randomization factor built in to avoid retries from multiple devices occurring simultaneously. Implementing retry logic with exponential backoff with jitter allows the IoT devices to more evenly distribute their traffic and help prevent them from creating unnecessary peak traffic. @@ -54,0 +55 @@ Cloud service providers have the same service in multiple regions. This architec +**Prescriptive guidance IOTREL11-BP01-02** _Use device edge software and the SDK to use built in exponential backoff logic._ @@ -55,0 +57 @@ Cloud service providers have the same service in multiple regions. This architec + * Exponential backoff logic is included in the AWS SDK, including the AWS IoT Device SDK, and edge software, such as AWS IoT Greengrass Core and FreeRTOS. @@ -57 +59 @@ Cloud service providers have the same service in multiple regions. This architec -**Recommendation IOTREL_9.2.2** – _Enable device authentication certificates in multiple regions_ + * [AWS SDK handles the exponential backoff](https://docs.aws.amazon.com/general/latest/gr/api-retries.html) @@ -59 +61 @@ Cloud service providers have the same service in multiple regions. This architec - * AWS IoT provides devices with authentication certificates to verify on connection. Deploy the device certificates in the Regions where the device will connect. + * [AWS IoT Device SDK C: MQTT](https://docs.aws.amazon.com/freertos/latest/lib-ref/c-sdk/mqtt/mqtt_config.html) uses IOT-MQTT-RETRY-MS-CEILING for setting maximum retry interval limit. @@ -61 +62,0 @@ Cloud service providers have the same service in multiple regions. This architec - * Setup the cloud side IoT data consumers to accept and process data in multiple regions. @@ -63 +63,0 @@ Cloud service providers have the same service in multiple regions. This architec - * [AWS IoT device registration](https://aws.amazon.com/about-aws/whats-new/2020/04/simplify-iot-device-registration-and-easily-move-devices-between-aws-accounts-with-aws-iot-core-multi-account-registration/) @@ -65,0 +66 @@ Cloud service providers have the same service in multiple regions. This architec +## IOTREL11-BP02 Design devices to use multiple methods of communication @@ -66,0 +68 @@ Cloud service providers have the same service in multiple regions. This architec +Devices hardware can be designed to make use of multiple networking interfaces. Consider a device that provides multiple network interface types when selecting device hardware according to the needs of your IoT application. @@ -68 +70 @@ Cloud service providers have the same service in multiple regions. This architec -**Recommendation IOTREL_9.2.3** – _Use device services in all the regions the device connects to_ +**Level of risk exposed if this best practice is not established:** Low @@ -70 +72 @@ Cloud service providers have the same service in multiple regions. This architec - * AWS IoT Rules Engine diverts device data to use multiple services. Set up AWS IoT Rules Engine in the respective Regions to divert traffic to the appropriate services. +**Prescriptive guidance IOTREL11-BP02-01** _Establish alternate network channels to meet requirements._ @@ -72 +74 @@ Cloud service providers have the same service in multiple regions. This architec - * [Rules for AWS IoT](https://docs.aws.amazon.com/iot/latest/developerguide/iot-rules.html) + * Have a separate failover network channel to deliver critical messages to AWS IoT. Failover channels can include Wi-Fi, cellular networks, or a wireless personal network. @@ -73,0 +76 @@ Cloud service providers have the same service in multiple regions. This architec + * For low latency workload, use [AWS Wavelength](https://aws.amazon.com/wavelength/) for 5G devices and [AWS Local Zones](https://aws.amazon.com/about-aws/global-infrastructure/localzones/) to keep your cloud services closer to the user. @@ -77 +79,0 @@ Cloud service providers have the same service in multiple regions. This architec -**Best practice IOTREL_9.3 – Use edge devices to store and analyze data** @@ -79 +81 @@ Cloud service providers have the same service in multiple regions. This architec -Edge storage can provide additional storage for device data. Data can be stored at the edge during large-scale network events and streamed later, when network is available. +## IOTREL11-BP03 Automate alerting for devices that are unable to reconnect @@ -81 +83 @@ Edge storage can provide additional storage for device data. Data can be stored -**Recommendation IOTREL_9.3.1** – _Use an edge device as a connection point to store and analyze data_ +In the event that devices are unable to reconnect, fleet operators are to be automatically notified to begin troubleshooting the device and to re-establish device connectivity. @@ -83 +85,5 @@ Edge storage can provide additional storage for device data. Data can be stored - * AWS IoT Greengrass can be used for local processing for serverless functions, containers, messaging, storage, and machine learning inference. +**Level of risk exposed if this best practice is not established:** Medium + +**Prescriptive guidance IOTREL11-BP03-01** _Implement logic in the cloud to notify the device operator if a device has not connected for an extended period of time._ + + * [Lifecycle events](https://docs.aws.amazon.com/iot/latest/developerguide/life-cycle-events.html) can be enabled to monitor device lifecycle events, including connect and disconnect events. @@ -85 +91 @@ Edge storage can provide additional storage for device data. Data can be stored - * Data can be stored in AWS IoT Greengrass and sent to the network when it’s available. + * AWS IoT Fleet Indexing can be used to identify device connectivity status @@ -87 +93 @@ Edge storage can provide additional storage for device data. Data can be stored - * [AWS IoT Greengrass Features](https://aws.amazon.com/greengrass/features/) + * AWS IoT Events can be used to monitor devices remotely. @@ -88,0 +95 @@ Edge storage can provide additional storage for device data. Data can be stored + * Remote monitoring using AWS IoT Events: [CloudWatch Metrics connector](https://docs.aws.amazon.com/greengrass/v1/developerguide/cloudwatch-metrics-connector.html) @@ -92 +99,2 @@ Edge storage can provide additional storage for device data. Data can be stored -IOTREL 10. How do you provision reliable storage for IoT data that has been sent to the cloud? + +IOTREL12: How do you verify that required data is transmitted to the cloud after a device has been disconnected? @@ -95 +103,42 @@ IOTREL 10. How do you provision reliable storage for IoT data that has been sent -IoT devices send a lot of small messages with no guarantee of delivery order. This data might not be immediately useful, but the data volume is typically low enough to economically store against a future need. It will be beneficial to store the data so that the data can processed in order. Stored data can be reprocessed as new requirements are developed. +Your IoT device must be able to operate without internet connectivity. To make sure that required data is not lost when devices become disconnected from the cloud, they should store important messages durably offline and, once reconnected, send those messages to AWS IoT Core. Connection to the cloud can be intermittent and devices should be designed to handle this. Choose devices with firmware designed for intermittent cloud connection and that have the ability to store data on the device if you cannot afford to lose the data. + +## IOTREL12-BP01 Provide adequate device storage for offline operations + +Store important messages durably offline and, once reconnected, send those messages to the cloud. Device hardware should have capabilities to store data locally for a finite period to help prevent loss of information. + +**Level of risk exposed if this best practice is not established:** Low + +**Prescriptive guidance IOTREL12-BP01-01** _Use the device edge software capabilities for storing data locally._ + + * Design your edge applications according to your device constraints to store and forward critical data when devices become disconnected from the cloud. + + * If your device has sufficient storage available, your application may implement a local cache of messages written to disk to make sure that data is not lost when the device is operating in a disconnected state. + + * To make sure that the disk is not accidentally filled with this persisted data, design your application to make use of only a set amount of total disk space, and consider implementing a FIFO overwrite strategy. + + * When the device comes back online, a background process should be implemented to transmit data that was stored locally to the cloud, emptying the local cache as messages are successfully published to the cloud. + + * If using AWS IoT Greengrass for device software, AWS IoT Greengrass components can help collect, process, and export data streams, including when devices are offline. + + * Messages collected on the device are queued and processed in FIFO order. + + * By default, AWS IoT Greengrass Core stores unprocessed messages destined for AWS Cloud targets in memory. + + * Configure AWS IoT Greengrass to cache messages to the local file system so that they persist across core restarts. + + * AWS IoT Greengrass stream manager makes it easier and more reliable to transfer high-volume IoT data to the AWS Cloud. + + * [Configure AWS IoT Greengrass core](https://docs.aws.amazon.com/greengrass/v1/developerguide/gg-core.html) + + * [Manage data streams on AWS IoT Greengrass Core](https://docs.aws.amazon.com/greengrass/v1/developerguide/stream-manager.html) + + * [AWS IoT Greengrass Developer Guide](https://docs.aws.amazon.com/greengrass/v1/developerguide/what-is-gg.html) + + * [Run Lambda functions on the AWS IoT Greengrass core](https://docs.aws.amazon.com/greengrass/v1/developerguide/lambda-functions.html) + + * The ETL with AWS IoT Greengrass solution accelerator (For more information, see [Unlock the value of embedded security IP to build secure IoT products at scale](https://aws.amazon.com/blogs/iot/unlock-the-value-of-embedded-security-ip-to-build-secure-iot-products-at-scale/))helps to quickly set up an edge device with AWS IoT Greengrass to perform extract, transform, and load (ETL) functions on data gathered from local devices before being sent to AWS. + + + + +**Prescriptive guidance IOTREL12-BP01-02** _Consider using AWS IoT SiteWise for data coming from disparate industrial equipment._ @@ -97 +146 @@ IoT devices send a lot of small messages with no guarantee of delivery order. Th -**Best practice IOTREL_10.1 – Store data before processing** +AWS IoT SiteWise Edge software collects local equipment data and sends it to AWS IoT SiteWise in the cloud. You can use SiteWise Edge gateways to collect data from multiple OPC Unified Architecture (UA) servers and publish it to AWS IoT SiteWise. The SiteWise Edge gateway runs on either AWS IoT Greengrass V2 or Siemens Industrial Edge can be used to cache data locally in the event of intermittent network connectivity. You can configure the maximum disk buffer size used for caching data. If the cache size exceeds the maximum disk buffer size, the connector discards the earliest data from the queue. For more information, see [Use AWS IoT SiteWise Edge gateways](https://docs.aws.amazon.com/iot-sitewise/latest/userguide/gateways-ggv2.html). @@ -99 +148 @@ IoT devices send a lot of small messages with no guarantee of delivery order. Th -Ensure that the data from the devices is stored before processing. As new requirements and capabilities are added, stored data can be analyzed to meet the new requirements. +## IOTREL12-BP02 Synchronize device states upon connection to the cloud @@ -101 +150 @@ Ensure that the data from the devices is stored before processing. As new requir -**Recommendation IOTREL_10.1.1** – _Use IoT Core Rules Engine to send data to Firehose to batch and store data on Amazon Simple Storage Service (Amazon S3)_ +IoT devices are not always connected to the cloud. Design a mechanism to synchronize device states every time the device has access to the cloud. Synchronizing the device state to the cloud allows the application to get and update device state easily, as the application doesn't have to wait for the device to come online. @@ -103 +152 @@ Ensure that the data from the devices is stored before processing. As new requir - * IoT Rules Engine can send data to Firehose to batch and store data on Amazon Simple Storage Service (Amazon S3). Intelligent tiering can be enabled on Amazon S3 to reduce storage costs. +**Level of risk exposed if this best practice is not established:** Medium @@ -105 +154 @@ Ensure that the data from the devices is stored before processing. As new requir - * Understand the latency to access data and choose the Region to store the data in based on device location. +**Prescriptive guidance IOTREL12-BP02-01** _Use a digital devices state representation to synchronize device state using the below capabilities._ @@ -107 +156 @@ Ensure that the data from the devices is stored before processing. As new requir - * If data will be processed in Amazon EC2 instances, consider using the highly available and low-latency Amazon Elastic Block Store (Amazon EBS). + * AWS provides device shadow capabilities that can be used to synchronize device state when the device connects to the cloud. The AWS IoT Device Shadow service maintains a shadow for each device that you connect to AWS IoT and is supported by the AWS IoT Device SDK, AWS IoT Greengrass core, and FreeRTOS. @@ -109 +158 @@ Ensure that the data from the devices is stored before processing. As new requir - * NoSQL data can be stored in Amazon DynamoDB, which is a key-value and document database that delivers single-digit millisecond performance at any scale. + * [Synchronizing device shadows](https://docs.aws.amazon.com/iot/latest/developerguide/using-device-shadows.html) \- Device SDKs and the AWS IoT Core take care of synchronizing property values between the connected device and its device shadow in AWS IoT Core. @@ -110,0 +160 @@ Ensure that the data from the devices is stored before processing. As new requir + * [AWS IoT Greengrass](https://docs.aws.amazon.com/iot/latest/developerguide/iot-rules.html) – AWS IoT Greengrass core software provides local shadow synchronization of devices and these shadows can be configured to sync with cloud. @@ -111,0 +162 @@ Ensure that the data from the devices is stored before processing. As new requir + * [FreeRTOS](https://docs.aws.amazon.com/greengrass/latest/developerguide/security.html) \- The FreeRTOS device shadow API operations define functions to create, update, and delete AWS IoT Device Shadow services. @@ -114 +164,0 @@ Ensure that the data from the devices is stored before processing. As new requir