AWS wellarchitected high security documentation change
Summary
Restructured content to include specific best practices such as NTP synchronization for device time accuracy, managing AWS service quotas, data downsampling, targeted messaging, and implementing retry logic.
Security assessment
The changes emphasize NTP synchronization (IOTREL01-BP01/BP02) to ensure accurate device time, which is critical for validating TLS certificates. Without proper time synchronization, devices may fail to authenticate due to expired/invalid certificates, directly impacting secure communication. This addresses a concrete security risk (certificate validation failures) and adds explicit guidance for secure device configuration.
Diff
diff --git a/wellarchitected/latest/iot-lens/foundations.md b/wellarchitected/latest/iot-lens/foundations.md index c06502d76..b156f7cc5 100644 --- a//wellarchitected/latest/iot-lens/foundations.md +++ b//wellarchitected/latest/iot-lens/foundations.md @@ -3 +3 @@ -[Documentation](/index.html)[AWS Well-Architected](https://aws.amazon.com/architecture/well-architected/)[AWS Well-Architected Framework](abstract-and-introduction.html) +[Documentation](/index.html)[AWS Well-Architected](https://aws.amazon.com/architecture/well-architected/)[AWS Well-Architected Framework](iot-lens.html) @@ -5,58 +5 @@ -# Foundations - -IoT devices must continue to operate in some capacity in the face of network or cloud errors. Design device firmware to handle intermittent connectivity or loss in connectivity in a way that is sensitive to memory and power constraints. IoT cloud applications must also be designed to handle remote devices that frequently transition between being online and offline to maintain data coherency and scale horizontally over time. Monitor overall IoT utilization and create a mechanism to automatically increase capacity to ensure that your application can manage peak IoT traffic. - -To prevent devices from creating unnecessary peak traffic, device firmware must be implemented that prevents the entire fleet of devices from attempting the same operations at the same time. For example, if an IoT application is composed of alarm systems and all the alarm systems send an activation event at 9am local time, the IoT application is inundated with an immediate spike from your entire fleet. Instead, you should incorporate a randomization factor into those scheduled activities, such as timed events and exponential back off, to permit the IoT devices to more evenly distribute their peak traffic within a window of time. - -IOTREL 01. How do you handle AWS service limits for peaks in your IoT application? ---- - -AWS IoT provides a set of soft and hard limits for different dimensions of usage. AWS IoT outlines all of the data plane limits on the [IoT limits page](https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html#limits_iot). Data plane operations (for example, MQTT Connect, MQTT Publish, and MQTT Subscribe) are the primary driver of your device connectivity. Therefore, it's important to review the IoT limits and ensure that your application adheres to any soft limits related to the data plane, while not exceeding any hard limits that are imposed by the data plane. - -The most important part of your IoT scaling approach is to ensure that you architect around any hard limits because exceeding limits that are not adjustable results in application errors, such as throttling and client errors. Hard limits are related to throughput on a single IoT connection. If you find your application exceeds a hard limit, we recommend redesigning your application to avoid those scenarios. This can be done in several ways, such as restructuring your MQTT topics, or implementing cloud-side logic to aggregate or filter messages before delivering the messages to the interested devices. - -Soft limits in AWS IoT traditionally correlate to account-level limits that are independent of a single device. For any account-level limits, you should calculate your IoT usage for a single device and then multiply that usage by the number of devices to determine the base IoT limits that your application will require for your initial product launch. AWS recommends that you have a ramp-up period where your limit increases align closely to your current production peak usage with an additional buffer. To ensure that the IoT application is not under provisioned: - - * Consult published AWS IoT CloudWatch metrics for all of the limits. - - * Monitor CloudWatch metrics in AWS IoT Core. - - * Alert on CloudWatch throttle metrics, which would signal if you need a limit increase. - - * Set alarms for all thresholds in IoT, including MQTT connect, publish, subscribe, receive, and rule engine actions. - - * Ensure that you request a limit increase in a timely fashion, before reaching 100% capacity. - - - - -In addition to data plane limits, the AWS IoT service has a control plane for administrative APIs. The control plane manages the process of creating and storing IoT policies and principals, creating the thing in the registry, and associating IoT principals including certificates and Amazon Cognito federated identities. Because bootstrapping and device registration is critical to the overall process, it's important to plan control plane operations and limits. Control plane API calls are based on throughput measured in requests per second. Control plane calls are normally in the order of magnitude of tens of requests per second. It’s important for you to work backward from peak expected registration usage to determine if any limit increases for control plane operations are needed. Plan for sustained ramp-up periods for onboarding devices so that the IoT limit increases align with regular day-to-day data plane usage. - -To protect against a burst in control plane requests, your architecture should limit the access to these APIs to only authorized users or internal applications. Implement back-off and retry logic, and queue inbound requests to control data rates to these APIs. - -IOTREL 02. What is the strategy for managing ingestion and processing throughput of IoT data to other applications? ---- - -Although IoT applications have communication that is only routed between other devices, there will be messages that are processed and stored in your application. In these cases, the rest of your IoT application must be prepared to respond to incoming data. All internal services that are dependent upon that data need a way to seamlessly scale the ingestion and processing of the data. In a well-architected IoT application, internal systems are decoupled from the connectivity layer of the IoT platform through the ingestion layer. The ingestion layer is composed of queues and streams that enable durable short-term storage while allowing compute resources to process data independent of the rate of ingestion. - -To optimize throughput, use AWS IoT rules to route inbound device data to services such as Amazon Kinesis Data Streams, Amazon Data Firehose, or Amazon Simple Queue Service before performing any compute operations. Ensure that all the intermediate streaming points are provisioned to handle peak capacity. This approach creates the queueing layer necessary for upstream applications to process data resiliently. - -IOTREL 03. How do you implement your IoT workload to withstand component and system faults? ---- - -Understanding and predicting the fault scenarios in the system helps you to architect for failure conditions and use service features to handle them. Therefore, the handling of such predicted system faults and recovering from them should be architected into the system. - -**Best practice IOTREL_3.1 – Use the services provided by your vendors for integration and error handling to withstand component failure** - -An IoT design consists of device software, connectivity and control services, and analytics services. Test the entire IoT environment for resiliency, starting with device firmware, data flow, the cloud services used, and error handling. Vendors have services integrated with each other to provide a simplified integration and fault handling. - -**Recommendation IOTREL_3.1.1** – _Understand and apply the standard libraries available to manage your device firmware_ - - * Devices can be built on [FreeRTOS](https://aws.amazon.com/freertos/), which provides connectivity, messaging, power management and device management libraries that are tested for reliability and designed for ease of use. - - - - -**Recommendation IOTREL_3.1.2** – _Use log levels appropriate to the lifecycle stage of your workload_ - - * AWS IoT logs can be set up per region and per account with the logging level set to DEBUG during product development phase to provide insights on data flow and resources used. This data can be used to improve the IoT system security and performance. +IOTREL01-BP01 Use NTP to maintain time synchronization on devicesIOTREL01-BP02 Provide devices access to NTP serversIOTREL02-BP01 Manage service quotas and constraintsIOTREL03-BP01 Down sample data to reduce storage requirements and network utilizationIOTREL04-BP01 Target messages to relevant devicesIOTREL04-BP02 Implement retry and back off logic to support throttling by device type @@ -64 +7 @@ An IoT design consists of device software, connectivity and control services, an - * [AWS IoT Secure Tunneling](https://aws.amazon.com/blogs/iot/securing-amazon-freertos-devices-at-scale-with-infineon-optiga-trust-x/) can be used to test and debug devices that are behind a restrictive firewall in the field. +# Foundations @@ -65,0 +9 @@ An IoT design consists of device software, connectivity and control services, an +IoT devices should continue to operate in some capacity in the face of network or cloud errors. Design device firmware to handle intermittent connectivity or loss in connectivity in a way that is sensitive to memory and power constraints. IoT cloud applications must also be designed to handle remote devices that frequently transition between being online and offline to maintain data coherency and scale horizontally over time. Monitor overall IoT utilization and create a mechanism to automatically increase capacity to make sure that your application can manage peak IoT traffic. @@ -66,0 +11 @@ An IoT design consists of device software, connectivity and control services, an +To help prevent devices from creating unnecessary peak traffic, device firmware must be implemented that helps prevent the entire fleet of devices from attempting the same operations at the same time. @@ -67,0 +13 @@ An IoT design consists of device software, connectivity and control services, an +For example, if an IoT application is composed of alarm systems and all the alarm systems send an activation event at 9am local time, the IoT application is inundated with an immediate spike from your entire fleet. Instead, you should incorporate a randomization factor into those scheduled activities, such as timed events and exponential backoff, to permit the IoT devices to more evenly distribute their peak traffic within a window of time. @@ -69 +15 @@ An IoT design consists of device software, connectivity and control services, an -IOTREL 04. How do you ensure that all IoT messages are processed? +IOTREL01: How do you make sure that your device consistently keeps its internal clock accurate? @@ -72,3 +18 @@ IOTREL 04. How do you ensure that all IoT messages are processed? -Data sent from devices should be processed and stored without excessive loss. Services that queue and deliver IoT data to compute and database services should be used to ensure the processing of data. IoT devices send lots of data in small sizes without order, and the cloud application should be able to handle this. - -**Best practice IOTREL_4.1 – Dynamically scale cloud resources with utilization** +A secure device should have a valid certificate. IoT devices use a server certificate to communicate to the cloud and the certificate presented uses time for certificate validity. Having reliable and accurate time is compulsory to be able to validate certificates. Because IoT data is not ordered, including an accurate timestamp with the data will enhance your analytic capabilities. @@ -76 +20 @@ Data sent from devices should be processed and stored without excessive loss. Se -The elastic nature of the cloud can be used to increase and decrease resources on demand. Use the ability to increase and decrease cloud resources based on data, number of messages and size of messages and number of devices. +## IOTREL01-BP01 Use NTP to maintain time synchronization on devices @@ -78 +22 @@ The elastic nature of the cloud can be used to increase and decrease resources o -**Recommendation IOTREL_4.1.1** – _Know the mechanisms that can be used to monitor cloud resource usage and methods to scale the resources_ +IoT devices need to have a client to keep track of time—either using Real Time Clock (RTC) or Network Time Protocol (NTP) to set the RTC on boot. Failure to provide accurate time to an IoT device could help prevent it from being able to connect to the cloud. @@ -80 +24 @@ The elastic nature of the cloud can be used to increase and decrease resources o - * Use Amazon CloudWatch Logs to trigger based on rate of data flow to auto-scale cloud resources as needed. +**Level of risk exposed if this best practice is not established:** Medium @@ -82 +26 @@ The elastic nature of the cloud can be used to increase and decrease resources o - * [Use AWS IoT Rules engine error actions](https://docs.aws.amazon.com/iot/latest/developerguide/rule-error-handling.html) to provision additional cloud resources and message retries as needed. +**Prescriptive guidance IOTREL01-BP01-01** _Prefer NTP to RTC when NTP synchronization is available._ @@ -84 +28 @@ The elastic nature of the cloud can be used to increase and decrease resources o - * Examine IoT logs for errors in communicating to resources and provision resources based on that data. +Many computers have an RTC peripheral that helps in keeping time. Consider that RTC is prone to clock drift of about 1 second a day, which can result in the device going offline because of certificate invalidity. @@ -86 +30 @@ The elastic nature of the cloud can be used to increase and decrease resources o - * Use AWS Lambda to automatically scale your application by running code in response to each event. +**Prescriptive guidance IOTREL01-BP01-02** _Use Network Time Protocol for connected applications._ @@ -88 +32 @@ The elastic nature of the cloud can be used to increase and decrease resources o - * Use automatic scaling where possible. Kinesis Data Streams and Amazon DynamoDB are two services that provide automatic scaling. + * Select a safe, reliable NTP pool to use, and a one that addresses your security design @@ -89,0 +34 @@ The elastic nature of the cloud can be used to increase and decrease resources o + * Many operating systems include an NTP client to sync with an NTP server @@ -90,0 +36 @@ The elastic nature of the cloud can be used to increase and decrease resources o + * If the IoT device is using GNU/Linux, it is likely to include the NTPD daemon @@ -91,0 +38 @@ The elastic nature of the cloud can be used to increase and decrease resources o + * You can import an NTP client to your system if using FreeRTOS @@ -93,2 +40 @@ The elastic nature of the cloud can be used to increase and decrease resources o -IOTREL 05. How do you ensure that your IoT device operates with intermittent connectivity to the cloud? ---- + * The device's software needs to include an NTP client and should wait until it has synchronized with an NTP server before attempting a connection with AWS IoT Core @@ -96 +42 @@ IOTREL 05. How do you ensure that your IoT device operates with intermittent con -IoT solution reliability must also encompass the device itself. Devices are deployed in remote locations and deal with intermittent connectivity, or loss in connectivity, due to a variety of external factors that are out of your IoT application’s control. For example, if an ISP is interrupted for several hours, how will the device behave and respond to these long periods of potential network outage? Implement a minimum set of embedded operations on the device to make it more resilient to the nuances of managing connectivity and communication to AWS IoT Core. + * The system should provide a way for a user to set the device's time so that subsequent connections can succeed @@ -98 +44 @@ IoT solution reliability must also encompass the device itself. Devices are depl -Your IoT device must be able to operate without internet connectivity. You must implement robust operations in your firmware to provide the following capabilities: + * Use NTP to synchronize RTC on the device to help prevent the device from deviating from UTC @@ -100 +46 @@ Your IoT device must be able to operate without internet connectivity. You must - * Store important messages durably offline and, once reconnected, send those messages to AWS IoT Core. + * Consider the following [The NTP Pool for vendors](https://www.pool.ntp.org/en/vendors.html) @@ -102 +48 @@ Your IoT device must be able to operate without internet connectivity. You must - * Implement exponential retry and back-off logic when connection attempts fail. + * [Chrony](https://chrony.tuxfamily.org/) is a different implementation of NTP than what NTPD uses and it is able to synchronize the system clock faster and with better accuracy than NTPD. Chrony can be set up as a client and server. @@ -104 +49,0 @@ Your IoT device must be able to operate without internet connectivity. You must - * If necessary, have a separate failover network channel to deliver critical messages to AWS IoT. This can include failing over from Wi-Fi to standby cellular network, or failing over to a wireless personal area network protocol (such as Bluetooth LE) to send messages to a connected device or gateway. @@ -106 +50,0 @@ Your IoT device must be able to operate without internet connectivity. You must - * Have a method to set the current time using an NTP client or low-drift real-time clock. A device should wait until it has synchronized its time before attempting a connection with AWS IoT Core. If this isn’t possible, the system provides a way for a user to set the device’s time so that subsequent connections can succeed. @@ -108 +51,0 @@ Your IoT device must be able to operate without internet connectivity. You must - * Send error codes and overall diagnostics messages to AWS IoT Core. @@ -110 +53 @@ Your IoT device must be able to operate without internet connectivity. You must - * Configure a AWS IoT Greengrass group to write logs to the local file system and to CloudWatch Log +## IOTREL01-BP02 Provide devices access to NTP servers @@ -111,0 +55 @@ Your IoT device must be able to operate without internet connectivity. You must +An NTP server should be available for clients to use for local time. NTP servers are required by NTP clients to synchronize device time and function properly. @@ -112,0 +57 @@ Your IoT device must be able to operate without internet connectivity. You must +**Level of risk exposed if this best practice is not established:** Low @@ -113,0 +59 @@ Your IoT device must be able to operate without internet connectivity. You must +**Prescriptive guidance IOTREL01-BP02-01** _Provide access to NTP services._ @@ -115 +61 @@ Your IoT device must be able to operate without internet connectivity. You must -Connection to the cloud can be intermittent and devices should be designed to handle this. Choose devices with firmware designed for intermittent cloud connection and that have the ability to store data on the device if you cannot afford to lose the data. + * ntp.org can be used to synchronize your computer clocks. @@ -117 +62,0 @@ Connection to the cloud can be intermittent and devices should be designed to ha -**Best practice IOTREL_5.1 – Synchronize device states upon connection to the cloud** @@ -119 +63,0 @@ Connection to the cloud can be intermittent and devices should be designed to ha -IoT devices are not always connected to the cloud. Design a mechanism to synchronize device states every time the device has access to the cloud. Synchronizing the device state to the cloud allows the application to get and update device state easily, as the application doesn’t have to wait for the device to come online. @@ -121 +64,0 @@ IoT devices are not always connected to the cloud. Design a mechanism to synchro -**Recommendation IOTREL_5.1.1** – _Use a digital devices state representation to synchronize device state using the below capabilities:_ @@ -123 +66 @@ IoT devices are not always connected to the cloud. Design a mechanism to synchro - * AWS provides device shadow capabilities that can be used to synchronize device state when the device connects to the cloud. The AWS IoT Device Shadow service maintains a shadow for each device that you connect to AWS IoT and is supported by the AWS IoT Device SDK, AWS IoT Greengrass core, and FreeRTOS. + * [Amazon Time Sync Service](https://aws.amazon.com/about-aws/whats-new/2017/11/introducing-the-amazon-time-sync-service/): a time synchronization service delivered over NTP, which uses a fleet of redundant satellite-connected and atomic clocks in each Region to deliver a highly accurate reference clock. This is natively accessible from Amazon EC2 instances and this can be pushed to edge devices. @@ -125 +68 @@ IoT devices are not always connected to the cloud. Design a mechanism to synchro - * [Synchronizing device shadows](https://docs.aws.amazon.com/iot/latest/developerguide/using-device-shadows.html) – Device SDKs and the AWS IoT Core take care of synchronizing property values between the connected device and its device shadow in AWS IoT Core. + * [Chrony](https://chrony.tuxfamily.org/) is a different implementation of NTP than what NTPD uses and it is able to synchronize the system clock faster and with better accuracy than NTPD. Chrony can be set up as a server and client. @@ -127 +69,0 @@ IoT devices are not always connected to the cloud. Design a mechanism to synchro - * [AWS IoT Greengrass](https://docs.aws.amazon.com/iot/latest/developerguide/iot-rules.html) – AWS IoT Greengrass core software provides local shadow synchronization of devices and these shadows can be configured to sync with cloud. @@ -129 +70,0 @@ IoT devices are not always connected to the cloud. Design a mechanism to synchro - * [FreeRTOS](https://docs.aws.amazon.com/greengrass/latest/developerguide/security.html) – The FreeRTOS device shadow API operations define functions to create, update, and delete AWS IoT Device Shadows. @@ -131,0 +73,2 @@ IoT devices are not always connected to the cloud. Design a mechanism to synchro +IOTREL02: How do you manage service quotas and limits for peaks in your IoT workload? +--- @@ -132,0 +76 @@ IoT devices are not always connected to the cloud. Design a mechanism to synchro +AWS IoT provides a set of soft and hard limits for different dimensions of usage. AWS IoT outlines the data plane limits on the IoT limits, see [AWS service quotas](https://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html#limits_iot). Data plane operations (for example, MQTT Connect, MQTT Publish, and MQTT Subscribe) are the primary driver of your device connectivity. Therefore, it's important to review the IoT limits and make sure that your application adheres to any soft limits related to the data plane, while not exceeding any hard limits that are imposed by the data plane. @@ -134 +78 @@ IoT devices are not always connected to the cloud. Design a mechanism to synchro -**Best practice IOTREL_5.2 – Use device hardware with sufficient capacity to meet your data retention requirements while disconnected** +## IOTREL02-BP01 Manage service quotas and constraints @@ -136 +80 @@ IoT devices are not always connected to the cloud. Design a mechanism to synchro -Store important messages durably offline and, once reconnected, send those messages to the cloud. Device hardware should have capabilities to store data locally for a finite period of time to prevent any loss of information. +For cloud-based workload architectures, there are service quotas (which are also referred to as service limits). These quotas exist to help prevent accidentally provisioning more resources than you need and to limit request rates on API operations so as to protect services from abuse. @@ -138 +82 @@ Store important messages durably offline and, once reconnected, send those messa -**Recommendation IOTREL_5.2.1** – _You can leverage the device edge software capabilities for storing data locally._ +**Level of risk exposed if this best practice is not established:** High @@ -140 +84 @@ Store important messages durably offline and, once reconnected, send those messa - * Using AWS IoT Greengrass for device software can help collect, process, and export data streams, including when devices are offline. +**Prescriptive guidance IOTREL02-BP01-01** _Follow the Reliability Foundations Best Practices defined in the AWS Well-Architected Framework._ @@ -142 +86 @@ Store important messages durably offline and, once reconnected, send those messa - * Messages collected on the device are queued and processed in FIFO order. +The most important part of your IoT scaling approach is to make sure that you architect around any hard limits because exceeding limits that are not adjustable results in application errors, such as throttling and client errors. Hard limits are related to throughput on a single IoT connection. Consider restructuring your MQTT topics, or implementing cloud-side logic to aggregate or filter messages before delivering the messages to the interested devices. @@ -144 +88 @@ Store important messages durably offline and, once reconnected, send those messa - * By default, the AWS IoT Greengrass Core stores unprocessed messages destined for AWS Cloud targets in memory. +Soft limits in AWS IoT traditionally correlate to account-level limits that are independent of a single device. For any account-level limits, you should calculate your IoT usage for a single device and then multiply that usage by the number of devices to determine the base IoT limits that your application will require for your initial product launch. AWS recommends that you have a ramp-up period where your limit increases align closely to your current production peak usage with an additional buffer. To make sure that the IoT application is not under provisioned: @@ -146 +90 @@ Store important messages durably offline and, once reconnected, send those messa - * Configure AWS IoT Greengrass to cache messages to the local file system so that they persist across core restarts. + * Consult published AWS IoT CloudWatch metrics for all limits: [AWS IoT metrics and dimensions](https://docs.aws.amazon.com/iot/latest/developerguide/metrics_dimensions.html) @@ -148 +92 @@ Store important messages durably offline and, once reconnected, send those messa - * AWS IoT Greengrass stream manager makes it easier and more reliable to transfer high-volume IoT data to the AWS Cloud. + * Monitor CloudWatch metrics in AWS IoT Core: [Logging and Monitoring](https://docs.aws.amazon.com/iot/latest/developerguide/security-logging.html) @@ -150 +94 @@ Store important messages durably offline and, once reconnected, send those messa - * [Configure AWS IoT Greengrass core](https://docs.aws.amazon.com/greengrass/v1/developerguide/gg-core.html) + * Alert on CloudWatch throttle metrics, which would signal if you need a limit increase. @@ -152 +96 @@ Store important messages durably offline and, once reconnected, send those messa - * [Manage data streams on AWS IoT Greengrass Core](https://docs.aws.amazon.com/greengrass/v1/developerguide/stream-manager.html) + * Set alarms for all thresholds in IoT, including MQTT connect, publish, subscribe, receive, and rule engine actions. @@ -154 +98 @@ Store important messages durably offline and, once reconnected, send those messa - * [AWS IoT Greengrass Developer Guide](https://docs.aws.amazon.com/greengrass/v1/developerguide/what-is-gg.html) + * Monitoring AWS IoT MQTT Traffic and Automating Quota and Throttling Notifications @@ -156 +100 @@ Store important messages durably offline and, once reconnected, send those messa