AWS glue documentation change
Summary
Restructured documentation to add IAM policy examples for serializers/deserializers and guidance for AWS PrivateLink connectivity. Removed technical implementation details about SerDe library installation and configuration.
Security assessment
The changes add IAM policy examples demonstrating least-privilege permissions for schema registry operations and guidance for using AWS PrivateLink to secure network traffic. While these are security best practices, there is no evidence they address a specific disclosed vulnerability.
Diff
diff --git a/glue/latest/dg/schema-registry-gs-serde.md b/glue/latest/dg/schema-registry-gs-serde.md index b0bcf0bdf..c39ef56c1 100644 --- a//glue/latest/dg/schema-registry-gs-serde.md +++ b//glue/latest/dg/schema-registry-gs-serde.md @@ -5,3 +5 @@ -# Installing SerDe Libraries - -###### Note +IAM examples for serializersIAM examples for deserializersPrivate connectivity using AWS PrivateLink @@ -9 +7 @@ -Prerequisites: Before completing the following steps, you will need to have a Amazon Managed Streaming for Apache Kafka (Amazon MSK) or Apache Kafka cluster running. Your producers and consumers need to be running on Java 8 or above. +# Installing SerDe Libraries @@ -15 +13 @@ You will install the open source serializer for your applications producing data -To install the libraries on producers and consumers: +## IAM examples for serializers @@ -17 +15 @@ To install the libraries on producers and consumers: - 1. Inside both the producers’ and consumers’ pom.xml files, add this dependency via the code below: +###### Note @@ -19,5 +17 @@ To install the libraries on producers and consumers: - <dependency> - <groupId>software.amazon.glue</groupId> - <artifactId>schema-registry-serde</artifactId> - <version>1.1.5</version> - </dependency> +AWS managed policies grant necessary permissions for common use cases. For information on using managed policies to manage the schema registry, see [AWS managed (predefined) policies for AWS Glue](./security-iam-awsmanpol.html#access-policy-examples-aws-managed). @@ -25 +19 @@ To install the libraries on producers and consumers: -Alternatively, you can clone the [AWS Glue Schema Registry Github repository](https://github.com/awslabs/aws-glue-schema-registry). +For serializers, you should create a minimal policy similar to that below to give you the ability to find the `schemaVersionId` for a given schema definition. Note, you should have read permissions on the registry in order to read the schemas in the registry. You can limit the registries that can be read by using the `Resource` clause. @@ -27 +21 @@ Alternatively, you can clone the [AWS Glue Schema Registry Github repository](ht - 2. Setup your producers with these required properties: +Code example 13: @@ -29,4 +22,0 @@ Alternatively, you can clone the [AWS Glue Schema Registry Github repository](ht - props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); // Can replace StringSerializer.class.getName()) with any other key serializer that you may use - props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, GlueSchemaRegistryKafkaSerializer.class.getName()); - props.put(AWSSchemaRegistryConstants.AWS_REGION, "us-east-2"); - properties.put(AWSSchemaRegistryConstants.DATA_FORMAT, "JSON"); // OR "AVRO" @@ -34 +24,12 @@ Alternatively, you can clone the [AWS Glue Schema Registry Github repository](ht -If there are no existing schemas, then auto-registration needs to be turned on (next step). If you do have a schema that you would like to apply, then replace "my-schema" with your schema name. Also the "registry-name" has to be provided if schema auto-registration is off. If the schema is created under the "default-registry" then registry name can be omitted. + { + "Sid" : "GetSchemaByDefinition", + "Effect" : "Allow", + "Action" : + [ + "glue:GetSchemaByDefinition" + ], + "Resource" : ["arn:aws:glue:us-east-2:012345678:registry/registryname-1", + "arn:aws:glue:us-east-2:012345678:schema/registryname-1/schemaname-1", + "arn:aws:glue:us-east-2:012345678:schema/registryname-1/schemaname-2" + ] + } @@ -36 +37 @@ If there are no existing schemas, then auto-registration needs to be turned on ( - 3. (Optional) Set any of these optional producer properties. For detailed property descriptions, see [the ReadMe file](https://github.com/awslabs/aws-glue-schema-registry/blob/master/README.md). +Further, you can also allow producers to create new schemas and versions by including the following extra methods. Note, you should be able to inspect the registry in order to add/remove/evolve the schemas inside it. You can limit the registries that can be inspected by using the `Resource` clause. @@ -38,8 +39 @@ If there are no existing schemas, then auto-registration needs to be turned on ( - props.put(AWSSchemaRegistryConstants.SCHEMA_AUTO_REGISTRATION_SETTING, "true"); // If not passed, uses "false" - props.put(AWSSchemaRegistryConstants.SCHEMA_NAME, "my-schema"); // If not passed, uses transport name (topic name in case of Kafka, or stream name in case of Kinesis Data Streams) - props.put(AWSSchemaRegistryConstants.REGISTRY_NAME, "my-registry"); // If not passed, uses "default-registry" - props.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, "86400000"); // If not passed, uses 86400000 (24 Hours) - props.put(AWSSchemaRegistryConstants.CACHE_SIZE, "10"); // default value is 200 - props.put(AWSSchemaRegistryConstants.COMPATIBILITY_SETTING, Compatibility.FULL); // Pass a compatibility mode. If not passed, uses Compatibility.BACKWARD - props.put(AWSSchemaRegistryConstants.DESCRIPTION, "This registry is used for several purposes."); // If not passed, constructs a description - props.put(AWSSchemaRegistryConstants.COMPRESSION_TYPE, AWSSchemaRegistryConstants.COMPRESSION.ZLIB); // If not passed, records are sent uncompressed +Code example 14: @@ -47 +40,0 @@ If there are no existing schemas, then auto-registration needs to be turned on ( -Auto-registration registers the schema version under the default registry ("default-registry"). If a `SCHEMA_NAME` is not specified in the previous step, then the topic name is inferred as `SCHEMA_NAME`. @@ -49 +42,15 @@ Auto-registration registers the schema version under the default registry ("defa -See [Schema versioning and compatibility](./schema-registry.html#schema-registry-compatibility) for more information on compatibility modes. + { + "Sid" : "RegisterSchemaWithMetadata", + "Effect" : "Allow", + "Action" : + [ + "glue:GetSchemaByDefinition", + "glue:CreateSchema", + "glue:RegisterSchemaVersion", + "glue:PutSchemaVersionMetadata", + ], + "Resource" : ["arn:aws:glue:aws-region:123456789012:registry/registryname-1", + "arn:aws:glue:aws-region:123456789012:schema/registryname-1/schemaname-1", + "arn:aws:glue:aws-region:123456789012:schema/registryname-1/schemaname-2" + ] + } @@ -51 +58 @@ See [Schema versioning and compatibility](./schema-registry.html#schema-registry - 4. Setup your consumers with these required properties: +## IAM examples for deserializers @@ -53,4 +60 @@ See [Schema versioning and compatibility](./schema-registry.html#schema-registry - props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName()); - props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, GlueSchemaRegistryKafkaDeserializer.class.getName()); - props.put(AWSSchemaRegistryConstants.AWS_REGION, "us-east-2"); // Pass an AWS Region - props.put(AWSSchemaRegistryConstants.AVRO_RECORD_TYPE, AvroRecordType.GENERIC_RECORD.getName()); // Only required for AVRO data format +For deserializers (consumer side), you should create a policy similar to that below to allow the deserializer to fetch the schema from the Schema Registry for deserialization. Note, you should be able to inspect the registry in order to fetch the schemas inside it. @@ -58 +62 @@ See [Schema versioning and compatibility](./schema-registry.html#schema-registry - 5. (Optional) Set these optional consumer properties. For detailed property descriptions, see [the ReadMe file](https://github.com/awslabs/aws-glue-schema-registry/blob/master/README.md). +Code example 15: @@ -60,3 +63,0 @@ See [Schema versioning and compatibility](./schema-registry.html#schema-registry - properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, "86400000"); // If not passed, uses 86400000 - props.put(AWSSchemaRegistryConstants.CACHE_SIZE, "10"); // default value is 200 - props.put(AWSSchemaRegistryConstants.SECONDARY_DESERIALIZER, "com.amazonaws.services.schemaregistry.deserializers.external.ThirdPartyDeserializer"); // For migration fall back scenario @@ -63,0 +65,9 @@ See [Schema versioning and compatibility](./schema-registry.html#schema-registry + { + "Sid" : "GetSchemaVersion", + "Effect" : "Allow", + "Action" : + [ + "glue:GetSchemaVersion" + ], + "Resource" : ["*"] + } @@ -64,0 +75 @@ See [Schema versioning and compatibility](./schema-registry.html#schema-registry +## Private connectivity using AWS PrivateLink @@ -65,0 +77 @@ See [Schema versioning and compatibility](./schema-registry.html#schema-registry +You can use AWS PrivateLink to connect your data producer’s VPC to AWS Glue by defining an interface VPC endpoint for AWS Glue. When you use a VPC interface endpoint, communication between your VPC and AWS Glue is conducted entirely within the AWS network. For more information, see [Using AWS Glue with VPC Endpoints](https://docs.aws.amazon.com/glue/latest/dg/vpc-endpoint.html). @@ -75 +87 @@ Getting started -Creating a registry +Java Implementation