AWS deadline-cloud: Rewrite job attachments overview and lifecycle docs
Summary
Replaces the job attachments introduction with a fuller description of how files move through Amazon S3 at four lifecycle points, notes that default queue creation configures the S3 bucket and IAM policies, and adds a topics list linking to related pages.
Security assessment
The change is a documentation rewrite of a file-transfer feature. It mentions IAM policies and S3 buckets only as configuration context, not as a fix for a vulnerability or as security guidance, so it is not security-related.
Evidence
When you create a queue in the Deadline Cloud console with default settings, the bucket and IAM policies that job attachments needs are configured for you.
Diff
diff --git a/deadline-cloud/latest/developerguide/build-job-attachments.md b/deadline-cloud/latest/developerguide/build-job-attachments.md index 44cdbbe..26070d7 100644 --- a//deadline-cloud/latest/developerguide/build-job-attachments.md +++ b//deadline-cloud/latest/developerguide/build-job-attachments.md @@ -7 +7 @@ -# Use job attachments to share files +# Job attachments @@ -9 +9 @@ -Use _job attachments_ to make files not in shared directories available for your jobs, and to capture the output files if they are not written to shared directories. Job attachments uses Amazon S3 to shuttle files between hosts. Files are stored in S3 buckets, and you don't need to upload a file if its content hasn't changed. +Job attachments is a Deadline Cloud feature that uses Amazon S3 to transfer a job's files, making the job's input files available to worker hosts and its output files available for download from the AWS Deadline Cloud monitor (Deadline Cloud monitor). The files are stored in an Amazon S3 bucket that belongs to the job's queue. When you create a queue in the Deadline Cloud console with default settings, the bucket and IAM policies that job attachments needs are configured for you. @@ -11 +11,24 @@ Use _job attachments_ to make files not in shared directories available for your -You must use job attachments when running jobs on [service-managed fleets](https://docs.aws.amazon.com/deadline-cloud/latest/userguide/smf-manage.html) because hosts don't share file system locations. Job attachments are also useful with [customer-managed fleets](https://docs.aws.amazon.com/deadline-cloud/latest/userguide/manage-cmf.html) when a job's input or output files stored on a shared network file system, such as when your [job bundle](https://docs.aws.amazon.com/deadline-cloud/latest/userguide/submit-job-bundle.html) contains shell or Python scripts. +Job attachments covers any job files that another shared storage solution doesn't cover. If your workers can reach a shared file system, define its locations as `SHARED` in a storage profile. For more information, see [Storage profiles and path mapping](./storage-profiles-and-path-mapping.html). Job attachments transfers every input and output file outside those `SHARED` locations. It works the same way on [service-managed fleets](https://docs.aws.amazon.com/deadline-cloud/latest/userguide/smf-manage.html) and [customer-managed fleets](https://docs.aws.amazon.com/deadline-cloud/latest/userguide/manage-cmf.html). + +Job attachments moves a job's files at four points in the job's lifecycle: + + 1. **When you submit the job.** The Deadline Cloud CLI or submitter reads the job bundle to find the job's input files. It hashes each file and compares the hash with the files already stored in the queue's bucket. It uploads only new or changed files. + + 2. **When a worker starts the job.** The worker downloads the job's input files into the session's working directory before the job's tasks run. If [virtual file system (VFS)](https://docs.aws.amazon.com/deadline-cloud/latest/userguide/storage-virtual.html) support is enabled for the job, the worker mounts the input files through the virtual file system instead. The virtual file system loads each file from Amazon S3 when a task reads it. + + 3. **When each task finishes.** The worker checks the output locations that the job bundle defines. It uploads any new files that it finds there to the queue's bucket. + + 4. **When the job's output is retrieved.** To download all of a job's output files, open the context menu on the job in the Deadline Cloud monitor and choose **Download output**. To browse and download individual input or output files, choose **Browse attachments**. For more information, see [Download finished output](https://docs.aws.amazon.com/deadline-cloud/latest/userguide/download-finished-output.html) and [Browse job attachments](https://docs.aws.amazon.com/deadline-cloud/latest/userguide/browse-job-attachments.html) in the _Deadline Cloud User Guide_. + + + + +The following topics describe these points in more detail. + +###### Topics + + * [Submitting files with a job](./submitting-files-with-a-job.html) + + * [Getting output files from a job](./getting-output-files-from-a-job.html) + + * [Using files from a step in a dependent step](./using-files-output-from-a-step-in-a-dependent-step.html) @@ -13 +35,0 @@ You must use job attachments when running jobs on [service-managed fleets](https -When you submit a job bundle with either the [Deadline Cloud CLI](https://pypi.org/project/deadline/) on the PyPI website or a Deadline Cloud submitter, job attachments use the job's storage profile and the queue's required file system locations to identify the input files that are not on a worker host and should be uploaded to Amazon S3 as part of job submission. These storage profiles also help Deadline Cloud identify the output files in worker host locations that must be uploaded to Amazon S3 so that they are available to your workstation. @@ -15 +36,0 @@ When you submit a job bundle with either the [Deadline Cloud CLI](https://pypi.o -The job attachments examples use the farm, fleet, queues, and storage profiles configurations from [Sample project infrastructure](./sample-project-infrastructure.html) and [Storage profiles and path mapping](./storage-profiles-and-path-mapping.html). You should go through those sections before this one. @@ -17 +37,0 @@ The job attachments examples use the farm, fleet, queues, and storage profiles c -In the following examples, you use a sample job bundle as a starting point, then modify it to explore job attachment's functionality. Job bundles are the best way for your jobs to use job attachments. They combine an Open Job Description job template in a directory with additional files that list the files and directories required by jobs using the job bundle. For more information about the [OpenJD specification](https://github.com/OpenJobDescription/openjd-specifications/wiki) on the GitHub website, see the OpenJD wiki. For more information about job bundles, see [Open Job Description (OpenJD) templates for Deadline Cloud](./build-job-bundle.html).