Skip to main content
 print this page

Write intermediate data using the ETL job staging directory

ETL job scripts often need somewhere to park intermediate output — a dataframe written between stages, a checkpoint, a set of files reused later in the same run. Amorphic gives every ETL job a staging directory for exactly this, and the job's execution role can delete objects there, so Spark's overwrite mode works.

Applicable from v3.4

Delete access on the staging folder is available starting v3.4. Jobs created from v3.4 onwards have the permission from the start; jobs that existed before the upgrade need the one-time update described in Enable it on an existing job.

The staging directory

Every ETL job is allocated a temporary location inside the Amorphic ETL bucket, passed to the Glue job through the reserved --TempDir argument:

s3://<amorphic-etl-bucket>/<JobId>/temp

The staging folder inside it is the part your script should write to:

s3://<amorphic-etl-bucket>/<JobId>/temp/staging/

The job's execution role holds s3:GetObject, s3:PutObject and s3:DeleteObject on this path.

Write with overwrite mode

Because delete is permitted on the staging path, Spark can clear previously written objects under the prefix, which is what overwrite mode requires:

temp_dir = args["TempDir"]

# Write intermediate output, replacing whatever the previous run left behind
df.write.mode( "overwrite" ).parquet( f"{temp_dir}/staging/intermediate_output" )

# Read it back later in the same run, or in a subsequent one
staged_df = spark.read.parquet( f"{temp_dir}/staging/intermediate_output" )

Without delete permission, an overwrite-mode write fails while attempting to remove the previous output.

Scope and retention
  • Delete access applies to the temp/staging/ prefix of the job's own temp folder. The rest of the temp directory remains read and write only, and other jobs' folders are not accessible.
  • Data written to the staging folder is not purged after each run. It persists until the ETL job is deleted, at which point the folder is cleaned up. Remove large intermediate output from within your script if you do not want it to accumulate.
  • The staging folder is scratch space. Data written here is not cataloged, validated, or queryable — final output should still be written to a dataset. See Writing to a Dataset using Jobs.

Enable it on an existing job

Delete access on the staging folder is applied when a job's execution role policy is generated, which happens when the job is created or updated. Jobs that already existed before the v3.4 upgrade keep running with their earlier policy, and overwrite-mode writes to the staging folder fail with an access denied error until the policy is refreshed.

To refresh it, perform a dummy update on the job:

  1. Open the ETL job and click Edit.
  2. Make a trivial change — for example, update the job description.
  3. Save the job.

Once the update completes, the regenerated role policy includes delete access on the staging folder and overwrite-mode writes succeed.

One-time action

This is only needed once per job that predates v3.4. The job does not need to be re-created, and its script, external libraries and resource access are untouched by the dummy update. Jobs created on v3.4 or later have the permission from the start.

Troubleshooting

SymptomCauseFix
AccessDenied on s3:DeleteObject during an overwrite writeJob role policy predates v3.4Perform the dummy update
AccessDenied writing outside temp/staging/Delete is scoped to the staging prefix onlyWrite intermediate output under temp/staging/
Old intermediate files still present after a runStaging data is not auto-purgedDelete them from your script, or let job deletion clean up