Write intermediate data using the ETL job staging directory
ETL job scripts often need somewhere to park intermediate output — a dataframe written between stages, a checkpoint, a set of files reused later in the same run. Amorphic gives every ETL job a staging directory for exactly this, and the job's execution role can delete objects there, so Spark's overwrite mode works.
Delete access on the staging folder is available starting v3.4. Jobs created from v3.4 onwards have the permission from the start; jobs that existed before the upgrade need the one-time update described in Enable it on an existing job.
The staging directory
Every ETL job is allocated a temporary location inside the Amorphic ETL bucket, passed to the Glue job through the reserved --TempDir argument:
s3://<amorphic-etl-bucket>/<JobId>/temp
The staging folder inside it is the part your script should write to:
s3://<amorphic-etl-bucket>/<JobId>/temp/staging/
The job's execution role holds s3:GetObject, s3:PutObject and s3:DeleteObject on this path.
Write with overwrite mode
Because delete is permitted on the staging path, Spark can clear previously written objects under the prefix, which is what overwrite mode requires:
temp_dir = args["TempDir"]
# Write intermediate output, replacing whatever the previous run left behind
df.write.mode( "overwrite" ).parquet( f"{temp_dir}/staging/intermediate_output" )
# Read it back later in the same run, or in a subsequent one
staged_df = spark.read.parquet( f"{temp_dir}/staging/intermediate_output" )
Without delete permission, an overwrite-mode write fails while attempting to remove the previous output.
- Delete access applies to the
temp/staging/prefix of the job's own temp folder. The rest of the temp directory remains read and write only, and other jobs' folders are not accessible. - Data written to the staging folder is not purged after each run. It persists until the ETL job is deleted, at which point the folder is cleaned up. Remove large intermediate output from within your script if you do not want it to accumulate.
- The staging folder is scratch space. Data written here is not cataloged, validated, or queryable — final output should still be written to a dataset. See Writing to a Dataset using Jobs.
Enable it on an existing job
Delete access on the staging folder is applied when a job's execution role policy is generated, which happens when the job is created or updated. Jobs that already existed before the v3.4 upgrade keep running with their earlier policy, and overwrite-mode writes to the staging folder fail with an access denied error until the policy is refreshed.
To refresh it, perform a dummy update on the job:
- Open the ETL job and click Edit.
- Make a trivial change — for example, update the job description.
- Save the job.
Once the update completes, the regenerated role policy includes delete access on the staging folder and overwrite-mode writes succeed.
This is only needed once per job that predates v3.4. The job does not need to be re-created, and its script, external libraries and resource access are untouched by the dummy update. Jobs created on v3.4 or later have the permission from the start.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
AccessDenied on s3:DeleteObject during an overwrite write | Job role policy predates v3.4 | Perform the dummy update |
AccessDenied writing outside temp/staging/ | Delete is scoped to the staging prefix only | Write intermediate output under temp/staging/ |
| Old intermediate files still present after a run | Staging data is not auto-purged | Delete them from your script, or let job deletion clean up |