Etl Jobs

APIs related to ETL/ELT Spark and Python jobs, shared job libraries, job executions, scripts, resource access, and job lineage.

📄️ List ETL jobs accessible to the user

Returns a paginated list of Glue ETL jobs the caller can access (owner, editor, or read-only). Use this to browse jobs before opening details, running an execution, or updating metadata.

📄️ Create an ETL job

Creates a Glue ETL job, IAM role, and Amorphic metadata. Use a full body for spark or pythonshell jobs, or quickCreate=yes to create from JobName and ETLJobType only. DatasetAccess is required on a full create.

📄️ Ingest Spark/OpenLineage execution plans

Spark agent callback that stores job-run lineage. Glue OpenLineage transport posts here (jobLauncher / jobExecutions set spark.openlineage.transport.endpoint to {env}jobs/lineage/execution-plans). The older Spline agent posts a plan object to the same path.

📄️ Ingest Spark execution lineage events

Spline agent callback that attaches read/write metrics to a plan previously stored by POST /jobs/lineage/execution-plans. Body must be a JSON array. Only payload[0] is used (planId plus extra.readMetrics / extra.writeMetrics). Byte fields filesSize, filesMaxSplitSize, and numOutputBytes are converted to human-readable strings before they are appended on the metrics item and written to {jobId}/execution-lineage/{executionId}.json.

📄️ Delete an ETL job

Deletes the Glue job, IAM role, S3 artifacts, and Amorphic metadata. Only the job owner can delete. If other resources depend on the job, deletion is blocked and DependentResources is returned.

📄️ Get job details or code generation status

Returns ETL job details for a job the caller can access. Use action=generate-code to retrieve AI code-generation status (requires AI services and Jobs AI enabled).

📄️ Update ETL job, bookmarks, or trigger code generation

Updates an ETL job based on the input payload. Use `action` query parameter to perform actions, such as code generation or job bookmark management (enable, disable, pause, or reset the job bookmark).

📄️ Get a presigned URL to upload a job library

Generates an S3 presigned PUT URL under the job's libs/python prefix. Requires editor or owner access. Spaces in FileName are replaced with underscores. Allowed extensions depend on ETLJobType.

📄️ Attach uploaded Python library paths to the job

Updates Glue --extra-py-files / --extra-jars from PythonLibPaths after files have been uploaded. Only PythonLibPaths is accepted. Shared libraries under the common-libs prefix cannot be modified here.

📄️ Get a presigned URL to download a job library

Returns a short-lived S3 GET URL for a library file already attached to the job. The filepath query is appended to s3://{etlBucket}/{jobId}.

📄️ List root executions for an ETL job

Returns paginated root executions only (rows without ParentExecutionId). Starting, stopping, and running rows are refreshed from Glue before they are returned. Failed runs expose ErrorMessage from Message.

📄️ Trigger an ETL job run

Starts a Glue job run and writes an executions row. Omit the body or send {} to use job defaults. Read-only callers cannot run the job. Inactive or pending-registration jobs are rejected.

📄️ Get job execution details

Returns details for a specific execution from a particular job, including optional Reruns for related re-executions. When OpenLineage is enabled, lineageType can request openlineage or columnlineage details.

📄️ Re-run a root ETL job execution

Starts a new Glue run using the same arguments and capacity as a root execution (no ParentExecutionId). The new run stores ParentExecutionId pointing at that root. Re-runs of re-runs, and currently running executions, are rejected. No request body.

📄️ Stop a running ETL job execution

Invokes Glue stop for an execution in starting or running state. Executions in any other state are rejected.

📄️ Download or trigger packaging of job execution logs

Returns a presigned URL for the latest CloudWatch logs, or with logs=all triggers async zip generation. servicename is required. logtype is required for glue and for logs=all.

📄️ Preview, generate, or download logs for a time window

Views, packages, or downloads execution logs for startTime/endTime. action is required: view-logs, generate-logs, or download-logs. servicename is required; logtype is required for glue.

📄️ Get a presigned URL for a job bookmark file

Returns a presigned GET URL for a bookmark manifest. Not allowed while the execution is starting, running, or stopping. filepath is required and must be a URL-decoded S3 URI.

📄️ Get AI error diagnosis for a job execution

Returns ErrorDiagnosisStatus and ErrorDiagnosisMetadata stored on the execution. Requires AI services and Jobs AI. Defaults to status pending and metadata {} when diagnosis has not run.

📄️ Trigger AI error diagnosis for a failed execution

Starts async diagnosis. The execution must be failed, error, or timeout, and ErrorDiagnosisStatus must not already be diagnosing or completed. No request body. Requires AI services and Jobs AI.

📄️ Retrieve the job script download URL

Returns a SigV4 presigned GET URL in Message to download the job script. Caller must send a valid role_id (AUTH-1001 if missing) and have at least read-only access on the job (AUTH-1012 if the access level is too low). Unknown JobId returns IPV-1002.

📄️ Get job script upload URL

Returns a SigV4 presigned PUT URL in Message to upload or replace the job script. Caller must send a valid role_id (AUTH-1001 if missing) and have at least editor access on the job. Read-only users are rejected with AUTH-1010 ('Read only users for the job cannot perform this action.'). Unknown JobId returns IPV-1002.

📄️ Get job resource access update status

Returns the latest extra resource-access grants on the job role (DatasetAccess, DomainAccess, ParameterAccess, SharedLibraries) plus Status and Message. Status is the job ResourceAccessStatus, defaulting to updated when unset (in_progress while the step function runs, failed if the last update did not complete).

📄️ Trigger resource access update for a job

Replaces extra dataset, domain, parameter, and shared-library grants on the job role and starts the resource-access step function. Success Message is 'Updating resource access' (HTTP 200); IAM and Lake Formation apply asynchronously. GET /jobs/{id}/resourceaccess to poll Status.

📄️ Repair and Clean Metadata for Executions

Performs a cleanup or report of metadata associated with Jobs, Schedules, Data Pipelines, Users, Datasets, Access Parity, and Datasources to ensure accurate and efficient data management. For jobs, schedules, and data-pipelines, triggers a state machine cleanup (no request body fields required). For users, datasets, access-parity, and datasources, use Mode to choose report (dry-run) or repair. For Jobs cleanup, it also removes dangling ENIs (network interfaces) in available state within the application VPC and Glue subnets that are left behind by Glue jobs.

📄️ List shared job libraries or KB sync status

Omit action (or any value other than sync-kb) to list libraries the caller can access. Default offset 1, limit 100 (max 1000), sortby LastModifiedTime, sortorder desc. action=sync-kb returns Jobs knowledge base ingestion status (Jobs AI must be enabled). No request body.

📄️ Create a shared job library

Creates a shared ETL library and grants the caller owner access. LibraryName and LibraryDescription are required (name is stored lowercase). Keywords default to []. Packages is accepted but not stored on create. Extra body fields are rejected.

📄️ Trigger knowledge base synchronization for shared libraries

Starts a Bedrock ingestion job for the shared libraries knowledge base. action must be sync-kb. Jobs AI must be enabled and the knowledge base must exist. No request body.

📄️ Get shared job library details

Returns the library the caller can access. Optional projectionExpression limits fields to LibraryId, LibraryName, AccessType, LibraryDescription, CreationTime, LastModifiedTime, LastModifiedBy, CreatedBy, Keywords. Missing library is returned as 500. No request body.

📄️ Update shared job library details

Owner or editor only. Omitted body keys keep stored values. LibraryName cannot change. Extra keys are rejected. Cannot run while LibraryStatus is update-in-progress. Package changes continue asynchronously. Returns Message when metadata update is accepted.