Ingest Spark/OpenLineage execution plans
POST/jobs/lineage/execution-plans
Spark agent callback that stores job-run lineage. Glue OpenLineage transport posts here (jobLauncher / jobExecutions set spark.openlineage.transport.endpoint to {env}jobs/lineage/execution-plans). The older Spline agent posts a plan object to the same path. Branching is by body shape, not a query param: if producer is present and contains the substring OpenLineage, handle_openlineage_execution_events writes the event JSON under {jobId}/execution-lineage/{executionId}/{lambdaRequestId}.json and, on eventType=COMPLETE, updates execution observability. Job name and execution id are parsed from run.facets.parent.root.job.name or spark.app.name (nativespark-{JobName}-{ExecutionId}). If those cannot be resolved, or the JobName is unknown, the handler logs and still returns HTTP 200. Otherwise handle_execution_plans treats the body as a Spline plan: extraInfo.appName is split on hyphen and the last token is the Glue JobRun id; operations.write / reads / other become Lineage vertices on the executions item; a 30-day metrics row is written; the job access token is deleted. Success Message is always "Updated lineage successfully". The handler has a single except Exception: typed failures (KeyError, GE-1018 metadata update, invalid JSON) surface as HTTP 500 GE-1008, not 400. role_id is read for audit logs only and is not validated.
Request
Responses
- 200
- 500
Lineage accepted. Message is "Updated lineage successfully" even when the OpenLineage job name cannot be resolved (handler returns early after logging).
Response Headers
Any handler exception, including invalid JSON, missing Spline keys, and GE-1018 metadata update failure. Body is {"Message": "GE-1008 - Could not complete the request. Please try again."}. HTTP 400 is not returned by this lambda.