Skip to main content
 print this page

Data Pipelines

APIs related to Data Pipelines functionality

📄️ Create a Glue or AI Flow data pipeline

Creates a data pipeline asynchronously. DataPipelineType=ai uses Bedrock Flows; omitted or glue uses Glue workflows. Glue requires DataPipelineName, Nodes, Graph, Description, and Keywords. Name is 3-50 alphanumeric/underscore (IPV-1004 / IPV-1018). Glue writes create_in_progress then starts Step Functions; poll GET /data-pipelines/{id} for PipelineStatus. AI create rejects extra body keys (IPV-1078) and may return LambdaUploadUrls for LambdaFunction nodes. Glue validate_data_pipeline_body failures return HTTP 501 DP-1010 (not 400). Empty body is GE-1001 (HTTP 403).

📄️ Update a Glue or AI Flow data pipeline

Updates by persisted DataPipelineType (body type is not used to switch). Requires editor access. Blocked while any execution is RUNNING or STOPPING (DP-1012) or while PipelineStatus is create_in_progress/update_in_progress (DP-1010). Glue fills omitted fields from the stored row and starts async update (Message 'Data pipeline update initiated.'). Cyclic child-pipeline graphs return DP-1037. Query publish is not read. AI requires at least one of Description, Nodes, DefaultExecutionProperties, Graph, Keywords (GE-1010 overwritten, HTTP 500).

📄️ Stop or resume a data pipeline execution

Requires editor access and query operation=stop|resume (IPV-1051 if missing, IPV-1041 if invalid). stop is allowed only when ExecutionStatus is RUNNING (DP-1012 otherwise) and returns Message 'Successfully stopped the data pipeline'. resume requires a body NodeNames array (GE-1001 if empty), matching Graph (DP-1015), and at least one named node that ran (DP-1016). Resume is blocked for RUNNING/STOPPING (DP-1012). Resume success includes a new ExecutionId.

📄️ Trigger or fetch generated node log files

Primary integration is v2-downloadLogs. The working GET branch is logs=all (requires servicename=data-pipelines and logtype). It starts log file generation (Message 'Triggered log file creation') or returns PresignedURL when a file already exists. Execution must be in a terminal state (LOG-1006). logs=latest is declared in the spec but is not a working GET unless action is also a POST action (view-logs|generate-logs|download-logs), which then expects a POST body. Prefer POST for view/generate/download.

📄️ View, generate, or download node logs for a time window

v2-downloadLogs. Required query action=view-logs|generate-logs|download-logs and servicename=data-pipelines. Body startTime and endTime are UTC YYYY-MM-DDThh:mm:ss (IPV-1008 / IPV-1054). view-logs returns logPreview and logTruncated (optional nextToken). generate-logs starts Glue retrieval or tells AI callers logs are natively available. download-logs returns Presignedurl when a file matches the window. logtype error is required for some Glue node types (GE-1034). Optional filter_pattern and encoding=base64.