Data Pipelines

Data Pipelines enable you to orchestrate and visualize complex analytical processes by combining various components including ETL jobs, machine learning model inference tasks, notifications, and AWS AI services.

Below is the sample resource definition file for Data Pipeline:

{
  "rDataPipeline": {
    "Type": "DataPipeline",
    "Properties": {
      "DataPipelineName": "cicddatapipeline",
      "Description": "Data pipeline created using CICD",
      "Keywords": [
        "Owner: john"
      ],
      "DefaultExecutionProperties": {
        "from": ""
      },
      "Nodes": [
        {
          "ModuleType": "start_node",
          "NodeName": "Start"
        },
        {
          "ModuleType": "end_node",
          "NodeName": "End"
        },
        {
          "ModuleType": "email",
          "NodeName": "email",
          "Arguments": {},
          "EmailToExecutionPropertyKey": "from",
          "EmailSubjectExecutionPropertyKey": "from",
          "EmailBodyExecutionPropertyKey": "from"
        }
      ],
      "Graph": {
        "Start": {
          "success": [
            "email"
          ],
          "failure": []
        },
        "End": {
          "success": [],
          "failure": []
        },
        "email": {
          "success": [
            "End"
          ],
          "failure": []
        }
      },
      "tags": {},
      "CostTags": []
    }
  }
}

Note: Deployment dependencies

DataPipeline has dependencies on Job and Tag.

Dependent resources should not be deleted before the primary resource; attempting to do so may lead to failures or inconsistencies during the deletion process.

Referencing this Resource

Below are the common keys that can be used in the DependsOn function to retrieve details of this resource.

Supported Keys

Key	Description
DataPipelineId	Returns the DataPipelineId value of this resource
DataPipelineName	Returns the DataPipelineName value of this resource

For additional supported keys, refer to the API definition document for the respective resource type.

Example

To reference this schedule from another resource, use !DependsOn with logical name and property in the format <logicalResourceName>.<PropertyName>.

Example: {"!DependsOn": "rDataPipeline.DataPipelineId"}