Dataflows

Dataflow refers to a unit of work that migrates data from a source database to a target database.

When defining a Dataflow in CICD, the Properties section must include a special key called DatasourceId.

• The DatasourceId should contain the ID of the datasource for which this particular Dataflow is being created.
• This key is mandatory for CICD to correctly associate the Dataflow with its datasource.

Below is the sample resource definition file for Dataflow:

{
  "rDataflow": {
    "Type": "Dataflow",
    "Properties": {
      "DatasourceId": {
        "!DependsOn": "mydatasource.DatasourceId"
      },
      "DataflowType": "tasks",
      "DatasetDetails": [
        {
          "TableName": "mytable",
          "SchemaName": "myschema",
          "Size": 2,
          "DatasetId": "8ff428e4-2f27-45c3-8501-9742daf2fe27"
        }
      ],
      "CreateDataset": false,
      "DataflowName": "cicid",
      "ProcessType": "full-load",
      "TargetLocation": "s3athena",
      "DataFormat": "csv",
      "DataflowConfig": {
        "AllocatedStorage": "50",
        "ServerlessReplication": false,
        "ExtraConnectionAttributes": "",
        "SharedInstance": "no",
        "InstanceAZ": "us-west-2b",
        "PubliclyAccessibleInstance": true,
        "DmsVersion": "3.6.1",
        "InstanceClass": "dms.t3.small",
        "TableMappings": {
          "rules": [
            {
              "rule-type": "transformation",
              "rule-id": "1",
              "rule-name": "1",
              "rule-target": "schema",
              "object-locator": {
                "schema-name": "myschema",
                "table-name": "mytable",
                "table-type": "all"
              },
              "rule-action": "rename",
              "value": "name",
              "old-value": null
            },
            {
              "rule-type": "transformation",
              "rule-id": "2",
              "rule-name": "2",
              "rule-target": "table",
              "object-locator": {
                "schema-name": "myschema",
                "table-name": "mytable",
                "table-type": "all"
              },
              "rule-action": "rename",
              "value": "license_number",
              "old-value": null
            },
            {
              "rule-type": "selection",
              "rule-id": "3",
              "rule-name": "3",
              "object-locator": {
                "schema-name": "myschema",
                "table-name": "mytable",
                "table-type": "all"
              },
              "rule-action": "include",
              "filters": []
            }
          ]
        }
      }
    }
  }
}

Note: Deployment dependencies

Dataflow has dependencies on Dataset, Datasource, and Tag.

Dependent resources should not be deleted before the primary resource; attempting to do so may lead to failures or inconsistencies during the deletion process.

Referencing this Resource

Below are the common keys that can be used in the DependsOn function to retrieve details of this resource.

Supported Keys

Key	Description
DataflowName	Returns the DataflowName value of this resource
DataflowId	Returns the DataflowId value of this resource

For additional supported keys, refer to the API definition document for the respective resource type.

Example

To reference this schedule from another resource, use !DependsOn with logical name and property in the format <logicalResourceName>.<PropertyName>.

Example: {"!DependsOn": "rDataflow.DataflowId"}

info

Amorphic CICD only supports the creation, updating, and deletion of Dataflows.
It does not support triggering, stopping, or performing other actions on a Dataflow.