Dataflows

Dataflow refers to a unit of work that migrates data from a source database to a target database.

When defining a Dataflow in CICD, the Properties section must include a special key called DatasourceId.

• The DatasourceId should contain the ID of the datasource for which this particular Dataflow is being created.
• This key is mandatory for CICD to correctly associate the Dataflow with its datasource.

Below is the sample resource definition file for Dataflow:

{
  "rDataflow": {
    "Type": "Dataflow",
    "Properties": {
      "DatasourceId": {
        "!DependsOn": "mydatasource.DatasourceId"
      },
      "DataflowType": "tasks",
      "DatasetDetails": [
        {
          "TableName": "mytable",
          "SchemaName": "myschema",
          "Size": 2,
          "DatasetId": "8ff428e4-2f27-45c3-8501-9742daf2fe27"
        }
      ],
      "CreateDataset": false,
      "DataflowName": "cicid",
      "ProcessType": "full-load",
      "TargetLocation": "s3athena",
      "DataFormat": "csv",
      "DataflowConfig": {
        "AllocatedStorage": "50",
        "ServerlessReplication": false,
        "ExtraConnectionAttributes": "",
        "SharedInstance": "no",
        "InstanceAZ": "us-west-2b",
        "PubliclyAccessibleInstance": true,
        "DmsVersion": "3.6.1",
        "InstanceClass": "dms.t3.small",
        "TableMappings": {
          "rules": [
            {
              "rule-type": "transformation",
              "rule-id": "1",
              "rule-name": "1",
              "rule-target": "schema",
              "object-locator": {
                "schema-name": "myschema",
                "table-name": "mytable",
                "table-type": "all"
              },
              "rule-action": "rename",
              "value": "name",
              "old-value": null
            },
            {
              "rule-type": "transformation",
              "rule-id": "2",
              "rule-name": "2",
              "rule-target": "table",
              "object-locator": {
                "schema-name": "myschema",
                "table-name": "mytable",
                "table-type": "all"
              },
              "rule-action": "rename",
              "value": "license_number",
              "old-value": null
            },
            {
              "rule-type": "selection",
              "rule-id": "3",
              "rule-name": "3",
              "object-locator": {
                "schema-name": "myschema",
                "table-name": "mytable",
                "table-type": "all"
              },
              "rule-action": "include",
              "filters": []
            }
          ]
        }
      }
    }
  }
}

info

Amorphic CICD only supports the creation, updating, and deletion of Dataflows.
It does not support triggering, stopping, or performing other actions on a Dataflow.