Intrinsic Functions in Amorphic CICD

The Amorphic CICD Utils module provides special functions called intrinsic functions. These functions allow you to create complex resource definitions, handle dependencies, and dynamically configure resource properties. They are especially useful for referencing key values and managing inter-resource dependencies in your CICD pipelines.

Below is a detailed overview of the supported intrinsic functions.

!DependsOn Function

The !DependsOn function defines an explicit dependency between two resources. This ensures that a resource is created only after the dependent resource is fully available.

For example, when creating a dataset and a job that uses this dataset, the Job definition requires the DatasetId. Since the DatasetId is metadata generated by Amorphic and is only available after the dataset is created, we can use the !DependsOn function to dynamically reference this metadata in the Job template.

Usage Format:

"!DependsOn": "LogicalId.AttributeName"

• LogicalId → The logical name of the dependency resource.
• AttributeName → The attribute value to fetch from the dependency resource.

Example:

{
  " rDataset": {
      "Type": "Dataset",
      "Properties": {
        "DatasetName": "healthinfodataset",
        "DatasetDescription": "Health information dataset created from CICD",
        "Domain": "health",
        "Keywords": ["Owner: john"],
        "DatasourceType": "api",
        "IsDataValidationEnabled": true,
        "SerDe": "OpenCSVSerde",
        "FileDelimiter": ",",
        "FileType": "csv",
        "IsDataCleanupEnabled": false,
        "IsDataProfilingEnabled": true,
        "LifeCyclePolicyStatus": "Disabled",
        "TargetLocation": "s3athena",
        "MalwareDetectionOptions": {
            "ScanForMalware": true,
            "AllowUnscannableFiles": false
        },
        "SkipFileHeader": true,
        "SkipRowCount": { "header": 1, "footer": 0 },
        "SkipLZProcess": false,
        "TableUpdate": "append",
        "DataMetricsCollectionOptions": { "IsMetricsCollectionEnabled": false },
        "DatasetType": "internal",
        "DatasetSchema": [
            {
            "name": "FirstName",
            "description": "",
            "type": "varchar(256)",
            "is_not_null": false
            }
        ]
        }
    },
  "rPythonJob": {
    "Type": "Job",
    "Artifacts": {
      "Script": "resources/jobs/pythonjob/job_script.py"
    },
    "Properties": {
      "JobName": "pythonshelljobc",
      "Description": "Python shell Job Created from CICD with a dataset attached",
      "ETLJobType": "pythonshell",
      "NetworkConfiguration": "general-public-network",
      "JobBookmarkOption": "disable",
      "Keywords": ["Owner: john"],
      "MaxCapacity": 0.0625,
      "ParameterAccess": [],
      "SharedLibraries": [],
      "DomainAccess": { "Owner": [], "ReadOnly": [] },
      "DatasetAccess": {
        "Owner": [
          {
            "DatasetId": {
                "!DependsOn": " rDataset.DatasetId"
              },
            "DatasetName": {
                "!DependsOn": " rDataset.DatasetName"
              }
          }
        ],
        "ReadOnly": [
            {
                "DatasetId": {
                    "!DependsOn": " rDataset.DatasetId"
                  },
                "DatasetName": {
                    "!DependsOn": " rDataset.DatasetName"
                  }
              }
        ]
      },
      "DefaultArguments": { "max_read": "10" },
      "IsDataLineageEnabled": "no",
      "IsAutoScalingEnabled": false
    }
  }
}

This allows the CICD module to safely create resources that rely on other resources’ metadata.

!Environ Function

The !Environ function loads environment variables into resource definitions. This is especially useful for securely managing credentials without hardcoding them in resource templates.

Usage Format:

"!Environ": "EnvironmentVariableName"

Example:

{
  "rDatasource": {
    "Type": "Datasource",
    "Properties": {
      "DatasourceConfig": {
        "JdbcURL": "",
        "Username": {
          "!Environ": "DB_USERNAME"
        },
        "Password":  {
          "!Environ": "DB_PASSWORD"
        },
        "PublicAccessibility": "yes"
      },
      "DatasourceType": "jdbc",
      "DatasourceName": "mysqldatabase",
      "Description": "MySQL database as datasource",
      "Keywords": ["Owner: john"],
      "IngestionType": "bulkdataload"
    }
  }
}

Tip: The environment variables referenced must exist at runtime. You can populate them dynamically from Amorphic Parameter Store using a driver script:

import os
import requests

def get_parameter_from_amorphic(parameter_name):
    response = requests.get(
        f"{AMORPHIC_BASE_URL}/parameters/{parameter_name}",
        headers={
            "Content-Type": "application/json",
            "Authorization": AMORPHIC_AUTH_TOKEN,
            "role_id": AMORPHIC_ROLE_ID
        },
        timeout=10,
    )
    return response.json()["ParameterValue"]

os.environ["DB_USERNAME"] = get_parameter_from_amorphic("adp-psql-username")
os.environ["DB_PASSWORD"] = get_parameter_from_amorphic("adp-psql-password")

!Base64 Function

The !Base64 function encodes a value into Base64. This is useful when certain resource attributes, such as SQL statements for view datasets, need to be encoded before deployment.

Usage Format:

"!Base64": "PlainTextToEncode"

Example:

{
    "rViewDataset": {
      "Type": "Dataset",
      "Properties": {
        "DatasetName": "viewdataset",
        "DatasetDescription": "Dataset of type view created from CICD",
        "Domain": {
            "!DependsOn": "rDomain.DomainName"
        },
        "Keywords": ["Owner: john"],
        "ViewType": "standard",
        "TargetLocation": "s3athena",
        "SqlStatement": {
            "!Bse64" : "create view viewdomain.viewdataset as select * from viewdomain.anotherdataset"
        },
        "DatasetType": "view"
        }
    }
}

This ensures the string is encoded during resource creation while the users can write the SQL query in plain text.

!Sub Function

The !Sub function dynamically generates a formatted string by replacing placeholders inside curley braces with corresponding environment variable values. All referenced variables must exist in the environment; otherwise, an error will occur.

This function is particularly useful for prefixing resource names with environment-specific identifiers.

Usage Format:

"!Sub": "{Placeholder}-String"

Example:

{
  "rDevelopAPIKey": {
    "Type": "Parameter",
    "Properties": {
      "ParameterKey": {
        "!Sub": "adp-{environment_name}-api-key"
      },
      "ParameterValue": "**************",
      "ParameterType": "String",
      "Description": "Parameter to store the API key",
      "Scope": "global",
      "TenantName": "adp"
    }
  }
}

Here, environment_name should be set as an environment variable in the driver script.