Import Feature

Amorphic CICD supports importing resources that were manually created in the Amorphic console into the CICD system. Once imported, these resources can be fully managed via CICD.

How Import Works

To import a resource, you must:

1. Define the resource in your CICD configuration with metadata that exactly matches the existing resource in Amorphic.
2. Add the resource identifier to the RESOURCES_TO_IMPORT section of your driver script.

If the resource is found and the metadata matches, CICD will import it into the system.

Example: Importing a Domain

Assume a domain was manually created in Amorphic with the following details:

• Domain Name: states
• Display Name: states
• Description: Domain for states data

Step 1: Define the resource in your resources directory

Create a new JSON file for the domain or add the definition to an existing file. You may use any logical name for the resource (e.g., rDomain), but the property values must exactly match the corresponding metadata in Amorphic.

"rDomain": {
  "Type": "Domain",
  "Properties": {
    "DomainName": "states",
    "DisplayName": "states",
    "DomainDescription": "Domain for states data"
  }
}

Step 2: Add the Resource to RESOURCES_TO_IMPORT

The ResourceManager class includes an argument called resources_to_import, which specifies the resources to be imported.

In the example below, rDomain is the logical name of the resource. Since it is of type Domain, the key field to provide is the DomainName.
In Amorphic, the actual domain name is states, so we include it in the payload.

RESOURCES_TO_IMPORT = {
    "rDomain": {
        "DomainName": "states",
    }
}

rm = ResourceManager(
    "./resources",
    amorphic_auth_token=AMORPHIC_AUTH_TOKEN,
    amorphic_base_url=AMORPHIC_BASE_URL,
    amorphic_role_id=AMORPHIC_ROLE_ID,
    validate_resource_properties=True,
    resources_to_import=RESOURCES_TO_IMPORT,
)

Each resource type requires specific keys for identification when importing. Some resources such as the Dataflow or Tag needs more than one key to be specified.

Resource Type	Resource Key
Datasource	DatasourceId
Domain	DomainName
Dataset	DatasetId
Dataflow	DataflowId, DatasourceId
Parameter	ParameterKey
Job	JobId
DataPipeline	DataPipelineId
Schedule	ScheduleId
SharedLibraries	libraryId
Tag	Key, Value
Role	RoleId
SAMLRoleMapping	SamlGroupId
SAMLTagMapping	SamlGroupId

Execution Behavior

• When you run the CICD pipeline, the execution plan will show the resources to be imported.
• If the resource is not found in Amorphic or if there is a mismatch in metadata, CICD will throw an error.
• Once a resource has been successfully imported, it will not be re-imported in subsequent CICD runs, even if it remains listed under resources_to_import. Therefore, it is safe to remove the resources_to_import argument after the import, though you may also choose to keep it without any impact.

Limitations

1. Artifact Validation is Skipped

• For resources with artifacts (e.g., Jobs, Shared Libraries), CICD does not compare the artifact in your definition with the one already in Amorphic.
• After import, CICD uploads the artifact defined in your resource file, overwriting the existing one.

Example:
If you import a Job and include a job script in the artifacts section, CICD will upload that script after import, regardless of whether it matches the script already in Amorphic.

2. Partial Metadata Validation

• Due to API limitations, some fields in resource definitions are not validated against existing metadata in Amorphic.
• This may result in partial validation for certain resource types.