Skip to main content
Version: v3.3 print this page

Amorphic CICD Architecture & Workflow

The Amorphic CICD framework automates the deployment of Amorphic resources through a well-defined and scalable architecture. The framework consists of two primary components: CICD Utils and CICD Infrastructure. These components work together to enable resource management as code, enforce validation, and deliver consistent deployments across environments.

Core Components​

CICD Utils Module​

The CICD Utils package is a Python module provided to customers for defining and deploying Amorphic resources. It reads JSON-based resource definitions and applies them to the target Amorphic environment. The module can be installed using the Python Package Manager (PIP) on any system with Python. It is flexible in its usage, as it can be executed locally by developers or integrated into the centralized CICD infrastructure for automated deployments.

A key feature of CICD Utils is its ability to generate a state file after every deployment. This file, stored in Amazon S3, serves as an internal blueprint of all resources managed by CICD. It is critical for maintaining consistency and tracking resource states. Manual modifications to this file are strongly discouraged, as corruption can result in the framework losing track of managed resources.

CICD Infrastructure​

The CICD Infrastructure provides the automation backbone required for centralized deployments. It is built on AWS services and can be deployed to an AWS account separate from the one hosting the Amorphic environment. The infrastructure relies on AWS CodePipeline to orchestrate the deployment process, with AWS CodeBuild executing the individual stages. A centralized Git repository such as GitHub, GitLab, or Bitbucket acts as the entry point for triggering these deployments.

Whenever contributors push code to the repository, the CICD infrastructure automatically responds by initiating the relevant pipeline. Each branch in the repository is typically mapped to a specific Amorphic environment. For example, code pushed to a development branch is deployed to the development environment, while a production branch is connected to the production environment. Separate pipelines are configured for each branch to ensure environment isolation.

CICD Architecture

Pipeline Execution​

Each pipeline in the CICD infrastructure consists of two sequential phases.

The first phase is the validation stage. During this stage, the pipeline performs static code analysis (SCA) and static application security testing (SAST) on the submitted code. These checks are designed to identify potential vulnerabilities and coding issues before deployment. Any findings are reported to AWS Security Hub for further review. If critical issues are detected, the pipeline fails, and affected contributors are notified to take corrective action.

The second phase is the deployment stage. Once validation is complete, the CICD Utils module is executed on the submitted JSON resource definitions. The module connects to the appropriate Amorphic environment and deploys the resources. Depending on the branch that triggered the pipeline, resources are applied to either the development or production environment. Following a successful deployment, the CICD Utils package generates the state file, which is then stored securely in Amazon S3.

This structured pipeline ensures that only validated and secure code is promoted into Amorphic environments, thereby reducing the risk of errors and vulnerabilities.

Resource Deployment Workflow​

The resource deployment process begins when a user defines Amorphic resources such as datasets, datasources, or ETL jobs in the form of JSON files. These files are organized under a project directory and committed to the Git repository. Once the changes are pushed to a branch, the CICD infrastructure is notified and begins execution of the corresponding pipeline.

In the case of a development branch, the pipeline first validates the code and then deploys the resources to the development environment using CICD Utils. The user can subsequently log in to Amorphic and verify whether the deployed resources are functioning as expected. If adjustments are necessary, further changes can be made and re-deployed through the same process until the configuration is satisfactory.

When the resources are validated in the development environment, the next step is to promote them to production. This is typically achieved by raising a pull request (PR) from the development branch to the production branch. Once the request is approved and merged, the CICD infrastructure is again triggered. The production pipeline executes in the same manner, running validation checks before deploying the changes to the production Amorphic environment.

This workflow ensures that resources move through a controlled path from development to production. Validation at each step, combined with automated deployments, minimizes risk and provides confidence in the stability and security of the deployed resources.