Skip to main content
Version: cicd.4.0 print this page

Prerequisites

Before setting up Amorphic CICD, ensure the following prerequisites are in place.

Repository Configuration​

Amorphic CICD currently supports remote Git repositories hosted on GitLab, GitHub, Bitbucket, and AWS CodeCommit.

  • Ensure the repository exists in one of these providers.
  • Create appropriate branches (e.g., dev, test, prod) to map to corresponding Amorphic environments.
  • Apply access controls or branch protection rules to prevent accidental deployments or modifications.

Amorphic Deployment​

  • Amorphic environments (dev, test, prod) must already be deployed in AWS accounts.
  • Each environment is linked to a specific repository branch during CICD deployment.

Service User​

A service user is required in each Amorphic environment to manage resource deployments via CICD. This user is dedicated to CICD operations and should not be used for regular user activity.

  1. Create a role with permissions to create, update, and delete resources.

    • We recommend assigning the Administrator role or an equivalent role with full permissions.
    • Assign this role to the service user.

  2. Collect the following details from each Amorphic environment:

    • Authentication token → CICD supports either a Personal Access Token (PAT) or a service user token. Ensure the token is active and has sufficient validity for resources deployments.
    • UserId → The identifier of the service user.
    • API Gateway URL → The base URL of the Amorphic API (different from the Amorphic UI URL). This can be obtained by inspecting API requests in the Amorphic UI.
    • RoleId → The ID of the role assigned to the service user.

Service User Token in Pipelines​

A service user token is a token generated for a non-human service account used specifically for automation. Users can also create a PAT Token (long lived) for Service User and can utilize that in CICD. In CICD, this token can be used anywhere the pipeline expects AMORPHIC_AUTHORIZATION_TOKEN.

For detailed guidance on creating and managing service users, see the Amorphic Service User documentation.

Use a service user token when:

  • You want deployments to run independently of an individual user's account lifecycle.
  • You want clearer ownership and auditability for automated pipeline actions.

Configuration steps:

  1. Create or identify a service user with the required deployment permissions.
  2. Generate a service user token in the target Amorphic environment.
  3. Store the token in your CI/CD secret manager (for example, AWS Secrets Manager, GitHub Actions Secrets, GitLab CI/CD Variables, or Bitbucket secured variables).
  4. Inject the token into pipeline runtime as AMORPHIC_AUTHORIZATION_TOKEN.

Example pipeline usage:

  • AMORPHIC_AUTHORIZATION_TOKEN=<service_user_token_value>

Alerting Configuration​

When deploying the Amorphic CICD framework, administrators must provide a list of email addresses.

  • These emails will receive alerts if the pipeline fails to deploy resources.
  • Each alert email contains a link to the Code Validation and Deployment Logs for troubleshooting.

CICD Deployment​

  • The Amorphic CICD framework is deployed in a separate AWS account after fulfilling the above prerequisites.
  • Ensure all developers subscribe to AWS SNS email notifications to stay informed about CICD events.

CICD Account Access​

All users should have at least read-only access to AWS CodeBuild and AWS CodePipeline.

This enables developers to:

  • View build and deployment logs.
  • Diagnose issues when deployments fail.
  • Redeploy resources if required.

How to Install Amorphic CICD​

Use this flow to install CICD infrastructure in AWS.

Step 1: Prerequisites​

Before deploying any stacks, ensure you have:

  • Access to the target AWS account with appropriate permissions.
  • Repository access credentials from the CloudWick support team.
  • AWS Console access for stack deployment.
  • Repository name: amorphic-cicd-infra.
  • Location of CloudFormation templates: amorphic-cicd-infra -> cf-templates -> amorphic-cicd-core-infra-stack.yaml (deploy once) and amorphic-cicd-infra -> cf-templates -> amorphic-cicd-codepipeline-stack.yaml (deploy per branch).

Step 2: Create SSM Parameter for Artifacts API Key​

Create the required SSM parameter for artifacts API access:

In AWS Console -> Systems Manager -> Parameter Store, click Create parameter and configure:

  • Name: AMORPHIC.CICD.ARTIFACTSAPIKEY
  • Type: SecureString
  • Value: Your API key from CloudWick team
  • Description: API key for accessing Amorphic CICD artifacts

Step 3: Deploy Core Infrastructure Stack (Once)​

Deploy the core infrastructure stack first using AWS Console:

  1. Go to AWS Console -> CloudFormation -> Stacks
  2. Click Create stack -> With new resources (standard)
  3. Upload template: amorphic-cicd-core-infra-stack.yaml
  4. Configure stack parameters:
    • pAppName: cicd (or your preferred app name)
    • pKMSKeyAdminRoleARN: KMS admin role ARN (provided by CloudWick)
    • pVPCEnabled: false (or true if VPC is needed)
    • pVPCCidrBlock: 10.0.0.0/26 (if VPC enabled)
    • pVCSProviderType: GitLab, GitHub, Bitbucket, Azure DevOps, CodeCommit, or S3Bucket
    • pArtifactsBaseURL: https://artifacts.int.amorphicdata.cloud/cicd
    • pCICDVersion: 3.1.1
    • pCodeScanningFlag: low, medium, or high
    • pCodeBuildImage: aws/codebuild/amazonlinux2-x86_64-standard:4.0
  5. Review and create the stack.

Step 4: Approve VCS Connection (Skip for CodeCommit and S3Bucket)​

After core stack deployment:

  • For GitLab/GitHub/Bitbucket/Azure DevOps:
    1. Go to AWS Console -> Developer Tools -> Settings -> Connections.
    2. Find the pending connection for your repository.
    3. Click Pending and approve the connection.
    4. This step is mandatory before deploying CodePipeline stacks.
  • For CodeCommit:
    • Skip this step.
    • CodeCommit is a native AWS service and does not require CodeStar Connections.
    • Ensure your CodeCommit repository exists before deploying pipeline stacks.
  • For S3Bucket:
    • Skip this step.
    • S3Bucket does not require CodeStar Connections.
    • The S3 source bucket is created automatically by the core stack.
    • Upload your source code as a zip file to trigger the pipeline (see S3Bucket Source Usage below).

Step 5: Deploy CodePipeline Stack for Each Branch​

Deploy the CodePipeline stack for each branch you want to track using AWS Console:

  1. Go to AWS Console -> CloudFormation -> Stacks
  2. Click Create stack -> With new resources (standard)
  3. Upload template: amorphic-cicd-codepipeline-stack.yaml
  4. Configure stack parameters:
    • pAppName: cicd (must match core stack)
    • pRepoName: Full repository identifier (see VCS Provider Repository Format below; not used for S3Bucket)
    • pBranchName: Branch name for this pipeline (for example, dev, main, cloud-featurebranch)
    • pApiGatewayUrl: Your Amorphic API Gateway URL
    • pUserId: Your Amorphic User ID
    • pRoleId: Your Amorphic Role ID
    • pPersonalAccessToken: Your Amorphic Personal Access Token
    • pIsDevelopmentEnvironment: false by default. Set to true to enable parallel development (see Parallel Development below).
  5. Review and create the stack.

Note: The first pipeline run can fail initially. This is expected and normal. Stack creation will still succeed.

VCS Provider Repository Format​

  • GitHub: owner/repository (example: myorg/myrepo)
  • GitLab: group/subgroup/.../repository (example: mygroup/mysubgroup/myrepo)
  • Bitbucket: workspace/repository (example: myworkspace/myrepo)
  • Azure DevOps: organization/project/repository (example: myorg/myproject/myrepo)
  • CodeCommit: repository-name (example: my-repo)
  • S3Bucket: N/A (not used)

Important notes:

  • GitHub preserves case.
  • Bitbucket normalizes to lowercase and may replace special characters with hyphens.
  • GitLab supports nested groups/subgroups.
  • Each provider has different organizational hierarchies.

S3Bucket Source Usage​

When using S3Bucket as the source:

  • Core stack creates source bucket: {pAppName}-{AccountId}-source
  • Bucket is configured with versioning, KMS encryption, EventBridge notifications, and SSL-only access.
  • Upload source as a zip file to {branch_name}/source.zip (example: dev/source.zip)
  • Uploading {branch_name}/source.zip automatically triggers the corresponding branch pipeline

Required naming:

  • Filename must be exactly source.zip.
  • Path must be {branch_name}/source.zip (for example, dev/source.zip, main/source.zip).
  • File must be a standard .zip containing source code.

For multiple branches, deploy one pipeline stack per branch and upload each branch artifact separately, for example:

  • dev/source.zip
  • main/source.zip
  • feature-branch/source.zip

Step 6: Pipeline Execution​

After CodePipeline stack deployment:

  • Secrets Manager secret is created automatically from the credentials provided during stack creation.
  • For Git-based sources (GitLab, GitHub, Bitbucket, Azure DevOps, CodeCommit), pushes to tracked branch trigger the pipeline.
  • For S3Bucket, uploading {branch_name}/source.zip triggers the pipeline.
  • The pipeline runs successfully with the configured credentials.

Parallel Development​

By default, the CodePipeline stack runs in SUPERSEDED execution mode (a newer push cancels an in-flight run) and triggers only on pBranchName (the branch name set in the CloudFormation stack during deployment).

Setting pIsDevelopmentEnvironment to true on the CodePipeline stack changes this behavior:

  • The pipeline switches to PARALLEL execution mode, so multiple runs can execute concurrently instead of cancelling each other.
  • The pipeline triggers on any branch matching the wildcard pattern cloud-*, in addition to pBranchName.

This allows multiple cloud-* feature branches to build at the same time, which is useful for shared development environments.

Credentials Per Branch​

At runtime, CodeBuild resolves the Amorphic credentials secret dynamically from the branch being built, using the name pattern <pAppName>-<BRANCH_NAME>-amorphic-credentials. The CodePipeline stack only creates this secret automatically for pBranchName.

Therefore, each additional cloud-* branch requires a matching secret to be created manually in the AWS account where CICD is deployed. The <BRANCH_NAME> portion of the secret name must exactly match the pushed branch name.

Create the secret with the AWS CLI (the name prefix must match pAppName, and the four credential fields take the same values used at stack deployment):

aws secretsmanager create-secret \
  --region <REGION> \
  --name "<pAppName>-<BRANCH_NAME>-amorphic-credentials" \
  --kms-key-id "$(aws ssm get-parameter --name AMORPHIC.CICD.CICDKMSKEYARN --query Parameter.Value --output text)" \
  --secret-string '{
    "api_gateway_url": "<AMORPHIC_API_GATEWAY_URL>",
    "user_id": "<AMORPHIC_USER_ID>",
    "role_id": "<AMORPHIC_ROLE_ID>",
    "personal_access_token": "<AMORPHIC_PERSONAL_ACCESS_TOKEN>"
  }'
warning

A missing per-branch secret causes that branch's pipeline run to fail at the deployment step. Create the matching secret before pushing a new cloud-* branch.

Recommendations​

  • Enable on only one stack per account — When multiple CodePipeline stacks live in the same CICD account (for example, dev, staging, and prod branch stacks), set pIsDevelopmentEnvironment to true on only one of them. The remaining stacks stay at the default false. Each dev-mode pipeline registers the same cloud-* wildcard trigger, so enabling it on more than one stack means a single push to a cloud-* branch fires every dev-mode pipeline at once, causing duplicate and potentially conflicting deployments into the same account.

  • Include the branch name in resource names — When working with parallel cloud-* branches, include the branch name in the names of all resources defined in the resource templates (for example, datasets, libraries etc). When multiple feature branches deploy resources concurrently into the same Amorphic environment, resources with identical names across branches will conflict, one branch's deployment will overwrite or clash with another's. More critically, after a cloud-* feature branch merges into pBranchName, the main pipeline run will attempt to reconcile the same resource definitions. If resource names do not include the branch name, the main pipeline may encounter conflicts with resources already created by the feature branch, causing deployment errors that are difficult to trace back to a specific branch. Including the branch name in resource names keeps each branch's resources isolated throughout their lifecycle, prevents cross-branch conflicts during concurrent runs, and ensures a clean merge path when the feature branch is integrated.

    Use the !Sub intrinsic function to inject the branch name dynamically at pipeline runtime:

    "DatasetName": {
      "!Sub": "{BRANCH_NAME}-my-dataset"
    }

    For a branch named cloud-feature-1, this resolves to cloud-feature-1-my-dataset at runtime.

  • Clean up feature branch resources before merging — Before merging a cloud-* feature branch into main branch, delete all resources that were created exclusively for that branch from the Amorphic environment. Feature branch resources with branch-specific names serve no purpose after the merge and will accumulate as orphaned resources in the environment.