Skip to main content
Version: cicd.5.0 print this page

Release Notes

Note

Compatibility Requirements:

  • Minimum supported Amorphic version is 3.4.
  • Minimum supported Python version is 3.12.

New Features​

  • Flexible agent packaging

    • Agent code can now be delivered either as a prebuilt archive or as a source folder that CICD packages during deployment, declared under Artifacts.Package with a File or Folder key.
    • Teams that keep agent code as a normal source tree no longer need a separate packaging step before the pipeline runs.

  • Folder based shared library artifacts

    • Artifacts.Library now accepts either File for individual files or Folder for a directory that is packaged before upload.
    • This makes it practical to ship larger shared libraries that span multiple modules. The existing plain list form continues to work.

  • Parameter Managers

    • Parameter resources accept a ParameterManagers list, so responsibility for a parameter can be delegated to named managers as part of the resource definition.
    • CICD keeps the recorded managers aligned with Amorphic after every create and update, which keeps deployment state accurate over time.

  • Many to many SAML mapping

    • A single SAML group can now be mapped to multiple roles or multiple tags in one definition.
    • This removes the need to duplicate mapping resources for each role or tag, and makes group based access easier to review.

  • Redshift availability pre-check

    • Before any resource is deployed, CICD confirms that the Redshift warehouse is available and accepting queries.
    • When the warehouse is idle or paused, the run stops before anything is deployed rather than failing partway through with an unrelated looking error. The check also wakes the warehouse, so a re-run shortly afterwards normally succeeds.

  • Graceful stop before the build times out

    • CICD is now aware of the configured build timeout and stops deliberately a short while before it is reached, so deployment state is saved.
    • A re-run continues from where the previous run stopped, instead of restarting a long release from the beginning.

  • AI data pipelines

    • DataPipeline supports an ai pipeline type, with nodes for agents, large language models, extraction, document splitting, and document analysis.
    • Notification targets can be configured at both pipeline and node level, so pipeline outcomes can be routed without additional wiring.

  • Automated reasoning policies on guardrails

    • Guardrail resources accept AutomatedReasoningPolicies (up to two) together with a confidence threshold, bringing guardrail definitions in line with the current Amorphic capability set.

Enhancements​

  • Access grants and revokes are now confirmed with Amorphic before the deployment continues, so recorded access always reflects what was actually applied.
  • Error reporting on asynchronous operations now surfaces the message returned by Amorphic first, with a clear fallback when no detail is returned.
  • Data pipeline create and update are tracked to completion rather than assumed successful once submitted.
  • Dataflow updates are tracked through a dedicated status field instead of matching message text, which adds a real failure path with the underlying reason attached.
  • Retry behaviour for blocked requests changed to fewer attempts with longer waits, and now also covers artifact uploads.
  • Datalabs are deployed after their lifecycle configuration, and shared libraries after their tags, without needing explicit dependencies in the definition.
  • Deleting and re-creating datalabs, agents, and shared libraries is now repeatable, because CICD recognises the responses Amorphic returns for resources that are already gone.
  • A guardrail service assignment failure no longer fails the whole deployment. The guardrail is kept and the assignment is retried on the next run.
  • MailServer Port and Enabled, and Datalab VolumeSizeInGB and MaxVolumeSizeInGB, now accept string values, which makes templated configuration easier.

Breaking Changes​

  • Agent artifacts are declared as Package instead of Script

    • What changed: Artifacts.Script has been renamed to Artifacts.Package, and its value is now an object containing exactly one of File (a path to a .zip) or Folder (a path to a source directory).
    • Why: Agents are deployed as packages rather than single scripts, and an explicit input type removes ambiguity about how the artifact is assembled.
    • Impact on existing pipelines: Agent definitions using Artifacts.Script, a bare string value, or a File path that is not a .zip will fail before upload.
    • Action required: Convert the artifact declaration. Example migration: "Artifacts": { "Script": "agents/agent.py" } -> "Artifacts": { "Package": { "Folder": "agents/agent_src" } }

  • Agent resources now require PythonVersion, EntryPoint, and AgentType

    • What changed: These three fields are mandatory. Instruction, Tools, and AgentResponseType have been removed, and KnowledgebaseAccess changed from a list of names to a list of objects containing KnowledgebaseId and KnowledgebaseName.
    • Why: Agent definitions now describe the runtime that Amorphic builds, so the language version and entry point must be stated explicitly.
    • Impact on existing pipelines: Existing agent definitions will fail validation with a missing required field. Because the Agent definition still tolerates unrecognised properties, leftover Instruction and Tools entries are ignored rather than reported, so the missing field is the error that appears.
    • Action required: Add PythonVersion, EntryPoint, and AgentType, and remove the retired fields. Example: "PythonVersion": "PYTHON_3_12", "EntryPoint": "agent_main.py", "AgentType": "CUSTOM"

  • SAML mapping keys renamed for many to many mapping

    • What changed: SAMLRoleMapping uses Roles (an array) in place of RoleId (a string), and SAMLTagMapping uses Tags (an array) in place of Tag (a string).
    • Why: A SAML group can now map to more than one role or tag, which requires a list rather than a single value.
    • Impact on existing pipelines: Both resource types reject unrecognised properties, so definitions using the old keys fail validation outright.
    • Action required: Convert single values to single item arrays. Example migration: { "RoleId": "abc123" } -> { "Roles": ["abc123"] } and { "Tag": "Team#DataEng" } -> { "Tags": ["Team#DataEng"] }

  • Fields that previously accepted any text now accept only a fixed set of values

    • What changed: A large number of properties across Dataset, Job, DataPipeline, Datalab, Parameter, KnowledgeBase, CostTag, and Datasource are now restricted to declared value lists. The complete lists are in the Validations and Schema Changes section below.
    • Why: Closed value lists catch typos and unsupported settings during validation instead of surfacing them as confusing backend failures during deployment.
    • Impact on existing pipelines: This is the most likely reason a definition that deployed successfully on 4.0 fails on 5.0. Values that differ only in case or spelling from the allowed value are now rejected.
    • Action required: Compare the affected properties against the allowed value lists below and correct any that do not match.

  • Job resources now require JobName and ETLJobType

    • What changed: Job definitions previously had no mandatory fields. Both JobName and ETLJobType are now required.
    • Why: Explicit naming and job type improve job identification and dependency resolution, and match how every other resource type is defined.
    • Impact on existing pipelines: Job definitions relying on defaults or on values supplied elsewhere will fail validation.
    • Action required: Add JobName and ETLJobType to every Job resource definition. ETLJobType accepts spark or pythonshell.

  • Descriptive fields are now mandatory on four resource types

    • What changed: CostTag requires TagName and TagDescription. DatalabLifecycleConfiguration, KnowledgeBase, and Schedule each require Description.
    • Why: Consistent naming and description across resource types improves discoverability and audit readiness in Amorphic.
    • Impact on existing pipelines: Definitions omitting these fields will fail validation.
    • Action required: Add the fields listed above to the affected resource definitions.

  • KnowledgeBase rejects unrecognised properties

    • What changed: Properties that are not part of the KnowledgeBase definition were previously ignored and are now reported as validation errors.
    • Why: Silently dropped properties hide typos and give the impression that a setting was applied when it was not.
    • Impact on existing pipelines: Knowledge base definitions carrying stale or misspelled keys will fail validation.
    • Action required: Remove properties that are not part of the current KnowledgeBase definition.

  • Datasource no longer accepts ScheduleConfig

    • What changed: The ScheduleConfig property has been removed from the Datasource definition.
    • Why: Scheduling is modelled as a dedicated Schedule resource, which keeps a single place to define and review schedules.
    • Impact on existing pipelines: Datasource definitions containing ScheduleConfig will fail validation, because the resource type rejects unrecognised properties.
    • Action required: Remove ScheduleConfig from datasource definitions and define a Schedule resource instead.

Validations & Schema Changes​

Newly added schema validations

Newly required fields, by resource type:

  • Agent: PythonVersion, EntryPoint, AgentType.
  • CostTag: TagName, TagDescription.
  • DatalabLifecycleConfiguration: Description.
  • Job: JobName, ETLJobType.
  • KnowledgeBase: Description.
  • Schedule: Description.
  • SAMLRoleMapping: Roles.
  • SAMLTagMapping: Tags.

Newly added allowed value lists:

  • Agent
    • PythonVersion: PYTHON_3_10, PYTHON_3_11, PYTHON_3_12, PYTHON_3_13, PYTHON_3_14.
    • AgentType: CUSTOM.
  • CostTag
    • TagScope: application, datasets, datasources, jobs, parameters, datalabs, code-repositories, models, data-pipelines, schedules, agents, ai-models, hcls-omics-workflows, hcls-omics-storage, hcls-health-imaging, hcls-healthlake.
  • Datalab
    • ComputeMetadata.DirectInternetAccess, ComputeMetadata.RootAccess, and AdditionalMetadata.RStudioAccessStatus: Enabled, Disabled.
  • DatalabLifecycleConfiguration
    • Type: notebook, studio.
  • DataPipeline
    • DataPipelineType: glue, ai.
    • Node ExecutionMode: basic, enrichment.
    • Node ModuleType is now restricted to the supported node types, including the new AI nodes (agent_node, llm_node, llm_extraction_node, bda_node, split_doc_node). See Data Pipelines for the full list.
  • Dataset
    • DatasetType: internal, external, view.
    • TargetLocation: s3, s3athena, redshift, lf, dynamodb.
    • DatasourceType: api, s3, jdbc, ext-api, email, arcgis, file-server, streams, saas.
    • IngestionType: normaldataload, bulkdataload, advanceddataload.
    • FileType: csv, tsv, json, ndjson, jsonl, parquet, xlsx, txt, pdf, jpg, png, mp3, wav, mp4, others.
    • SerDe: OpenCSVSerde, LazySimpleSerDe, RegexSerDe, ParquetHiveSerDe, OpenXJSONSerDe.
    • ViewType: standard, materialized. Encoding: none, base64. AssumeRole: yes, no.
  • Datasource
    • SslMode: none, require, verify-ca, verify-full.
    • PasswordEncoding: base64, none.
  • Guardrail
    • Tier: CLASSIC, STANDARD. Scope: global, private.
  • Job
    • ETLJobType: spark, pythonshell.
    • GlueVersion: 3.0, 4.0, 5.0.
    • WorkerType: Standard, G.1X, G.2X, G.4X, G.8X, G.12X, G.16X, R.1X, R.2X, R.4X, R.8X.
    • NetworkConfiguration: app-private-network, app-public-network, app-private-network-2, app-public-network-2, general-public-network, N/A.
    • IsDataLineageEnabled: yes, no. JobBookmarkOption: enable, disable, pause.
  • KnowledgeBase
    • DataStoreType: s3athena, redshift.
  • Parameter
    • ParameterType: String, StringList, SecureString. Scope: global, user.
  • Schedule
    • State: ENABLED, DISABLED.

Current required fields by resource type

Resource typeRequired fields
AgentAgentName, Description, ModelId, PythonVersion, EntryPoint, AgentType
CostTagTagKey, TagValues, TagName, TagDescription
DashboardDashboardName
DataClassificationDataClassificationName
DataflowDataflowName, DatasourceId
DatalabDatalabName, DatalabType
DatalabLifecycleConfigurationLifecycleName, Description
DataPipelineDataPipelineName
DatasetDomain, DatasetName, DatasetType, TargetLocation
DomainDomainName
GuardrailGuardRailName, Description, Services
JobJobName, ETLJobType
KnowledgeBaseKnowledgebaseName, Description
MailServerServerName, ServerDescription, Port, Enabled, UserName, Password, AccountEmail
ParameterParameterKey, ParameterValue, ParameterType, Description, Scope
ProjectProjectName
RoleRoleName, RolePermissions, UsersAttached, RoleManagers
SAMLRoleMappingSamlGroupId, Roles
SAMLTagMappingSamlGroupId, Tags
ScheduleJobName, ScheduleType, ScheduleParam, JobType, Resource, Arguments, Description
SharedLibrariesLibraryName, LibraryDescription, Packages
DatasourceNone
TagNone

Nested structures also carry required fields. Notable examples: DataPipeline nodes require ModuleType and NodeName; Dataset MalwareDetectionOptions requires ScanForMalware and AllowUnscannableFiles; Job and Schedule cost tag entries require TagKey and TagValue; Agent and KnowledgeBase guardrail entries require Id and Version.

Updated existing schema validations

  • Fields that are no longer required, so existing definitions remain valid:
    • Agent: Instruction, Tools, ResourceAccessMetadata.
    • DataClassification: Description.
    • Dataset: DatasetDescription, Keywords.
    • KnowledgeBase: GuardRails.
  • Removed fields: Datasource ScheduleConfig, Project EnableSmartOrchestration, and the Agent fields listed under Breaking Changes.
  • Newly supported optional fields:
    • Agent: AgentId, DisplayName, session idle timeout and maximum lifetime.
    • CostTag: TagScope.
    • Dashboard: DashboardId.
    • DataClassification: DisplayName.
    • Datalab: EnableCustomR, code repository access, and studio lifecycle configuration.
    • DataPipeline: notification configuration at pipeline and node level, node level guardrails, compute resource, dataset access, shared library access, and the AI node settings.
    • Dataset: IsGovernedTable, SerdeProperties, TableProperties, IngestionType, CostTags, and a spatial type on schema columns.
    • Datasource: DisplayName, DatasourceConnectorName, certificate key, SslMode, PasswordEncoding, and OAuth fields.
    • Guardrail: AutomatedReasoningPolicies and PoliciesConfidenceThreshold.
    • KnowledgeBase: SmartChunking.
    • Parameter: ParameterManagers, CostTags, and attached code repositories.
    • Project: ProjectId.
    • Schedule: State.
    • SharedLibraries: Keywords.
    • Tag: DisplayName, UsersAttached.
  • Value types widened, which makes templated and environment driven configuration easier:
    • MailServer: Port accepts an integer or a string, and Enabled accepts a boolean or a string.
    • Datalab: VolumeSizeInGB and MaxVolumeSizeInGB accept an integer or a string.

Validation engine behavior updates

  • Most resource types now reject properties that are not part of the definition. Typos and stale keys are reported during validation instead of being silently dropped, which removes the case where a setting appears to be applied but is not.
  • Validation errors remain location aware, so failures include the exact path to the problematic field (for example: ResourceName.Properties.field). This keeps debugging fast in CICD logs, because the failing key can be corrected directly and the pipeline rerun.

Behavior Changes​

  • Deployment stops before any resource is created when the Redshift warehouse is not available, and the message explains that the warehouse has been woken up and the run can be retried shortly. When the state cannot be determined, a warning is logged and the deployment continues. The check is a safeguard and not a new requirement.
  • Adding a Share block to an already deployed resource and rerunning applies the sharing. No property change is needed to trigger it.
  • When sharing partially fails, the resource is recorded with exactly the access that was applied, and the remainder is retried on the next run instead of being lost or reapplied incorrectly.
  • Agent packages are uploaded before the property update, so a new entry point is in place when Amorphic rebuilds the agent runtime.
  • Schedule ScheduleParam and Agent AgentType are applied at creation only and are ignored on update.
  • Import comparison ignores values that Amorphic maintains automatically, including the dataset metrics collection timestamp and data pipeline status and message. This removes spurious mismatches during import.
  • Deleting a dataflow that has already been removed completes cleanly instead of failing the run.

Infrastructure & Pipeline Changes​

  • Longer builds are supported by default

    • A new pCodeBuildTimeOut stack parameter sets the build timeout, with a default of 300 minutes. Both the security validation and deployment builds previously used the AWS default of 60 minutes, so long releases that stopped at the one hour mark now have room to complete.

  • The deployment respects the configured timeout

    • The deployment build reports the configured timeout to CICD, which stops gracefully shortly before it is reached so deployment state is saved and the run can be resumed.

  • Failed deployments are now reported as failed

    • Previously a deployment could fail while the pipeline still reported success. The build now fails when the deployment exits with an error, when deployment state is missing after the run, or when the state or log upload fails.
    • After upgrading, expect failures that were previously hidden to become visible. These are existing problems now being surfaced rather than new failures.

  • CodePipeline upgraded to V2 for Git provider pipelines

    • Pipelines sourced from GitHub, GitLab, Bitbucket, and Azure DevOps now run as CodePipeline V2, with an explicit push trigger for the configured branch, and the branch that triggered the run is the branch that is deployed.
    • Two points to note before upgrading: AWS bills V2 pipelines differently from V1, and updating the stack converts the pipeline in place. Pipelines sourced from CodeCommit or an S3 bucket are unchanged.

  • Deployment state is no longer written to build logs by default

    • Deployment state was previously printed in full to the build logs. To opt in, set CICD_STATE_LOGS=true in a default.conf file at the repository root. default.conf also accepts LOG_LEVEL and LOG_FILE.

  • New build role permission

    • The build role requires codebuild:BatchGetBuilds, scoped to the deployment build project, so the build can read its own configured timeout. Teams whose stack updates go through a security review should account for this when upgrading.

  • New pIsDevelopmentEnvironment parameter

    • This parameter enables parallel pipeline execution and wildcard branch triggers and is intended for internal development use. Leave it set to false for all normal deployments.

Bug Fixes​

  • Reapplying access that a user or tag already holds no longer fails the deployment, so reruns with unchanged sharing are safe. Revoke attempts for access that is already absent are also recognised as harmless.
  • Fixed a race condition where an agent property update could be applied before its new package was available, which caused failures whenever the new entry point existed only in the new package.
  • Guardrail service assignment no longer resends model configuration, which was causing failures in environments still holding retired models.
  • Fixed stopping a datalab before deletion.
  • Fixed reading the access management directory path.
  • Fixed parameter key handling during create and update.
  • Improved reliability of artifact uploads by retrying when requests are transiently blocked.