Repository Organization

This section describes the recommended structure for organizing repositories when working with the Amorphic CICD framework. The following conventions are intended to promote consistency and maintainability across projects. While these practices are strongly recommended, they are not enforced by the framework and can be adapted to meet specific project needs.

All resource definitions must be stored as JSON files within a root directory. In the example provided below, this directory is named resources. Within this root directory, it is considered good practice to create subdirectories by resource type. For example, all dataset-related definitions may be grouped within a datasets folder, while data sources, jobs, and other resource types each maintain their own dedicated directories.

While there is no restriction on the number of resources that can be defined in one JSON file, we recommend creating individual JSON files for each resource. For instance, to define a dataset, create a folder for the dataset, then include a JSON file containing the dataset definition within that folder. This approach improves readability and helps isolate resources for easier management.

Some resources, such as ETL jobs, may require additional supporting assets such as scripts. These assets should be stored alongside the JSON definition in the same resource-specific folder. This ensures that all components required for deploying a resource are logically grouped together.

In addition to resource definitions, projects must include a special driver file called driver_file.py, and this file name must not be changed.. This driver file serves as the entry point for the Amorphic CICD Utils module during deployments. It is also recommended that each repository include a README.md file containing documentation for contributors, such as usage notes or instructions relevant to the project.

The CICD framework imposes no restrictions on the naming of files or folders. However, following a consistent and descriptive naming convention is highly encouraged to improve collaboration across teams.

The example below illustrates a recommended directory structure for organizing Amorphic CICD repositories:

.
├── driver_file.py
├── README.md
└── resources
    ├── datasources
    │   └── bulkload_v2
    │       └── bulkload_v2.json
    ├── datasets
    │   ├── cicd_dataset_1
    │   │   └── cicd_dataset_1.json
    │   └── cicd_dataset_2
    │       └── cicd_dataset_2.json
    ├── domains
    │   └── cicd_domain_1
    │       └── cicd_domain_1.json
    ├── jobs
    │   └── cicd_job_1
    │       ├── job_script.py
    │       └── cicd_job_1.json
    ├── parameters
    │   └── api_key_param
    │       └── api_key_param.json
    ├── schedules
    │   └── schedule_1
    │       └── schedule_1.json
    ├── roles
    │   └── role_1
    │       └── role_1.json
    ├── tags
    │   └── tag_1
    │       └── tag_1.json
    ├── saml_mappings
    │   └── saml_mapping_1
    │       └── saml_mapping_1.json
    ├── shared_libraries
    │   └── library_1
    │       └── library_1.json
    ├── data_pipelines
    │   └── data_pipeline_1
    │       └── data_pipeline_1.json
    ├── dataflows
    │   └── dataflow_1
    │       └── dataflow_1.json
    └── guardrails
        └── guardrail_1
            └── guardrail_1.json