Datasources

APIs related to Datasources functionality

📄️ Authenticate an AWS Transfer file-server user

AWS Transfer Family identity-provider callback (SigV4, no role_id). Authenticates username with the Password header (Cognito) or, if Password is omitted, SSH keys from SSM. The user must have at least read on the file-server datasource. 200 returns a LOGICAL home directory for datasets they own or can edit on that server. Password auth is not idempotent.

📄️ Get all datasources user access to

Returns a list of datasources of which the user is owner, editor or has viewer access. Use action=get-ca-upload-url to get a CA certificate upload presigned URL response. Use request_type=versions to get available datasource versions, or request_type=ingestion_types to get supported ingestion types per datasource type.

📄️ Create a new datasource

Creates a new datasource. Request body must include DatasourceName and DatasourceType. IngestionType is optional for most types (defaults to the first supported type) but required for saas. For IngestionType=customconnector, DatasourceConnectorName is also required. Response includes Message and DatasourceId.

📄️ Get list of all dataflow of type tasks in system

Returns JDBC bulkdataload/advanceddataload dataflow tasks across datasources the user can access. Excludes normaldataload and deleted/failed/registration flows. Optional projectionExpression limits returned attributes.

📄️ Get datasource details

Returns a single datasource's details (view access required). Use action=get-ca-upload-url for a DMS CA certificate upload presigned URL. Use request_type=get_dependent_resources, get_entity_info, or get_rules_config for alternate payloads. For saas datasources, request_type may also be mappings, list_connector_entities, or describe_connector_entity (entity_type required for describe_connector_entity; api_version required when IngestionType is customconnector).

📄️ Update/upgrade/downgrade datasource details

Updates datasource details or upgrades/downgrades/toggles based on action (edit, upgrade, downgrade, start, stop). Requires edit access. DatasourceName, DatasourceType, and IngestionType cannot be changed. For arcgis upgrade/downgrade, DatasourceVersion is required in the body.

📄️ Delete the datasource

Deletes a datasource (owner access required). Fails if dependent schedules or insight dashboards still exist for applicable types. Optional delete-lineage=true also removes lineage associations.

📄️ Run a read-only datasource operation

Dispatches on the action query parameter. Caller needs at least read/view on the datasource (else AUTH-1012). Safe to retry (read-only). Actions: testDatasource (jdbc, email, s3, ext-api, amorphic, arcgis — other types fail with GE-1004); getSampleData (ext-api only; other types return 200 Message); getSupportedItemTypes (arcgis only; other types return 200 Message); getSchemas, getSchemaTables, getTableMetadata (JDBC bulkdataload or advanceddataload only — otherwise the handler leaves the response unset and can fail with 500).

📄️ Enable or disable datasource schedule

Enables or disables EventBridge schedule for amorphic and jdbc datasources only. Caller needs editor access (else AUTH-1012). Other datasource types or unsupported actions raise a generic Exception (500). Not idempotent: enabling when already enabled or disabling when already disabled returns IPV-1045.

📄️ List ArcGIS items for ingestion

ArcGIS only. With action=getItems, returns a page of items for the given ingestionType and itemType. Caller needs editor access (else AUTH-1012). Non-arcgis types return 200 with a not-supported Message. Pagination: offset defaults to 0, limit defaults to 100 (max 100); last page when next_available is false.

📄️ Get datasource flows: replication tasks, consumers, flows..

Returns paginated dataflows for a datasource (view access required). Supported types: jdbc bulkdataload, jdbc advanceddataload, saas, streams, and arcgis. For jdbc bulkdataload, action=get-presigned-url returns a presigned PUT URL string for uploading a large create payload. MetadataUpdateStatus and MetadataUpdateMessage are defaulted when absent.

📄️ Create dataflows

Creates a dataflow for the datasource (edit access required). Supported types: jdbc bulkdataload, jdbc advanceddataload (not GovCloud), saas, streams, and arcgis. Request body shape varies by datasource type (DatasourceFlowsDetails). For jdbc bulkdataload, required fields include DataflowName, ProcessType, TargetLocation, DataflowConfig, and CreateDataset. Saas flows require DataflowName plus CreateDataset/DatasetDetails. Response includes Message and DataflowId.

📄️ Get a single datasource dataflow

Returns details of one dataflow under an existing datasource. The caller (identified by role_id) must have at least read/view permission on the datasource (else AUTH-1012). Supports JDBC Bulkload, JDBC Advanced Load, SaaS, Streams, and ArcGIS. Response fields vary by type (see DatasourceFlowsDetails).

📄️ Start, stop, or control a dataflow run

Runs a control action on an existing dataflow. The caller (identified by role_id) must have at least edit permission on the datasource (else AUTH-1012). Streams datasources are not supported. Supported DataflowAction values by type: JDBC Bulkload start|stop|resume| restart|terminate; JDBC Advanced Load start|terminate (optional action=reload-tables with start on full-load only); SaaS start|stop (stop not allowed for ondemand); ArcGIS start|stop.

📄️ Update a datasource dataflow

Updates an existing dataflow. The caller (identified by role_id) must have at least edit permission on the datasource (else AUTH-1012). Supports JDBC Bulkload, JDBC Advanced Load, SaaS, Streams, and ArcGIS. For JDBC Bulkload and Advanced Load, query action=update-dataflow is required. JDBC Bulkload also supports action=reload-tables. SaaS, Streams, and ArcGIS do not use the action query param.

📄️ Delete a datasource dataflow

Deletes an existing dataflow. The caller (identified by role_id) must have at least owner permission on the datasource (else AUTH-1012). Supports JDBC Bulkload, JDBC Advanced Load, SaaS, Streams, and ArcGIS. Soft vs permanent delete and whether action is used depend on type (see action). Not idempotent for soft delete when the flow is already deleted (SaaS/Streams return GE-1034).

📄️ List datasource entities

Lists entities for a datasource. The caller (identified by role_id) must have at least read/view permission (else AUTH-1012). Supports JDBC Bulkload (DMS instances), JDBC Advanced Load (Kafka clusters), and Streams (data transformation functions). Other types return GE-1007. Streams action=get_presigned_url returns an upload URL and EntityId instead of a list.

📄️ Create a datasource entity

Creates an entity for a datasource. The caller (identified by role_id) must have at least edit permission (else AUTH-1012). Supports JDBC Bulkload (shared DMS instance), JDBC Advanced Load (shared Kafka cluster; blocked in GovCloud with GE-1034), and Streams (data transformation function after get_presigned_url upload). Other types fail at runtime. Not idempotent.

📄️ Get a datasource entity

Returns one entity. Non-global entities require at least read/view on the path datasource (else AUTH-1012). Global entities (IsGlobalEntity=true) skip the datasource permission check. Supports JDBC Bulkload shared instances, JDBC Advanced Load shared clusters, and Streams transformation functions.

📄️ Update a datasource entity

Updates an entity. Requires at least edit permission on the path datasource (else AUTH-1012). Shared entities use a UUID entity_id and must belong to the path datasource. Non-UUID entity_id updates a JDBC Bulkload dedicated DMS instance and requires EntityConfig.DataflowId. Advanced dedicated entity_id equal to datasource_id cannot be edited (GE-1034). Not idempotent.

📄️ Delete a datasource entity

Deletes a shared entity. Requires at least edit permission on the path datasource (else AUTH-1012). Entity must belong to the path datasource. Advanced dedicated entity_id equal to datasource_id cannot be deleted (GE-1034). Supports instances, functions, and kafka-cluster entity types. Not idempotent when already deleted.

📄️ List assets in a datasource

Returns a paginated list of assets for amorphic, jdbc, or arcgis datasources.

📄️ Get details of an asset in a datasource

Returns details of an asset for amorphic, jdbc, or arcgis datasources. Use action=get_dependencies to return DependentResources instead (internal amorphic datasources only). Response includes AccessType and RequestStatus; AssetSchema is omitted for users without owner/editor access.

📄️ Update asset metadata

Updates asset metadata for amorphic, jdbc, or arcgis datasources (edit access required). Default body updates CustomMetadata and/or BusinessMetadata.Keywords. Use action=update-glossary-details to set per-column glossary terms (body is a map of column name to TermId/GlossaryId objects). Use action=review_all or review_suggestion to approve/decline AI suggestions on internal amorphic dataset assets (body requires ReviewAction; Columns required for review_suggestion). Response is Message.

📄️ Get list of comments for an asset

Returns a paginated list of comments for an asset (view access on the datasource required). For internal amorphic assets, the caller must also have at least read access on the underlying asset. Comments are sorted by CommentTime.

📄️ Add a comment to an asset

Adds a comment to an asset (edit access on the datasource required). For internal amorphic assets, the caller must also have at least read access on the underlying asset. Request body must include Type and Description. Description max length is 1000 characters. Response includes Message and CommentId (CommentTime timestamp).

📄️ Delete the comment on an asset

Deletes a comment on an asset (edit access on the datasource required). For internal amorphic assets, asset-level permission is also checked. The caller must be the comment author, or an owner/editor of the asset. comment_id is the CommentTime value returned as CommentId on create (URL-encoded values are accepted). Response is Message only.

📄️ List datasource sync jobs

Lists sync job runs for a datasource. Caller needs at least read/view (else AUTH-1012). Supported types: amorphic, arcgis, and jdbc with IngestionType metadataonly (other types raise a generic Exception → 500). Pagination: offset is 1-based (converted to offset-1 internally; omitted starts at 0); limit defaults to 100, max 1000 (else GE-1028). Last page when next_available is 'no'.

📄️ Trigger or resume a datasource sync job

Starts a sync job for a datasource. The caller must have editor access. Supported datasource types are amorphic and jdbc with IngestionType set to metadataonly. Only one RUNNING or STARTING job can exist for a datasource at a time. External Amorphic and metadata-only ingestions use the external-assets Step Functions flow; other supported syncs either resume an eligible STOPPED/FAILED Glue job or start a new catalog Glue sync. Each request creates a new JobId.

📄️ Get a datasource sync job

Returns one sync job run. Caller needs at least read/view (else AUTH-1012). Supported types: amorphic, or jdbc with IngestionType metadataonly (arcgis and others → 500). ResourcesToSync is normalized to 'All' or a comma-separated string. DatasourceId and JobReportLocation are omitted from the response. LogsDownloadURL is set when LogStatus is AVAILABLE.

📄️ Stop a running datasource sync job

Stops an in-flight catalog Glue sync job. Caller needs editor access (else AUTH-1012). Supported types: amorphic, or jdbc with IngestionType metadataonly. Job must be RUNNING or STARTING in DynamoDB (else IPV-1078). If Glue is still RUNNING, stop is requested and JobStatus becomes STOPPING; if Glue already terminal, DynamoDB is synced to that status. Intended for catalog Glue runs (GlueJobRunId 'n/a' on the external path can fail with GE-1034). Not idempotent for an already-stopped job.

📄️ View, generate, or download dataflow logs

Retrieves CloudWatch logs for a dataflow. The caller (identified by role_id) needs datasources.view and at least read-only access on the datasource (else AUTH-1010). Supports JDBC Bulkload (servicename=jdbc-dataflow), JDBC Advanced Load (jdbc-advanced-dataflow), Streams (streams-dataflow), and ArcGIS (arcgis-dataflow). Not supported for SaaS. JDBC uses async view/generate/download; Streams and ArcGIS return a sync download URL.

📄️ View, generate, or download JDBC execution logs

Retrieves CloudWatch logs for one JDBC dataflow execution. The caller (identified by role_id) needs datasources.view and at least read-only access on the datasource (else AUTH-1010). Supported only for full-load JDBC Bulkload (servicename=jdbc-dataflow) and JDBC Advanced Load (jdbc-advanced-dataflow). Not supported for cdc / full-load-and-cdc (GE-1034), SaaS, Streams, or ArcGIS. Time window comes from the execution StartTime/EndTime; do not send startTime/endTime in the body.

📄️ View, generate, or download Streams entity logs

Retrieves CloudWatch logs for a Streams data transformation entity. The caller (identified by role_id) needs datasources.view and at least read-only access on the path datasource (else AUTH-1010). Use servicename=streams-entity. Actions: view-logs, generate-logs, download-logs. Body startTime and endTime are required (YYYY-MM-DDTHH:MM:SS UTC).

📄️ List dataflow executions or table statistics

Returns execution history for a dataflow, or JDBC table statistics when request_type is set. The caller (identified by role_id) must have at least read/view permission on the datasource (else AUTH-1012). Supports JDBC Bulkload, JDBC Advanced Load, SaaS, and ArcGIS. Not supported for Streams (no handler branch). JDBC list history is full-load only; CDC Bulkload can still use request_type stats modes.

📄️ Get a specific ArcGIS dataflow execution

Returns one ArcGIS dataflow run by ExecutionId. The caller (identified by role_id) must have at least read/view permission on the datasource (else AUTH-1012). ArcGIS only; other datasource types have no handler. When MetadataIngestionJobRunId or DataIngestionJobRunId is present, short-lived S3 presigned URLs for output/error logs are added if those files exist.