Skip to main content
 print this page

Datasets

APIs related to datasets, which are logical collections of files(data).

📄️ List ML models applicable to a dataset file

Returns ML models the caller can access that are compatible with the given dataset file. Caller must be owner or editor of the dataset (else AUTH-1002). For DWH targets with a supported file type, models are filtered by InputSchema match against DatasetSchema; otherwise models whose SupportedFileFormats include the dataset FileType are returned. Optional projectionExpression limits returned model fields (comma-separated attribute names). Safe to retry (read-only).

📄️ Trigger AI summarization for dataset files

Starts asynchronous AI summarization for one or more successfully loaded text/pdf files in an s3-target dataset. Caller must have editor or owner access. Datasets AI service must be enabled (else AI-1004). Sets SummaryGenerationStatus to summarizing and starts the AI operations step function. Success Message is 'Successfully triggered AI operation: 'summarize' on the dataset.' Not idempotent while files are already summarizing/summarized (IPV-1020).

📄️ Run an ML model on a dataset file

Invokes an ML model run for a selected file (or file set) on the dataset. Caller must have access to the model and owner/editor on the dataset (else AUTH-1002). For OutputType DatasetData, TargetDatasetId is required and must be an active dataset the caller can write to. Body FileName may be a single path or list depending on DatasetProcessingMode. On success starts the invoke-model step function and returns Message 'success' with ModelRunId and FileName. Not idempotent: each call creates a new ModelRunId.

📄️ Sync partitions from source for lf and s3athena datasets

Triggers an asynchronous partition sync from the dataset S3 source via Step Functions. Supported only for TargetLocation lf or s3athena. Not supported for views, Iceberg tables, or Delta Lake tables (GE-1106: 'sync partitions is/are not supported for {views| <target> datasets|deltalake/iceberg datasets}'). External datasets without partition keys return DS-1054. Internal datasets default the partition key to upload_date when none are defined. On success an email notification with more details is sent when the sync completes. No request body.

📄️ Batch update dataset metadata

Triggers asynchronous bulk update of dataset metadata. The caller must be owner or editor of each dataset. Processing happens via a Step Functions state machine in batches of 50 datasets. On completion, an email notification with a PDF report is sent. Use the operation query parameter to target lifecycle policy updates or deletes instead of general metadata updates. There is no dataset count limit for general metadata updates. Lifecycle operations (update_lifecycle_policy, delete_lifecycle_policy) are limited to 100 datasets per request.

📄️ Get user Lake Formation data permissions on a dataset

Returns fine-grained Lake Formation column/row permissions for the path userid on the dataset. LF (TargetLocation=lf) datasets only (else IPV-1058). Response is UserDataPermissions with TablePermissions[] and DataFilterPermissions[]. Each permission entry includes Columns, ColumnCondition (Include|Exclude), and for data filters optional Rows and Name. Dataset must exist, be active, and registration completed; path userid must have access to the dataset. Query params limit/offset/sortby/sortorder/projectionExpression are accepted by API Gateway but not applied by the handler (full permission set returned). Safe to retry (read-only).