Glue File Processing

The system supports an additional API parameter GlueFileProcessing that can be used in conjunction with DataLoadThrottling to enable faster processing using Glue jobs.

This is an API-only feature that can be configured with the following parameters:

  Path: /datasets/dataload-throttling
  Method: PUT
  {
    "GlueFileProcessing":"disable" or "enable”
    "DataLoadThrottling":"disable" or "enable",
  }

Note:

• This is ONLY applicable for append and update type datasets.
• When GlueFileProcessing is enabled, data validation steps will be skipped for datasets targeting LakeFormation and S3 Athena.

For dataload limits and throttling behavior in the UI, see Dataload Limits.