Skip to main content
Version: v3.4 print this page

Dataload Limits

The way data load limits work is mostly determined by the Data load throttling setting.

  • If enabled: dataset files are processed one by one, with completion time depending on the number of files.
  • If disabled: dataset files are processed and completed immediately.

Dataload Limits

Important Information: No action is required to enable/disable data load throttling. The system automatically adjusts based on the number of files being processed. Users can manually enable/disable if needed.

If throttling is enabled manually by any user, it will not turn off automatically.

Throttling Automatic Process

  • If processing files are within the throttle limit: system disables throttling.
  • If processing files exceed the limit: system enables throttling and processes files in a queue.

Setting Dataload Limits

Dataload limits let users set batch limits for processing uploaded files. Different limits apply to each target location (S3, S3Athena, Lakeformation, Dynamodb, AuroraMySQL).

Example: If users upload 1000 files to an S3 dataset with a 300 file limit, up to 300 files will process in parallel, with remaining files queued. The system polls the queue every 3 minutes to trigger processing according to the specified limits.

The ranges for each target location are calculated based on AWS limits and performance tests.

Users can view counts of recent dataload executions and messages waiting in SQS queues on the Infrastructure page.

Updating Limits

Users can update data load limits for all applicable target locations using "Set Limits":

  1. Enter new values in the respective target location fields
  2. Click "Update Limits" to apply changes
info
  • If updated limits don't appear immediately after successful update, refresh after a few seconds.
  • Files uploaded directly from the Amorphic UI dataset details page are processed immediately, even if throttling is enabled.
  • Files uploaded through jobs, ingestion, etc., still follow normal throttling procedures.
  • For use cases with large numbers of files, the best data load limit for Redshift is 90-100.

For Glue-based file processing used with dataload throttling, see Glue File Processing.