6 posts tagged with "v3.0"

Serverless Dataflows Occasionally Stuck in Deleting State​

Affected Versions: 2.7 3.0 3.0.3 3.0.6

Fix Version: 3.1

JDBC Bulkdataload Dataflows Issues​

Affected Versions: 2.7 3.0 3.0.3 3.0.6

Fix Version: 3.1

This report covers two related issues affecting JDBC bulkdataload dataflows:

• Individual source table reload not working for full load dataflows
• Endpoint testing failing immediately without retry on timeout.

Dataflow auto-retry mechanism is not working as expected

Affected Versions: 3.0 3.0.3 3.0.6

Fix Version: 3.1

Files with identical names from different folders may not upload correctly, causing some of your data to be lost during S3 ingestion.

Affected Versions: 2.7 3.0

Fix Version: 3.1

Root cause(s)​

When you upload files with the same name from different folders in your S3 bucket, Amorphic doesn't always process all of them correctly.

For example, if your S3 bucket has these files:

• reports/2025-01-01/daily_sales.csv
• reports/2025-01-02/daily_sales.csv
• reports/2025-01-03/daily_sales.csv

Amorphic sees them all as just daily_sales.csv and treats them as the same file. This means only the last file gets processed successfully - the others get overwritten and lost.

This happens because the system focuses on the filename only and ignores which folder each file came from. When multiple files are processed at the same time, they conflict with each other and some data gets lost.

Impact​

• Your data goes missing: Files with the same name get overwritten, and you lose data without any warning
• Unpredictable results: Which files get processed depends on timing, so results vary each time you run ingestion
• Misleading error messages: System provides file not found error message for the ingestion exception.
• Affects all S3 connections: This happens whether you use access keys or bucket policies to connect to S3
• Historical data loss: You might lose important historical records stored in date-organized folders

Mitigation​

What you can do right now:

1. Make filenames unique: Add dates, timestamps, or folder names to your filenames (like daily_sales_2025-01-01.csv)
2. Check your data: Review your ingestion results to make sure all expected data is present
3. Process folders one at a time: If possible, upload files from one folder at a time instead of all at once
4. Monitor carefully: Keep a close eye on your data uploads until the fix is available

What's available in version 3.1:

• All files will be processed correctly, even if they have the same name
• Better tracking to ensure no data is lost

Timeline​

• 2025-08-05: Users reported missing data after S3 ingestion
• 2025-08-07: Problem identified and analyzed - files with same names overwriting each other
• 2025-08-14: Solution developed and completed to handle identical filenames properly
• 2025-08-15: Solution available in version 3.1

Parquet file uploads fail when you change column data types in Redshift datasets.

Affected Versions: 2.7 3.0 3.0.3 3.0.6

Fix Version: 3.1

Datalab post custom R image changes fail due to corrupted backend image corresponding to the required R configurations

Affected Versions: 3.0 3.0.3 3.0.6

Fix Version: 3.1