Skip to main content
print this page

Shared libraries updation making the associated ETL Spark jobs to fail

· 2 min read
Fix Available
Fix Available
This bug has been fixed

After updating the shared libraries, the associated ETL spark job exeuctions are getting failed

Affected Versions: 2.2 2.1 2.0 1.14 1.13

Fix Version: 2.2* 2.1* 2.0* 1.14* 1.13*

Root cause(s)​

When updating the shared libraries, the associated ETL spark job attributes are resetting to defaults with version 0.9 in AWS console, which leads to failure of ETL job executions.

List of ETL job attributes resetting are:

  • Glue Version
  • Max Retries
  • Max Concurrent Runs
  • Timeout
  • Notify Delay After
  • Number Of Workers
  • Worker Type

Impact​

If user updates any shared library which is attached to ETL spark job, the job execution will fail.

Mitigation​

Fix available

A fix is available in Amorphic version 2.2. Please upgrade to the latest version to resolve this issue.

Timeline​

gantt
    title Timeline
    dateFormat YYYY-MM-DD
    tickInterval 1day
    axisFormat %b-%d
    todayMarker off
    section Tracker
        %% update the ticket number and date of bug report
        CLOUD-3539 : done, 2023-05-29, 0d
    section Identification
        Reported : crit, des1, 2023-05-29, 1d
    section Mitigation
        %% Update number of days took for each step below
        Bug fixed: milestone, 2023-05-29, 1d
    section Delivery
        %% update the date of each step below
        testing complete: milestone, 2023-05-29, 1d
  • 2023-06-16: Bug reported/identified (CLOUD-3539)
  • 2023-06-16: Bug triaged
  • 2023-06-16: Bug fixed
  • 2023-06-17: Testing completed
  • 2023-06-19: Fix is available