FAQs
Frequently asked questions (FAQs)
Answers to the questions raised most often about MDM workflows: what a run costs, the service limits that apply to AWS Entity Resolution, how survivorship rules behave, and how to size the Glue compute for a large workload.
Cost
An MDM workflow run bills against two AWS services. AWS Entity Resolution is charged per record processed and covers the matching stage, which runs on AWS managed capacity. AWS Glue is charged for the post-processing job that applies the survivorship rules and writes the unified master records to the output dataset.
Policy Engine flows also run on AWS Glue, so the same Glue pricing applies to them. A Policy Engine flow evaluates unified master records that an Entity Resolution workflow has already produced, so it incurs no Entity Resolution charge: its cost is the Glue cost of the evaluation job alone.
All prices on this page are list prices for the US East (N. Virginia) (us-east-1) Region, in USD, and exclude taxes. Prices differ by Region and change over time. Confirm the current figures on the AWS Entity Resolution pricing and AWS Glue pricing pages before committing to a budget. Amorphic does not add a markup; these charges appear directly on the AWS bill for the account the platform is deployed into.
What does AWS Entity Resolution cost per run?
AWS Entity Resolution is priced at $0.25 per 1,000 source records processed in a matching workflow. The charge applies to the records submitted to the workflow, not to the number of unified master records it produces, and it applies to every run, so a workflow on a monthly cadence incurs the charge each month.
| Records processed | Approximate cost per run |
|---|---|
| 100,000 | $25 |
| 1 million | $250 |
| 20 million | $5,000 |
| 100 million | $25,000 |
| 600 million | $150,000 |
The same rate applies to rule-based and ML-based matching.
At $0.25 per 1,000 records, cost scales linearly and without warning: a single 100 million record run costs roughly $25,000, and a misconfigured matching rule or survivorship strategy costs exactly the same as a correct one. Validate every workflow against a small sample first (for example 50,000 to 100,000 records, roughly $12 to $25), review the results in the Unified Record Hub, and only then point the workflow at the full dataset.
How is the AWS Glue cost calculated?
This applies to both MDM workflows and Policy Engine flows: both use an AWS Glue job and are billed the same way. For a workflow, the Glue charge covers the survivorship and output stage only. The matching stage runs on AWS managed capacity and is covered by the Entity Resolution per-record charge instead.
AWS Glue bills on DPU-hours: the number of Data Processing Units allocated to the job multiplied by how long the job runs. In us-east-1, a Glue ETL job costs approximately $0.44 per DPU-hour, billed per second.
The DPU count is the number of workers multiplied by the DPUs per worker:
Glue cost = workers × DPUs per worker × $0.44 × job duration in hours
| Worker type | DPUs per worker |
|---|---|
G.1X, R.1X | 1 |
G.2X, R.2X | 2 |
G.4X, R.4X | 4 |
G.8X, R.8X | 8 |
G.12X | 12 |
G.16X | 16 |
Applying that formula gives the following approximate hourly costs:
| Worker configuration | Total DPUs | Approximate cost per hour |
|---|---|---|
10 × G.1X | 10 | $4.40 |
20 × G.1X | 20 | $8.80 |
10 × G.2X | 20 | $8.80 |
20 × G.2X | 40 | $17.60 |
10 × G.4X | 40 | $17.60 |
10 × R.2X | 20 | $8.80 |
Raising the worker count or moving to a larger worker type increases the hourly rate but usually shortens the run, so the total Glue cost for a job often stays comparable. The larger benefit is operational: more memory per worker avoids the out-of-memory and timeout failures that large workloads hit on smaller configurations, and a failed run still bills for the time it consumed before failing.
R.x worker types are memory-optimised and are priced differently from the G.x types in some Regions. Check the AWS Glue pricing page for the rate that applies to the deployment Region.
How do I estimate the total cost of a workflow run?
Add the Entity Resolution charge to the Glue charge. Two worked examples, both in us-east-1:
A proof of concept on 100,000 records, run on 10 × G.1X workers, completing in 30 minutes:
| Component | Calculation | Cost |
|---|---|---|
| AWS Entity Resolution | 100 × $0.25 | $25.00 |
| AWS Glue | 10 DPU × $0.44 × 0.5 h | $2.20 |
| Total | ≈ $27.20 |
A production run on 20 million records, run on 20 × G.2X workers, completing in 3 hours:
| Component | Calculation | Cost |
|---|---|---|
| AWS Entity Resolution | 20,000 × $0.25 | $5,000.00 |
| AWS Glue | 40 DPU × $0.44 × 3 h | $52.80 |
| Total | ≈ $5,052.80 |
Entity Resolution dominates the bill at scale; Glue is a rounding error by comparison. Optimize for record volume first: deduplicate and filter the input datasets, and use Incremental Processing as the processing cadence so subsequent runs only process new and updated records instead of the full dataset.
These figures cover the matching workflow itself. A Policy Engine flow run against the resulting unified master records is billed separately, on the same Glue rates. Storage of the input and output datasets in Amorphic, and any downstream querying of the unified master dataset, are billed separately by their respective AWS services.
What does a Policy Engine flow cost?
A Policy Engine flow is executed by an AWS Glue ETL job, so its cost is the Glue cost of that job and nothing more. The rates and the DPU-hour formula are identical to the ones in the Glue question above.
There is no AWS Entity Resolution charge for a Policy Engine flow. The flow evaluates unified master records that an Entity Resolution workflow has already produced, so no records are submitted to a matching workflow and the $0.25 per 1,000 records rate does not apply.
For example, a flow that evaluates 20 million unified master records on 10 × G.2X workers in 1 hour costs approximately $8.80 in total (20 DPU × $0.44 × 1 h).
The capacity for the rule evaluation job is configurable on the rule set and can be overridden for a single evaluation. See Compute Capacity below.
Because the Entity Resolution charge dominates a matching workflow but does not apply here, Policy Engine flows are inexpensive by comparison. Iterating on rule sets in the Policy Engine is a much cheaper way to refine data quality than re-running the matching workflow.
How can the cost of an MDM workflow be reduced?
- Size the input. The Entity Resolution charge is driven entirely by record count. Restrict the input datasets to the records and sources that genuinely need mastering.
- Use Incremental Processing. A cadence of Incremental Processing limits each run to new or updated records rather than reprocessing the whole dataset, which is the single largest saving on a recurring workflow.
- Prove the configuration on a sample. Matching rules, comparison types, and survivorship strategies are far cheaper to get wrong on 50,000 records than on 50 million.
- Avoid failed full-scale runs. A run that fails on memory or timeout after two hours still bills for both the Glue time and the records already processed. Right-size the worker configuration before the production run; see Compute Capacity.
- Reapply survivorship rules instead of re-running the workflow. A survivorship rule run reuses the last full run's matching output, so it is billed for Glue time only, with no Entity Resolution charge. See the survivorship section below.
- Request quota increases early. Hitting a service quota mid-run wastes the work already done.
AWS Entity Resolution limits
What are the service limits for a matching workflow?
| Limit | Value |
|---|---|
| Input datasets (schema mappings) per matching workflow | 20 |
| Records per rule-based matching workflow | 100 million |
| Records per ML-based matching workflow | 600 million in most Regions; 150 million in a few Regions |
Most of these are soft quotas and can be raised by requesting a quota increase from AWS where the quota is adjustable. See AWS Entity Resolution quotas for the authoritative list and the current per-Region values.
The ML-based record limit is Region-dependent. Confirm the limit for the Region the deployment runs in before planning a workflow above 150 million records. The workflow fails at submission time if the input exceeds the quota, and the Glue time spent preparing the input is still billed.
Amorphic applies its own limits on matching rules within a workflow: up to 15 rules for the Simple rule type and 25 rules for the Advanced rule type. See Workflows for details.
Survivorship rules
How are survivorship rules applied?
Survivorship rules are configured per attribute. Select the strategy that should decide the surviving value for each attribute from the available options: Source Priority, Frequency, Most Recent/Recency, Aggregation, Minimum Value, or Maximum Value.
The Default Governance Strategy set on the workflow prefills every attribute-level rule; override it on the individual attributes that need different handling. There is no automatic per-attribute inference; an attribute keeps whichever strategy is assigned to it.
For the full strategy reference and configuration steps, see Workflows.
Changing a survivorship strategy does not retroactively alter the unified master records already written. To apply the new strategy to records from the last run, trigger a survivorship rule run, described below.
Can survivorship rules be reapplied without re-running the whole workflow?
Yes. A survivorship rule run reapplies the current survivorship configuration to the matching output of the workflow's most recent successful full run, without submitting the records to AWS Entity Resolution again. This is the way to change a strategy and see its effect on existing unified master records.
The behaviour and its constraints:
- The run reuses the Entity Resolution job output from the latest successful full run of that workflow. Input records added since that run are not included; a full run is required to pick those up.
- At least one successful full workflow run must have completed first. Without one there is no matching output to reapply rules to, and the request is rejected.
- The workflow must be in AVAILABLE status.
- Only one survivorship rule run per workflow can be in progress at a time.
- These runs are recorded as separate executions in the workflow's run history, flagged as survivorship runs, and are never themselves used as the source for a later survivorship rule run.
A survivorship rule run skips AWS Entity Resolution entirely, so it incurs no $0.25 per 1,000 records charge. Its only cost is the AWS Glue time for the job. Tuning survivorship strategies against 100 million already-matched records therefore costs a few dollars in Glue rather than another $25,000 in Entity Resolution. Get the matching right in a full run, then iterate on survivorship with these runs.
Compute Capacity
Can the worker type, worker count, and job timeout be changed?
Yes. Compute Capacity is an optional panel available on both ER workflows and Policy Engine rule sets. On a workflow it sets the capacity for the survivorship job; on a policy engine it sets the capacity for the rule evaluation job. It has three settings:
| Setting | Accepted values | Default |
|---|---|---|
| Worker Type | G.1X, G.2X, G.4X, G.8X, G.12X, G.16X, R.1X, R.2X, R.4X, R.8X. The dropdown lists each type with its DPU count, for example Standard (G.1X - 1DPU). | G.1X |
| No. of Workers | A whole number from 2 to 100 | 10 |
| Timeout (minutes) | A whole number from 1 to 2880 (48 hours) | 300 |
Compute Capacity applies to the survivorship and output stage. Entity resolution matching runs on AWS managed capacity and is unaffected by these settings, so raising the worker count does not make the matching step faster. It shortens the survivorship and output stage, and prevents that stage failing on memory or timeout.
Values set on the workflow or rule set become its stored capacity, used by every run. They can be changed at any time by editing the resource.
The run dialog, Run Workflow? or Run Evaluation?, shows the capacity the run will use, summarised as Machine Size, Machines, and Stops After. Select Edit to change Worker Type, No. of Workers, or Timeout (minutes), then Save. Each setting is resolved independently: a value supplied for the run takes precedence, then the stored value, then the default above. Changing No. of Workers alone therefore keeps the stored Worker Type.
The capacity each run actually used, including any override, is recorded against the execution and shown in the run history.
Choosing a worker type. The G series is general-purpose compute. The R series provides the same DPU count per worker with double the memory, so it is the right choice when a job fails on executor memory rather than on CPU. A larger worker type within a series raises both the DPU count and the memory per worker.
G.12X, G.16X, and the R series are not available in every Region. Glue rejects an unsupported worker type when the job is created or updated, so an unavailable combination fails at configuration time rather than part-way through a run. G.025X is not accepted at all, since it is streaming-only.
For where this appears, see Workflows and Policy Engine.
Does editing Compute Capacity in the run dialog change it for future runs?
No. An edit made in the Run Workflow? or Run Evaluation? dialog applies to that run alone, as the This run only label in the dialog indicates. The stored capacity on the workflow or rule set is left unchanged, and the next run reverts to it.
This is the intended way to give a single run a larger machine size or more machines, for example a one-off backfill or an unusually large evaluation, without permanently raising the cost of routine runs. To change the capacity for every run instead, edit the workflow or rule set itself.
For an ER workflow, the per-run capacity applies to both run options, Run Workflow and Run Survivorship Rules.
A large workflow fails on memory or times out. What should be changed?
If the failure is in the survivorship and output stage, adjust Compute Capacity rather than reducing the input:
- Out-of-memory failures. Move to a larger worker type, or to the equivalent
Rseries type for double the memory per worker at the same DPU count. For example, move fromG.2XtoG.4X, or fromG.2XtoR.2X. - Timeouts. Raise Timeout (minutes) towards its 2880 ceiling, and raise No. of Workers so the job completes in less wall-clock time. Both together are usually needed for the largest workloads.
- Slow but successful runs. Raise No. of Workers. This increases the hourly rate but shortens the run, so the total Glue cost often stays comparable while the run finishes sooner.
A run that fails on memory or timeout still bills for the Glue time it consumed and for every record already submitted to AWS Entity Resolution. Sizing the capacity correctly before a large run is cheaper than a failed attempt.
If the matching stage itself is the problem, Compute Capacity will not help. Check the input against the AWS Entity Resolution limits above instead.