← Back to writing

· 13 min read

Fairness and Bias in Synthetic Data: Evidence Before Assurance

A practical framework for examining representation, label quality, group outcomes, uncertainty, and privacy before claiming that synthetic data makes a system fairer.

By Shamiul Shifat
Lead AI Platform Engineer · Betterdata AI

In this article

A synthetic dataset can contain equal numbers of records from two groups and still train a model that misses important cases in one group more often. It can reproduce historical labels faithfully even when those labels encode an unfair process. It can improve an aggregate metric while reducing usefulness for a small population that was already poorly represented.

The number of generated rows does not resolve these problems. Fairness depends on the intended decision, the people affected, the data-generating process, and the consequences of error.

As a Lead AI Platform Engineer at Betterdata AI, I approach synthetic data through both platform engineering and privacy responsibilities. My recommendation is to make fairness claims conditional on evidence: what was evaluated, for whom, against which objective, and with what uncertainty. The examples below are illustrative. They do not describe Betterdata’s systems, customer data, or measured outcomes.

Start with the decision and the affected population

“Is this dataset fair?” is incomplete. A dataset can support one use adequately and be inappropriate for another. Begin with the intended decision, target population, deployment setting, and likely harms. Identify who receives the benefits, who bears the errors, and who has authority to challenge an outcome.

NIST SP 1270 discusses bias in AI as more than a statistical property of a dataset, including systemic and human contributions. That framing is useful because a technically balanced sample can still participate in a biased institutional process. 1

For a screening model, a false negative may withhold a beneficial intervention. In another workflow, a false positive may impose an unnecessary investigation. These consequences should shape the evaluation rather than being inferred from whichever metric is easiest to calculate.

Write a short evaluation charter: intended use, populations in scope, relevant attributes, decision threshold or ranking rule, outcome definition, material harms, chosen metrics, uncertainty requirements, and decision owners. Include uses that the evidence does not support. NIST’s AI Risk Management Framework provides a useful foundation for connecting this context to measurement and governance. 2

Affected stakeholders and domain experts should help define the problem. Statistical analysis can reveal differences; it cannot independently determine which differences an organization should accept. Legal requirements also depend on jurisdiction and context, so an engineering assessment should not claim universal legal approval.

Trace bias through the entire data pathway

Synthetic generation learns from a source and from the objectives and constraints imposed on it. It can preserve source patterns, smooth them, amplify them, or introduce new distortions. It should not be assumed to repair bias merely because the resulting records are artificial.

I would trace at least five pathways. Selection determines who appears in the source. Measurement determines what is recorded and how consistently. Labels express an observed outcome or proxy. Generation changes the representation of those patterns. Deployment changes how the data influences decisions and future observations.

StageExample questionWhy a favorable aggregate can mislead
Source selectionWhich eligible people never entered the dataset?High performance on observed users says little about excluded users
MeasurementIs a feature missing or noisier for particular groups?A complete generated column can conceal uneven source quality
LabelsDoes the target measure the actual need or a historical proxy?Predicting the proxy accurately can reproduce the wrong objective
GenerationAre rare combinations and relationships preserved appropriately?Similar overall distributions can hide intersectional distortion
Deployment feedbackDo model decisions affect which outcomes are later observed?Future labels can reflect the previous model’s own choices

Obermeyer and colleagues’ study of a health-care algorithm is a candidate primary example of how using cost as a proxy for health need can produce consequential bias. The general lesson is to interrogate the target itself, not to assume that a more accurate predictor of the existing label is a fairer system. 3

Document missingness before generation. Replacing missing fields with plausible synthetic values can improve surface completeness while hiding uncertainty or differences in how records were collected. Where appropriate, preserve meaningful missingness indicators and evaluate the consequences of imputation separately.

Separate representation from downstream outcomes

Representation asks who and what appears in the dataset. Outcome analysis asks how a system behaves for the relevant people when that data is used. Both matter, but they are not interchangeable.

Adding synthetic examples for an underrepresented group can help an engineering experiment. It does not create new independent observations about that group’s real-world experience. If the source contains few examples, generating many more rows can reproduce the same limited information with greater apparent confidence.

Changing group proportions also changes the training distribution. That may be an intentional intervention, but its effect depends on model training, labels, feature relationships, weighting, and deployment prevalence. Evaluate the downstream system against a representative, governed real-world holdout where possible. A synthetic test set generated from the same process as training can share its blind spots.

Field guide From source coverage to a defensible fairness claim An illustrative evaluation path. More synthetic records do not create independent evidence about populations that were poorly observed in the source.
  1. Define the decision

    Identify the target population, material harms, outcome label, and decision owner.

  2. Inspect the source

    Examine selection, missingness, labels, and relevant group and intersection coverage.

  3. Evaluate generation

    Measure changed distributions and relationships, including rare but consequential cases.

  4. Test downstream outcomes

    Assess group performance on independent evidence, report uncertainty, and review the actual decision policy.

I recommend maintaining separate reports for source quality, generated-data quality, and downstream performance. That separation makes it possible to identify whether an intervention improved coverage, changed an error rate, or merely made a report look more balanced.

Define the fairness metrics before comparing them

For a binary decision, let Y be the observed outcome label, Ŷ the model’s binary decision, S a predicted probability or risk score, and A the group attribute used in the assessment. The labels themselves must be examined for validity; these definitions do not make them objective ground truth.

Demographic parity compares positive decision rates across groups: P(Ŷ = 1 | A = a). It does not condition on the observed label. This can be relevant when examining allocation of opportunities, but it does not reveal who was correctly or incorrectly selected.

Equalized odds asks for matching positive decision rates conditional on the true label across groups. For a binary classifier, this means matching both the true-positive rate and the false-positive rate. Equal opportunity, as described by Hardt, Price, and Srebro, focuses on the true-positive-rate condition for the relevant favorable outcome. 4

Calibration within groups asks whether a given predicted probability corresponds to the observed outcome frequency within each group. If records assigned a score near 0.7 have an outcome rate near 70% in each sufficiently supported group, that is evidence relevant to calibration. Calibration is a property of scores and observations; equal precision at one decision threshold does not demonstrate it.

These criteria answer different questions. Research by Kleinberg, Mullainathan, and Raghavan examines inherent tradeoffs among fairness conditions in risk assessment. With differing base rates and imperfect prediction, reasonable requirements can conflict; the precise conflict depends on the definitions and assumptions. 5

I would not choose a metric because it produces the smallest disparity. Choose it because it corresponds to the harm and decision being assessed, then report other relevant measures so the tradeoffs remain visible. Changing a threshold may improve one group’s recall while increasing false positives and changing precision. It does not, by itself, recalibrate unchanged probability scores. That is a decision requiring analysis, not a dashboard defect to hide.

Work through a small, explicitly hypothetical example

Assume an illustrative screening task in which a positive decision means someone is selected for a beneficial follow-up. The following counts are invented solely to explain the metrics. They are not observations from Betterdata, a customer, or a benchmark.

GroupTrue positiveFalse positiveTrue negativeFalse negativeTotal
A828220
B4111420

Group A has 10 positive outcomes and 10 negative outcomes. Group B has 8 positive outcomes and 12 negative outcomes. The formulas and results are:

MeasureDefinitionGroup AGroup B
Observed positive-label rate(TP + FN) / total10/20 = 50%8/20 = 40%
Positive decision rate(TP + FP) / total10/20 = 50%5/20 = 25%
True-positive rate / recallTP / (TP + FN)8/10 = 80%4/8 = 50%
False-positive rateFP / (FP + TN)2/10 = 20%1/12 ≈ 8.3%
Precision / positive predictive valueTP / (TP + FP)8/10 = 80%4/5 = 80%
Accuracy(TP + TN) / total16/20 = 80%15/20 = 75%

Equal precision hides a substantial observed recall difference. Among those with a positive label, Group B receives the beneficial follow-up less often in this sample. The positive decision rates differ too, and the false-positive rates move in the opposite direction. We cannot infer score calibration because the table contains no probability scores.

The counts are very small. For illustration, approximate 95% Wilson intervals for the individual recall rates are 49%–94% for Group A’s 8/10 and 22%–78% for Group B’s 4/8. These wide intervals communicate uncertainty; they are not a formal comparison of the two groups, and interval overlap is not a reliable stand-alone significance test.

The correct response is to investigate and gather appropriate evidence, not to declare the system fair because a test lacks power or unfair solely because this toy table contains a gap. Decide what disparity would be consequential, then design the assessment to estimate it with useful precision. Also investigate whether the positive labels mean the same thing across groups.

Examine intersections and uncertainty without manufacturing certainty

Aggregate groups can conceal patterns at intersections such as age and disability status or language and geography, when those attributes are relevant and legitimately available for the assessment. A system can perform acceptably across each broad category while failing a smaller combination.

Report the numerator and denominator behind each metric. Mark undefined rates explicitly: if a group has no observed positive labels, its recall cannot be estimated from that sample. Do not fill the gap with zero, a population average, or a confident green indicator.

Use uncertainty methods suited to the sampling process. Repeated records from one individual or linked events can violate independence assumptions. Bootstrap procedures may need to resample at the person or cluster level. Many exploratory subgroup comparisons also create opportunities to discover unstable apparent differences; separate exploratory findings from pre-specified release criteria and confirm material findings with additional evidence.

More synthetic evaluation rows do not solve these statistical problems. If they derive from a limited source or shared generator, they are not independent observations of deployment outcomes. A generator’s confidence about an underobserved population is not a replacement for evidence about that population.

When support is inadequate, report the limitation and define a response: collect appropriate data, narrow the intended use, require additional review, or delay a release. “Insufficient evidence” is a useful decision state. It should not be silently converted to “no issue found.”

Investigate interventions as changes to a system

Possible interventions include improving collection, changing an invalid proxy label, correcting measurement practices, reweighting training examples, generating targeted scenarios, changing the model, or modifying the decision process. Each addresses a different mechanism.

I would compare interventions against a baseline using the same independent evaluation design. Track group outcomes, overall utility, false positives, false negatives, and operational consequences. Investigate whether gains persist across time periods and relevant deployment conditions, rather than optimizing only for one evaluation snapshot.

Targeted synthetic generation can be especially useful for testing defined scenarios and examining whether code or workflows behave consistently. Distinguish these scenario tests from estimates of real-world prevalence or performance. A deliberately constructed edge case can reveal a defect without telling us how common it is.

Be cautious about interventions that simply remove an attribute. Other features can encode related information, and removing an attribute can make certain audits harder. Conversely, adding or inferring sensitive attributes without a lawful, governed purpose can create new harm. The appropriate choice requires domain, privacy, legal, and affected-stakeholder input; there is no universal engineering shortcut.

Treat privacy and fairness as connected responsibilities

Group-level evaluation can require sensitive attributes. Establish why they are needed, who may access them, how they are obtained, and how long they are retained. Do not infer protected attributes from names, photographs, or other proxies simply to complete a fairness dashboard. Proxy construction can introduce error and change the legal and ethical character of the analysis.

Rare groups create both statistical and privacy challenges. Publishing a detailed intersectional report can expose small populations or individuals even if the main dataset is synthetic. Separate internal restricted analysis from externally released reports, and assess disclosure risks for tables, examples, and evaluation artifacts as well as generated rows.

Differential privacy can provide a formal privacy framework under specified assumptions, but it does not automatically improve fairness. Noise and other choices can affect groups differently, especially when support is limited. NIST SP 800-226 is the candidate reference for evaluating differential privacy guarantees; utility and fairness still require their own assessment. 6

I recommend evaluating the actual privacy mechanism, parameters, protected unit, and workflow used in production. Do not assume a result from a non-private generator transfers unchanged to its private counterpart. Repeated source-accessing analyses and releases may also require composition accounting. Keep claims precise enough that reviewers can understand both the protection and the remaining limitations.

Publish a record of evidence and keep it current

Datasheets for Datasets and Model Cards for Model Reporting offer primary research references for structured documentation of datasets and models. They help make context, intended uses, and evaluation visible; completing a template does not independently establish fairness. 7 8

For a synthetic-data release, I recommend recording source scope, collection limitations, generation version and configuration, intended uses, evaluation design, group definitions, sample counts, metrics, uncertainty, known failures, privacy assessment, and the owner of the release decision. Document which interventions were tested and which tradeoffs were accepted.

Provide a channel for contesting decisions and reporting harm. Affected users may reveal problems that the available data could not show. Define who triages reports, who can pause a use, and how a correction reaches downstream consumers. A dataset’s version history should connect to the models and decisions that depend on it.

Field guide A fairness review that continues after release An illustrative governance loop. Release approval is tied to an intended use and current evidence; feedback and changed conditions can trigger reassessment.
  1. Document and review

    Record population scope, data limitations, metrics, uncertainty, privacy constraints, and accountable ownership.

  2. Release with conditions

    State permitted uses, unresolved limitations, review dates, and escalation paths.

  3. Observe and hear challenges

    Monitor relevant outcomes and provide meaningful reporting and appeal mechanisms.

  4. Reassess and correct

    Investigate changing populations, labels, and feedback loops; update or suspend affected uses.

Ongoing monitoring should examine population shifts, missingness, label availability, and outcome differences. Be careful with delayed or selectively observed labels: the system’s decisions can determine whose outcomes become visible. Stable measured performance under that feedback loop can hide deterioration elsewhere.

Fairness work belongs alongside Guardrails and Evaluation for Synthetic Data and LLM Platforms and Enterprise AI Control Towers and Governance. My standard for a useful claim is straightforward: a reviewer should be able to identify the people and decision in scope, reproduce the relevant measurements, see the uncertainty and tradeoffs, and find the person responsible for acting on them.

References and further reading

  1. NIST. Towards a Standard for Identifying and Managing Bias in Artificial Intelligence, SP 1270. 2022. A broad framing of bias across technical, human, and systemic factors.
  1. NIST. Artificial Intelligence Risk Management Framework (AI RMF 1.0). 2023. Risk context, measurement, management, and accountability.
  1. Ziad Obermeyer and colleagues. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 2019. A primary study of consequential bias associated with a proxy target.
  1. Moritz Hardt, Eric Price, and Nati Srebro. Equality of Opportunity in Supervised Learning. 2016. Equalized odds and equal-opportunity criteria.
  1. Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent Trade-Offs in the Fair Determination of Risk Scores. 2016. Conditions under which risk-score fairness requirements conflict.
  1. NIST. Guidelines for Evaluating Differential Privacy Guarantees, SP 800-226. 2025. Formal privacy guarantees and evaluation assumptions, distinct from fairness assurance.
  1. Timnit Gebru and colleagues. Datasheets for Datasets. 2018 preprint. Structured documentation of dataset motivation, composition, collection, and uses.
  1. Margaret Mitchell and colleagues. Model Cards for Model Reporting. 2018 preprint. Reporting model context, evaluation, and intended uses.

Department of small adventures

Pip’s pocket passport

Six little souvenirs. One very well-travelled duck.

Kept in this browser, just for you. Visit each section and collect its stamp here—or catch one from Pip.