Empirical Privacy Risk Assessment in Tabular Synthetic Data

The Article 29 Working Party, predecessor of the European Data Protection Board, identified three types of privacy risk as the core indicators of whether data has been effectively anonymized: singling out, linkability, and inference. These categories were established in the Opinion 05/2014 on Anonymization Techniques¹ and remain the standard reference for assessing factual anonymization under EU data protection law. For synthetic data, while most attention focused on fit-for-use evaluations, the question of which risk types matter and how to measure them empirically had not been answered in a unified, practically usable way before Anonymeter.

In 2023, we published Anonymeter in the Proceedings on Privacy Enhancing Technologies symposium (PoPETs), a leading venue for privacy research.² The paper presents a unified framework for quantifying all three privacy risk indicators for synthetic tabular data. The library is open source,³ and the approach has been evaluated positively by France's data protection authority (CNIL).⁴

Three risk indicators, one framework

Instead of relying on structural properties of the synthesis model, Anonymeter looks at the output of the model: the generated synthetic dataset. It evaluates all three risk types through a single, unified methodology.

For each risk type, the framework models an attacker who possesses the full synthetic dataset and optionally some auxiliary information about individuals in the original dataset. The attacker attempts to make inferences about records in the original data:

• Singling out: there is exactly one person in the dataset with attributes X, Y, and Z.
• Linkability: records A and B belong to the same person, with A and B representing subsets of the dataset's attributes.
• Inference: a person with attributes X and Y also has attribute Z.

Each risk evaluation proceeds in three phases. In the attack phase, the attacker makes a large number of guesses about target records. In the evaluation phase, the success rates of those guesses are measured against the ground truth. In the risk quantification phase, a risk value and confidence interval are derived. The result is a set of empirical risk measurements that can be compared across algorithms, datasets, and configurations.

Separating population-level utility from individual-level leakage

The key methodological contribution is how the framework distinguishes between general population-level information and private information about specific individuals. Synthetic data must carry population-level information to be useful. The question is whether it also carries information specific to particular individuals in the source dataset, which constitutes a privacy risk.

In order to achieve this, three attacks are carried out:

• A main attack uses the synthetic data to make guesses about records in the original dataset.
• A control attack uses the same attacker against records drawn from the same distribution as the original data, but not used in synthesis.
• A naive attack guesses at random, providing a lower bound.

The success rate of the control attack measures what a well-informed attacker can achieve using only population-level information, since control records were not part of synthesis and no individual-specific information about them is present in the synthetic data. If the main attack does no better than the control attack, the information in the synthetic data is explainable by population statistics alone. If the main attack succeeds at a higher rate, private information about original records was leaked during synthesis. The privacy risk reported by Anonymeter is this excess success rate, normalized by the maximum possible improvement over the control baseline.

Key findings

The paper evaluated the framework across a range of synthetic datasets generated with and without differential privacy, varying the amount of auxiliary information available to the attacker.

Synthetic data consistently shows the lowest vulnerability to linkability among the three risk types. This is structurally expected: in most generative synthesis processes, original records are replaced with generated samples, conditioned on the whole dataset rather than single datapoints. This breaks the one-to-one mapping between original records and synthetic ones. A linkage attack that relies on matching synthetic records to original ones is therefore weakened at the source.

Inference and singling out are the dominant residual risks. These risk types do not require record-to-record matching and persist even when the generation process is designed to avoid direct reproduction of original records. They are harder to eliminate through synthesis alone and require careful configuration of the synthesis parameters or supplementary privacy-enhancing techniques following data minimization principles.

Privacy risks scale with the amount of information leaked by the synthesis process. For differentially private algorithms, the empirical risks measured by Anonymeter are consistent with the theoretical privacy guarantee when the budget is tight enough, providing a practical bridge between theoretical bounds and observed behavior.

CNIL evaluation

The framework was evaluated positively by the technical experts at the Commission Nationale de l'Informatique et des Libertés (CNIL), France's national data protection authority.⁴ For organizations deploying synthetic data in the EU, CNIL's position is a significant practical signal: it indicates that empirical risk evaluation using Anonymeter is a meaningful basis for GDPR compliance assessment. It also means that organizations seeking to demonstrate the privacy properties of synthetic data to European regulators have a defensible evaluation methodology available to them.

Anonymeter is designed to be algorithm-agnostic: it evaluates the output of any synthesis process, regardless of the model used to generate it. It accepts any synthetic tabular dataset as input and returns empirical risk estimates with confidence intervals. A large-scale application of the framework to over 150 real-world datasets from the NIST Collaborative Research Cycle is described in NIST's Privacy Data Management Conference 2023⁵.