Towards technical evaluation of AI model anonymity

Introduction

Foundation models are pre-trained on text scraped from various sources, including the open web. Some data filtering is applied, removing duplicates or low quality content, but a careful review of what goes into the models is not part of the process. Personal and sensitive data present in any of those scraped pages likely remains in the training set, and ultimately finds its way into the trained model.

The same applies inside organisations, at a smaller scale. Models are fine-tuned on support tickets, call transcripts, clinical notes, or email archives. This personal data often ends up in the model training, whether or not it is required by the use case.

The anonymity of a model can therefore rarely be settled by examining the ingestion pipeline. This means that most model providers, and many organizations that fine-tune their own models, cannot demonstrate that personal data was absent from their training data. The assessment has to be made on the model itself.

From legal to technical

In December 2024, the European Data Protection Board (EDPB) adopted its opinion on personal data in AI models, stating that a model trained on personal data cannot be presumed anonymous in all cases.¹ The assessment has to be made case by case. For a model to qualify as anonymous, the likelihood of extracting personal data relating to training-data subjects from the model itself, or obtaining this data via queries, should be insignificant.

Moreover, given their complexity and the scale and variety of training data, the EDPB deems that AI models are very likely to require thorough evaluation of the risks of identification. In July 2025, the CNIL produced a guide on the status of an AI model under the GDPR, including a detailed description of the tests and measures that providers need to perform, in order to assess the anonymity of their models and AI systems.²

The CNIL lists the possible causes that can lead a model to memorize some of its training data. These include, for example, having direct identifiers (such as names, emails, addresses) in the training set, or the presence of rare records and outliers. Models with very large numbers of parameters are also more prone to memorization, especially if they are trained without proper regularization or dedicated privacy preserving algorithms.

The CNIL also covers the technical and organisational measures that can be taken during development and training to reduce the likelihood of re-identification, such as, for example, sanitizing the training data to remove identifiers and outliers or employing strong regularization techniques and privacy preserving training algorithms.

In most cases though, these measures won't be sufficient to rule out, a priori, the possibility of successful data reconstruction and the consequent re-identification of data subjects in the training set. For this reason, reconstruction attacks must be performed.

This adversarial evaluation must make use of all the means reasonably likely to be employed, such as additional information on the attacker's targets, significant time and money, and the access to state of the art methodology and attack algorithms. In particular, one needs to consider attacks aimed at reconstructing exact fragments of training data, membership inference, exfiltration and reconstruction attacks, as well as empirical measures of the model memorization.

Such tests are highly technical and require specialized expertise and tools.

Indeed, many of these tests use refined attack algorithms that are continuously improved and complemented by other methods by the research community. To centralize this effort and facilitate the analysis, the CNIL has started the PANAME project.³

Testing PANAME

PANAME (Privacy Auditing of AI Models) is a software library for running extraction and re-identification tests on AI models, planned for release as open source by the end of 2026. The project was launched in June 2025 together with PEReN, ANSSI, and INRIA.

We participated in the first test phase of the library⁴ and presented our results to the consortium in June 2026, together with four other participants coming from adtech, energy and utilities, consulting, and frontier AI labs.

In our tests, we used the library to evaluate and measure the extent to which sanitizing fine tuning data reduced the success of extraction attacks.

We used public emails from the Enron corpus, with 100 emails as members of the training set and 100 held out as non-members. We fine-tuned a small instruction-tuned Llama 3.2 1B model with LoRA, in two variants.

The first variant was fine-tuned on cleartext. The second used sanitised text, with direct identifiers detected and replaced with placeholders before training. We used Presidio⁵ to cover names, dates, addresses, cities and countries, email addresses, phone numbers and IP addresses.

We then ran membership inference attacks against the base model and both fine-tuned variants. In essence, membership inference is a classification task where the goal is to infer, given a record (an email in our case), whether or not it was used to fine tune the model. PANAME implements algorithms from the published literature to perform this task. We have used the perplexity-based attack and the Min-K% probability attack.

What we found

We measure attack success as the area under the ROC curve. A value of 0.5 means the attack does no better than guessing.

The attacks fail on the base model. Both attacks score near the random guessing threshold of 0.5 before fine-tuning. This is what we expect (if the model was not trained on the fine-tuning data). The signal we want to measure is created by the fine-tuning step, not inherited from pre-training.

Fine-tuning on one hundred emails creates a clear signal. Both attacks move well above random guessing, demonstrating that using personal data in fine-tuning is enough to make the model carry information about who was in the set.

Sanitisation reduces the signal. Redacting direct identifiers reduces the attack success by 0.1 of AUC. This effect extends in the region that matters most for privacy, the low false positive regime, where an attacker makes confident claims about a small number of records. Up to a false positive rate of one percent, the AUC of both attacks drops by roughly 60%.

What it means in practice

Our experiment supports that data sanitization before training or fine-tuning lowers measured privacy risk without eliminating it entirely. Membership inference compares how confidently the model handles a training record against how it handles records it never saw. Even if identifiers are removed, a large part of the training email body is left untouched, leaving enough signal in the model weights for the attack to pick up on.

As we have demonstrated, a model fine-tuned on redacted personal data may still tell an attacker that a record was in the training set. The extent to which this happens depends on how the fine-tuning is done, the size of the dataset, etc. For a precise characterization of the impact of all these factors in a real world deployment, the evaluation needs to cover a wider range of attack families, repeated configurations, and realistic threat models.

Our small experiment demonstrates that redacting fine tuning data is a useful mitigation but does not resolve the question of model anonymity. It is a measure, rather than an outcome.

Looking ahead

By providing a centralized, open source repository of state of the art attacks, PANAME makes evaluating model anonymity easier, allowing teams to perform the kind of analysis requested by regulators. This becomes more important as agentic systems spread. In their July 2026 note, the CNIL and the CIANum describe systems where personal data circulates between services, interaction history and persistent memory accumulate, and responsibility becomes harder to allocate across a decentralised chain.⁶ Models sit inside those systems, and every system built on a model inherits its status.

We are looking forward to the next phases of PANAME and to the publication of the library.