LeakPro 2: Operational privacy risk management for AI systems

LeakPro 2 is a framework for assessing and mitigating privacy risks in machine learning models. It combines privacy attacks, PET evaluation, and structured workflows to support practical privacy risk assessment.

Challenges

Machine learning models are algorithms that internally encode the capability to identify patterns in a data source. In many domains, e.g., in life science or finance, the data may be sensitive. It is, therefore, paramount to assess the difficulty in extracting sensitive information under realistic adversary settings.

Data and AI models drive modern innovation across business, research, and technology. Their transformative potential often depends on the ability to share and collaborate across organizational and sectoral boundaries. Yet datasets and models frequently contain sensitive information, making them difficult to exchange. Concerns about privacy, intellectual property, regulatory compliance, and competitive advantage all act as barriers, limiting the opportunities for collaboration and slowing the pace of innovation.

The Penrose triangle
CC (Attribution-Share Alike 3.0 Unported)

"The Privacy Triangle"

The Penrose triangle is an ‘impossible’ geometric figure. It can work as a metaphor for the conflicting demands that AI developers, lawyers, and organizations grapple with:

The technology demands precision, the regulations are open for interpretation, and organizations need practical guidelines.

Continuing and expanding the scope from LeakPro 1, LeakPro 2 continue the building on a platform to assess the information leakage of:

Trained machine learning models
The risk of leaking information during training with federated learning, and
The risk of leaking information in synthetic data,

In addition to assessing the risk for leakage, LeakPro 2 align technical findings with legal and organizational auditing standards, adding frameworks for organizational auditing, and legal regulation. This evolution aims to bridge the gap between abstract privacy regulations, such as the EU AI Act, and the practical need for robust, data-driven security assessments.

Project purpose

Current privacy evaluations produce technical metrics that are difficult to interpret and act upon. Organizations need clear, decision-ready evidence to manage privacy risks and meet regulatory requirements. LeakPro II addresses this by:

Operationalizing the "privacy triangle": The core purpose of LeakPro 2 is to evolve the LeakPro tool into a comprehensive reference framework that systematically links technical attack outcomes with organizational harm quantification and regulatory compliance.
Objective data protective impact assessments (DPIA): By directly connecting the outcomes of empirical privacy attacks with quantifiable measurements of harm to data subjects, the project aims to transform DPIAs from subjective guesses into objective, evidence-based assessments.
Expanding to generative AI: The project will expand LeakPro "horizontally" to also cover generative models, such as Large Language Models (LLMs) and diffusion models, alongside strengthening the existing support for image, tabular, and time-series data.
Privacy enhancing technologies (PET) optimization: The project will expand "vertically" to include systematic optimization and validation of PETs, using empirical attacks to help organizations perfectly balance privacy protection with data utility.