LeakPro 2 expands successful AI privacy assessment tool

“A probability score for attack success is a great start, but it doesn’t tell you how serious the consequences could be,” says Fazeleh Hoseini, research scientist, and project manager for LeakPro 2. She continues:

Addressing the privacy paradox

For industries subject to strict regulations, like healthcare, finance, and government, the "Privacy Paradox" has long been a barrier to innovation: how can you leverage the power of AI without risking the exposure of sensitive data? Addressing this question, LeakPro was launched in 2023 to develop a tool to assess privacy risks in machine learning.

LeakPro 1 focused on developing technical attacks against AI models to be able to quantify the risk of attacks leading to data leakage. The result was an open-source tool that is already being used by partners to evaluate the security of AI models.

Covering GenAI, compliance, and impact quantification

Fazeleh Hoseini likens the project’s expansion in phase two to the Penrose triangle, an ‘impossible’ geometric figure metaphor for the conflicting demands that AI developers, lawyers, and organizations grapple with: The technology demands precision, the regulations are open for interpretation, and organizations need practical guidelines.

With partners such as Scaleout productifying the tool, Syndata working with synthetic data, and Recorded Future adding cyber security expertise, LeakPro 2 becomes a catalyst for privacy aware AI in Sweden. Swedish. The project's aim to open-source the code is a conscious strategy to democratize AI privacy and make advanced risk assessment accessible to everyone, from startups to authorities.

One of the most requested features in this second phase of LeakPro is the mentioned support for LLMs. In the first phase, generative models were deemed possible to evaluate, but excluded due to their complexity. Now they are a high priority.

By including LLMs in LeakPro 2's attack suite, organizations can assess whether the AI models they use or share inadvertently expose sensitive data, whether that's personal information, proprietary research, or confidential records.

“LeakPro 2 is not just a continuation of a successful project; it's an expansion that turns a useful tool into a practical framework for responsible AI. Beyond measuring risk, it also helps organizations actively mitigate it by optimizing privacy-enhancing technologies (PETs) like federated learning and synthetic data to find the right balance between privacy and utility. By integrating risk assessment, PET optimization, legal compliance, and organizational workflows, we're building a resource for safer, more trustworthy AI adoption” says Fazeleh Hoseini.