US-Sweden Pre-Study: Securing Agentic AI

About the pre-study

The pre-study (Step 1), "From Networks and Talent Programs to International Collaborations," aimed to leverage AI Sweden's ecosystem to align the needs of Swedish industry with US stakeholders. Through intensive meetings, digital workshops, and a visit from Harvard University to AI Sweden, the most urgent research needs were identified: securing applied LLM solutions and agentic systems in critical societal sectors.

The pre-study successfully resulted in a drafted Step 2 proposal called "SAFE-CARE"—a project utilizing a digital healthcare assistant at Västra Götalandregonien as a concrete test case for AI-driven red-teaming. Bilateral co-funding for a new student exchange program was also explored, though interest from funders was insufficient.

During the pre-study, a wide range of collaborative projects were discussed to evaluate their strategic fit:

• Agentic dynamics
• Decision-support systems
• Examining long-interaction dynamics of LLMs
• Context-enhancement for large multimodal models
• Multilingual Uses of LLMs
• AACR Support/Data Research Proposals
• Transportation Security Generalized SVD (GSVD) applications
• Privacy-Preserving Model Training (ExpM+NF)
• Dynamical Structure Function (DSF) Modeling of System Security
• Cybersecurity in the Age of LLMs: Predictive Defense and Attack Study
• Custom LLM Systems Security

Of these, the final two—Cybersecurity in the Age of LLMs: Predictive Defense and Attack Study and Custom LLM Systems Security—emerged as the most relevant with the highest potential for US-SE overlap, forming the backbone of the ongoing collaboration.

Ecosystem insights: Swedish partner survey

To validate the needs of the Swedish ecosystem, AI Sweden distributed a digital survey to dozens of partners. The results highlight a clear demand for security frameworks in applied AI:

• Primary Focus: Organizations are heavily focused on Securing Custom LLM Systems (RAG, Local LLMaaS, and Agents), with 50% of respondents identifying this as their top priority and the other 50% viewing it as equally critical to other security tracks.

• Priority Use Cases: The most important use cases identified were Agentic systems (80%), followed closely by LLM assistants (70%) and RAG systems (70%).

• Critical Concerns: The highest-ranked priorities for the ecosystem include Data Privacy/Leakage, Securing Agentic Tools (APIs/Architecture), and the need for dedicated red-team audits of internal systems.

• Collaborative Engagement: Partners expressed a strong preference for contributing through Subject Matter Expertise (e.g., providing specific use cases in Healthcare or Government) and providing Technical Resources (compute and data samples) to support the research group.

Challenges

Agentic AI and autonomous systems are being rapidly integrated into society, but frameworks for assessing their security remain underdeveloped. The combination of inherent vulnerabilities within AI models and their granted authority to execute actions creates complex security risks, which are further exacerbated by the growing threat of autonomous AI hacker agents.

Project purpose

To rapidly distill project ideas and pair US and Swedish expertise to draft a robust Step 2 proposal. The objective was to transition from general matchmaking to designing an applied, bilateral research and innovation project focused on proactive cyber defense and LLM security.

Outcomes

The primary deliverable was a comprehensive report summarizing the findings, alongside a complete Step 2 application to Vinnova.

Additional outcomes included establishing a bilateral research pipeline for continuous AI security collaboration and exploring potential student exchange programs between the US and Sweden.

For more information, please reach out to: