Experts converged on three main critical areas requiring action to prevent likely national security risks from AI.
The Institute for Security and Technology, with support from the Future of Life Institute (FLI), recently released a first-of-its-kind survey to capture how national security professionals perceive the implications of artificial intelligence. More than 100 survey respondents, ranging from former Under Secretaries to retired flag officers, provided their thinking about the impact of powerful new AI models on military capabilities and decision making, cybersecurity, and weapons of mass destruction.
Bringing the Findings to the Table
To mark the report’s release, IST convened 24 experts from industry, government, think tanks, and the research community on September 18 to digest the AI Risk Barometer Survey results and identify the necessary paths forward. The following summary presents key takeaways and observations from the discussion, which was held under Chatham House Rule.
Based on the findings, participants agreed on three primary next steps:
- Evaluation science is lacking: there needs to be a dramatic increase of investment in AI evaluation science, especially evaluation practices beyond in silico
- Prioritizing research conducted by scientists, not just technologists, on how AI risks are affecting the physical world today. Conducting deep research on the real risks posed by AI and engaging in relevant red teaming practices with the help of subject matter experts
- Rapid technical literacy: the U.S. faces an onslaught of high velocity technical change and must expand efforts to ensure the development of a technologically literate workforce in the domains where AI presents the most significant national security risks.
IST CEO Philip Reiner and FLI AI and National Security Lead Hamza Chaudhry delivered welcome remarks, after which IST Director for AI Security Policy Mariami Tkeshelashvili presented the survey’s findings and its unique attributes.
“The point of getting everyone in a room was to move from measuring the problem to figuring out how to solve it,” IST’s Philip Reiner said.
“AI has gone from being a problem on the margins of national security to being at the center of the most important national security debates in the century,” said Hamza Chaudhry, AI and National Security Lead at FLI. This is why we decided to hear from the community itself on where it sees AI’s greatest risk and opportunities, both through the survey and by convening stakeholders across the national security enterprise in the same room. As we move to secure our future in an era of unprecedented technological uncertainty, it is vital that we hear and learn from those who have long been charged to protect it.”
The headline finding, that 61 percent of respondents rate the risk of an AI-driven catastrophe at or above the level they themselves deem acceptable, was generally unsurprising to the roundtable. Some participants expressed they had expected an even higher figure. But on how the government, industry, and research communities can meet each other’s needs to act on this and other findings, there was less consensus.
Access to the Model and Evaluation
Survey results showed that evaluation of AI models is the mitigation experts trust most, ranking above technical safeguards, operational governance, and international coordination. Roundtable participants shared this view and focused on the conditions that would make evaluation work.
Throughout the discussion, access emerged as a central barrier to independent evaluation. Some participants urged model developers to provide evaluators access to their weights and full chain of thought (CoT), overriding privacy concerns. Others noted that sharing CoT with institutions such as the military presents its own security and privacy risks, and questioned what a reasonable ask of companies might look like. Some participants suggested a trusted-access arrangement.
Participants also called for greater funding and capacity for evaluation science and standards. Pointing to the gap between in silico and real-world testing, they suggested comprehensive support for red teaming, rather than capability evaluations alone. Proposals included mandatory capability-based evaluation, authorization and funding for the Center for AI Standards and Innovation (CAISI), and a self-regulatory organization to set benchmarks for government use of AI.
Biology drew particular concern from the roundtable. According to one expert, most AI safety evaluations today only assess large language models. Biological models, on the other hand, lack proper evaluations, with the greatest risk seen in a biological model looped into a harness. Roundtable participants also flagged the open-weight model question as a growing area of importance and observed that nothing currently stops someone from running an experiment that combines AI and biology. This prompted calls for intervention, and the observation that second-line defenses, much of which don’t depend on AI at all, are underrated by organizations across industries.
Aligning the Military, Policymakers, and Developers
Participants also flagged how the military, government, and AI developers are all operating on different clocks. The military, they observed, has historically been postured to work in the now and tends to focus on imminent rather than existential threats. Meanwhile, frontier AI developers are racing toward tomorrow’s capabilities. The government is playing catch-up, even as some participants stressed the need for strategic foresight.
To make progress, several urged grounding the conversation in what is currently being deployed today, since machine learning and autonomous systems have long been an element of military systems, and focusing on specific policies that can be implemented now. Additionally, a participant argued that AI regulation should not slow innovation to the point that small AI companies cannot enter contracts with the DoD.
Workforce and Policymaker Attention
In the survey, 32 percent of respondents named workforce expertise and training among the most significant gaps in preparing for AI integration and defending against AI threats.
The lack of a technologically literate workforce was also a recurring theme during the roundtable, especially among the policymakers gathered in the room. Participants pointed to gaps in technical expertise in the military and at the State Department, and to a need for more education among policy leaders. Some saw barriers to entry, such as how long it takes to get hired and bureaucracy resource cuts, as the core problem. Others pointed instead to a lack of desire to go into government and the difficulty of returning to government after a period of private sector work.
In considering where to direct policymaker attention to be most effective, participants suggested meeting people where they can act, such as Members of Congress responding to AI-related concerns from their home districts, and preparing people for whatever technology comes next by educating them on how technology can shape society. Participants also noted a growing fatigue around AI, which many experience as something happening to them rather than a tool they use, and which the media, with its incentive to dramatize, may be contributing to.
Explore the Findings Further
The AI and the Future of National Security survey produced 58 findings, 31 quantitative and 27 qualitative. These findings offer valuable data points for several distinct audiences: the U.S. national security community, as well as multilateral organizations working on AI security and safety issues; technical and AI governance researchers and the cybersecurity community; U.S. policymakers; and AI industry stakeholders, specifically frontier AI companies. The areas of consensus and disagreement highlighted in the report show where momentum already exists for specific action, where defenders need to pay closer attention, and where more work remains to address open questions and disagreements.
To discuss further analysis or schedule a briefing, reach out to Mariami Tkeshelashvili ([email protected] ) or Sophia Mauro ([email protected]).
