The Institute for Security and Technology, a 501(c)(3) non-profit critical action think tank, today released “AI and the Future of National Security,” a report outlining findings from its novel survey of national security professionals about the implications of artificial intelligence. Survey respondents, who ranged from former Under Secretaries in the U.S. government to retired flag officers, answered 70 qualitative and quantitative questions across 5 sections intended to gauge how current and former decisionmakers and policymakers are thinking about the impact of powerful new AI models on the military, cybersecurity, and weapons of mass destruction.
Adversarial misuse of AI, for instance through the creation of biological or cyber weapons, ranked as the most important threat to U.S. national security (72% of respondents ranked it as one of their top three concerns). On the defensive side, cybersecurity (78%) and intelligence analysis (73%) were considered top areas of promise.
Respondents were bullish on the rapid growth of AI capabilities. The median respondent felt with 90% confidence that artificial general intelligence (AGI) would be achieved by 2040, with their best guess being AGI by 2032. When asked about AI operating beyond human control with no way to regain it, 33% expected that scenario to occur within 10 years. And when asked whether AI would cause a catastrophe–described in the survey as one resulting in the deaths of at least 10 million people—40% of the respondents thought there was at least a five percent chance of such a catastrophe occurring by 2050.
In their foreword to the report, IST CEO Philip J. Reiner and Lieutenant General John (Jack) N.T. Shanahan, United States Air Force, Retired, wrote: “Andy Marshall spent four decades arguing that the most dangerous strategic error is not misjudging an adversary’s capability, but misjudging your own institution’s capacity to adapt to a shift already underway. The survey results and this summary report are aimed at that second failure. Not what AI can do — in fact, every finding here assumes the answer is that AI has the potential to be extraordinarily powerful — but whether the people responsible for national security actually understand what they are managing, and if they are organized to do so properly and effectively.”
Survey respondents were not confident in current or proposed measures to constrain the capabilities of these systems that would be commensurate with their risk. 61% felt that the probability of an AI-induced catastrophe was at or above the threshold they would set for pushing to continue to maintain U.S. leadership in AI. In open-ended responses captured by the survey team, one pointed to “little to no legal frameworks or guidelines for use,” while another warned that, absent a clear risk assessment and against domestic pressure to deregulate, “safety will likely continue to occupy a back seat to rapid development and deployment of AI tools.”
“National security leaders are taking AI seriously,” said Mariami Tkeshelashvili, lead author of the report. “At the same time, many respondents noted that they have had to update their own predictions in the last few months. The recent cybersecurity incidents involving loss of control of frontier models all happened after the survey closed, but I’m sure they would have influenced the results. These findings can serve as a baseline for the field as we continue to examine how policymakers are approaching this vital national security issue.”
“This first-of-its-kind survey demonstrates just how concerned the national security community is about the threat AI could pose to American national interests,” said Hamza Chaudhry, the AI and National Security Lead at the Future of Life Institute. “IST’s invaluable work should serve as a wake up call for policymakers. It’s past time to take AI seriously as a core national security priority.”
The survey was conducted in partnership with the Forecasting Research Institute and was made possible thanks to the support of the Future of Life Institute.
Survey Results: Deep Dive
Strong consensus on intelligence applications; fears over nuclear integration and lack of norms; unreadiness of the policymaking apparatus
87% of respondents thought that AI can infer sensitive information from open-source or commercially-available intelligence, yet only 6% ranked the power of inference as a top three national security threat. This mismatch in translating capability into risk may be, in part, a result of a lack of technical expertise available to the community: One respondent noted that policymakers are limited by “expertise, capability, dependence on industry voices [with] no independent capacity for evaluation.”
The survey also saw consensus with respect to nuclear weapons integration. Most survey respondents (96%) agreed that any decision to use nuclear weapons should be made by humans rather than AI, and 89% agreed that AI systems should not autonomously control nuclear command, control, and communication (NC3) systems. With respect to the policy environment surrounding such decisions, more than a third of respondents rated the lack of international norms for military use as “extremely concerning,” while no other risk received a single extremely concerning rating.
In open-ended responses, survey participants were blunt about the readiness of the national security policymaking apparatus to address national security AI risks. They specifically cited a “lack of technical expertise among senior officials” and “lack of humans with the right technical skills.” Respondents connected that deficit directly to dependence on vendors, claiming there is an “expertise [and] capability dependence on industry voices, no independent capacity for evaluation,” and that the “policy community… [is] poorly equipped to separate promotional hype from technical reality.” Several framed the risk as political as well as technical: “This knowledge gap can also make policymakers overly reliant on industry claims, creating opportunities for companies to shape policy in ways that prioritize profit over the public interest.”
Domain-specific results
In addition to the main survey, experts also had the opportunity to answer questions specific to their areas of expertise.
The 42 experts who completed the cybersecurity portion of the survey saw significant opportunity for AI tools improving cyber defense, with 81% saying all defensive capabilities are very likely to improve and 88% expecting AI to deliver significant or transformational gains. Yet, in the near term, they were much more pessimistic: 57% expect the offense-defense balance to shift in attackers’ favor. 93% of respondents were in favor of mandatory regulations to mitigate cybersecurity risks from AI.
For the nuclear-specific questions, 42 nuclear and chemical, biological, radiological and nuclear (CBRN) experts were aligned that AI will affect nuclear deterrence or strategic stability (76%). Asked to rank their top three risks relating to nuclear weapon systems, 62% ranked erroneous or ambiguous AI-assisted recommendations, and 62% included automation bias under time compression. Solutions-wise, 72% wanted to see specific AI-related confidence-building measures, such as information sharing about AI integration with NC3.
Finally, 25 CBRN experts answered domain-specific questions regarding chemical and biological weapons risks. There was a significant gap between concern about biological dual-use (64%) versus chemical (41%), with a small majority (52%) indicating that AI is more helpful for offense. Concerningly, only 12% thought that defense had the upper hand, with the remainder unsure or believing that offensive and defensive gains would balance out. 70% expect that AI will meaningfully increase bioweapon development risk within three years, while only 12% believe it is unlikely to ever do so.
Work to be done
Taken as a whole, the survey underlines significant uncertainty in the national security community when it comes to AI risk. Strong consensus in anonymized survey results that mandatory regulation is necessary has not translated into broad public calls for the same. Experts are simultaneously skeptical of the ability of norms and other diplomatic measures to reduce risk and eager to see confidence-building measures put in place to avoid catastrophe. Near unanimity about the ability of current models to infer classified information did not rate as one of the most urgent risks we face, according to the respondents surveyed.
“Current AI systems still face real limitations: forgetting previously learned tasks after being trained on new data, real-time environment constraints, limits on automated research and development, and more broadly, gaps in adaptability, reliability, and performance in complex environments,” continued Tkeshelashvili. “Given this gulf between near-term capability limits and long-term uncertainty about catastrophic risks, the question remains: What are specific, well-defined capability thresholds for advanced AI that society as a whole should watch for?”
This survey represents a continuation of IST’s research on significant AI risks, including recent publications on loss of control indications and warnings and cybersecurity risks associated with AI agents. Researchers interested in collaborating, including by conducting additional analysis of the unique dataset collected for this project, should reach out to [email protected].
* * *
About IST:
The Institute for Security and Technology (IST) is the 501(c)(3) critical action think tank that unites technology and policy leaders to create solutions to emerging security challenges. IST stands at the forefront of convening policymakers, technology experts, and industry leaders to identify and translate discourse into impact. We take collaborative action to advance national security and global stability through technology built on trust, guiding businesses and governments with hands-on expertise, in-depth analysis, and a global network. Since our foundation in 2020, IST has launched an array of impactful initiatives addressing challenges such as ransomware, the resilience of lifeline critical infrastructure, nuclear risk reduction, artificial intelligence security, and more. Visit us at https://securityandtechnology.org/.
Media Contact:
Sophia Mauro
Senior Director for Strategic Communications
