Director for AI Security Research Krystal Jackson to lead a new line of effort bringing together industry, academia, and government experts, anchored by a $875,000 grant
The Institute for Security and Technology is announcing a new research initiative to establish governance frameworks and control mechanisms for AI systems with recursive self-improvement (RSI) capabilities. Supported by an initial grant of $875,000, the project will address the urgent need for frameworks and response mechanisms to protect against behavior consistent with self-improvement, which represents a significant security risk. Although RSI is acknowledged as a risk requiring oversight, there are no agreed-upon signals for when self-improvement behavior should trigger closer monitoring, coordinated action, or a pause in development.
Frontier AI systems are increasingly capable of automating portions of their own development pipelines, from code generation to architecture optimization. Unlike narrower capabilities, RSI is not likely to arrive in a single breakthrough moment. It will instead unfold through compounding improvements across model capabilities, tooling, orchestration, and the underlying architectures that connect agentic systems. As such, RSI could be one of the most important sources of risk—and opportunity—faced in the development of advanced AI.
“The RSI initiative extends IST’s ongoing research into AI risk dynamics and leverages our strong relationships with AI labs, independent evaluators, and government AI safety bodies,” said Philip Reiner, CEO of IST. “IST’s work as the connective tissue between technologists and policymakers has never been more important, especially at the frontier of human accomplishment. I am excited to welcome Krystal Jackson to the team as she continues our work developing pragmatic solutions for both industry and government, and I’m grateful to the Future of Life Institute for their support.”
“Recursive self-improvement will compress years of AI development into days, and no one is quite sure what governance or control frameworks would effectively mitigate the risks this poses,” said Hamza Chaudhry, AI and National Security Lead at the Future of Life Institute. “IST is uniquely situated to bring together labs, evaluators, and key government officials to address challenging technical questions that result in practical, usable, answers. We’re proud to support this work.”
Krystal Jackson, Director for AI Security Research, recently joined IST from the University of California, Berkeley’s Center for Long-Term Cybersecurity, where her research focused on developing risk modeling, management, and threshold-setting frameworks for frontier AI systems. She previously worked at the Frontier Model Forum and as an AI Capabilities Analyst at the Cybersecurity and Infrastructure Security Agency (CISA). Krystal explains the project’s goals as: “There’s broad agreement that systems capable of RSI require a new kind of oversight, but no consensus on what that should look like. We’re working with developers and researchers to co-create the risk management guidance that will enable safe and secure development as we enter into this new phase where AI systems are capable of self-modification and improvement. It is critical that we start these efforts now before we reach a point where we begin to lose meaningful control.”
