On September 15, 2008, Lehman Brothers Inc.–once the fourth largest-investment bank in the United States–declared bankruptcy. Marking a pivotal moment in the 2008 financial crisis, the collapse led to widespread economic downturn, loss of jobs, and erosion of public trust in financial institutions.
In September 2018, Theranos, a health technology company that claimed to have developed a revolutionary blood testing technology that could run hundreds of tests using only a few drops of blood, was dissolved and liquidated. As a result, founder Elizabeth Holmes faced criminal charges, its patients faced potential harm from inaccurate testing results, and the general public lost confidence in health technology startups.
What do these two stories have in common? And what can they tell us about the artificial intelligence compliance landscape? In IST’s latest report, authors Mariami Tkeshelashvili and Tiffany Saade analyze 11 cases of compliance failure to illuminate potential pitfalls in the AI ecosystem.
Thanks to the time, dedication, and expertise of IST’s AI Risk Reduction multi stakeholder working group, as well as the generous support of the Patrick J. McGovern Foundation, whose funding allowed us to continue this project through the lens of IST’s Applied Trust & Safety program, this report puts forward definitions, frameworks, and lessons learned to help AI builders and users navigate today’s complex compliance landscape.
In this month’s newsletter, we sat down with Senior Associate for Artificial Intelligence Security Policy Mariami Tkeshelashvili to learn more about the research process, their findings, and what’s next for this effort.
Read the full interview in the December edition of IST’s newsletter, The TechnologIST.
Q: So, how exactly does the collapse of Lehman Brothers or Theranos relate to AI?
On the surface, these might not seem related. But this report argues that we have a lot to learn from the specific failures that took place–and the ways in which they could have been prevented.
In the case of Lehman Brothers, we found an internal culture that rewarded high risk investments, executives that were hyper-focused on gains, regulatory oversight lapses, and even fraudulent “Repo 105” accounting maneuvers–all of which helped lead to its bankruptcy, widespread loss of jobs, and the subsequent negative impact on domestic and international financial markets. These findings have important implications for the AI ecosystem. Nowadays, AI builders might face pressure to prioritize production over safety and security, or cut corners to obtain go-to-market advantage. As the Lehman Brothers bankruptcy case demonstrates, institutional guardrails that create a culture of compliance, establish necessary risk management policies, or prevent cutting corners in order to get ahead are important safeguards against failure.
Likewise, in the case of the Theranos scandal, we found inadequate testing and validation of the technology and violations of clinical laboratory regulations that, among other key failures, led to its dissolution and liquidation. In the AI context, subjecting AI to proper validation, testing, and quality assurance can protect against a failure like the one that occurred in the Theranos case.
Q: How did you go about selecting these particular case studies?
Some of the case studies, like the 1979 Three Mile Island accident, the 1986 Challenger space shuttle disaster, and the 2008 Lehman Brothers bankruptcy are well-documented and familiar to a broader audience, so it was important for us to analyze these to understand the full extent of their impact and distill lessons learned for the AI ecosystem. Reviewing more modern cases like the 2018 Cambridge Analytica scandal or the 2024 CrowdStrike outage was helpful to draw parallels between historical and current issues in the technology landscape.
Q: What were some common threads of the compliance failures that you investigated?
Through our investigation, we found three main categories of compliance failures that recur throughout history. First are institutional failures, which are exemplified by a lack of executive commitment to creating a culture of compliance, establishing necessary policies, or empowering success through the organizational structure. Next, procedural failures occur when there are misalignments between an institution’s established policies and its internal procedures and the staff training required to adhere to those policies. Lastly, we define performance failures as an individual employee’s failure to follow an established process, or an automated system’s failure to perform as intended, leading to an undesirable result.
Q: This report assesses the current AI governance landscape, looking at laws & regulations, guidance, norms, standards, and organizational policies. AI governance is a rapidly-shifting field—how did you ensure that you captured an accurate view of the landscape?
Instead of covering all existing and emerging frameworks and presenting them quantitatively or mapping them out for each individual institution or country, I chose to focus on five main sources or types of AI governance, and presented examples of each.
For instance, in the case of laws & regulations, I tracked state laws on privacy that could have an impact on AI systems, looking at examples in California, Utah, and Colorado; analyzed existing U.S. federal laws and regulations that do not explicitly mention AI, but could have important implications for its compliance needs, like the Fair Housing Act, Equal Credit Opportunity Act, Equal Employment Opportunity Commission guidelines, and the Health Insurance Portability and Accountability Act; and looked at laws in the European context, such as GDPR.
In focusing on these 5 primary categories, I defined each in a way that makes them easy to understand and helps AI builders and users track what might emerge in the future.
Q: How do you expect the AI governance landscape to evolve in the coming months—and years?
Risks, opportunities and challenges associated with AI are not going away. Among the most prominent concerns, as articulated in IST’s December 2023 report, are the malicious use of AI, a dangerous “race to the bottom,” compliance failure, and the risk that humans are left out of the decision-making loop. Additionally, we are tracking other emerging risk categories associated with AI agents. Therefore, it’s essential that industry leaders, researchers, and civil society remain vigilant in monitoring these risks and, more importantly, take proactive measures to address them.
Q: How will this research play a role in shaping the governance landscape? What’s next for the effort?
I believe that a historical perspective provides valuable insights for governance efforts. History often repeats itself—or at least echoes in the present and future—making it especially enlightening to analyze past case studies. Working on these case studies was a fascinating experience, and I hope this report inspires others to explore similar approaches. Building on the lessons outlined in Part 1, as well as the definitions and categories of failures identified, Part 2—slated for release in early 2025—will present a comprehensive list of risk mitigation strategies. These strategies are designed to help AI builders and users avoid compliance failures and navigate the challenges of responsible AI development and deployment.