Safety Critical AI

Given AI’s potential for misuse, how do we develop and deploy algorithmic systems responsibly?

Forecast future risks 

Develop best practices

Improve preparedness

Create foundations for governance

Increasingly, AI systems are being deployed in contexts where safety risks can have widespread consequences, including medicine, finance, transportation, and social media. This makes anticipating and mitigating such risks — in both the near and long term — an urgent societal need.

Our Safety Critical AI Program convenes Partners and other stakeholders to develop best practices that can help us avert likely accidents, misuses, and unintended consequences of AI technologies. We don’t have to wait for such incidents to arise. As our work shows, precaution can be taken as early as the research stage to ensure the development of safe AI systems.

Our Safety-Critical AI Work

The Safety Critical AI Program has developed norms for responsible publication of AI research, supported the launch of the AI Incident Database, and created SafeLife, a novel AI learning environment for training non-destructive agents.

In 2021, PAI released the white paper “Managing the Risks of AI Research,” offering six recommendations for anticipating potential harms when publishing AI research. Following its publication, Nature Machine Intelligence published an editorial endorsing the paper’s recommendations. More recently, Meta AI was informed by and cited the white paper in the release of their large language model OPT-175B.

In 2022, PAI partnered with CIFAR and the Ada Lovelace Institute to bring together recent ML conference organizers and AI ethics researchers to consider how existing ethics review practices like impact statements are faring. “A Culture of Ethical AI,” a report co-authored by all three organizations, synthesizes insights gathered from this convening.

Steering Committee

Anthony Aguirre

Vice President & Secretary of the Board

Future of Life Institute

Shahar Avin

Senior Research Associate


Wafa Ben-Hassine

Principal, Responsible Technology

Omidyar Network

Esha Bhandari

Deputy Director

ACLU Speech, Privacy, and Technology Project

Jack Clark



Iason Gabriel

Staff Research Scientist


Gillian Hadfield


Schwartz Reisman Institute for Technology and Society, University of Toronto

Tadayoshi Kohno

Professor of Computer Science

University of Washington Department of Computer Science & Engineering

Christina Montgomery

Vice President and Chief Privacy & Trust Officer


Joelle Pineau

Vice President of AI Research


Adrian Weller

Programme Director for Safe and Ethical AI

The Alan Turing Institute

Jess Whittlestone

Head of AI Policy

Centre for Long-Term Resilience

Program Workstreams

Program: Safety Critical AI
Publication Norms for Responsible AI
Program: Safety Critical AI
AI Incident Database
Program: Safety Critical AI
Explainable AI in Practice