Safety Critical AI

Given AI’s potential for misuse, how do we develop and deploy algorithmic systems responsibly?

Increasingly, AI systems are being deployed in contexts where safety risks can have widespread consequences, including medicine, finance, transportation, and social media. This makes anticipating and mitigating such risks — in both the near and long term — an urgent societal need.

Our Safety Critical AI Program convenes Partners and other stakeholders to develop best practices that can help us avert likely accidents, misuses, and unintended consequences of AI technologies. We don’t have to wait for such incidents to arise. As our work shows, precaution can be taken as early as the research stage to ensure the development of safe AI systems.

Our Safety-Critical AI Work

The Safety Critical AI Program has developed norms for responsible publication of AI research, supported the launch of the AI Incident Database, and created SafeLife, a novel AI learning environment for training non-destructive agents.

In 2021, PAI released the white paper “Managing the Risks of AI Research,” offering six recommendations for anticipating potential harms when publishing AI research. Following its publication, Nature Machine Intelligence published an editorial endorsing the paper’s recommendations. More recently, Meta AI was informed by and cited the white paper in the release of their large language model OPT-175B.

In 2022, PAI partnered with CIFAR and the Ada Lovelace Institute to bring together recent ML conference organizers and AI ethics researchers to consider how existing ethics review practices like impact statements are faring. “A Culture of Ethical AI,” a report co-authored by all three organizations, synthesizes insights gathered from this convening.