Balancing Safety and Accessibility for Open Foundation Models  

The release of Meta’s Llama 3 and the findings from the Stanford AI Index Report 2024 underscore an unmistakable trend – foundation models are not just advancing rapidly but also increasingly being released with open access. Open access refers to models publicly released with access to key components, notably model weights. In 2023 alone, 149 new foundation models were released, with a majority 65.7% being open, up from just 33.3% in 2021.

The coming months and years may see the release of more powerful open models. Even as we scrutinize the evidence on the potential risks of open access AI, we need to develop tailored risk mitigation strategies in parallel. While research is yet to find that open models significantly increase safety risks compared to closed models, they may present unique challenges around lowering barriers to generating non-consensual intimate imagery (NCII) and child sexual abuse material (CSAM) compared to closed models or pre-existing technologies.

In October 2023, Partnership on AI released its Guidance for Safe Foundation Model Deployment. Developed through a participatory process with over 40 institutions, the Guidance offers customized sets of responsible practices for model providers to safely develop and deploy models, specific to the capabilities of the model they are developing and their release approach, which include Research Release, Closed Development, Restricted Access, or Open Access.

The Guidance calls for some form of openness from all model providers regardless of their release approach. For open access models, the guidelines span interventions from pre-deployment testing, risk identification and mitigation, to monitoring and reporting usage.

In the months since the release of the Guidance, stakeholders across the AI ecosystem have debated the merits of closed vs. open access models. PAI and GitHub jointly hosted an in-person workshop on April 11, 2024, convening 50 experts across industry, academia, and civil society to explore safeguards for state-of-the-art open foundation models and roles and responsibilities within the value chain.

The workshop opened with remarks from Bertram Lee, policy specialist at the National Telecommunications and Information Administration (NTIA), following the agency’s far-reaching public comments process on open foundation model governance that solicited inputs widely.

By bringing together diverse voices, PAI is able to integrate insights, recommendations, and concerns from key actors and thoughtfully approach potential updates to its Model Deployment Guidance for the upcoming year. Two primary topics discussed at the workshop were (1) risk mitigation strategies for open foundation models and (2) roles within the open foundation model value chain.

Risk Mitigation Strategies for Open Foundation Models

Open foundation models offer significant benefits, such as increased transparency, accountability, innovation, and competition in downstream markets. However, the open nature of these models can make it more challenging to assess and mitigate these risks. Concerns arise when the key “building blocks” of AI models known as “model weights”, are released openly. As models become increasingly capable of generating realistic audio, video and images, or even executing complex tasks, chances of malicious use when model weights are released are also heightened. This includes risks like generating non-consensual intimate imagery or fake content for harassment or fraud.

While these risks are not necessarily unique to open models and also exist in closed models, releasing the “building blocks” or model weights lowers the barriers for malicious actors to repurpose and fine-tune the models for harmful applications. This compared to closed models where providers have more control over access to these foundational components.

The challenges of limited post-deployment control and the complexities of monitoring and moderating usage don’t negate the importance of implementing and evaluating responsible AI practices. It’s crucial to invest in new safeguards, whether technical or policy, across the value chain for the open foundation model ecosystem. These efforts can help reduce potential harms associated with the open availability of these models, even if complete control and oversight are not feasible. A more balanced approach would involve proactively developing risk mitigations while enabling open access models to drive innovation and competition in downstream markets.

The core question addressed at the workshop was: How can we actively pursue harm reduction strategies for open foundation models without hindering their accessibility? And what can we learn from other fields that have advanced safety measures while preserving the underlying technology’s benefits? This approach to harm reduction can go beyond scrutinizing just the release decision by upstream model providers; we must also examine the roles and responsibilities of other key actors throughout the open foundation model lifecycle.

Roles in the Open Foundation Model Value Chain

The current generation of foundation models involves multiple handoffs, from model development to hosting and integration into applications. When we released the Model Deployment Guidance last year, PAI focused solely on safety practices applicable to those training and releasing foundation models: the model provider layer.

However, it is increasingly apparent that responsibility must be distributed across the value chain, since each layer has certain access and control over the resulting technology. Another key question at the workshop was: What is the value chain for open foundation model governance, and which actor(s) are best suited to implement specific guidance to address risks?

What’s Next

In the run-up to the second AI Safety Summit in South Korea, we will publish detailed takeaways expanding on stakeholder perspectives around the two core questions explored at the workshop. We appreciate the invaluable inputs provided by experts at the workshop.

To stay up-to-date on the latest news from PAI, including updates to the Guidance for Safe Foundation Model Deployment, you can sign up for our newsletter.

Thanks to the Partnership on AI team for their contributions to the workshop: Kasia Chmielinksi, Jiyoo Chang, Neil Uhl, Jessica Gratton,and Stephanie Bell. We’re grateful to GitHub for co-hosting this important convening and providing insights that helped shape the discussions.