Operationalizing AI Ethics Through Documentation: ABOUT ML in 2021 and Beyond

The following is an update on the ABOUT ML Research Portfolio for the year. However, before we share some exciting details about the work we plan to do, we would like to encourage you to follow this link to review some information about the origin of this work and the amazing researchers who helped from the very beginning.

Notably, Hanna Wallach, Meg Mitchell, Jenn Wortman Vaughan, Timnit Gebru, Lassana Magassa, and Jingying Yang were instrumental in the foundations of the work and we thank them for their significant contributions. Francesa Rossi and Kush Varshney, both from IBM, and Eric Horvitz at Microsoft were also key contributors in making this work possible. Please read more about the origin story here.

We highlight this as an example of where Partners doing work across the ecosystem saw PAI as a place where various organizations could work together toward a common goal with specific outcomes. All of our work is informed, supported, contributed to, and communicated by our Partner organizations. Being a multistakeholder organization allows us as a community to spot trends and move forward with essential work that will benefit people and society.

 Technical standards form the backbone of many of the world’s most trusted technologies. Few people think about this when they get on a plane, plug in a new appliance, or securely enter information into a website: they just know it works. Ultimately, this confidence is made possible by specifications which codify industry best practices. With ABOUT ML, the Partnership on AI (PAI) is leading a multistakeholder effort to develop similar guidelines for the documentation of machine learning (ML) systems, setting new industry norms for transparency in artificial intelligence (AI). This means not just identifying the necessary components of transparency, but developing actionable resources for implementing transparency at scale.

In addition to the core ABOUT ML reference document, which provides starting principles for achieving transparency through documentation, this work has already resulted in workshops, scientific investigations, conference showcases, and published papers. The coming year will see the release of even more resources to help organizations operationalize transparency. Developed through an iterative, multistakeholder process, these artifacts pool the collective efforts and insights of academic researchers, industry practitioners, civil society organizations, and the impacted public.

Having established, with our previous work, a foundational basis of what ML transparency means to various stakeholders, we are advancing this project in 2021 by determining how people, processes, and tools must adjust to meet this definition. This will serve to prepare organizations for standards, regulation, and even certification in this space. In this post, you will learn more about our completed, in-progress, and future plans for ABOUT ML (Annotation and Benchmarking on Understanding and Transparency of Machine learning Lifecycles) as well as how to get involved.

Why documentation?

Amid calls across the AI community for greater transparency, responsibility, and accountability, many organizations are exploring ethical principles to support these goals. Fully operationalizing these principles, however, is a complex problem that is difficult to solve alone. Moreover, any uncoordinated effort is unlikely to satisfy the diverse needs of those creating, using, and impacted by ML systems.

A full lifecycle strategy for documenting ML systems can help bridge the gap between principle and practice. Properly conceived, documentation can both anticipate potential issues before they arise and produce the artifacts needed for future transparency. Through collaboration, we can further ensure that these documentation processes satisfy actual needs while reducing redundant workstreams between organizations.

By bringing together a diverse range of perspectives to develop, test, and implement machine learning system documentation practices, ABOUT ML is creating both scalable solutions for operationalizing AI ethics and the guides necessary to implement them.

About ABOUT ML

At the core of this project is the ABOUT ML reference document, created and updated through an iterative, multistakeholder process. Version 0 (v0) of this document, released in 2019, provided initial recommendations for ML documentation drawn from the latest research. The purpose of v0 was to promote community discussion, captured through public comment, PAI Partner input, and a Diverse Voices panel collecting perspectives that might not otherwise be included.

The process of updating the document has been guided by the ABOUT ML Steering Committee, which has included dozens of experts, researchers, and practitioners drawn from a diverse set of PAI Partner organizations. In the coming months, PAI will be releasing Version 1 (v1) of the ABOUT ML reference document, a stable resource synthesizing community feedback on v0.

PAI is also currently engaged in multiple ABOUT ML research workstreams. One of these is focussed on challenges for implementing documentation practices within ML development and deployment teams. Another focuses on documentation templates currently in use and how well they are working for various stakeholder audiences. These workstreams, along with other ongoing research covering related topics in the Fairness, Transparency, and Accountability (FTA) Program at PAI, will provide further guidance on how to operationalize the principles of the reference document.

Together, the research workstreams and the reference document will inform the future ABOUT ML Playbook, a collection of documentation specifications, guides, and templates that can be used by ML practitioners, procurers, policy developers, and other stakeholders. These artifacts can then be implemented in ABOUT ML Pilots which will test promising interventions.

2021 is poised to be a productive year for this exciting approach to achieving AI transparency. As we work toward creating actionable resources for industry change, we welcome all stakeholders in AI transparency to get involved with ABOUT ML.

ABOUT ML v1

In the spring of 2019, PAI launched the ABOUT ML project, aimed at building transparency into the AI development process, industry-wide, through full lifecycle documentation. At the foundation of this project lies the ABOUT ML reference document, which both identifies transparency goals and offers suggestions on how they might be achieved.

Created and updated through an iterative, multistakeholder process, this document synthesizes the latest research, practitioner experience, and community insights. In the coming months, PAI will release the latest iteration of this text, ABOUT ML v1, which incorporates feedback on the initial draft from, among others, a Diverse Voices panel conducted by the ABOUT ML Steering Committee.

ABOUT ML v1 will serve as a stable resource for individual champions who wish to advocate for documentation of ML systems, whatever their role. This includes developers, external auditors, internal compliance teams, ML system procurers, and end users of all types. Additionally, this text will inform subsequent ABOUT ML artifacts, such as documentation guides, templates, and recommendations, as well as open research questions surrounding implementation.

Research Workstreams

ABOUT ML seeks to provide not just strong principles for the documentation of ML systems, but ones that will successfully increase ML transparency in practice. In pursuit of this goal, PAI is currently pursuing multiple research workstreams in support of the ABOUT ML project.

This year, two new Research Fellows are joining PAI to study questions directly related to ML transparency. The first of these workstreams asks, “What are the organizational, technological, and other challenges to implementing ML documentation?” Through multistakeholder research conducted with PAI Partners, Research Fellow Jiyoo Chang is currently identifying these obstacles, work which will inform future evidence-based recommendations.

The second of these workstreams asks, “What visual layouts are most effective for presenting documentation information to different audiences?” To provide beneficial transparency, any ML documentation effort must consider the varying needs of stakeholders, a group which includes everyone from users, procurers, and deployers to public advocates, policymakers, and impacted nonusers. Research Fellow Surya Karunagaran will study highly usable documentation templates that meet the needs of these audiences.

In addition to this research, other, ongoing PAI workstreams will further inform our ABOUT ML and FTA work:

• By identifying unapparent risks of AI systems, research relevant to the AI Incidents Database will help shape what questions should be asked during the documentation process.
• In addition to drawing lessons from other industries that have successfully implemented documentation, the ABOUT ML Team will incorporate findings from the Demographic Data workstream within the FTA Program to suggest best practices for handling sensitive data.
• By broadening the scope of stakeholder inputs, the Methods for Inclusion workstream will further illuminate what information different parties deserve to know.
• And by investigating pervasive challenges to ethnic, gender, and cultural diversity in the field of AI, the Diversity, Equity, and Inclusion workstream will help organizations pursue transparency efforts that are shaped by inclusive workspaces.

ABOUT ML Playbook and Pilots

In the future, the current and completed work described above will become the basis of functional guides for implementing ML documentation — and ultimately lead to testable pilots of the ABOUT ML principles.

Using principles provided by the reference document and insights about implementation gathered through our research workstreams, PAI plans to begin releasing ML documentation guides, templates, recommendations, and other artifacts later this year. Together, these will form the ABOUT ML Playbook, a collection of actionable resources for increasing the transparency of AI systems.

These Playbook artifacts, tailored to specific actors and contexts, will provide individuals with concrete instructions for championing ML documentation. Ultimately, various Playbook resources will be packaged together for application to individual use-cases, creating ABOUT ML Pilots for organizations to test ML documentation in practice. Feedback from these real-world Pilots will help identify what practices work best, allowing ABOUT ML to iterate as additional Pilots are launched.

How to get involved

As we work toward translating ABOUT ML into actionable resources for industry change, there are multiple ways you can get involved:

• Stay connected with the work by signing up on our Expression of Interest form.
• Comment on and share your Organizational Challenges.
• Give feedback on deployed examples of ML documentation.

Together, we change the AI industry through equitable machine learning practices