As a growing number of technology companies realize the importance of AI ethics and create AI principles that embody their values, it remains a struggle to uphold these principles in practice. Resource-constrained teams, engineers, product managers, and others involved in AI development – all working towards tight deadlines and ambitious quarterly goals – are often unrewarded for the due diligence they invested in the development of best practices for operationalizing ethical principles. This tension between developing best practices and delivering on business demands can lead to burn out for the individuals who champion responsible AI initiatives.
Ultimately, AI practitioners are faced with a choice: develop best practices which uphold ethical principles at the risk of business deadlines, or prioritize business deliverables without the application of ethical AI practices. Either scenario leaves a gap between a company’s good intentions (principles) and real-world outcomes (practice).
One way to close the gap between principles and practice in AI is through documentation for machine learning systems at scale. To produce documentation at scale means that accurate and useful documentation will be created for every ML system, product, and API that an organization ships. By asking the right question at the right time via documentation in the AI development process, teams are more likely to identify potential issues and take appropriate mitigating actions. For example, data documentation asking “Have users given consent to be in this dataset?” before beginning data collection reminds the data collector to ensure that proper consent will be obtained. Asking the same question after a full ML system has been trained and tested leads to a different set of mitigation options, but would still help to avoid the situation of data misuse that may violate company policy or even regulations.
Thus, documentation is both an artifact (e.g. what questions should be asked for which types of ML systems, how should this information be presented to maximize usefulness to developers, consumers, and regulators, etc.) and a process (e.g. when should those questions be asked, how does a team incorporate documentation into their workflow, how can organizations incentivize people to prioritize documentation, what is the review process for making sure documentation is filled out correctly, etc.). These questions are challenging and resource-intensive to answer in any context. The disparate efforts at different organizations to solve this question can create redundant work, which is why PAI launched the ABOUT ML initiative to connect the existing efforts and accelerate progress.
Since launching the version 0 draft of ABOUT ML in July, many of our Partners across the extended AI community have expressed interest in the initiative. Partners articulated a resounding need to create an industry norm around documentation in machine learning systems that increase adoption and scaling of documentation practices, and to do so with the input of people impacted by their technologies.
Many organizations have responsible AI principles. But turning these principles into practice is a challenging task that, at its core, requires a fundamental culture change. One way of bringing about this culture change, industry-wide, is by prioritizing shared mental frameworks and effective communication, as in the ABOUT ML project.
Our ABOUT ML initiative is tasked with creating an actionable set of recommendations to enable organizations implement documentation at scale. In accordance with how PAI operates, ABOUT ML is a multistakeholder process that incorporates feedback from not only experts but also people directly impacted by the technologies and the general public at large so that the recommendations reflect the concerns and priorities of a truly diverse and multidisciplinary set of perspectives.
To reflect that documentation is both an artifact and a process, ABOUT ML’s end products will be split into 2 topics for resources: 1) an artifact, which will include templates and a database of documentation questions, adapted by features like domain of application and type of model and 2) a process, which will include a research-based guide to initiating and scaling a documentation pilot.
Documentation can help bridge the gap between principles and practice because it is adaptable to an organization’s needs. Teams will be able to prioritize sections in documentation templates. Moreover, we see implementing ABOUT ML’s process recommendations as a first step towards creating organizational cultural infrastructure for upholding responsible AI principles because the process will include org-wide incentive structures, review processes, and buy-in from every level, all of which can serve as the foundation for expanding other AI ethics initiatives.
Over the course of 2020, PAI is excited to continue research on organizational enablers for documentation practices, gather public comment on which questions to include in documentation, adapt questions for different domains, and finally pilot ABOUT ML recommendations with Partners. As we work to release “version 1” of ABOUT ML in early 2020, we will need your unique perspectives to ensure its success. Ahead of the next draft of ABOUT ML, we ask that the extended community share with us successful examples of documentation and sign up to become involved in this work.
At PAI, and with ABOUT ML, we endeavor to spread a culture of responsibility for the field of artificial intelligence. This culture is made possible when we elevate the unglamorous yet deeply important task of documentation.
