As part of PAI’s ABOUT ML initiative, Research Fellow Jiyoo Chang has conducted research to better understand organizational challenges of implementing machine learning (ML) documentation practices at scale. Through conversations with stakeholders, she has found that one of the biggest implementation challenges is getting buy-in and understanding of ML documentation as a source of value for their organization. PAI recently hosted a webinar to explore the value of ML documentation from a return on investment perspective.
Implementation Challenges for ML Documentation: Q&A with PAI Research Fellow Jiyoo Chang
During the webinar, Jiyoo presented findings from her current research, which focuses on understanding ML documentation practices on the ground and developing implementation strategies.
Her presentation highlighted three of the key barriers organizations face when adopting and implementing documentation practices at scale: (1) Organizational culture, (2) technical barriers, and (3) practitioner challenges. Some important questions raised during the webinar were:
1. What was the most surprising thing you found so far in your interviews?
Implementing and scaling ML documentation across an organization cannot be done by one person or team, but requires a company-wide approach. It is crucial to get buy-in from executives to encourage adoption across the organization, managers to allocate time into the project scope, designers to make documentation readable and usable, and engineers to document their work through the ML development lifecycle.
2. What is one of the best methods or approaches to overcoming the challenges you identified?
One of the best approaches to integrate documentation in the day-to-day practice is to scope it out in the project plan. For instance, establish documentation as part of the deliverables or milestones for the projects, or have meetings around shared documentation amongst the team. Other practitioners also noted that creating documentation on collaborative knowledge management tools allow cross-functional collaboration and provide a centralized location for knowledge sharing within the organization.
3. In your presentation, you noted that one Practitioner challenge is predicting downstream risks and harms in AI systems, which arises due to the complexity of the systems. How are folks currently tackling this challenge?
Complex systems produce complex consequences. While predicting downstream risks of any emerging technology can be challenging, AI Incident Database provides insights into problems experienced in the real world so that future researchers and developers may mitigate or avoid repeated bad outcomes.
4. Is there anyone or any organization that is doing an exemplary job at ML documentation?
Numerous efforts have happened across the field with several organizations proposing templates and frameworks for ML documentation, such as Google’s Model Cards, IBM Factsheets, and the Data Nutrition Labels. The ABOUT ML Resources Library provides resources and recommendations for teams implementing documentation and transparency practices at scale.