Appendix A: Initial Discovery Process and Getting Reactions to PAI Responsible Sourcing Recommendations

Appendix A: Initial Discovery Process and Getting Reactions to PAI Responsible Sourcing Recommendations

To understand the types of resources that would realistically help DeepMind’s research and development teams implement the recommendations from PAI’s white paper, we wanted to understand how the company and individual teams within it were approaching data enrichment and, more specifically, how they were working with data enrichment workers.

We began by having conversations with the Responsible Development and Innovation team, a team with a mandate to ensure research is done responsibly. As the team that oversees the review process for internal teams setting up projects involving human subjects, they were also driving the company’s initiative to build a parallel set of guidelines for projects requiring enriched data. Given this team’s mandate and knowledge of DeepMind’s internal research teams, they were able to share deep insights into how teams currently approached data enrichment and how they may respond to any proposed changes. Being able to incorporate their feedback into the guidelines early on allowed us to make early adjustments and present more refined resources to researchers. Doing this upfront saved us time because it allowed us to get more substantive and targeted feedback from the researchers who would be using the guidelines. During this initial review process, the implementation team re-ordered the guidelines to make them more user-friendly, adjusted the language on some of the guidelines to make it clearer to the target audience, learned more about how the teams were currently approaching data enrichment, identified potential areas where additional guidance may be needed to effectively implement the guidelines, and identified specific follow-ups where the implementation team needed direct feedback from researchers.

After incorporating this initial feedback into the guidelines, the implementation team began engaging with a broader group of stakeholders who would be able to provide different perspectives on what organizational changes might be required to operationalize these recommendations. Additionally, involving a broad range of stakeholders from across the company helped get people acquainted with the guidelines prior to the official roll out, allowed the implementation team to address concerns up front, and helped us get early buy-in from the teams who would be using the new guidelines. We sought feedback by presenting the guidelines to the internal group who would make up the Human Data Review Group for any project involving enriched data, presenting the guidelines to a broader group of researchers working with human data, making the guidelines available for people to leave comments and questions, and conducting one-on-one interviews with various stakeholders to have more targeted conversations. Along the way, the implementation team continued to incorporate feedback, resolve questions, and reach out to relevant stakeholders to resolve uncertainties as they came up.

One of the primary reasons we wanted to collect feedback from researchers and other stakeholders was to identify how to bridge the gap between current practice and recommended practice at an organizational level. By talking to those involved with setting up data enrichment projects at DeepMind, the implementation team was able to develop a deeper understanding of the types of resources that would enable researchers to consistently meet the recommended guidance. This feedback informed the content created, either as a part of the guidance itself or in the form of additional resources linked out of the guidance. In some cases, the feedback led us to identify external (to the company) barriers that would make it difficult for researchers to adopt aspects of our recommendations.