Policy Alignment on AI Transparency

Analyzing Interoperability of Documentation Requirements across Eight Frameworks

As governments and organizations worldwide race to develop policy frameworks for foundation models, we face a juncture that demands both swift action and careful coordination. However, without coordination, we risk creating an inconsistent patchwork of frameworks and divergent understandings of best practices.

Ensuring these frameworks work together is critical.

Partnership on AI’s Policy Alignment on AI Transparency conducts a comparative analysis of eight leading policy frameworks for foundation models, with a particular focus on documentation requirements, which are a critical lever for achieving transparency and safety.

In this report, we analyze current and potential near-term interoperability challenges between the documentation requirements in leading policy frameworks, and offer recommendations that aim to promote interoperability as well as establish a common baseline for best practices for accountability and transparency. However, we recognize that this is only the beginning of a much larger conversation. Achieving global interoperability will require ongoing efforts and substantial input, particularly from the Global Majority, to reflect diverse perspectives and priorities.

The full report provides a comprehensive exploration of interoperability challenges, including the nuances of our methodology, key findings, and detailed recommendations. This summary serves to highlight our most salient conclusions, aiming to inform and guide ongoing policy discussions and decision-making in this rapidly evolving field.

As we navigate the complex landscape of AI governance, the need for coordinated, interoperable policy frameworks becomes increasingly clear. By working together across borders and sectors, we can create a more coherent, effective approach to managing the risks and harnessing the potential of foundation models, ensuring accountability, transparency and fostering innovation in the global AI ecosystem.

⇩ Jump to Summary of Recommendations ⇩

Read the Report

Download the complete Policy Alignment on AI Transparency report (41 pages) as a PDF.
Download the full report

Without coordination, we risk creating an inconsistent patchwork of frameworks and divergent understandings of best practices.

Frameworks Reviewed

Documentation and transparency play a key role in managing risk for foundation models, and are a common feature of policy frameworks.

To explore how documentation guidance is being incorporated in current policy frameworks, we compared key frameworks from the US, EU, UK, and select multilateral initiatives (see Table 1). The table outlines each framework’s provisions for foundation model documentation, ranging from high-level transparency guidelines to specific artifact requirements. It also indicates ongoing development processes for each framework.

Table 1. Frameworks reviewed in this report

	A	B	C	D	E
Multilateral
OECD AI Principles	•
Seoul Frontier AI Safety Commitments	•	•			•
Hiroshima AI Process Code of Conduct		•	•	•	•
Council of Europe AI Convention		•
Regional
EU AI Act			•	•	•
National
US AI Executive Order 14110			•	•	•
NIST AI RMF (with Gen-AI Profile)		•		•	•
UK AI White Paper and followup	•			•

A: Contains high-level commitments to transparency
B: Requires/recommends documentation practices
C: Requires documentation artifacts
D: Further/more detailed provisions proposed or in development
E: Specifically addresses foundation models

Mapping Documentation Requirements Across the Frameworks

Documentation guidance and requirements under the reviewed frameworks are summarized in Table 2A below. Key findings include:

Documentation is a common feature of the frameworks, though this is couched in various terms. Several of the high-level frameworks recommend providing certain kinds of information to various actors; some require recording and/or reporting of information, and some require the preparation of specific documentation artifacts.
The most commonly referenced artifacts are technical documentation, instructions for use, information about datasets, and incident reports. However, there is little detail in most of the frameworks about what should be included in each of these documents and the form each document should take.
This analysis suggests that there is an opportunity to develop standardized requirements for some of the key documentation artifacts required across frameworks – provided that agreement can be reached about what the content of these artifacts should be.

Tables 2A, 2B and 2C contain a comparison of documentation requirements across in-scope frameworks. Specific documentation artifacts are shown in red. The principal documentation guidelines from PAI’s Model Deployment Guidance are included as a comparator.

Table 2A. Comparison of documentation requirements across in-scope frameworks

Stage in AI Lifecycle	Framework
Stage in AI Lifecycle	PAI Model Deployment Guidance	EU AI Act	AI Executive Order	NIST RMF and Generative AI Companion	Hiroshima Code of Conduct	Seoul Frontier AI Commitments	COE Convention	OECD AI Principles	UK AI White Paper, AI Principles, Response
R&D	•	•	•
Pre-deployment/ on deployment	• •	• •	•	•	• •
Post-deployment	•	• •		•	• •
Across lifecycle	•			•	• •
Unspecified						•	•	•	•

• Documentation requirements for specific stage in the AI lifecycle
• Specific documentation artifacts
• General documentation requirements

Table 2B. Comparison of documentation requirements across in-scope frameworks
Specific documentation artifacts are shown in red. The principal documentation guidelines from PAI’s Model Deployment Guidance are included as a comparator.

Stage in AI Lifecycle	Framework
Stage in AI Lifecycle	PAI Model Deployment Guidance	EU AI Act	AI Executive Order	NIST RMF and Generative AI Companion	Hiroshima Code of Conduct
R&D	Pre-system card: Planned testing, evaluation, and risk management procedures for foundation/ frontier models prior to development. Including: Intended training data approach Responsible AI practices Development Team Written “safety case”	Notify EU Commission of models with systemic risk	Report dual-use models to Department of Commerce; report cybersecurity protections	N/A	N/A
Pre-deployment/ on deployment	Publicly report model impacts “Key ingredient list”: including details of evaluations, limitations, risks, compute, parameters, architecture, training data approach, model documentation Disclose performance benchmarks, intended use, risks and mitigations, testing and evaluation methodologies, environmental and labor impacts Downstream use documentation: including appropriate uses, limitations, mitigations, safe development practices Share red-teaming findings	Technical documentation: including information about training, testing, and evaluations Documentation for downstream developers: including information about capabilities, limitations, and to aid downstream compliance Public summary of training data	Report red-teaming results to Department of Commerce	Multiple guidelines for documentation, including of: Risks and potential impacts Knowledge limits TEVV considerations & tools Measures of trustworthiness Residual risks after mitigations Model details Data curation policies Environmental impacts	Technical documentation Transparency reports: with “meaningful information” Instructions for Use Technical Documentation Documentation to include details of evaluations, capabilities/ limitations re: domains of use; risks to safety and society; red-teaming results
Post-deployment	Incident reporting Transparency reporting (frontier model usage and policy violations)	Serious incident reports	N/A	Incident and performance reporting Transparency reports with steps taken to update generative AI systems	Maintain “appropriate documentation” of reported incidents
Across lifecycle	Iteration of model development Provide documentation to government as required Environmental Impacts Severe labor market risks Human rights impact assessments	N/A	N/A	Multiple guidelines to document processes and management systems	“Work towards” information sharing and incident reporting, including on: Evaluation reports Safety & security risks “ensuring appropriate and relevant documentation and transparency across the AI lifecycle” Document datasets, processes and decisions during development Regularly update Technical Documentation

Table 2C. Comparison of more general documentation and transparency requirements, at unspecified stages of the AI lifecycle

Framework
Publicly report model or system capabilities, limitations, and domains of appropriate and inappropriate use Provide public transparency on implementation of commitments, including on: Risk assessments, effectiveness of mitigations, evaluation results Risk thresholds Approach to mitigations Processes to follow if risk thresholds are met/ exceeded	Countries ratifying the convention must have frameworks (such as national laws) that: Contain documentation requirements that will allow people to seek remedies for human rights violations Require developers to adopt measures to identify, prevent, and mitigate risk. These measures are to include documentation of risks and mitigations	Principles include: Transparency and Explainability: “Provide meaningful information” to “foster understanding of AI Systems” “Provide plain and easy-to-understand information on the sources of data/input, factors, processes and/or logic” “Provide information [to] enable those adversely affected by an AI system to challenge its output.” Accountability: “Ensure traceability, including in relation to datasets, processes and decisions made during the AI system lifecycle”	Provide transparency and accountability, including “documentation on key decisions throughout the AI system life cycle”

Framework

Seoul Frontier AI Commitments

COE Convention

OECD AI Principles

UK AI White Paper, AI Principles, Response

Publicly report model or system capabilities, limitations, and domains of appropriate and inappropriate use

Provide public transparency on implementation of commitments, including on:

Risk assessments, effectiveness of mitigations, evaluation results
Risk thresholds
Approach to mitigations
Processes to follow if risk thresholds are met/ exceeded

Countries ratifying the convention must have frameworks (such as national laws) that:

Contain documentation requirements that will allow people to seek remedies for human rights violations
Require developers to adopt measures to identify, prevent, and mitigate risk. These measures are to include documentation of risks and mitigations

Principles include:

Transparency and Explainability:

“Provide meaningful information” to “foster understanding of AI Systems”
“Provide plain and easy-to-understand information on the sources of data/input, factors, processes and/or logic”
“Provide information [to] enable those adversely affected by an AI system to challenge its output.”

Accountability:

“Ensure traceability, including in relation to datasets, processes and decisions made during the AI system lifecycle”

Provide transparency and accountability, including “documentation on key decisions throughout the AI system life cycle”

Other Features of the Frameworks Relevant to Interoperability

In reviewing the in-scope frameworks, a number of additional factors emerged including their binding nature, enforcement mechanisms, scope of applicable models, overseeing institutions, development processes, and emphasis on collaboration, as detailed in Table 3.

Table 3. In-scope frameworks
Normative status, coverage/thresholds, reference to international standardization processes and collaboration/interoperability

Framework	PAI Model Deployment Guidance	EU AI Act	AI Executive Order	NIST RMF and Generative AI Companion	Hiroshima Code of Conduct	UK AI White Paper, Consultation Response	OECD	Seoul frontier AI Commitments
Binding or Voluntary?	Voluntary	Binding	Partly binding	Voluntary	Voluntary	Voluntary (guidance for sectoral regulators)	Voluntary	Voluntary
Coverage	Foundation models (with guidance tailored according to three capability bands and four release strategies). The most stringent guidance applies to “paradigm-shifting or frontier” models	General- purpose AI models (baseline requirements) “General- purpose AI models with systemic risk”	“Dual-use foundation models”	AI systems (NIST AI RMF) Generative foundation models (Gen-AI Profile)	“The most advanced AI systems, including the most advanced foundation models and generative AI systems”	AI systems; generally a sectoral approach Initial focus of UK AISI on advanced systems Planned laws for “the most powerful AI systems”	AI Systems	“Frontier AI” – “highly capable general- purpose AI models or systems that can perform a wide variety of tasks and match or exceed the capabilities present in the most advanced models”
Initial Threshold	N/A	None (baseline requirements) 10^25 FLOPs (models “with systemic risk”)	10^26 FLOPs (10^23 FLOPs for models trained on biological sequence data)	N/A	N/A	N/A	N/A	N/A
Institutions/ Oversight	N/A	AI Office	Dept of Commerce (for reporting requirements)	N/A	OECD (monitoring mechanism under development)	AISI	N/A	N/A
Next Steps	N/A	Codes of Practice for GPAI due August 2025 Templates for training data (AI Office) Harmonized standards Delegated acts – thresholds for GPAI with systemic risk; documentation requirements	Various, including: OMB materials for federal procurement Copyright guidance Dept of Commerce can change threshold for dual-use model reporting	NIST/the NIST AISI have a broad work plan including developing tools, evaluations, metrics	COC to be iterated by G7 HAIP OECD developing monitoring mechanism	Intention to legislate announced re advanced models, and to place AISI on statutory footing	OECD developing Due Diligence Guidance (DDG) for AI under OECD Responsible Business Conduct (RBC) guidelines	AI Action Summit February 2025 (France) AI Safety Science Report to be published at AI Action Summit
Commitment to cooperation/ collaboration?	Collaborate with cross-sector Al stakeholders re risk identification, methodologies, best practices, standardization	Mandates creation of AI Board, Advisory Forum; multistakeholder participation in development of Codes of Practice and harmonized standards	Under EO, NIST released plan for global engagement on AI standards; Secretary of State is developing Global Development Playbook; EO contained several consultation requirements	Several references to collaboration e.g. with external researchers, industry experts, and community representatives about best risk measurement and management practices NIST is committed to collaboration/ cooperation, e.g. through AISI Consortium and pending Network of AISIs	Across sectors, including on research to assess/ adopt risk mitigations, document incidents, and share information with the public to promote safety	Focus on collaboration across government, stakeholder groups, and internationally	OECD convenes the Network of Experts	Information sharing, collaboration on safety research (Seoul AI Principles)
Commitment to standards	Development and adoption of standards	EU harmonized Standards—though EU committed to adopting international standards where possible	NIST is required to develop standards Under EO, NIST has released plan for global engagement on promoting and developing AI standards	Contains references to considering relevance of standards (including NIST frameworks) NIST will continue to align AI RMF with international standards	Advance development and adoption of standards	Support for work on assurance techniques and technical standards	Governments should promote standards development	Contribute to/ take account of international standards

Summary of Findings

Interoperability and collaboration are explicitly included as policy goals in a number of the international frameworks, though there is no current agreement on how those goals will be achieved.
The frameworks emphasize the importance of documentation for foundation models, but remain light on detail about the form and content it should take.
There are a number of steps that could be taken to advance interoperability now and in the future, leveraging existing and proposed forums, mechanisms and processes.
An early focus should be on agreeing upon thresholds for regulation, to provide international consistency about which foundation models are captured by regulatory and policy frameworks.
While there are some challenges to relying on international standardization processes to align AI policy frameworks, standards remain an important plank in that effort.
Harmonizing key documentation requirements across national, regional, and international foundation model policy frameworks – and in particular, harmonizing the form and content of documentation artifacts – should be made a priority.
The lack of consensus on the best approaches to manage AI risks is a significant challenge to developing interoperable frameworks, including for documentation.
The existing and newly announced AI Safety Institutes can establish a foundation for AI safety consensus through research, evaluations, scientific advancement, and collaborative development of safety standards and documentation practices.
Participation by civil society and the global community is needed in all major foundation model policy initiatives if we are to ensure that they lead to alignment around best practices, and that the agenda for global interoperability is not set by a comparatively small group of nations from the Global North.

Participation by civil society and the global community is needed in all major foundation model policy initiatives.

Summary of Recommendations

National governments and the EU should prioritize cooperation in setting thresholds for identifying which foundation models require additional governance measures, including through supporting the OECD’s work on this issue. The AI Summit Series could also be used to take this forward. Agreeing on a common definition, and thresholds, for the models covered by policy frameworks, should flow through to greater alignment between the frameworks, including in relation to documentation requirements.
The G7 Presidency should continue developing the Hiroshima Code of Conduct into a more detailed framework to provide more detail about thresholds, relevant risks, and the form and content of documentation artifacts. This work should be a focus of Canada’s G7 Presidency in 2025, including aligning closely with the EU Code of Practice development timeline. In doing this, it should seek input from foundation model providers, civil society, academia and other stakeholder groups equally.
When creating and approving initial Codes of Practice for the EU AI Act, all involved parties should prioritize compatibility with other major AI governance frameworks where possible. The involvement of non-EU model providers, experts and civil society organizations will help advance this objective.
To support the development of standardized documentation artifacts, Standards Development Organizations should ensure that their processes are informed by socio-technical expertise and diverse perspectives as well as required resources. To that end, SDOs, industry, governments, and other bodies should invest in capacity building for civil society and academic stakeholders to engage in standards-making processes, including to ensure participation from the Global South.
The development of standardized documentation artifacts for foundation models should be a priority in AI standardization efforts. This would promote internationally comparable documentation requirements for foundation models – promoting interoperability and establishing a baseline for best practice internationally.
International collaboration and research initiatives should prioritize research that will support the development of standards for foundation model documentation artifacts. Documentation is a key feature of foundation model policy requirements, and common requirements for artifacts will directly improve interoperability. It will also make comparisons between models from different countries easier, promoting accountability and innovation.
National governments should continue to prioritize both international dialogue and collaboration on the science of AI Safety, however with more specificity and tracking of progress on commitments that will foster good practice. This work will inform a common understanding of what should be included in documentation artifacts to promote accountability and address foundation model risks.
National governments should support the creation/development of AI Safety Institutes (or equivalent bodies), and ensure they have the resources, functions, and powers necessary to fulfill their core tasks. Efforts should be made to align the functions of these bodies with those common among existing AISIs. This will promote efforts to develop trusted mechanisms to evaluate advanced foundation models, and may, at a later stage, lead to the potential to work towards “institutional interoperability.”
The Network of AISIs (and bodies with equivalent or overlapping functions such as the EU AI Office) should be supported and efforts should be made to expand its membership. Consideration should be given to how the Network could support broader AI Safety research initiatives.

Background and Methodology

The work plan leading to this report was developed with guidance from PAI’s Policy Steering Committee. This report has been informed through desk research and consultations with experts from industry, civil society, academia, and non-profit organizations, drawn from PAI’s partner and wider stakeholder networks. We tested our initial thinking in a virtual multistakeholder workshop in August 2024. The views and recommendations in this report remain those of PAI.

For a comprehensive exploration of interoperability challenges, including our methodology, key findings, detailed recommendations, and more, please download the full report. To stay in touch with our latest policy work, sign up here.

DOWNLOAD THE FULL REPORT

The development of standardized documentation artifacts for foundation models should be a priority in AI standardization efforts.

ABOUT ML in Practice: The Example of Intuit

Neil Uhl

June 6, 2024

Eyes Off My Data

Exploring Differentially Private Federated Statistics To Support Algorithmic Bias Assessments Across Demographic Groups

PAI Staff

December 13, 2023

Executive Summary

Designing and deploying algorithmic systems that work as expected every time for all people and situations remains a challenge and a priority. Rigorous pre- and post-deployment fairness assessments are necessary to surface any potential bias in algorithmic systems. As they often involve collecting new user data, including sensitive demographic data, post-deployment fairness assessments to observe whether the algorithm is operating in ways that disadvantage any specific group of people can pose additional challenges to organizations. The collection and use of demographic data is difficult for organizations because it is entwined with highly contested social, regulatory, privacy, and economic considerations. Over the past several years, Partnership on AI (PAI) has investigated key risks and harms individuals and communities face when companies collect and use demographic data. In addition to well-known data privacy and security risks, such harms can stem from having one’s social identity being miscategorized or data being used beyond data subjects’ expectations, which PAI has explored through our demographic data workstream. These risks and harms are particularly acute for socially marginalized groups, such as people of color, women, and LGBTQIA+ people.

Given these risks and concerns, organizations developing digital technology are invested in the responsible collection and use of demographic data to identify and address algorithmic bias. For example, in an effort to deploy algorithmically driven features responsibly, Apple introduced IDs in Apple Wallet with mechanisms in place to help Apple and their partner issuing state authorities (e.g., departments of motor vehicles) identify any potential biases users may experience when adding their IDs to their iPhones.＊IDs in Wallet, in partnership with state identification-issuing authorities (e.g., departments of motor vehicles), were only available in select US states at the time of the writing of this report.

In addition to pre-deployment algorithmic fairness testing, Apple followed a post-deployment assessment strategy as well. As part of IDs in Wallet, Apple applied differentially private federated statistics as a way to protect users’ data, including their demographic data. The main benefit of using differentially private federated statistics is the preservation of data privacy by combining the features of differential privacy (e.g., adding statistical noise to data to prevent re-identification) and federated statistics (e.g., analyzing user data on individual devices, rather than on a central server, to avoid the creation and transfer of datasets that can be hacked or otherwise misused). What is less clear is whether differentially private federated statistics can attend to some of the other risks and harms associated with the collection and analysis of demographic data. To understand this, a sociotechnical lens is necessary to understand the potential social impact of the application of a technical approach.

This report is the result of two expert convenings independently organized and hosted by PAI. As a partner organization of PAI, Apple shared details about the use of differentially private federated statistics as part of their post-deployment algorithmic bias assessment for the release of this new feature.

During the convenings, responsible AI, algorithmic fairness, and social inequality experts discussed how algorithmic fairness assessments can be strengthened, challenged, or otherwise unaffected by the use of differentially private federated statistics. While the IDs in Wallet use case is limited to the US context, the participants expanded the scope of their discussion to consider differential private federated statistics in different contexts. Recognizing that data privacy and security are not the only concerns people have regarding the collection and use of their demographic data, participants were directed to consider whether differentially private federated statistics could also be leveraged to attend to some of the other social risks that can arise, particularly for marginalized demographic groups.

The multi-disciplinary participant group repeatedly emphasized the importance of having both pre- and post-deployment algorithmic fairness assessments throughout the development and deployment of an AI-driven system or product/feature. Post-deployment assessments are especially important as they enable organizations to monitor algorithmic systems once deployed in real-life social, political, and economic contexts. They also recognized the importance of thoughtfully collecting key demographic data in order to help identify group-level algorithmic harms.

The expert participants, however, clearly stated that a secure and privacy-preserving way of collecting and analyzing sensitive user data is, on its own, insufficient to deal with the risks and harms of algorithmic bias. In fact, they expressed that such a technique is not entirely sufficient for dealing with the risks and harms of collecting demographic data. Instead, the convening participants identified key choice points facing AI-developing organizations to ensure the use of differentially private federated statistics contributes to overall alignment with responsible AI principles and ethical demographic data collection and use.

This report provides an overview of differentially private federated statistics and the different choice points facing AI-developing organizations in applying differentially private federated statistics in their overall algorithmic fairness assessment strategies. Recommendations for best practices are organized into two parts:

General considerations that any AI-developing organization should factor into their post-deployment algorithmic fairness assessment
Design choices specifically related to the use of differentially private federated statistics within a post-deployment algorithmic fairness strategy

The choice points identified by the expert participants emphasize the importance of carefully applying differentially private federated statistics in the context of algorithmic bias assessment. For example, several features of the technique can be determined in such a way that reduces the efficacy of the privacy-preserving and security-enhancing aspects of differentially private federated statistics. Apple’s approach to using differentially private federated statistics aligned with some of the practices suggested during the expert convenings: the decision to limit the data retention period (90 days), allowing users to actively opt-into data sharing (rather than creating an opt-out model), clearly and simply sharing what data the user will be providing for the assessment, and maintaining organizational oversight of the query process and parameters.

The second set of recommendations surfaced by the expert participants primarily focus on the resources (e.g., financial, time allocation, and staffing) necessary to achieve a level of alignment and clarity on the nature of “fairness” and “equity” AI-developing organizations are seeking for their AI-driven tools and products/features. While these considerations may seem tangential, expert participants emphasized the importance of establishing a robust foundation on which differentially private federated statistics could be effectively utilized. Differentially private federated statistics, in and of itself, does not mitigate all the potential risks and harms related to collecting and analyzing sensitive demographic data. It can, however, strengthen overall algorithmic fairness assessment strategies by supporting better data privacy and security throughout the assessment process.

Executive Summary

Introduction

The Challenges of Algorithmic Fairness Assessments

Prioritization of Data Privacy: An Incomplete Approach for Demographic Data Collection?

Premise of the Project

A Sociotechnical Framework for Assessing Demographic Data Collection

Differentially Private Federated Statistics

Differential Privacy

Federated Statistics

Differentially Private Federated Statistics

A Sociotechnical Examination of Differentially Private Federated Statistics as an Algorithmic Fairness Technique

General Considerations for Algorithmic Fairness Assessment Strategies

Design Considerations for Differentially Private Federated Statistics

Conclusion

Acknowledgments

Funding Disclosure

Appendices

Appendix 1: Fairness, Transparency and Accountability Program Area at Partnership on AI

Appendix 2: Case Study Details

Appendix 3: Multistakeholder Convenings

Appendix 4: Glossary

Appendix 5: Detailed Summary of Challenges and Risks Associated with Demographic Data Collection and Analysis

ABOUT ML in Practice: The Example of UNOCHA

Neil Uhl

January 18, 2023

ABOUT ML in Practice: The Example of Biologit

Penelope Sosa

January 17, 2023

AI and Job Quality

Insights from Frontline Workers

PAI Staff

September 28, 2022

Executive Summary

Based on an international study of on-the-job experiences with AI, this report draws from workers’ insights to point the way toward a better future for workplace AI. In addition to identifying common themes among workers’ stories, it provides guidance for key stakeholders who want to make a positive impact. These opportunities for impact can be downloaded individually as audience-specific summaries below.

Opportunities for impact for:

Across industries and around the world, AI is changing work. In the coming years, this rapidly advancing technology has the potential to fundamentally reshape humanity’s relationship with labor. As highlighted by previous Partnership on AI (PAI) research, however, the development and deployment of workplace AI often lacks input from an essential group of experts: the people who directly interact with these systems in their jobs.

Bringing the perspectives of workers into this conversation is both a moral and pragmatic imperative. Despite the direct impact of workplace AI on them, workers rarely have direct influence in AI’s creation or decisions about its implementation. This neglect raises clear concerns about unforeseen or overlooked negative impacts on workers. It also undermines the optimal use of AI from a corporate perspective.

This PAI report, based on an international study of on-the-job experiences with AI, seeks to address this gap. Through journals and interviews, workers in India, sub-Saharan Africa, and the United States shared their stories about workplace AI. From their reflections, PAI identified five common themes:

Executive and managerial decisions shape AI’s impacts on workers, for better and worse. This starts with decisions about business models and operating models, continues through technology acquisitions and implementations, and finally manifests in direct impacts to workers.
Workers have a genuine appreciation for some aspects of AI in their work and how it helps them in their jobs. Their spotlights here point the way to more mutually beneficial approaches to workplace AI.
Workplace AI’s harms are not new or novel — they are repetitions or extensions of harms from earlier technologies and, as such, should be possible to anticipate, mitigate, and eliminate.
Current implementations of AI often serve to reduce workers’ ability to exercise their human skills and talents. Skills like judgment, empathy, and creativity are heavily constrained in these implementations. To the extent that the future of AI is intended to increase humans’ ability to use these talents, the present of AI is sending many workers in the opposite direction.
Empowering workers early in AI development and implementation increases the opportunities to attain the aforementioned benefits and avoid the harms. Workers’ deep experience in their own roles means they should be treated as subject-matter experts throughout the design and implementation process.

In addition, PAI drew from these themes to offer opportunities for impact for the major stakeholders in this space:

AI-implementing companies, who can commit to AI deployments that do not decrease employee job quality.
AI-creating companies, who can center worker well-being and participation in their values, practices, and product designs.
Workers, unions, and worker organizers, who can work to influence and participate in decisions about technology purchases and implementations.
Policymakers, who can shape the environments in which AI products are developed, sold, and implemented.
Investors, who can account for the downside risks posed by practices harmful to workers and the potential value created by worker-friendly technologies.

The actions of each of these groups have the potential to both increase the prosperity enabled by AI technologies and share it more broadly. Together, we can steer AI in a direction that ensures it will benefit workers and society as a whole.

Executive Summary

Introduction

The need for workers’ perspectives on workplace AI

The contributions of this report

Our Approach

Key research questions

Research methods

Site selection

Who we learned from

Participant recruitment

Major Themes and Findings

Theme 1: Executive and managerial decisions shape AI’s impacts on workers, for better and worse

Theme 2: Workers appreciate how some uses of AI have positively changed their jobs

Theme 3: Workplace AI harms repeat, continue, or intensify known possible harms from earlier technologies

Theme 4: Current implementations of AI in work are reducing workers’ opportunities for autonomy, judgment, empathy, and creativity

Theme 5: Empowering workers early in AI development and implementation increases opportunities to implement AI that benefits workers as well as their employers

Opportunities for Impact

Stakeholder 1: AI-implementing companies

Stakeholder Group 2: AI-creating companies

Stakeholder Group 3: Workers, unions, and worker organizers

Stakeholder Group 4: Policymakers

Stakeholder Group 5: Investors

Conclusion

Acknowledgements

Appendix 1: Detailed Site and Technology Descriptions

Appendix 2: Research Methods

Sources Cited

Making AI Inclusive: 4 Guiding Principles for Ethical Engagement

Tina Park

July 20, 2022

Introduction

While the concept of “human-centered design” is hardly new to the technology sector, recent years have seen growing efforts to build inclusive artificial intelligence (AI) and machine learning (ML) products. Broadly, inclusive AI/ML refers to algorithmic systems which are created with the active engagement of and input from people who are not on AI/ML development teams. This includes both end users of the systems and non-users who are impacted by the systems.✳“Impacted non-user” refers to people who are impacted by the deployment of an AI/ML system, but are not the direct user or customer of that system. For example, in the case of students in the United Kingdom in 2020 whose A-level grades were determined by an algorithm, the “user” of the algorithmic system is Ofqual, the official exam regulator in England, and the students are “impacted non-users.” To collect this input, practitioners are increasingly turning to engagement practices like user experience (UX) research and participatory design.

Amid rising awareness of structural inequalities in our society, embracing inclusive research and design principles helps signal a commitment to equitable practices. As many proponents have pointed out, it also makes for good business: Understanding the needs of a more diverse set of people expands the market for a given product or service. Once engaged, these people can then further improve an AI/ML product, identifying issues like bias in algorithmic systems.

Despite these benefits, however, there remain significant challenges to greater adoption of inclusive development in the AI/ML field. There are also important opportunities. For AI practitioners, AI ethics researchers, and others interested in learning more about responsible AI, this Partnership on AI (PAI) white paper provides guidance to help better understand and overcome the challenges related to engaging stakeholders in AI/ML development.

Ambiguities around the meaning and goals of “inclusion” present one of the central challenges to AI/ML inclusion efforts. To make the changes needed for a more inclusive AI that centers equity, the field must first find agreement on foundational premises regarding inclusion. Recognizing this, this white paper provides four guiding principles for ethical engagement grounded in best practices:

All participation is a form of labor that should be recognized
Stakeholder engagement must address inherent power asymmetries
Inclusion and participation can be integrated across all stages of the development lifecycle
Inclusion and participation must be integrated to the application of other responsible AI principles

To realize ethical participatory engagement in practice, this white paper also offers three recommendations aligned with these principles for building inclusive AI:

Allocate time and resources to promote inclusive development
Adopt inclusive strategies before development begins
Train towards an integrated understanding of ethics

This white paper’s insights are derived from the research study “Towards An Inclusive AI: Challenges and Opportunities for Public Engagement in AI Development.” That study drew upon discussions with industry experts, a multidisciplinary review of existing research on stakeholder and public engagement, and nearly 70 interviews with AI practitioners and researchers, as well as data scientists, UX researchers, and technologists working on AI and ML projects, over a third of whom were based in areas outside of the US, EU, UK, or Canada. Supplemental interviews with social equity and Diversity, Equity, and Inclusion (DEI) advocates contributed to the development of recommendations for individual practitioners, business team leaders, and the field of AI and ML more broadly.

This white paper does not provide a step-by-step guide for implementing specific participatory practices. It is intended to renew discussions on how to integrate a wider range of insights and experiences into AI/ML technologies, including those of both users and the people impacted (either directly or indirectly) by these technologies. Such conversations — between individuals, inside teams, and within organizations — must be had to spur the changes needed to develop truly inclusive AI.

Introduction

Guiding Principles for Ethical Participatory Engagement

Principle 1: All Participation Is a Form of Labor That Should Be Recognized

Principle 2: Stakeholder Engagement Must Address Inherent Power Asymmetries

Principle 3: Inclusion and Participation Can Be Integrated Across All Stages of the Development Lifecycle

Principle 4: Inclusion and Participation Must Be Integrated to the Application of Other Responsible AI Principles

Recommendations for Ethical Engagement in Practice

Recommendation 1: Allocate Time and Resources to Promote Inclusive Development

Recommendation 2: Adopt Inclusive Development Strategies Before Development Begins

Recommendation 3: Train Towards an Integrated Understanding of Ethics

Conclusion

Acknowledgements

Sources Cited

Jean-Baptiste, A. (2020). Building for Everyone: Expand Your Market with Design Practices from Google’s Product Inclusion Team. John Wiley and Sons, Inc.
Romao, M. (2019, June 27). “A vision for AI: Innovative, Trusted and Inclusive.” Policy@Intel. https://community.intel.com/t5/Blogs/Intel/Policy-Intel/A-vision-for-AI-Innovative-Trusted-and-Inclusive/post/1333103
Zhou, A., Madras, D., Raji, D., Milli, S., Kulynych, B. and Zemel, R. (2020, July 17). “Participatory Approaches to Machine Learning.” (Workshop). International Conference on Machine Learning 2020.
Lewis, J. E., Abdilla, A., Arista, N., Baker, K., Benesiinaabandan, S., Brown, M., ... and Whaanga, H. (2020). Indigenous protocol and artificial intelligence position paper. Indigenous AI. https://www.indigenous-ai.net/position-paper
Costanza-Chock, S. (2020). Design justice: Community-led practices to build the worlds we need. The MIT Press.
Hamraie, A., and Fritsch, K. (2019). “Crip technoscience manifesto.” Catalyst: Feminism, Theory, Technoscience, 5(1), 1-33. https://catalystjournal.org/index.php/catalyst/article/view/29607
Taylor, L. (2017). “What is data justice? The case for connecting digital rights and freedoms globally.” Big Data and Society, 4(2), 2053951717736335. https://doi.org/10.1177/2053951717736335
Benjamin, Ruha. (2019). Race After Technology: Abolitionist Tools for the New Jim Code. Polity. https://www.ruhabenjamin.com/race-after-technology
Hanna, A., Denton, E., Smart, A., and Smith-Loud, J. (2020). “Towards a critical race methodology in algorithmic fairness.” In Proceedings of the 2020 conference on fairness, accountability, and transparency (pp. 501-512). https://arxiv.org/abs/1912.03593
Sloane, M., Moss, E., Awomolo, O. and Forlano, L. (2020). ''Participation is not a design fix for machine learning.'' arXiv. https://arxiv.org/abs/2007.02423
Cifor, M., Garcia, P., Cowan, T.L., Rault, J., Sutherland, T., Chan, A., Rode, J., Hoffmann, A.L., Salehi, N. and Nakamura, L. (2019). “Feminist Data Manifest-No.” Feminist Data Manifest-No. Retrieved October 1, 2020 from https://www.manifestno.com/home
Harrington, C., Erete, S. and Piper, A.M. (2019). “Deconstructing Community-Based Collaborative Design: Towards More Equitable Participatory Design Engagements.” In Proceedings of the ACM on Human-Computer Interaction 3(CSCW):1–25. https://doi.org/10.1145/3359318
Freimuth V.S., Quinn, S.C., Thomas, S.B., Cole, G., Zook, E and Duncan, T. (2001). “African Americans’ Views on Research and the Tuskegee Syphilis Study.” Social Science and Medicine 52(5):797–808. https://doi.org/10.1016/S0277-9536(00)00178-7
George, S., Duran, N. and Norris, K. (2014). “A Systematic Review of Barriers and Facilitators to Minority Research Participation Among African Americans, Latinos, Asian Americans, and Pacific Islanders.” American Journal of Public Health 104(2):e16–31. https://doi.org/10.2105/AJPH.2013.301706
Barabas, C., Doyle, C., Rubinovitz, J.B., and Dinakar, K. (2020). “Studying Up: Reorienting the Study of Algorithmic Fairness around Issues of Power.” In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (pp. 167-176).
Harrington, C., Erete, S. and Piper, A.M.. (2019). “Deconstructing Community-Based Collaborative Design: Towards More Equitable Participatory Design Engagements.” In Proceedings of the ACM on Human-Computer Interaction 3(CSCW):1–25. https://dl.acm.org/doi/10.1145/3359318.
Chan, A., Okolo, C. T., Terner, Z., and Wang, A. (2021). “The Limits of Global Inclusion in AI Development.” arXiv. https://arxiv.org/abs/2102.01265
Sanders, E. B. N. (2002). “From user-centered to participatory design approaches.” In Design and the social sciences (pp. 18-25). CRC Press. https://www.taylorfrancis.com/chapters/edit/10.1201/9780203301302-8/user-centered-participatory-design-approaches-elizabeth-sanders
Leslie, D., Katell, M., Aitken, M., Singh, J., Briggs, M., Powell, R., ... and Burr, C. (2022). “Data Justice in Practice: A Guide for Developers.” arXiv. https://arxiv.org/ftp/arxiv/papers/2205/2205.01037.pdf
Zdanowska, S., and Taylor, A. S. (2022). “A study of UX practitioners roles in designing real-world, enterprise ML systems.” In CHI Conference on Human Factors in Computing Systems (pp. 1-15). https://dl.acm.org/doi/abs/10.1145/3491102.3517607
Leslie, D., Katell, M., Aitken, M., Singh, J., Briggs, M., Powell, R., ... and Burr, C. (2022). “Data Justice in Practice: A Guide for Developers.” arXiv. https://arxiv.org/ftp/arxiv/papers/2205/2205.01037.pdf
Saulnier, L., Karamcheti, S., Laurençon, H., Tronchon, L., Wang, T., Sanh, V., Singh, A., Pistilli, G., Luccioni, S., Jernite, Y., Mitchell, M. and Kiela, D. (2022). “Putting Ethical Principles at the Core of the Research Lifecycle.” Hugging Face Blog. Retrieved from https://huggingface.co/blog/ethical-charter-multimodal
Ada Lovelace Institute. (2021). “Participatory data stewardship: A framework for involving people in the use of data.” Ada Lovelace Institute. https://www.adalovelaceinstitute.org/report/participatory-data-stewardship/
Delgado, F., Yang, S., Madaio, M., and Yang, Q. (2021). “Stakeholder Participation in AI: Beyond ‘Add Diverse Stakeholders and Stir.’” arXiv. https://arxiv.org/pdf/2111.01122.pdf
Sloane, M., Moss, E., Awomolo, O. and Forlano, L. (2020). ''Participation is not a design fix for machine learning.'' arXiv. https://arxiv.org/abs/2007.02423

After the Offer: The Role of Attrition in AI’s ‘Diversity Problem’

Jeffrey Brown

April 13, 2022

Executive Summary

As a field, AI struggles to retain team members from diverse backgrounds. Given the far-reaching effects of algorithmic systems and the documented harms to marginalized communities, the fact that these communities are not represented on AI teams is particularly troubling. Why is this such a widespread phenomenon and what can be done to close the gap? This research paper, “After the Offer: The Role of Attrition in AI’s ‘Diversity Problem’” seeks to answer these questions, providing four recommendations for how organizations can make the AI field more inclusive. Click the button below to download a summary of these recommendations or continue on to read the paper in full.

Summary of Recommendations

Amid heightened attention to society-wide racial and social injustice, organizations in the AI space have been urged to investigate the harmful effects that AI has had on marginalized populations. It’s an issue that engineers, researchers, project managers, and various leaders in both tech companies and civil society organizations have devoted significant time and resources to in recent years. In examining the effects of AI, organizations must consider who exactly has been designing these technologies.

Diversity reports have revealed that the people working at the organizations that develop and deploy AI lack diversity across several dimensions. While organizations have blamed pipeline problems in the past, research has increasingly shown that once workers belonging to minoritized identities get hired in these spaces, systemic difficulties affect their experiences in ways that their peers from dominant groups do not have to worry about.

Attrition in the tech industry is a problem that disproportionately affects minoritized workers. In AI, where technologies already have a disproportionately negative impact on these communities, this is especially troublesome.

Minoritized Workers

This report uses minoritized workers as an umbrella term to refer to people whose identities (in categories such as race, ethnicity, gender, or ability) have been historically marginalized by those in dominant social groups. The minoritized workers in this study include people who identified as minoritized within the identity categories of race and ethnicity, gender identity, sexual orientation, ability, and immigration status. Because this study was international in scope, it is important to note that these categories are relative to their social context.

We are left wondering: What leads to these folks leaving their teams, organizations, or even the AI field more broadly? What about the AI field in particular influences these people to stay or leave? And what can organizations do to stem this attrition to make their environments more inclusive?

The current study uses interviews with folks belonging to minoritized identities across the AI field, managers, and DEI (diversity, equity, and inclusion)- leaders in tech to get rich information about what aspects of cultures within an organization promote inclusion or contribute to attrition. Themes that emerged during these interviews formed 3 key takeaways:

Diversity makes for better team climates
Systemic supports are difficult but necessary to undo the current harms to minoritized workers
Individual efforts to change organizational culture fall disproportionately on minoritized folks who are usually not professionally rewarded for their efforts

In line with these takeaways, the study makes 4 recommendations about what can be done to make the AI field more inclusive for workers:

Organizations must systemically support ERGs
Organizations must intentionally diversify leadership and managers
DEI trainings must be specific in order to be effective and be more connected to the content of AI work
Organizations must interrogate their values as practiced and fundamentally alter them to include the perspectives of people who are not White, cis, or male

These takeaways and recommendations are explored in more depth below.

Key Takeaways

1. Diversity makes for better team climates

Across interviews, participants consistently expressed that managers who belonged to minoritized identities or who took the time to learn about working with diverse identities were more supportive of their needs and career goals. Such efforts reportedly resulted in teams that were also more diverse, inclusive, interdisciplinary, and engendering of a positive team culture/climate. In these environments, workers belonging to minoritized identities thrived. A diversity in backgrounds and perspectives was particularly important for AI teams that needed to solve interdisciplinary problems.

Conversely, the negative impact of work environments that were sexist or where participants experienced acts of prejudice such as microaggressions was also a recurring theme.

While collaborative or positive work environments were also a common theme, such environments did not in themselves negate predominant cultures which deprioritized “DEI-focused” work, work that was highly interdisciplinary, or work that did not serve the dominant group. Negative organizational cultures seemed to exacerbate experiences of prejudice or discrimination on AI teams.

2. Systemic supports are difficult but necessary to undo the current harms to minoritized workers

Participants belonging to minoritized identities said that they either left or intended to leave organizations that did not support their continued career growth or possessed values that did not align with their own. Consistent with this, participants described examples of their organizations not valuing the content of their work.

Participants also tied their desires to leave with instances of prejudice or discrimination, which may also be related to “toxic” work environments. Some participants reported instances of being tokenized or being subject to negative stereotypes about their identity groups, somewhat reflective of wider contexts in tech beyond AI.

Systemic supports include incentive structures that allow minoritized workers to succeed at every level, from the teams that they work with actively validating their experiences to their managers finding the best ways for them to deliver work products in accordance with both individual and institutional needs. Guidelines for promotion that recognize the barriers these workers face in environments mostly occupied by dominant group norms are another important support.

3. Individual efforts to change organizational culture fall disproportionately on minoritized folks who are usually not professionally rewarded for their efforts

Individuals discussed ways in which they tried to make their workplaces or teams more inclusive or otherwise sought to incorporate diverse perspectives into their work around AI. Participants sometimes had to contend with bias against DEI efforts, reporting that other workers in their organizations would dismiss their efforts as lacking rigor or focus on the product.

There were some institutional efforts to foster a more inclusive culture, most commonly DEI trainings. DEI trainings that were very specific to some groups (e.g., gender diverse folks, Black people) were reported as being the most effective. However, even when they were specific, DEI trainings seemed to be disconnected from some aspects of the workplace climate or the content of what teams were working on.

Participants who mentioned Employee Resource Groups (ERGs) uniformly praised them, discussing the huge positive impact they had on a personal level, forming the bases of their social support networks in their organizations and having a strong impact on their ability to integrate aspects of their identities or other “DEI topics” they were passionate about into their work.

Recommendations

1. Organizations must systemically support ERGs

Employees specifically named ERGs as one of their main sources of support even in work environments that were otherwise toxic.. Additionally, ERGs provided built-in mentorship for those who did not have ready access to mentors or whose supervisors had not done the work to understand the kinds of support needed for those of minoritized identities to thrive in predominantly White and male environments.

What makes this recommendation work?

Within these ERGs, there existed other grass-roots initiatives that supported workers, such as informal talking circles and networks of employees that essentially provided peer mentoring that participants found crucial to navigating White- and male-dominated spaces. The mentorship provided by ERGs was also essential when HR failed to provide systemic support for staff and instead prioritized protecting the organization.

What must be in place?

While participants uniformly praised ERGs, they required large amounts of time from staff members that detracted from their work. Such groups also ran the risk of getting taken over by leadership and having their original mission derailed. Institutions should seek a balance between supporting these groups and giving them the freedom to organize in pursuit of their own best interests.

What won’t this solve?

ERGs will not necessarily make an organization’s AI or tech more inclusive. Rather, systematically supporting ERGs will provide more support and community for minoritized workers, which is meant to promote a more inclusive workplace in general.

2. Organizations must intentionally diversify leadership and managers

What makes this recommendation work?

Participants repeatedly pointed to managers and upper-level leaders who belonged to minoritized identities (especially racial ones) as important influences, changing policy that permeated through various levels of their organizations. A diverse workforce may also bring with it multiple perspectives, including those belonging to people from different disciplines who may be interested in working in the AI field due to the opportunity for interdisciplinary collaboration, research, and product development. Bringing in folks from various academic, professional, and technical backgrounds to solve problems is especially crucial for AI teams.

What must be in place?

There must be understanding about the reasons behind the lack of diversity and the “bigger picture” of how powerful groups more easily perpetuate power structures already in place. Participants spoke of managers who did not belong to minoritized identities themselves but who took the time to learn in depth about differences in power and privilege in the tech ecosystem, appreciating the diverse perspectives that workers brought. These managers, while not perfect, tended to take advocating for their reports very seriously, particularly female reports who often went overlooked.

What won’t this solve?

Intentionally diversifying leadership and managers will not automatically create a pipeline for diversity at the leadership level, nor will it automatically override institutional culture or policies that ignore DEI best practices.

3. DEI trainings must be specific in order to be effective and be more connected to the content of AI work

What makes this recommendation work?

Almost all participants reported that their organizations mandated some form of DEI training for all staff. These ranged widely, from very general ones to very specific trainings that discussed cultural competency about more specific groups of people (e.g., participants reported that there were trainings on anti-Black racism). Participants discussed that the more specific trainings tended to be more impactful.

What must be in place?

Organizations must invest in employees who see the importance of inclusive values in AI research and product design. Participants pointed to the importance of managers who had an ability to foster inclusive team values, which was not something that HR could mandate.

What won’t this solve?

As several participants observed, DEI trainings will not uproot or counteract institutional stigmas against DEI. It would take sustained effort and deliberate alignment of values for an organization to emphasize DEI in its work.

4. Organizations must interrogate their values as practiced and fundamentally alter them to include the perspectives of people who are not White, cis, or male

What makes this recommendation work?

Participants frequently reported that a misalignment of values was a primary reason for them leaving their organizations or wanting to leave their organizations. Participants in this sample discussed joining the AI field to create a positive impact while growing professionally. This led them to feeling disappointed when their organizations did not prioritize these goals (despite them being among their stated values).

What must be in place?

Participants found it frustrating when organizations stated that they valued diversity and then failed to live up to this value with hiring, promotion, and day-to-day operations, ignoring the voices of minoritized individuals. If diversity is truly a value, organizations may have to investigate their systems of norms and expectations that are fundamentally male, Eurocentric, and do not make space for those from diverse backgrounds. They then must take additional steps to consider how such systems influence their work in AI.

What won’t this solve?

Because achieving a fundamental re-alignment like this is a more comprehensive solution, it cannot satisfy the most immediate and urgent needs for reform. Short-term, organizations must work with DEI professionals to recognize how they are perpetuating potentially harmful norms of the dominant group and work to create policies that are more equitable. Longer term fixes may not, for instance, satisfy the immediate and urgent need for more diversity in leadership and teams in general.

Executive Summary

Key Takeaways

Recommendations

Introduction

Why Study Attrition of Minoritized Workers in AI?

Background

Problems Due to Lack of Diversity of AI Teams

More Diverse Teams Yield Better Outcomes

Current Level of Diversity in Tech

Diversity in AI

What Has Been Done

Attrition in Tech

Current Study and Methodology

Recruitment

Participants

Measure

Procedure

Analysis

Results

Attrition

Culture

Efforts to Improve Inclusivity

Summary and the Path Forward

Acknowledgements

Appendices

Appendix 1: Recruitment Document

Appendix 2: Privacy Document

Appendix 3: Research Protocol

Appendix 4: Important Terms

Sources Cited

Fairer Algorithmic Decision-Making and Its Consequences: Interrogating the Risks and Benefits of Demographic Data Collection, Use, and Non-Use

PAI Staff

December 2, 2021

Introduction and Background

Introduction

Algorithmic decision-making has been widely accepted as a novel approach to overcoming the purported cognitive and subjective limitations of human decision makers by providing “objective” data-driven recommendations. Yet, as organizations adopt algorithmic decision-making systems (ADMS), countless examples of algorithmic discrimination continue to emerge. Harmful biases have been found in algorithmic decision-making systems in contexts such as healthcare, hiring, criminal justice, and education, prompting increasing social concern regarding the impact these systems are having on the wellbeing and livelihood of individuals and groups across society. In response, algorithmic fairness strategies attempt to understand how ADMS treat certain individuals and groups, often with the explicit purpose of detecting and mitigating harmful biases.

Many current algorithmic fairness techniques require access to data on a “sensitive attribute” or “protected category” (such as race, gender, or sexuality) in order to make performance comparisons and standardizations across groups. These demographic-based algorithmic fairness techniques assume that discrimination and social inequality can be overcome with clever algorithms and collection of the requisite data, removing broader questions of governance and politics from the equation. This paper seeks to challenge this assumption, arguing instead that collecting more data in support of fairness is not always the answer and can actually exacerbate or introduce harm for marginalized individuals and groups. We believe more discussion is needed in the machine learning community around the consequences of “fairer” algorithmic decision-making. This involves acknowledging the value assumptions and trade-offs associated with the use and non-use of demographic data in algorithmic systems. To advance this discussion, this white paper provides a preliminary perspective on these trade-offs derived from workshops and conversations with experts in industry, academia, government, and advocacy organizations as well as literature across relevant domains. In doing so, we hope that readers will better understand the affordances and limitations of using demographic data to detect and mitigate discrimination in institutional decision-making more broadly

Background

Demographic-based algorithmic fairness techniques presuppose the availability of data on sensitive attributes or protected categories. However, previous research has highlighted that data on demographic categories, such as race and sexuality, are often unavailable due to a range of organizational challenges, legal barriers, and practical concerns Andrus, M., Spitzer, E., Brown, J., & Xiang, A. (2021). “What We Can’t Measure, We Can’t Understand”: Challenges to Demographic Data Procurement in the Pursuit of Fairness. ArXiv:2011.02282 (Cs). http://arxiv.org/abs/2011.02282. Some privacy laws, such as the EU’s GDPR, not only require
data subjects to provide meaningful consent when their data is collected, but also prohibit the collection of sensitive data such as race, religion, and sexuality. Some corporate privacy policies and standards, such as Privacy By Design, call for organizations to be intentional with their data collection practices, only collecting data they require and can specify a use for. Given the uncertainty around whether or not it is acceptable to ask users and customers for their sensitive demographic information, most legal and policy teams urge their corporations to err on the side of caution and not collect these types of data unless legally required to do so. As a
result, concerns over privacy often take precedence over ensuring product fairness since the trade-offs between mitigating bias and ensuring individual or group privacy are unclear Andrus et al., 2021.

In cases where sensitive demographic data can be collected, organizations must navigate a number of practical challenges throughout its procurement. For many organizations, sensitive demographic data is collected through self-reporting mechanisms. However, self reported data is often incomplete, unreliable, and unrepresentative, due in part to a lack of incentives for individuals to provide accurate
and full information Andrus et al., 2021. In some cases, practitioners choose to infer protected categories of individuals based on proxy information, a method which is largely inaccurate. Organizations also face difficulty capturing unobserved characteristics, such as disability, sexuality, and religion, as these categories are frequently missing and often unmeasurable Tomasev, N., McKee, K. R., Kay, J., & Mohamed, S. (2021). Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities. ArXiv:2102.04257 (Cs). https://doi.org/10.1145/3461702.3462540. Overall, deciding on how to classify and categorize demographic data is an ongoing challenge, as demographic categories continue to shift and change over time and between contexts. Once demographic data is collected, antidiscrimination law and policies largely inhibit organizations from using this data since knowledge of sensitive categories opens the door to legal liability if discrimination is uncovered without a plan to successfully mitigate it Andrus et al., 2021.

In the face of these barriers, corporations looking to apply demographic-based algorithmic fairness techniques have called for guidance on how to responsibly collect and use demographic data. However, prescribing statistical definitions of fairness on algorithmic systems without accounting for the social, economic, and political systems in which they are embedded can fail to benefit marginalized
groups and undermine fairness efforts Bakalar, C., Barreto, R., Bogen, M., Corbett-Davies, S., Hall, M., Kloumann, I., Lam, M., Candela, J. Q., Raghavan, M., Simons, J., Tannen, J., Tong, E., Vredenburgh, K., & Zhao, J. (2021). Fairness On The Ground: Applying Algorithmic Fairness Approaches To Production Systems. 12.. Therefore, developing guidance requires a deeper understanding of the risks and trade-offs inherent to the use and non-use of demographic data. Efforts to detect and mitigate harms must account for the wider contexts and power structures that algorithmic systems, and the data that they draw on, are embedded in.

Finally, though this work is motivated by the documented unfairness of ADMS, it is critical to recognize that bias and discrimination are not the only possible harms stemming directly from ADMS. As recent papers and reports have forcefully argued, focusing on debiasing datasets and algorithms is (1) often misguided because proposed debiasing methods are only relevant for a subset of the kinds of bias ADMS introduce or reinforce, and (2) likely to draw attention away from other, possibly more salient harms Balayn, A., & Gürses, S. (2021). Beyond Debiasing. European Digital Rights. https://edri.org/wp-content/ uploads/2021/09/EDRi_Beyond-Debiasing-Report_Online.pdf. In the first case, harms from tools such as recommendation systems, content moderation systems, and computer vision systems might be characterized as a result of various forms of bias, but resolving bias in those systems generally involves adding in more context to better understand differences between groups, not just trying to treat groups more similarly. In the second case, there are many ADMS that are clearly susceptible to bias, yet the greater source of harm could arguably be the deployment of the system in the first place. Pre-trial detention risk scores provide one such example. Using statistical correlations to determine if someone should be held without bail, or, in other words, potentially punishing individuals for attributes outside of their control and past decisions unrelated to what they are currently being charged for, is itself a significant deviation from legal standards and norms, yet most of the debate has focused around how biased the predictions are. Attempting to collect demographic data in these cases will likely do more harm than good, as demographic data will
draw attention away from harms inherent to the system and towards seemingly resolvable issues around bias.

Introduction and Background

Introduction

Background

Social Risks of Non-Use

Hidden Discrimination

''Colorblind'' Decision-Making

Invisibility to Institutions of Importance

Social Risks of Use

Risks to Individuals

Encroachments on Privacy and Personal Life

Individual Misrepresentation

Data Misuse and Use Beyond Informed Consent

Risks to Communities

Expanding Surveillance Infrastructure in the Pursuit of Fairness

Misrepresentation and Reinforcing Oppressive or Overly Prescriptive Categories

Private Control Over Scoping Bias and Discrimination

Conclusion and Acknowledgements

Conclusion

Acknowledgements

Sources Cited

Andrus, M., Spitzer, E., Brown, J., & Xiang, A. (2021). “What We Can’t Measure, We Can’t Understand”: Challenges to Demographic Data Procurement in the Pursuit of Fairness. ArXiv:2011.02282 (Cs). http://arxiv.org/abs/2011.02282
Andrus et al., 2021
Andrus et al., 2021
Tomasev, N., McKee, K. R., Kay, J., & Mohamed, S. (2021). Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities. ArXiv:2102.04257 (Cs). https://doi.org/10.1145/3461702.3462540
Andrus et al., 2021
Bakalar, C., Barreto, R., Bogen, M., Corbett-Davies, S., Hall, M., Kloumann, I., Lam, M., Candela, J. Q., Raghavan, M., Simons, J., Tannen, J., Tong, E., Vredenburgh, K., & Zhao, J. (2021). Fairness On The Ground: Applying Algorithmic Fairness Approaches To Production Systems. 12.
Balayn, A., & Gürses, S. (2021). Beyond Debiasing. European Digital Rights. https://edri.org/wp-content/ uploads/2021/09/EDRi_Beyond-Debiasing-Report_Online.pdf
Ntoutsi, E., Fafalios, P., Gadiraju, U., Iosifidis, V., Nejdl, W., Vidal, M., Ruggieri, S., Turini, F., Papadopoulos, S., Krasanakis, E., Kompatsiaris, I., Kinder‐Kurlanda, K., Wagner, C., Karimi, F., Fernandez, M., Alani, H., Berendt, B., Kruegel, T., Heinze, C., … Staab, S. (2020). Bias in data‐driven artificial intelligence systems—An introductory survey. WIREs Data Mining and Knowledge Discovery, 10(3). https://doi.org/10.1002/widm.1356
Olteanu, A., Castillo, C., Diaz, F., & Kıcıman, E. (2019). Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries. Frontiers in Big Data, 2, 13. https://doi.org/10.3389/fdata.2019.00013
Rimfeld, K., & Malanchini, M. (2020, August 21). The A-Level and GCSE scandal shows teachers should be trusted over exams results. Inews.Co.Uk. https://inews.co.uk/opinion/a-level-gcse-results-trust-teachers-exams-592499
Davidson, T., Warmsley, D., Macy, M., & Weber, I. (2017). Automated Hate Speech Detection and the Problem of Offensive Language. Proceedings of the International AAAI Conference on Web and Social Media, 11(1), 512–515.
Davidson, T., Bhattacharya, D., & Weber, I. (2019). Racial Bias in Hate Speech and Abusive Language Detection Datasets. ArXiv:1905.12516 (Cs). http://arxiv.org/abs/1905.12516
Bogen, M., Rieke, A., & Ahmed, S. (2020). Awareness in Practice: Tensions in Access to Sensitive Attribute Data for Antidiscrimination. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 492–500. https://doi.org/10.1145/3351095.3372877
Executive Order On Advancing Racial Equity and Support for Underserved Communities Through the Federal Government. (2021, January 21). The White House. https://www.whitehouse.gov/briefing-room/presidential-actions/2021/01/20/executive-order-advancing-racial-equity-and-support-for-underserved-communities-through-the-federal-government/
Executive Order on Diversity, Equity, Inclusion, and Accessibility in the Federal Workforce. (2021, June 25). The White House. https://www.whitehouse.gov/briefing-room/presidential-actions/2021/06/25/executive-order-on-diversity-equity-inclusion-and-accessibility-in-the-federal-workforce/
Kusner, M. J., Loftus, J., Russell, C., & Silva, R. (2017). Counterfactual Fairness. Advances in Neural Information Processing Systems, 30. https://papers.nips.cc/paper/2017/hash/a486cd07e4ac3d270571622f4f316ec5-Abstract.html
Harned, Z., & Wallach, H. (2019). Stretching human laws to apply to machines: The dangers of a ’Colorblind’ Computer. Florida State University Law Review, Forthcoming.
Washington, A. L. (2018). How to Argue with an Algorithm: Lessons from the COMPAS-ProPublica Debate. Colorado Technology Law Journal, 17, 131.
Rodriguez, L. (2020). All Data Is Not Credit Data: Closing the Gap Between the Fair Housing Act and Algorithmic Decisionmaking in the Lending Industry. Columbia Law Review, 120(7), 1843–1884.
Hu, L. (2021, February 22). Law, Liberation, and Causal Inference. LPE Project. https://lpeproject.org/blog/law-liberation-and-causal-inference/
Bonilla-Silva, E. (2010). Racism Without Racists: Color-blind Racism and the Persistence of Racial Inequality in the United States. Rowman & Littlefield.
Plaut, V. C., Thomas, K. M., Hurd, K., & Romano, C. A. (2018). Do Color Blindness and Multiculturalism Remedy or Foster Discrimination and Racism? Current Directions in Psychological Science, 27(3), 200–206. https://doi.org/10.1177/0963721418766068
Eubanks, V. (2017). Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. St. Martin’s Press
Banco, E., & Tahir, D. (2021, March 9). CDC under scrutiny after struggling to report Covid race, ethnicity data. POLITICO. https://www.politico.com/news/2021/03/09/hhs-cdc-covid-race-data-474554
Banco, E., & Tahir, D. (2021, March 9). CDC under scrutiny after struggling to report Covid race, ethnicity data. POLITICO. https://www.politico.com/news/2021/03/09/hhs-cdc-covid-race-data-474554
Elliott, M. N., Morrison, P. A., Fremont, A., McCaffrey, D. F., Pantoja, P., & Lurie, N. (2009). Using the Census Bureau’s surname list to improve estimates of race/ethnicity and associated disparities. Health Services and Outcomes Research Methodology, 9(2), 69.
Shimkhada, R., Scheitler, A. J., & Ponce, N. A. (2021). Capturing Racial/Ethnic Diversity in Population-Based Surveys: Data Disaggregation of Health Data for Asian American, Native Hawaiian, and Pacific Islanders (AANHPIs). Population Research and Policy Review, 40(1), 81–102. https://doi.org/10.1007/s11113-020-09634-3
Poon, O. A., Dizon, J. P. M., & Squire, D. (2017). Count Me In!: Ethnic Data Disaggregation Advocacy, Racial Mattering, and Lessons for Racial Justice Coalitions. JCSCORE, 3(1), 91–124. https://doi.org/10.15763/issn.2642-2387.2017.3.1.91-124
Fosch-Villaronga, E., Poulsen, A., Søraa, R. A., & Custers, B. H. M. (2021). A little bird told me your gender: Gender inferences in social media. Information Processing & Management, 58(3), 102541. https://doi.org/10.1016/j.ipm.2021.102541
Browne, S. (2015). Dark Matters: On the Surveillance of Blackness. In Dark Matters. Duke University Press. https://doi.org/10.1515/9780822375302
Eubanks, 2017
Farrand, T., Mireshghallah, F., Singh, S., & Trask, A. (2020). Neither Private Nor Fair: Impact of Data Imbalance on Utility and Fairness in Differential Privacy. Proceedings of the 2020 Workshop on Privacy-Preserving Machine Learning in Practice, 15–19. https://doi.org/10.1145/3411501.3419419
Jagielski, M., Kearns, M., Mao, J., Oprea, A., Roth, A., Sharifi -Malvajerdi, S., & Ullman, J. (2019). Differentially Private Fair Learning. Proceedings of the 36th International Conference on Machine Learning, 3000–3008. https://bit.ly/3rmhET0
Kuppam, S., Mckenna, R., Pujol, D., Hay, M., Machanavajjhala, A., & Miklau, G. (2020). Fair Decision Making using Privacy-Protected Data. ArXiv:1905.12744 (Cs). http://arxiv.org/abs/1905.12744
Quillian, L., Pager, D., Hexel, O., & Midtbøen, A. H. (2017). Meta-analysis of field experiments shows no change in racial discrimination in hiring over time. Proceedings of the National Academy of Sciences, 114(41), 10870–10875. https://doi.org/10.1073/pnas.1706255114
Quillian, L., Lee, J. J., & Oliver, M. (2020). Evidence from Field Experiments in Hiring Shows Substantial Additional Racial Discrimination after the Callback. Social Forces, 99(2), 732–759. https://doi.org/10.1093/sf/soaa026
Cabañas, J. G., Cuevas, Á., Arrate, A., & Cuevas, R. (2021). Does Facebook use sensitive data for advertising purposes? Communications of the ACM, 64(1), 62–69. https://doi.org/10.1145/3426361
Datta, A., Tschantz, M. C., & Datta, A. (2015). Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination. Proceedings on Privacy Enhancing Technologies, 2015(1), 92–112. https://doi.org/10.1515/popets-2015-0007
Hupperich, T., Tatang, D., Wilkop, N., & Holz, T. (2018). An Empirical Study on Online Price Differentiation. Proceedings of the Eighth ACM Conference on Data and Application Security and Privacy, 76–83. https://doi.org/10.1145/3176258.3176338
Mikians, J., Gyarmati, L., Erramilli, V., & Laoutaris, N. (2013). Crowd-assisted search for price discrimination in e-commerce: First results. Proceedings of the Ninth ACM Conference on Emerging Networking Experiments and Technologies, 1–6. https://doi.org/10.1145/2535372.2535415
Cabañas et al., 2021
Leetaru, K. (2018, July 20). Facebook As The Ultimate Government Surveillance Tool? Forbes. https://www.forbes.com/sites/kalevleetaru/2018/07/20/facebook-as-the-ultimate-government-surveillance-tool/
Rozenshtein, A. Z. (2018). Surveillance Intermediaries (SSRN Scholarly Paper ID 2935321). Social Science Research Network. https://papers.ssrn.com/abstract=2935321
Rocher, L., Hendrickx, J. M., & de Montjoye, Y.-A. (2019). Estimating the success of re-identifications in incomplete datasets using generative models. Nature Communications, 10(1), 3069. https://doi.org/10.1038/s41467-019-10933-3
Cummings, R., Gupta, V., Kimpara, D., & Morgenstern, J. (2019). On the Compatibility of Privacy and Fairness. Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization - UMAP’19 Adjunct, 309–315. https://doi.org/10.1145/3314183.3323847
Kuppam et al., 2020
Mavriki, P., & Karyda, M. (2019). Automated data-driven profiling: Threats for group privacy. Information & Computer Security, 28(2), 183–197. https://doi.org/10.1108/ICS-04-2019-0048
Barocas, S., & Levy, K. (2019). Privacy Dependencies (SSRN Scholarly Paper ID 3447384). Social Science Research Network. https://papers.ssrn.com/abstract=3447384
Bivens, R. (2017). The gender binary will not be deprogrammed: Ten years of coding gender on Facebook. New Media & Society, 19(6), 880–898. https://doi.org/10.1177/1461444815621527
Mittelstadt, B. (2017). From Individual to Group Privacy in Big Data Analytics. Philosophy & Technology, 30(4), 475–494. https://doi.org/10.1007/s13347-017-0253-7
Taylor, 2021
Draper and Turow, 2019
Hanna, A., Denton, E., Smart, A., & Smith-Loud, J. (2020). Towards a Critical Race Methodology in Algorithmic Fairness. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 501–512. https://doi.org/10.1145/3351095.3372826
Keyes, O., Hitzig, Z., & Blell, M. (2021). Truth from the machine: Artificial intelligence and the materialization of identity. Interdisciplinary Science Reviews, 46(1–2), 158–175. https://doi.org/10.1080/03080188.2020.1840224
Scheuerman, M. K., Wade, K., Lustig, C., & Brubaker, J. R. (2020). How We’ve Taught Algorithms to See Identity: Constructing Race and Gender in Image Databases for Facial Analysis. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW1), 1–35. https://doi.org/10.1145/3392866
Roth, W. D. (2016). The multiple dimensions of race. Ethnic and Racial Studies, 39(8), 1310–1338. https://doi.org/10.1080/01419870.2016.1140793
Hanna et al., 2020
Keyes, O. (2018). The Misgendering Machines: Trans/HCI Implications of Automatic Gender Recognition. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW), 88:1-88:22. https://doi.org/10.1145/3274357
Keyes, O. (2019, April 8). Counting the Countless. Real Life. https://reallifemag.com/counting-the-countless/
Keyes, O., Hitzig, Z., & Blell, M. (2021). Truth from the machine: Artificial intelligence and the materialization of identity. Interdisciplinary Science Reviews, 46(1–2), 158–175. https://doi.org/10.1080/03080188.2020.1840224
Scheuerman et al., 2020
Scheuerman et al., 2020
Stark, L., & Hutson, J. (2021). Physiognomic Artificial Intelligence (SSRN Scholarly Paper ID 3927300). Social Science Research Network. https://doi.org/10.2139/ssrn.3927300
U.S. Department of Justice. (2019). The First Step Act of 2018: Risk and Needs Assessment System. Office of the Attorney General.
Partnership on AI. (2020). Algorithmic Risk Assessment and COVID-19: Why PATTERN Should Not Be Used. Partnership on AI. http://partnershiponai.org/wp-content/uploads/2021/07/Why-PATTERN-Should-Not-Be-Used.pdf
Hill, K. (2020, January 18). The Secretive Company That Might End Privacy as We Know It. The New York Times. https://www.nytimes.com/2020/01/18/technology/clearview-privacy-facial-recognition.html
Porter, J. (2020, February 6). Facebook and LinkedIn are latest to demand Clearview stop scraping images for facial recognition tech. The Verge. https://www.theverge.com/2020/2/6/21126063/facebook-clearview-ai-image-scraping-facial-recognition-database-terms-of-service-twitter-youtube
Regulation (EU) 2016/679 (General Data Protection Regulation), (2016) (testimony of European Parliament and Council of European Union). https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32016R0679&from=EN
Obar, J. A. (2020). Sunlight alone is not a disinfectant: Consent and the futility of opening Big Data black boxes (without assistance). Big Data & Society, 7(1), 2053951720935615. https://doi.org/10.1177/2053951720935615
Obar, J. A. (2020). Sunlight alone is not a disinfectant: Consent and the futility of opening Big Data black boxes (without assistance). Big Data & Society, 7(1), 2053951720935615. https://doi.org/10.1177/2053951720935615
Obar, 2020
Angwin, J., & Parris, T. (2016, October 28). Facebook Lets Advertisers Exclude Users by Race. ProPublica. https://www.propublica.org/article/facebook-lets-advertisers-exclude-users-by-race
Benjamin, R. (2019). Race after technology: Abolitionist tools for the new Jim code. Polity.
Browne, S. (2015). Dark Matters: On the Surveillance of Blackness. In Dark Matters. Duke University Press. https://doi.org/10.1515/9780822375302
Eubanks, V. (2017). Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor. St. Martin’s Press.
Hoffmann, 2020
Rainie, S. C., Kukutai, T., Walter, M., Figueroa-Rodríguez, O. L., Walker, J., & Axelsson, P. (2019). Indigenous data sovereignty.
Ricaurte, P. (2019). Data Epistemologies, Coloniality of Power, and Resistance. Television & New Media, 16.
Walter, M. (2020, October 7). Delivering Indigenous Data Sovereignty. https://www.youtube.com/watch?v=NCsCZJ8ugPA
See, for example: Bowker, G. C., & Star, S. L. (1999). Sorting things out: Classification and its consequences. MIT Press.
See, for example: Dembroff, R. (2018). Real Talk on the Metaphysics of Gender. Philosophical Topics, 46(2), 21–50. https://doi.org/10.5840/philtopics201846212
See, for example: Hacking, I. (1995). The looping effects of human kinds. In Causal cognition: A multidisciplinary debate (pp. 351–394). Clarendon Press/Oxford University Press.
See, for example: Hanna et al., 2020
See, for example: Hu, L., & Kohler-Hausmann, I. (2020). What’s Sex Got to Do With Fair Machine Learning? 11.
See, for example: Keyes (2019)
See, for example: Zuberi, T., & Bonilla-Silva, E. (2008). White Logic, White Methods: Racism and Methodology. Rowman & Littlefield Publishers.
Hanna et al., 2020
Andrus et al., 2021
Bivens, 2017
Hamidi, F., Scheuerman, M. K., & Branham, S. M. (2018). Gender Recognition or Gender Reductionism?: The Social Implications of Embedded Gender Recognition Systems. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems - CHI ’18, 1–13. https://doi.org/10.1145/3173574.3173582
Keyes, 2018
Keyes, 2021
Fu, S., & King, K. (2021). Data disaggregation and its discontents: Discourses of civil rights, efficiency and ethnic registry. Discourse: Studies in the Cultural Politics of Education, 42(2), 199–214. https://doi.org/10.1080/01596306.2019.1602507
Poon et al., 2017
Hanna et al., 2020
Saperstein, A. (2012). Capturing complexity in the United States: Which aspects of race matter and when? Ethnic and Racial Studies, 35(8), 1484–1502. https://doi.org/10.1080/01419870.2011.607504
Keyes, 2019
Ruberg, B., & Ruelos, S. (2020). Data for queer lives: How LGBTQ gender and sexuality identities challenge norms of demographics. Big Data & Society, 7(1), 2053951720933286. https://doi.org/10.1177/2053951720933286
Tomasev et al., 2021
Pauker et al., 2018
Ruberg & Ruelos, 2020
Braun, L., Fausto-Sterling, A., Fullwiley, D., Hammonds, E. M., Nelson, A., Quivers, W., Reverby, S. M., & Shields, A. E. (2007). Racial Categories in Medical Practice: How Useful Are They? PLOS Medicine, 4(9), e271. https://doi.org/10.1371/journal.pmed.0040271
Hanna et al., 2020
Morning, A. (2014). Does Genomics Challenge the Social Construction of Race?: Sociological Theory. https://doi.org/10.1177/0735275114550881
Barabas, C. (2019). Beyond Bias: Re-Imagining the Terms of ‘Ethical AI’ in Criminal Law. SSRN Electronic Journal. https://doi.org/10.2139/ssrn.3377921
Barabas, 2019
Hacking, 1995
Hacking, 1995
Dembroff, 2018
Andrus et al., 2021
Holstein, K., Vaughan, J. W., Daumé III, H., Dudík, M., & Wallach, H. (2019). Improving fairness in machine learning systems: What do industry practitioners need? Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19, 1–16. https://doi.org/10.1145/3290605.3300830
Rakova, B., Yang, J., Cramer, H., & Chowdhury, R. (2021). Where Responsible AI meets Reality: Practitioner Perspectives on Enablers for shifting Organizational Practices. ArXiv:2006.12358 (Cs). https://doi.org/10.1145/3449081
Wachter, S., Mittelstadt, B., & Russell, C. (2021). Why Fairness Cannot Be Automated: Bridging the Gap Between EU Non-Discrimination Law and AI. Computer Law & Security Review, 41. https://doi.org/10.2139/ssrn.3547922
Xenidis, R. (2021). Tuning EU Equality Law to Algorithmic Discrimination: Three Pathways to Resilience. Maastricht Journal of European and Comparative Law, 27, 1023263X2098217. https://doi.org/10.1177/1023263X20982173
Xiang, A. (2021). Reconciling legal and technical approaches to algorithmic bias. Tennessee Law Review, 88(3).
Balayn & Gürses, 2021
Fazelpour, S., & Lipton, Z. C. (2020). Algorithmic Fairness from a Non-ideal Perspective. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 57–63. https://doi.org/10.1145/3375627.3375828
Green & Viljoen, 2020
Green, B., & Viljoen, S. (2020). Algorithmic realism: Expanding the boundaries of algorithmic thought. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 19–31. https://doi.org/10.1145/3351095.3372840
Gitelman, L. (2013). Raw Data Is an Oxymoron. MIT Press.
Barabas, C., Doyle, C., Rubinovitz, J., & Dinakar, K. (2020). Studying Up: Reorienting the study of algorithmic fairness around issues of power. 10.
Crooks, R., & Currie, M. (2021). Numbers will not save us: Agonistic data practices. The Information Society, 0(0), 1–19. https://doi.org/10.1080/01972243.2021.1920081
Muhammad, K. G. (2019). The Condemnation of Blackness: Race, Crime, and the Making of Modern Urban America, With a New Preface. Harvard University Press.
Ochigame, R., Barabas, C., Dinakar, K., Virza, M., & Ito, J. (2018). Beyond Legitimation: Rethinking Fairness, Interpretability, and Accuracy in Machine Learning. International Conference on Machine Learning, 6.
Ochigame et al., 2018
Basu, S., Berman, R., Bloomston, A., Cambell, J., Diaz, A., Era, N., Evans, B., Palkar, S., & Wharton, S. (2020). Measuring discrepancies in Airbnb guest acceptance rates using anonymized demographic data. AirBnB. https://news.airbnb.com/wp-content/uploads/sites/4/2020/06/Project-Lighthouse-Airbnb-2020-06-12.pdf

ABOUT ML Foundational Resource

Overview

ABOUT ML (Annotation and Benchmarking on Understanding and Transparency of Machine learning Lifecycles) is a multi-year, multi-stakeholder initiative aimed at building transparency into the AI development process, industry-wide, through full lifecycle documentation. On this page, you will find the collected outputs of ABOUT ML, a library of resources designed to help organizations and individuals begin implementing transparency at scale. To further increase the usability of these resources, recommended reading plans for different readers are provided below.

Learn more about the origins of ABOUT ML and contributors to the project here.

Content Type: Report

Policy Alignment on AI Transparency

Frameworks Reviewed

Mapping Documentation Requirements Across the Frameworks

Other Features of the Frameworks Relevant to Interoperability

Summary of Findings

Summary of Recommendations

Background and Methodology

Eyes Off My Data

Exploring Differentially Private Federated Statistics To Support Algorithmic Bias Assessments Across Demographic Groups

Executive Summary

Executive Summary

Eyes Off My Data

Executive Summary

Introduction

Premise of the Project

A Sociotechnical Framework for Assessing Demographic Data Collection

Differentially Private Federated Statistics

A Sociotechnical Examination of Differentially Private Federated Statistics as an Algorithmic Fairness Technique

Conclusion

Acknowledgments

Funding Disclosure

Appendices

AI and Job Quality

Insights from Frontline Workers

Executive Summary

AI and Job Quality

Executive Summary

Introduction

Our Approach

Major Themes and Findings

Opportunities for Impact

Conclusion

Acknowledgements

Appendix 1: Detailed Site and Technology Descriptions

Appendix 2: Research Methods

Sources Cited

Making AI Inclusive: 4 Guiding Principles for Ethical Engagement

Introduction

Making AI Inclusive: 4 Guiding Principles for Ethical Engagement

Introduction

Guiding Principles for Ethical Participatory Engagement

Recommendations for Ethical Engagement in Practice

Conclusion

Acknowledgements

Sources Cited

After the Offer: The Role of Attrition in AI’s ‘Diversity Problem’

Executive Summary

Key Takeaways

Key Takeaways

1. Diversity makes for better team climates

2. Systemic supports are difficult but necessary to undo the current harms to minoritized workers

3. Individual efforts to change organizational culture fall disproportionately on minoritized folks who are usually not professionally rewarded for their efforts

Recommendations

Recommendations

1. Organizations must systemically support ERGs

2. Organizations must intentionally diversify leadership and managers

3. DEI trainings must be specific in order to be effective and be more connected to the content of AI work

4. Organizations must interrogate their values as practiced and fundamentally alter them to include the perspectives of people who are not White, cis, or male

After the Offer: The Role of Attrition in AI’s ‘Diversity Problem’

Executive Summary

Introduction

Background

Current Study and Methodology

Results

Summary and the Path Forward

Acknowledgements

Appendices

Sources Cited

Fairer Algorithmic Decision-Making and Its Consequences: Interrogating the Risks and Benefits of Demographic Data Collection, Use, and Non-Use

Introduction and Background

Introduction

Introduction

Background

Background

Fairer Algorithmic Decision-Making and Its Consequences: Interrogating the Risks and Benefits of Demographic Data Collection, Use, and Non-Use

Introduction and Background

Social Risks of Non-Use

Social Risks of Use

Conclusion and Acknowledgements