Conclusion

Conslusion

An important part of responsible AI development is recognizing that it is difficult, if not impossible, to release an algorithmically-driven feature or product that is guaranteed to work every time for all people and situations. Rigorous pre- and post-deployment fairness assessments are necessary to surface any potential bias in algorithmic systems. Post-deployment fairness assessments can pose additional challenges to organizations, as they often involve collecting new user data, including sensitive demographic data, to observe whether the algorithm is operating in ways that disadvantage any specific group of people. The collection and use of demographic data is recognized to be challenging for organizations due to concerns related to data privacy, data security, and legal barriers. Demographic data collection also poses key risks to data subjects and communities such as data misuse or abuse of data (including potential discriminatory uses), as well as harms stemming from misrepresentation and miscategorization in datasets.

In an effort to deploy algorithmically driven features responsibly, Apple introduced IDs in Apple Wallet with mechanisms in place for Apple (and the identification card issuing state authority) to identify any potential biases users may experience when setting up or using their new digital ID. Currently only available in the United States, Apple applied differentially private federated statistics as a way to protect users’ data, including their demographic data, as part of IDs in Apple Wallet. The main benefit of using differentially private federated statistics is the preservation of data privacy by combining the features of differential privacy (e.g., adding statistical noise to data to prevent re-identification) and federated statistics (e.g., analyzing user data on individual devices, rather than on a central server, to avoid the creation of datasets that can be hacked or otherwise misused).

A member organization of Partnership on AI (PAI), Apple shared details about the use of differentially private federated statistics in a US context for discussion by responsible AI, algorithmic fairness, and social inequality experts across two convenings. Independently organized and hosted by PAI, the two expert convenings discussed how algorithmic fairness assessments are strengthened, challenged, or otherwise unaffected by the use of differentially private federated statistics. PAI applies a sociotechnical lens to various AI issues, including algorithmic fairness and bias issues, in order to draw attention to the complex ways AI can have social impact, particularly for marginalized demographic groups.

Expert participants were asked to consider not only the specific technical strengths or weaknesses of differentially private federated statistics but how this approach interacts with an overall algorithmic fairness strategy. Recognizing that data privacy and security are not the only concerns people have regarding the collection and use of their demographic data, participants were directed to consider whether differentially private federated statistics could also be leveraged to attend to some of the other social risks that can arise.

The expert participants — drawn from commercial AI companies, research institutions, and civil society organizations — emphasized the importance of having both pre- and post-deployment algorithmic fairness assessments throughout the development and deployment of an AI-driven system or product/feature. Post-deployment assessments are especially important as they enable organizations to monitor algorithmic systems once deployed in real-life social, political, and economic contexts. They also recognized the importance of thoughtfully collecting some demographic data in order to help identify group-level algorithmic harms.

The expert participants, however, clearly noted that a secure and privacy-preserving way of collecting and analyzing sensitive user data is, on its own, insufficient to deal with the risks and harms of algorithmic bias. In fact, they expressed that such a technique is not entirely sufficient for dealing with the risks and harms of collecting demographic data. Instead, the convening participants identified key choice points facing AI-developing organizations to ensure the use of differentially private federated statistics contributes to overall alignment with responsible AI principles and ethical demographic data collection and use.

The following tables (Tables 2 and 3) summarize the different choice points and recommendations for best practices identified by the expert participants. Recommendations are organized into two types:

1. general considerations that any AI-developing organization should consider for their post-deployment algorithmic fairness assessment (Table 2)
2. design choices specifically related to the use of differentially private federated statistics within a post-deployment algorithmic fairness strategy (Table 3)

The choice points identified by the expert participants summarized in Table 2 emphasize the importance of carefully applying differentially private federated statistics in the context of algorithmic bias assessment. They noted that several features of the technique can be determined in such a way that reduces the efficacy of the privacy-preserving and security-enhancing aspects of differentially private federated statistics. Several expert participants highlighted Apple’s decision to limit the data retention period (90 days), clearly and simply sharing what data the user will be providing for the assessment, and maintaining organizational oversight of the query process and parameters as aligning with the best practices they would recommend.

Many of the recommendations surfaced by the expert participants focus on the resources (e.g., financial, time allocation, and staffing) necessary to achieve a level of alignment and clarity on the nature of “fairness” and “equity” AI-developing organizations are seeking for their AI-driven tools and products/features before integrating differentially private federated statistics into their overall bias mitigation strategy. While these considerations may seem tangential, the experts emphasized the importance of establishing a robust foundation on which differentially private federated statistics could be effectively utilized. Any form of demographic data collection or use can expose people to potential risk or harm. Regardless of the steps taken to minimize such risk, the collection of demographic data without an explicit purpose or effective plan for its responsible usage is not justifiable given the potential individual or societal cost. Differentially private federated statistics, in and of itself, does not mitigate all the potential risks and harms related to collecting and analyzing sensitive demographic data. It can, however, strengthen overall algorithmic fairness assessment strategies by supporting better data privacy and security throughout the assessment process.

TABLE 2: General Considerations for Algorithmic Fairness Assessment Strategies

TABLE 3: Design Considerations for Differential Private Federated Statistics