Social Risks of Use

Social Risks of Use

When demographic data is used, it carries risks for both individuals and groups. Here, we discuss both the affordances and limitations of using demographic data to detect and mitigate discrimination in institutional decision-making more broadly. Our goal is not to suggest that demographic data shouldn’t be used, but rather to build out a clearer picture of what it is we are trying to use it for so as to outline the minimum conditions we expect our demographic data governance strategies to enable.

Risks to Individuals

Risks to Individuals

Encroachments on Privacy and Personal Life

Likely the first concern that many would have when it comes to collecting or using sensitive demographic data are the risks from breaching individual privacy. Demographic attributes such as race, ethnicity, country of birth, gender, and sexuality are rarely inconsequential aspects of one’s identity that can be shared or learned without risk. Quite to the contrary, sharing or otherwise determining these attributes can expose individuals to various forms of direct or indirect harm, especially already marginalized and vulnerable individuals. Though there are numerous proposed methods for ensuring the privacy and security of sensitive attributes, the strategies for assessing (let alone mitigating) fairness or discrimination under privacy constraints are still very experimental Farrand, T., Mireshghallah, F., Singh, S., & Trask, A. (2020). Neither Private Nor Fair: Impact of Data Imbalance on Utility and Fairness in Differential Privacy. Proceedings of the 2020 Workshop on Privacy-Preserving Machine Learning in Practice, 15–19. https://doi.org/10.1145/3411501.3419419 Jagielski, M., Kearns, M., Mao, J., Oprea, A., Roth, A., Sharifi -Malvajerdi, S., & Ullman, J. (2019). Differentially Private Fair Learning. Proceedings of the 36th International Conference on Machine Learning, 3000–3008. https://bit.ly/3rmhET0 Kuppam, S., Mckenna, R., Pujol, D., Hay, M., Machanavajjhala, A., & Miklau, G. (2020). Fair Decision Making using Privacy-Protected Data. ArXiv:1905.12744 (Cs). http://arxiv.org/abs/1905.12744. As such, we should anticipate that any current efforts to collect sensitive demographic attributes will at some point in the pipeline require tying the attributes to individuals, risking individuals’ privacy.

One clear privacy risk of obtaining an individual’s demographics is that these attributes are still the basis for many types of discrimination.

One clear privacy risk of obtaining an individual’s demographics is that these attributes are still the basis for many types of discrimination. Though many countries have laws against direct discrimination. Though many countries have laws against direct discrimination, it is still a common occurrence due to the difficulty of proving discrimination in individual cases. In domains such as hiring Quillian, L., Pager, D., Hexel, O., & Midtbøen, A. H. (2017). Meta-analysis of field experiments shows no change in racial discrimination in hiring over time. Proceedings of the National Academy of Sciences, 114(41), 10870–10875. https://doi.org/10.1073/pnas.1706255114 Quillian, L., Lee, J. J., & Oliver, M. (2020). Evidence from Field Experiments in Hiring Shows Substantial Additional Racial Discrimination after the Callback. Social Forces, 99(2), 732–759. https://doi.org/10.1093/sf/soaa026, advertising Cabañas, J. G., Cuevas, Á., Arrate, A., & Cuevas, R. (2021). Does Facebook use sensitive data for advertising purposes? Communications of the ACM, 64(1), 62–69. https://doi.org/10.1145/3426361 Datta, A., Tschantz, M. C., & Datta, A. (2015). Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination. Proceedings on Privacy Enhancing Technologies, 2015(1), 92–112. https://doi.org/10.1515/popets-2015-0007, and pricing Hupperich, T., Tatang, D., Wilkop, N., & Holz, T. (2018). An Empirical Study on Online Price Differentiation. Proceedings of the Eighth ACM Conference on Data and Application Security and Privacy, 76–83. https://doi.org/10.1145/3176258.3176338 Mikians, J., Gyarmati, L., Erramilli, V., & Laoutaris, N. (2013). Crowd-assisted search for price discrimination in e-commerce: First results. Proceedings of the Ninth ACM Conference on Emerging Networking Experiments and Technologies, 1–6. https://doi.org/10.1145/2535372.2535415, direct forms of discrimination, algorithmically mediated or not, are relatively common. For domains like advertising, discriminatory practices are often justified by claims that differential treatment results in better services, which may in fact be true. However, in a recent survey study of Facebook users, most were still uncomfortable with sensitive attributes being used as the basis for decisions around what they are being shown Cabañas et al., 2021.

In the most pernicious cases, demographic attributes can be used as the criteria for various forms of state or societally enacted violence, such as detainment and deportation based on documentation status in the United States. Even in cases where the sensitive attribute (e.g., documentation status) is not collected, other collected attributes (e.g., country of birth and spoken language) can be used to help infer the targeted attribute. As corporate data becomes increasingly requested by and made available to state agencies Leetaru, K. (2018, July 20). Facebook As The Ultimate Government Surveillance Tool? Forbes. https://www.forbes.com/sites/kalevleetaru/2018/07/20/facebook-as-the-ultimate-government-surveillance-tool/ Rozenshtein, A. Z. (2018). Surveillance Intermediaries (SSRN Scholarly Paper ID 2935321). Social Science Research Network. https://papers.ssrn.com/abstract=2935321, it is critical that practitioners consider what types of identity-based violence individuals might be exposed to by sharing certain attributes.

A commonly suggested approach to reducing these forms of direct targeting risk is to “anonymize” or “de-identify” datasets. Experimental methods, however, have achieved high “re-identification” accuracy for datasets with numerous demographic attributes Rocher, L., Hendrickx, J. M., & de Montjoye, Y.-A. (2019). Estimating the success of re-identifications in incomplete datasets using generative models. Nature Communications, 10(1), 3069. https://doi.org/10.1038/s41467-019-10933-3. Marginalized individuals are especially vulnerable to these types of re-identification strategies, as there tend to be fewer data subjects in datasets that share their demographic attributes. Attempting to address this problem, researchers have proposed various differential privacy techniques for ensuring both a technical definition of fairness and non-identifiability, but these approaches are experimental and can inhibit other types of demographic analysis Cummings, R., Gupta, V., Kimpara, D., & Morgenstern, J. (2019). On the Compatibility of Privacy and Fairness. Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization – UMAP’19 Adjunct, 309–315. https://doi.org/10.1145/3314183.3323847 Kuppam et al., 2020.

Finally, another salient privacy risk to consider is the possible loss of autonomy over one’s identity and interactions when demographic data is collected or used. Machine learning and AI systems are often built with the intention of making generalizations across groups in order to categorize individuals, meaning that it is not even necessary for an individual to share their demographic attributes in order for the system to decide to treat them as a “Black woman” or “Asian man.” Simply by matching patterns of behavior, algorithmic systems can categorize individuals, even if the categories are not explicitly labeled “Black woman” or “Asian man”* Mavriki, P., & Karyda, M. (2019). Automated data-driven profiling: Threats for group privacy. Information & Computer Security, 28(2), 183–197. https://doi.org/10.1108/ICS-04-2019-0048. Barocas and Levy (2019) Barocas, S., & Levy, K. (2019). Privacy Dependencies (SSRN Scholarly Paper ID 3447384). Social Science Research Network. https://papers.ssrn.com/abstract=3447384 refer to these types of associations between individuals as privacy dependencies, as an individual’s privacy quite literally depends on the privacy of the people like them. In other cases, even when users provide sensitive data about themselves, platforms may not take that data into account when making decisions for that user, subverting their agency around self-presentation Bivens, R. (2017). The gender binary will not be deprogrammed: Ten years of coding gender on Facebook. New Media & Society, 19(6), 880–898. https://doi.org/10.1177/1461444815621527.

For many of these privacy risks, we might expect privacy regulation such as the GDPR or California’s CCPA to prevent the worst abuses. Privacy regulation to date, however, has largely focused on the individual’s “right to privacy” and agency over their own personal data Mittelstadt, B. (2017). From Individual to Group Privacy in Big Data Analytics. Philosophy & Technology, 30(4), 475–494. https://doi.org/10.1007/s13347-017-0253-7. As we just discussed, an individual’s sensitive attributes need not be explicitly collected or inferred in order for algorithmic systems to treat them as part of a specific group. Even when it comes to an individuals’ agency over data about them specifically, the relationship between individuals and the tech firms collecting their data is frequently one of “convention consent.” Taylor, 2021 In other words, users are resigned to provide data even when they do not agree with how it is being used because it is the cost of accessing platforms and services and they do not see any reasonable alternative Draper and Turow, 2019. While there is technically always the option of not using platforms or services that require personal data, many have come to serve as essential infrastructure, calling into question how much someone can afford to hold onto their privacy by withholding their consent.

Individual Misrepresentation

In an effort to mitigate bias, some organizations seek to make their datasets more “representative” by including more data on different demographic categories such as race and gender. However, this is often done without a deeper engagement with the categories themselves or the collection methods used. How demographic data is coded and represented in datasets — specifically, what categories are being used to define individual characteristics — can have an enormous impact on the representation of marginalized individuals. In the context of ADMS, individual misrepresentation can lead to discrimination and disparate impacts.

How demographic data is coded & represented in datasets can have an enormous impact on the representation of marginalized individuals.

Gender and race are two demographic categories that have long and complex socio-political histories of classification. Yet, many current algorithmic fairness methodologies fail to account for the socially constructed nature of race and gender, instead treating these categories as fixed, indisputable, apolitical attributes Hanna, A., Denton, E., Smart, A., & Smith-Loud, J. (2020). Towards a Critical Race Methodology in Algorithmic Fairness. Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, 501–512. https://doi.org/10.1145/3351095.3372826 Keyes, O., Hitzig, Z., & Blell, M. (2021). Truth from the machine: Artificial intelligence and the materialization of identity. Interdisciplinary Science Reviews, 46(1–2), 158–175. https://doi.org/10.1080/03080188.2020.1840224 Scheuerman, M. K., Wade, K., Lustig, C., & Brubaker, J. R. (2020). How We’ve Taught Algorithms to See Identity: Constructing Race and Gender in Image Databases for Facial Analysis. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW1), 1–35. https://doi.org/10.1145/3392866. These two types of demographic categories are highly contextual, and debates and legislation around gender and race classification are constantly evolving.

Misrepresentation can occur both when the categories used do not adequately represent individuals as they self-identify and when an individual is misclassified despite there being a representative category that they could have been classified as. To better understand the implications of misrepresentation, it’s important to understand the different dimensions of identity and how these can lead to misrepresentation. With respect to racial identity, Roth (2016) Roth, W. D. (2016). The multiple dimensions of race. Ethnic and Racial Studies, 39(8), 1310–1338. https://doi.org/10.1080/01419870.2016.1140793 distinguishes between multiple dimensions of the concept of race, highlighting how an individual’s racial identity can be represented differently depending on the observer or method of data collection. Dimensions of racial identity include self-identity (the race an individual self-identifies as), self-classification (the racial category an individual identifies with on an official form), observed race (the race others believe you to be), appearance-based (observed race based on readily observable characteristics), interaction-based (observed race based on characteristics revealed through interaction such as language, accent, surname), reflected race (the race you believe others assume you to be), and phenotype (racial appearance) Hanna et al., 2020. When racial data collection is conducted by observation, either by person or machine, there is the risk that an individual’s observed race does not align with their self-identification and can lead to individual misrepresentation. Moreover, treating the notion of identity as a quality that can be “inferred” externally produces new forms of control over an individual’s agency to define themselves Keyes, O. (2018). The Misgendering Machines: Trans/HCI Implications of Automatic Gender Recognition. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW), 88:1-88:22. https://doi.org/10.1145/3274357 Keyes, O. (2019, April 8). Counting the Countless. Real Life. https://reallifemag.com/counting-the-countless/ Keyes, O., Hitzig, Z., & Blell, M. (2021). Truth from the machine: Artificial intelligence and the materialization of identity. Interdisciplinary Science Reviews, 46(1–2), 158–175. https://doi.org/10.1080/03080188.2020.1840224.

Facial recognition technologies are a prominent case where the harm of misrepresentation occurs, since categorization is often based solely on observable characteristics. Additionally, many databases include a binary, physiological perspective of female and male, and consequently misrepresent individuals who do not self-identify with those categories Scheuerman et al., 2020. Continuing to build databases that assume identity is a fixed, observable trait risks reinforcing harmful practices of marginalization. Additionally, doing so can further entrench pseudoscientific practices which assume invisible aspects of one’s identity from visible characteristics such as physiognomy Scheuerman et al., 2020 Stark, L., & Hutson, J. (2021). Physiognomic Artificial Intelligence (SSRN Scholarly Paper ID 3927300). Social Science Research Network. https://doi.org/10.2139/ssrn.3927300.

Data Misuse and Use Beyond Informed Consent

Once collected, sensitive demographic data can be susceptible to misuse. Misuse refers to the use of data for a purpose other than that for which it was collected or consent was obtained. Specifically in the context of ADMS, this could involve collecting and using data to train models that may be deployed in unexpected contexts or re-purposed for other goals. In practice, it is difficult for organizations to specify clear data uses at the point of collection. Sensitive data, in this case, can go on to inform systems beyond the initial scope defined during collection. For example, in 2019 the U.S. government developed the Prisoner Assessment Tool Targeting Estimated Risk and Needs (PATTERN). PATTERN was trained on data including demographic characteristics and criminal history for the purpose of assessing recidivism risk and providing guidance on recidivism reduction programming and productive activities for incarcerated people U.S. Department of Justice. (2019). The First Step Act of 2018: Risk and Needs Assessment System. Office of the Attorney General.. Then, in March 2020 the Bureau of Prisons was directed to begin using PATTERN to determine which individuals to transfer from federal prison to home confinement in the wake of the COVID-19 pandemic Partnership on AI. (2020). Algorithmic Risk Assessment and COVID-19: Why PATTERN Should Not Be Used. Partnership on AI. http://partnershiponai.org/wp-content/uploads/2021/07/Why-PATTERN-Should-Not-Be-Used.pdf. However, the data used to inform PATTERN was not intended to inform inmate transfers, let alone during a global pandemic which introduced a number of unprecedented social and economic variables.

Data misuse could also refer to instances where data is shared with third parties or packaged and sold to other organizations. A notable example of data misuse in this respect can be seen in Clearview AI’s facial recognition dataset, which the company claims contains over three billion images scraped from social media platforms such as Facebook, Instagram, LinkedIn, and Twitter, along with personal attribute data listed on people’s social media profiles Hill, K. (2020, January 18). The Secretive Company That Might End Privacy as We Know It. The New York Times. https://www.nytimes.com/2020/01/18/technology/clearview-privacy-facial-recognition.html. With this data, Clearview AI developed the world’s most comprehensive facial recognition system, with a dataset beyond the scope of any government agency. Following public backlash, many of the social media platforms claimed that Clearview AI violated their policies. For example, LinkedIn sent Clearview AI a cease-and-desist letter stating that scraping personal data was not permitted under their terms of service, and Facebook released a statement saying it demanded Clearview AI stop scraping data from its platforms in violation of company policy Porter, J. (2020, February 6). Facebook and LinkedIn are latest to demand Clearview stop scraping images for facial recognition tech. The Verge. https://www.theverge.com/2020/2/6/21126063/facebook-clearview-ai-image-scraping-facial-recognition-database-terms-of-service-twitter-youtube. Individuals who have made profiles on these social media platforms and shared their images were not aware that their data was going to be used to develop a facial recognition system used by law enforcement agencies, nor were they asked for their consent.

Corporations collecting and using people’s data to train and deploy ADMS face increased pressure (from both the public and regulatory bodies) for transparency on how such data is collected and used. For example, Article 13 of the GDPR requires companies collecting personal data from a data subject to provide the data subject with information such as the purpose of the data processing, where the data is being processed and by which entity, recipients of the data, the period for which the data will be stored, the existence of algorithmic decision-making and the logic involved, and the right to withdraw data Regulation (EU) 2016/679 (General Data Protection Regulation), (2016) (testimony of European Parliament and Council of European Union). https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32016R0679&from=EN. Companies have begun to incorporate this informational requirement into their data collection practices, often in the form of click wraps, digital banners that appear on users’ screens and require them to “accept all” or “decline” a company’s digital policies. Yet, providing individuals with transparency and information about how data will be used is generally not sufficient to ensure adequate privacy and reputational protections Obar, J. A. (2020). Sunlight alone is not a disinfectant: Consent and the futility of opening Big Data black boxes (without assistance). Big Data & Society, 7(1), 2053951720935615. https://doi.org/10.1177/2053951720935615. Overloading people with descriptions of how their data is used and shared and by what mechanisms is not a way to meaningfully acquire data subjects’ consent, especially in cases where they are sharing sensitive, personal information. Rather, the goals of data use and the network of actors expected to have access to the data are what need to be clearly outlined and agreed upon by the data subject.

Collecting sensitive data consensually requires clear, specific, & limited use as well as strong security & protection following collection. Current consent practices are not meeting this standard.

Additionally, while it may be difficult for organizations to specify clear data uses at the point of collection, companies may consider providing updates as the use cases for that data becomes clearer. In following with this more rigorous notion of consent, we would expect check-ins on how the data was used to assess or mitigate discrimination and on whether the data subjects would still like for their sensitive data to be used towards these ends.

Collecting sensitive data consensually requires clear, specific, and limited use as well as strong security and protection following collection. Current consent practices, including clickwraps and notice and consent frameworks, are not meeting this standard. Instead, these approaches overload individuals with descriptions and information that users see as boring and time-intensive Obar, J. A. (2020). Sunlight alone is not a disinfectant: Consent and the futility of opening Big Data black boxes (without assistance). Big Data & Society, 7(1), 2053951720935615. https://doi.org/10.1177/2053951720935615 Obar, 2020.