Guidance for Inclusive AI

The use of LLMs to generate summaries of raw data is seen as a way to reduce the amount of human labor needed to review, process, and analyze large amounts of information or data quickly. This may be particularly attractive to small teams with limited capacity and time, or teams that are less familiar with qualitative research methodologies.

• Generating summaries: When a corpus of data is text-based and highly structured (e.g., journal articles), LLM-based tools can be useful for generating more succinct summaries of the text. This can be useful, too, with interview transcripts that have been processed and cleaned to be more legible.
• Brainstorming tool: Particularly for teams that are small in size (or for a public engagement lead or a researcher working alone), LLM tools (e.g., ChatGPT) may be useful for brainstorming prompts and activities for public engagement. As when working with a synthetic participant (AI agent), LLM-based tools may be useful for fine-tuning details of public engagement activities and the prompts used to solicit feedback from human participants.

• Conflating summaries for analyses: LLMs are presently able to identify patterns in text, thereby providing good summaries. To distinguish a significant point that is an outlier worth addressing more directly (versus a blip) requires human critical analysis, as when additional context is needed to make sense of a discussion. Users should beware the limitations of AI tools and should use them within the scope of their capacities; it is risky to presume that these tools are capable of human-level (highly trained) analysis.
• Replacement of human expertise: Domain and subject-matter expertise are important requirements for both analyzing data and designing materials and strategies for public engagement. While LLMs may be good brainstorming tools to help supplement such expertise, it should not be assumed that they are suitable replacements for people.
• Real cost of using AI tools: One of the main selling points for AI-driven tools of all kinds is that they are faster and cost less than working with human beings. Such claims often ignore or underestimate the cost of the computing needed for the synthetic data. Computing costs entail not only expense for hardware but also the cost of such natural resources as electricity (to power computing systems), water (to cool them), and minerals (to construct the hardware). Cost comparisons between working with real humans and AI tools should include the true total cost of running AI models.

The use of AI agents as synthetic participants or facilitators is seen as cutting the expense and time needed to recruit humans and work with them to solicit the feedback needed for projects. It is also seen as a way to avoid challenges around choosing the appropriate or most relevant groups to work with; the lower financial cost and time commitment required to use synthetic participants allows researchers to experiment with many different personas.

• High-level overview of different communities: AI-generated user personas are not a substitute for conducting the necessary literature review and expert discussions to build a foundational awareness of different socially marginalized groups and the histories/issues facing individuals in these communities. But AI-generated user personas trained on data drawn from real humans can provide high-level overviews of different types of users by synthesizing large volumes of available data about specific user groups. The responses generated by these personas may be able to succinctly capture known behaviors and experiences of real people.
• Refine public engagement activities and strategies: Synthetic users can serve as useful starting places for designing public engagement activities and strategies; early drafts can be refined through feedback provided by AI-generated user personas to identify any glaring issues with activities and survey instruments, or even to brainstorm additional social identities and characteristics that should be considered.
• Supplement public engagement interviewers: In situations where it is best for participants to engage via written text chats, it may be possible to supplement the number of human interviewers with the use of AI chatbots to ask participants questions. However, in alignment with other best practices about AI use, this should be done with full transparency that the participant will be working with an AI chatbot. It should not be used for highly sensitive topics on which interviewer empathy is critical (e.g., topics related to violent experiences, mental health issues, etc.) and should offer participants the option to work with a human interviewer if they so choose.
• Make public engagement feel more accessible to practitioners: Working with synthetic users can feel less risky, particularly for AI practitioners who are unfamiliar with public engagement and are first interacting with a synthetic participant to learn how to conduct public engagement activities and build meaningful relationships with different communities. Using synthetic participants could help train new practitioners to hone talking points and different approaches, as well as receive feedback about how they could adjust their work to become more effective.

• Neglects the purpose of interacting with the public: Human behavior and the relationship between humans and technology (and one another) is complex and ever-changing. Working with synthetic participants limits what we can learn because they are based on specific snapshots of behavior; they are limited in terms of what is captured, when, and where. One of the main purposes of conducting public engagement is to discover human experiences and interactions that have yet to be explored, understood, or documented.
• Can erode trust with human participants: If humans are part of the interaction but are unaware they are engaging with an AI system, this may erode trust not only in the engagement activity but in the organization sponsoring the engagement.
• Over-generalization of human experience: Synthetic users may be effective at reflecting the average user, but may not be able to capture the nuances or important differences that outlier users bring to public engagement; atypical experiences are important signals to identify issues. Also, replacing genuine lived experiences with synthesized amalgams of experiences can create ambiguity about what is a “ground truth” (really happening) versus what could happen (but has not). AI chatbots, as they currently exist, are not good models of human behavior, as they feature a tendency toward sycophancy (giving responses aligned with what the user wants to be told).
• Real cost of running synthetic participants and facilitators: One of the main selling points of using AI-driven tools of all kinds is that it is faster and less costly than working with other human beings. However, these claims often ignore or underestimate the cost of computing needed for the synthetic data. Computing costs not only involve buying the necessary hardware but also entail expenses for natural resources like electricity (to power computing systems), water (to cool them), and minerals (to construct the hardware). Cost comparisons between working with real humans should be compared against the total cost of running AI models.

The use of synthetic data is often seen as a way to address issues of consent, fair use, and intense surveillance of populations by bypassing the collection of data from people. It is also perceived to be more cost efficient to generate synthetic data rather than pay for the use of an individual’s data and the data enrichment labor required to prepare it for training or testing. The usage of synthetic data has been extended across several domains (e.g., health, privacy, red teaming) and formats such as visuals, audio, and text. Synthetic data may be used to create datasets:

• with adversarial examples to detect and understand vulnerabilities in models
• to cover sensitive topics that are highly regulated (e.g., sensitive personal characteristics)
• to improve a model’s ability to effectively handle real-world inputs instead of working with genuine users or experts to collect data from real-world circumstances.

• Overcome limitations of real-world data: Synthetic data helps overcome the limitations of real-world data by providing an abundant, scalable source of training and testing data, which is particularly useful in areas with scarce or hard-to-obtain real-world data.
• Address some concerns around data consent and privacy: Synthesizing data is an alternative to collecting it from seemingly public domains (e.g., public webpages or social media) without consent, thereby addressing such critical issues as data scarcity and privacy.
• Address some concerns around data enrichment labor conditions: Synthetic data is thought to require less human labor and time to generate the datasets needed, as well enabling organizations to bypass some of the harms that can arise from human workers having to work with violent and other hurtful content (e.g., child sexual abuse material, hate speech). Currently, as more attention is drawn to the working conditions and low pay of data enrichment workers, synthetic data may provide a way to create new datasets without violating labor policies.
• Semi-automated approaches: Semiautomated approaches to developing synthetic data — whereby data is synthesized but also reviewed by human annotators and experts — may help fill data gaps while incorporating important contextual knowledge from the human reviewers.

• Generating synthetic datasets requires good “data seeds”: Curating data seeds for synthetic data generation is crucial. If it is not undertaken thoughtfully, it will be difficult for the synthesized data to address any of the issues that arise in using existing, human-generated datasets:
• Data Quality: If the seed data is of poor quality, it can lead to unrealistic or harmfully biased synthetic data.
• Bias: Seed data might reflect inherent biases that can be perpetuated or even amplified in the synthetic data.
• Data Representativeness: The seed data needs to be representative of the real-world scenarios the synthetic data aims to model.
• Domain Expertise: Understanding the domain from which the seed data has been drawn is essential. Inaccurate or incomplete domain knowledge can lead to unrealistic or misleading synthetic data.
• Data Diversity: Lack of diversity in seed data can result in synthetic data that does not capture the full spectrum of potential scenarios or variations, leading to limited applicability.
• Lack of documentation/“lineage” and “circumvented consent”: The provenance (and documentation) of synthetic data is as important as the provenance of human-generated datasets. Particularly in cases in which synthetic datasets are used to create models that themselves generate more synthetic data, poor documentation can lead to a potentially long “feedforward” chain that becomes more and more distant from the original data source.
• Difficult to establish “ground truth”: Synthetic data is often generated from models and algorithms that may not fully capture the complexities of real-world data. This makes it difficult to validate it against actual ground truth. For example, in cases in which synthetic datasets include offensive content or adversary prompts, it becomes more difficult to determine what constitutes ground truth: Does the offensive content reflect what is actually circulating in the real world or did the model “hallucinate” because there is an issue with the model? It is also possible to engage in “diversity-washing” wherein dataset bias may be addressed by enhancing representation of data points for underrepresented groups of people; if the data is imputed on assumptions and stereotypes, it will reflect aggregate biases present in the seed datasets. Outlier data points, which are particularly important for understanding the experiences of those who are socially marginalized, may be ignored or disregarded — when these data points might generate important insights for the project.
• Real cost of generating synthetic data: While one of the major selling points of synthesizing data is that it will cost less than ethically collecting and enriching human-generated data, these calculations often ignore or underestimate the cost of the computing needed for the synthetic data. Computing costs involve not only hardware but expenses for such natural resources as electricity (to power computing systems), water (to cool them), and minerals (to construct the hardware). There is also an environmental cost for diminishing important resources. Moreover, at this stage of development, human labor is still required to refine the queries to generate the synthetic data and validate the datasets.
• Unclear regulation: It remains unclear how to regulate the use of synthetic data. While some regulations and standards for synthetic media are already in place, the industry lacks standards to develop and enforce appropriate practices to define parameter settings on the synthetic generation of data, as well as the assessment of the correlation of synthetic and real data.

This tool is often applied to a distinct thread of participatory AI research and governance that is found in AI safety and alignment research and tooling. Many AI research labs and technical researchers are concerned about the possibility that future powerful “superintelligent” systems (those with intelligence far beyond human capacity and networked into physical infrastructures) pose the risk of massive — even existential — harm to the well-being of humanity and Earth’s ecological condition. Some technology developers are striving to build AI systems that share general social and political values with the public. To establish these sets of values, they will use surveying platforms to solicit general, non-specific ideas from thousands of people about how they feel about AI and its use. These questions may be very general (not for a specific use case) because the values need to be broad enough to suit many different situations. For such questions, understanding the sentiments held by a broad array of people is useful.

• Can match the global scale of the technology: Because it is optimized to work with large audiences, these platforms can make it possible to reach the full breadth of people who are ultimately impacted by AI technology. By surveying large groups of people, it is possible to see group patterns emerge or be identified in real time. This addresses one of the issues with smaller-scale approaches to public engagement: Being able to distinguish between individual personal experience and patterns that operate at a group/community level is challenging when working with only a small group of participants.
• Can bolster the capacity of small teams: By offering a platform for feedback solicitation, engagements with different communities could be replicated and re-implemented many times. Platforms that offer real-time clustering or grouping of feedback generated by participants can support more dynamic facilitation of the discussion in ways that small sponsor groups would be unable to do without technological support.

• Fixation on scale: While these platforms are generally agnostic about what is the “right” kind of feedback to help guide technology development, the emphasis on their ability to scale to large audiences may orient users toward trying to find consensus on the experiences of “average” people rather than further exploring those of “outliers.”
• Scoping matters: Given the ability to solicit feedback from a large body of people, it is possible to exploit the platform to show “evidence” for broad-based acceptance or agreement with ideas outlined by the host, and this can lead to “participant-washing.” How the topic for discussion is scoped and how questions are posed are both very important issues that can t strongly affect the conclusions that are generated.
• Selection bias: The platforms often focus on written survey questions and written text responses from participants, as well as access to computing devices that can run the platform applications. This may limit participation to people who are literate, have access to computing devices with sufficient power to run the application, and have data access that is consistent enough to participate. When using these platforms, it is important for organizations to consider who may lack the means to access the platform and whether adaptations can be made to render it more accessible (e.g., questions delivered and answered vocally for those with limited literacy, working with local partners to offer physical sites for meetings that are equipped with the necessary equipment).