1

Identify the outcomes and objectives of adding an AI tool

What opportunities exist in your newsroom for greater efficiencies, faster news production, or better audience engagement? What are the current gaps or improvements that your newsroom is hoping to address?

First, identify the key objectives or outcomes your newsroom is attempting to improve upon. Then, where additional technology and automation can support addressing those objectives, bolster the work already being done in the newsroom or provide avenues for new coverage in the newsroom.

Consult the journalists in your newsroom who will be using the tool as a first step to identifying gaps or pain points that can be supported by AI tools. In addition, bringing together members of the product, tech, and editorial teams (where applicable) in consultation is important to ensure that all views are represented when considering an AI tool. Doing so ensures that the technology is in fact needed by the newsroom, and that it will address an existing problem or aid in the newsroom’s sustainability or growth.

Choose a technology based on clear indications that it: a) will be an investment in the newsroom’s sustainability or growth, and b) is addressing clear objectives. This ensures that the time spent choosing the tool, adding it to the news production process, and any accompanying training required for its use is time well spent. Throughout the tool’s lifecycle, return to the objectives identified in this step to determine if the tool continues to meet its original objectives, or if it is no longer needed and can be retired (see Step 10: Determine when you should retire an AI tool).

The following questions can help identify the objectives and opportunities an AI tool addresses:

Is there an area of work where your newsroom does not have enough support?

In many newsrooms, this may be social media distribution or creating content in multiple formats to expand the reach of the newsroom from written content to podcasts or videos that can be shared on social media. There might not be a dedicated person responsible for content creation, but an AI tool might support that capability in a newsroom (and should include human oversight).

Are there repetitive tasks that take place in the news production or distribution process that can otherwise be automated?

Repetitive tasks may be a starting point for automation as they can alleviate the burden for journalists and give them time back to accomplish more complex work.

Is there a category of tools that your newsroom is particularly excited about? Why?

Sometimes it may be easier to look at what tools your newsroom would be excited to integrate (as long as they still fulfill a need in your newsroom). These tools may not address your newsroom’s most pressing needs. However, it may enhance your newsroom’s adoption and acceptance of AI tools more broadly, and help demonstrate their efficacy in other, perhaps related, contexts.

2

Map out your news production cycle and where an AI tool might fit into existing systems

Start by mapping out what the typical news production pipeline looks like for your newsroom, from ideation to publishing and marketing. This process mapping exercise will support the adoption of an AI tool by allowing you to consider where the tool would be added in your news production cycle before procurement. To streamline the tool’s implementation, determine where the tool will be added to the news production process and communicate that with team members positioned to use or make decisions related to the use of the tool. This is an important step that will set you up for success for the remainder of the adoption process.

In addition, mapping the news production cycle and where an AI tool will be added helps identify the primary use case for the AI tool. Consequently, narrowing down the primary use case for the tool will make it much easier to find the tool that makes sense for your newsroom. It’s not necessarily the cheapest, best-marketed, or most feature-rich tool, but rather the one that makes the most sense for your team’s needs at the moment.

For example, your newsroom might have identified a need for a translation tool to reach a wider audience in your community. Mapping out the news production cycle will help determine when the translation should occur and who will be verifying that translation is accurate.

3

Pinpoint the category of tools you’ll be considering and understand the associated risks

Choose a category of tools to help you more easily identify potential benefits and risks of its use while also narrowing down your choices from the variety of AI tools available to newsrooms.

We have identified five broad categories of AI tools that may assist newsrooms. By starting with these broad categories, newsrooms can begin narrowing down the appropriate AI tools to consider.

Brainerd Dispatch and Michigan Radio use AI tools to cut down monitoring and transcription time

In March 2023, Renee Richardson of the Brainerd Dispatch told The Monitor about her newsroom’s motivation for adopting AI.

“We’re constantly asking our staff to do more and provide more information in many more ways,” said Richardson. “Whether that be social media, video podcasts, audio segments, all of our photography, or all those pieces that go into it. Rarely do we do much that gives them time back. The benefit I see for this is finally giving them that time back.”

Richardson told the Monitor that the Brainerd Dispatch is building an AI program to automate police public safety announcements, a system intended to give journalists time back to work on reporting.

The hope is that the Brainerd Dispatch can cut down on time required to monitor police blotters in the same way that reporters at Michigan Radio WUOM-FM have been able to use an AI transcription tool to create transcripts of City Council and Subcommittee meetings in over 40 cities. “It allows a reporter like me to get a lot more coverage to the audience than I could just doing it by myself,” Dustin Dwyer told The Monitor. The transcriptions aren’t perfect and need to be reviewed by the reporters for accuracy, but they provide a starting point for reporting on important City Council issues without having to sit through hours of meetings.

4

Establish performance benchmarks

If you’ve made it to Step 4, you’ve identified the objectives you’d like an AI tool to support, where it would fit into your news production cycle, and the broad category of tools under consideration. The next step is to benchmark your newsroom’s current performance in the area the tool will be used so you can compare a few tool options.

Establishing a baseline or benchmark for your current newsroom’s performance will help you determine if those tools can support your newsroom’s improvement in that area. Benchmarks also help you measure the efficacy of the tool once you’ve implemented it because they set a baseline for your newsroom’s performance in a particular area prior to AI adoption. The benchmarks selected should also flow from the objectives identified for the use of an AI tool in Step 1. What are the measurable ways you can assess whether the identified objectives are being fulfilled? How do these compare to industry standards or those of other newsrooms?

For example, the following questions can be a great starting point and can lead you to metrics that can be developed into benchmarks moving forward:

• How does your audience come across your content at the moment?
• How does your audience interact with your content? Do they click on links within it, read through to the end, or share the content with others?
• What type of content is your audience most or least interested in?
• What is the demographic diversity of your audience?
• How much time does it take to produce your current content?
• How many content items are produced for your audience daily, weekly, or monthly?
• How does your newsroom’s performance in this area compare to other newsrooms in your geographical area or of your size?

5

Shortlist three to five potential AI tools and interview the tool developers

After establishing benchmarks for your newsroom, consult PAI’s AI Tools for Local Newsrooms Database as well as conduct your own research as to which tools might be best-suited to meet the needs identified in Steps 1 through 3.

From the AI tools available, choose three to five potential tools you’d like to explore.

The database provides a great starting point, particularly given the many sorting filters available. However, you would also benefit from consulting with other newsrooms to find out which tools they’re using and conducting your own research. JournalismAI has surveyed and collected use cases of newsrooms that have used AI to support their work.

Shortlisting three to five tools will help make it easier to really vet each tool, interview its developers, and make an informed decision about which is best-suited for the needs of your newsroom. After you’ve created that list, reach out to each of the vendors for a demo before you make any decisions. Most of their websites will provide a way to request a demo and further discuss how the tool can work for you.

Make sure to take note of the various uses of any tool considered, since it likely can achieve various outcomes. Compare the uses provided to the objective or outcome you’ve identified in Step 1.

What are the gaps between the needs you’ve identified for your newsroom and the services that the tool provides? Some tools require further customization to fit your newsroom’s needs. It is imperative that you spend time asking all the questions and understanding all the ways in which the tool can work for you, the associated costs, and any ethical considerations you should keep in mind during implementation. Ethical considerations about AI implementation — such as fairness, transparency and accountability — should be thought of as an extension of journalistic ethics. As such, understanding a tool completely by asking the right questions will allow you to make the appropriate evaluation of whether it embodies the standards and values of the news operation.

Procurements can take three calls or more with the developer to truly understand the tool and ask all of your questions, so make sure that you factor that time into the process.

Below, we’ve included a list of questions to ask in a demo. Depending on the type of tool you’re considering, its level of sophistication, and how embedded it will be in your newsroom, omit or add to them as necessary.

1. What is the core offering of this tool? What was the primary use case it was designed for?
   • Most tools will have a core offering that it was originally intended for along with add-on features that were added later. It is important that you understand the original purpose of the AI tool because that is what the tool will be optimized for.
2. Are there additional customizations that will be needed for the tool to be used or are you able to put it to work right away?
   • What control does your newsroom have over customizations?
   • What control levers can your newsrooms continue to adjust in the use of the tool after its launch?
3. If additional customization/programming is needed, is the expertise to do so available on your team? Are the AI tool developers able to aid with the process?
4. Are tool customizations saved locally or deployed globally in the tool?
   • Are any customizations made by you to the AI tool then deployed across the various users of the tool in your organization or are they kept specific to each use?
   • Can any customizations made by your organizations be reused by the tool developers?
5. What is the cost of procurement? Is it a recurring or one-time cost?
6. Does the tool require frequent updates?
   • This question will help you determine how often you might need to stop and update the tool and how that might affect your workflow.
7. How do you monitor its efficacy? Are there metrics that are provided by the tool to track its outcomes?
8. Has the tool been used by other newsrooms? Can the developers provide use cases or examples?
   • This question will provide practical insight into the reality of using this tool. If neither can be provided by the developers, then perhaps it might be worth questioning the efficacy and claims made by the tool developers.
   • Are you able to get in touch with these newsrooms to ask about their experience?
9. What liability might the newsroom incur when using this AI tool?

   This is an important question to help identify any privacy or copyright implications of using this tool and glean if the tool developers have actively worked to address and/or support users with such concerns.

10. What data does the AI-developing company collect through the tool when it’s deployed? Is any of the data collected personal information? How might that data be reused?
    • Questions on data collection, storage, and reuse will help you determine if this tool meets the ethical values set by your newsroom.
    • If the data is being reused for other products made by the same company, it would be helpful to know that upfront and decide how to be transparent about that with your newsroom staff and audiences.
11. What is the source of the training data? How often is the tool updated?
    • The source of the training data will let you know if the data was obtained ethically, if it’s diverse enough and includes your primary audience, and if there are potential “blind spots” in the data that you need to account for in its use.
    • For tools that utilize machine learning, PAI has developed a useful resource on understanding and transparency of machine learning life cycles.
12. Is there any consent that should be obtained from the audience if and when they’re interacting with the tool? Is this collected and embedded in the tool, or should you develop your own consent protocol?
13. How is the company funded and how does the company make money? What other industries does it serve?
14. What notification will the AI company provide in the event that there are shifts to the business model?
    • The company may choose to switch its business model or focus down the line from being a newsroom-dedicated product to targeting a different industry. It is important to know if they will notify you of such change and if that industry is acceptable to your newsroom standards. For example, sometimes the AI tool can be used for surveillance purposes or policing as an alternative use case.
15. Are there any ethical considerations that were explored by the developer when creating this tool? What mitigations, if any, have they put in place as a result?

6

Select one or two tools that you would like to procure

Use the responses to the questions posed in Step 5 to eliminate tools that do not meet your requirements and that should narrow your focus to one to two tools.

There will likely be additional considerations that are unique to your newsroom that will factor into your decision-making, such as the cost of the tool, the capacity for and anticipated speed of implementation, or if your newsroom has the expertise for oversight. From there, land on one to two tools that you’d like to shortlist for adoption, from the three to five in earlier stages of the Guide.

If no tools meet your requirements, it might be best to hold off on proceeding further instead of just “making a tool work” for the time being. “Technological path dependency” is when organizations continue using a tool because they’ve invested money and time in adoption, training, and incorporating it — even when it is not the best option. Technological path dependency makes it increasingly difficult to swap tools after you’ve invested time and money in adoption, training, and incorporating one effectively, making your newsroom operation dependent on it. This is a hazard for newsrooms and therefore it is better to hold off implementing a tool until you find one that is best for your organization in the long run.

7

Outline the potential benefits and drawbacks of implementing this tool

Now that you’ve chosen your top tool, it’s time to turn to implementation considerations within your newsroom. To plan for implementation, it is important to be clear about how the tool will be used within the newsroom and the oversight that will be required. Even the most responsible and ethical tool cannot be implemented without continuous oversight of its use and outputs. In order to appropriately gauge all the above, and put in the mechanisms to mitigate any negative outcomes, you must first understand what those outcomes might be.

A thorough vetting process ensures that leadership and staff are aligned when it comes to opportunities, objectives, and risks. Furthermore, it provides clarity on how any risks can be mitigated by your newsroom.

Questions to consider when creating the list of opportunities and risks to mitigate:

Ultimately, the newsroom should consider the tradeoffs between the business opportunities that using the AI tool provides and the ethical considerations of how the use of the tool will impact the newsroom staff, audiences, and news production.

Lost Coast Outpost limited the use of its AI tool to mitigate risks

According to The Monitor, Lost Coast Outpost — a local newsroom in Humboldt County, California — built LoCOBot, an AI tool capable of scraping, summarizing and writing articles (using ChatGPT) from City Council meeting minutes. Lost Coast elected to use the bot exclusively to scrape and summarize the meeting minutes. Lost Coast told the Monitor that it has chosen to continue being cautious and transparent with its audiences as well, labeling any work that is performed by the program.

CNET’s implementation of an AI tool without appropriate oversight led to widespread misreporting

News outlets, including Futurism, reported extensively on CNET’s use of AI to produce written articles and its failure to disclose this approach to its audiences. When the news first broke in January of 2023, many journalists documented their dismay at the lack of disclosure and the misuse of AI in the journalistic context. In particular, CNET was using automated writers for detailed explainers and long-form writing as opposed to regular reporting on more data-driven insights, like sports results or real estate prices as some other outlets do. Articles written using factual data points such as sports scores tend to be short, providing only information on the game results and standings. This makes them less prone to mistakes as the AI writer is simply reformatting the data provided into fact-based sentences on game results.

CNET responded by adding bylines referencing AI as the author to previous articles where AI had been involved in writing the content as well as disclaimers at the onset of the article to that effect. Later, The Verge reported that CNET also used AI-generated content to re-write some of its published articles to increase their visibility on search engines, since optimizing language for algorithmic distribution can support engagement. Oftentimes, this left the article with disjointed pieces of content. According to The Verge, an editor from CNET would review the content to make sure the article makes sense and to remove any misinformation, but often did so months after publishing, limiting the effectiveness of the review. The Verge conducted a thorough investigation on CNET outlining the many ways AI tools were mismanaged by the news outlet, leading to misreporting and overall mistrust by its journalists and audiences.

If CNET had better governance in place for how the tool ought to be used and disclosed to the audiences when an article was written or updated by an automated writer, they would have operated more responsibly and likely received less backlash. Transparency with their audiences would have also allowed them to receive meaningful, productive feedback from readers and experts in the field alike on how they are using AI in the newsroom and where using automated writing is acceptable.

8

Set up your newsroom for success after procurement

Now that you’ve decided on the tool, procured it, and outlined benefits and drawbacks to its use, there are a few more governance and measurement mechanisms to put in place in your organization. This step focuses on the questions to ask to establish appropriate governance and develop sound metrics to determine the AI tool’s impact.

Decide:

1. Who is primarily responsible for the tool and its use
   • It is imperative that a single person be identified in the newsroom as responsible for decision-making and oversight of the use of an AI tool. This ensures that there are clear lines of accountability within the newsroom and, should the AI tool make mistakes, it is clear who to contact to make changes.
2. Who will be able to use the tool in your newsroom
3. What metrics you will track to determine the efficacy of the tool
4. What happens when things go wrong, what is the plan for course correction, and what is the communication plan
5. If you can pilot the AI tool before implementing it across the board
   • This ensures that any governance questions that have not been addressed are considered in the pilot and any previously unaccounted for risks are mitigated. Furthermore, this ensures that staff get a chance to familiarize themselves with the tool before it’s implemented across all news stories.
6. How you will measure the efficacy of the transparency measures adopted in Step 7
   • The transparency measures adopted in Step 7 should be monitored for their efficacy to understand how your audience is responding to them. It is important to measure if they are effective in conveying to your audience when AI is being used, especially as it impacts the content they’re seeing.
   • Putting in place avenues for feedback from both the newsroom staff and from the public will help glean how they’re receiving those transparency measures.
7. What the necessary changes are in your newsroom (such as workflow redesign and retraining) to ensure the successful launch of the tool
8. What mechanisms you will have to collect feedback on the use of the tool from the users in your newsroom or from audiences
9. What methods of redress you will provide for your audience and staff members to provide feedback/report errors on the use (or misuse) of the tool

Governance helps set a newsroom up for success, helping newsrooms measure the efficacy of the tool and its impact. The next section outlines various ways a newsroom can measure the efficacy and impact of a tool on a continual basis.

Metrics and efficacy

Measuring the tool’s efficacy serves dual purposes: it helps ensure that the tool is operating ethically and responsibly using quantifiable metrics and it also allows you to track the business outcomes your newsroom hoped to achieve from adopting this tool. This is in contrast to benchmarks set in Step 4, which establish a baseline on how your newsroom is performing on this objective.

To get started on measuring the efficacy of the tool, you should consider two pieces of information that you should have been previously collecting (see Steps 4 and 5):

1. The baseline benchmarks of how your newsroom previously performed prior to using the tool
2. The metrics and data points that the tool provider has built in or tracks as part of the tool

Once considered, decide what additional metrics you’d like to collect to provide insight into the tool’s performance and how it’s impacting your overall newsroom goals. These metrics will flow from the initial outcome you’ve identified for the use of this AI tool in Step 1. This section focuses on new metrics that can be collected due to the use of a particular AI tool which provides new opportunities for oversight and measurement, in contrast to benchmarks which help you measure the baseline of where your newsroom has started and how that compares to the improvements made as you use a new AI tool.

Start by asking what the best way to measure the outcome is and what evidence there may be in your newsroom (both qualitative and quantitative) to prove this. Below are some of the areas of measurement used by other newsrooms:

• Accuracy measures how well the AI tool is able to perform its intended function without errors. For example, an AI tool designed to write articles using data inputs should be able to do so without misinforming the audience or adding inaccurate context to the written article.
• Precision measures how precise the AI tool is in its results. For example, how often does it correctly identify a piece of information as fake? The rate of both false positives and negatives should be considered when assessing precision.
• Recall measures the portion of relevant results that the AI tool is able to return. For example, if you’re querying an AI tool for a particular piece of information out of a large data set, is it able to accurately recall all the relevant information related to the query in the database?
• Efficiency measures the time savings from using an AI tool.
• User satisfaction measures how satisfied the users of the AI tool are with its performance. This can be collected directly from journalists who are using the tool in their day-to-day work, perhaps through qualitative surveys.
• Cost-effectiveness measures how cost-effective the AI tool is compared to other solutions. Here, cost encompasses both the direct cost of the tool and the indirect cost of training staff and integrating the tool into existing workflows.
• Audience engagement measures how well a tool directed at the audience is increasing engagement with the news. This can include online comments, emails, and newsletter signups.
• Conversion measures how the AI tool is increasing subscriptions and overall revenue for the newsroom.

In addition to the above categories, there might be cascading effects worth tracking to get a better idea of a tool’s impact. For example, if you are using a translation tool in order to reach a wider audience, the tool itself might not track your new readership as a result of using the tool. However, your newsroom likely internally tracks the readership and their broad geographical location. Comparing the baseline readership metrics prior to translating the content and after might give you an idea of the additional audience engagement provided by the tool. Going a step further might include adding a survey for the translated article asking for feedback on the accuracy of the translation and how it can be improved. That way, you’re getting a deeper understanding not just of the impact of the AI tool on your audience, but also its accuracy in delivering the correct interpretation and news to your audience.

Lastly, we’d recommend that you limit the number of vanity metrics used to measure efficacy. Vanity metrics are metrics that are easily obtainable, easily understandable, and can seem to showcase success over time — such as likes and shares on a social media post. However, they do not provide an accurate portrayal of the return on investment from using a particular tool, as they only reflect surface-level audience engagement. Metrics such as open rates for newsletters or traffic on a website look great on paper, but alone they might not provide a complete picture of how an AI tool is actually supporting a newsroom or meeting the expectations set out in Step 1.

9

Understand the lifecycle of an AI tool

AI tools regularly incorporate new data inputs, can be customized to the needs of each newsroom, and develop over time. This means that their outputs and behavior can also change. As your newsroom continues to use a tool, it’s important to audit the inputs, data collection, and output of the tool to ensure that it continues to yield the intended results. Mitigating risk early on helps support the continued use of a tool and can help ensure that negative or unwanted outputs don’t appear later on. However, that doesn’t take away the need to continually monitor the tool, any updates that have been issued by the developer, and any changes to the ways in which the data is being collected or used.

This is where the role of the person designated with oversight in Step 8 becomes particularly crucial. They will be responsible for monitoring not just the outputs of the tool, but also any changes or updates issued by the provider. This includes developer update emails that can unfortunately be quite lengthy and include a lot of fine print. Buried within them might be important information about how the collection or use of data has changed, therefore implicating your newsroom.

In addition, continual maintenance of the tool — keeping it up-to-date and ensuring its outcomes continue to match its intended purpose — avoids future “technical debt.” Technical debt occurs when teams delay updating features, fail to maintain functionality, or settle for suboptimal performance over time. These outcomes can be the result of a “if it’s not broken, don’t fix it” mentality.

Questions you might want to ask in your newsroom at this stage include:

• What does continual maintenance look like?
• Who will be responsible for future and continued communication with the tool provider?
• How can you ensure that the AI tool continues to operate as intended? At what frequency will you conduct audits of performance?
  • If the tool has a regular subscription payment, tying the audit to the renewal of the subscription can build in a natural trigger for regular audits and ensure regular oversight before renewing the subscription.
• How will you be accounting for unintended consequences outlined in Step 7?
• Are there any other processes that need to be implemented to account for negative impacts of the tool or mitigate their impact?

10

Determine when you should retire an AI tool

Part of stewarding AI tools in your newsroom is knowing when they are no longer serving their original purpose or supporting your newsroom as defined in earlier steps.

The questions about the tool, its use, metrics, governance, and ethical guidelines in previous sections can be called upon perpetually when reviewing the tool. If the intended goals are no longer being met, or new ethical concerns arise, then there should be an established off-ramp process for retiring a tool from your newsroom.

It might not even be the functionality of the tool that has changed, but rather the intended use by its provider that has shifted or, perhaps, data being collected and reused has changed. In addition, the priorities or needs of your newsroom may have changed and therefore the tool is no longer serving those needs.

Some questions you might want to ask in your assessment:

• Has the primary use of this tool changed by the developer? Is that use in line with journalistic ethics?
• Is the governance and oversight of the AI tool by our newsroom sufficient to mitigate its potential adverse impacts? If not, are we able to adjust our oversight to meet these challenges?
• Does the cost-benefit analysis of using this tool still make sense for our newsroom? Are there other considerations we should now take into account?

All of these elements should be a cause for pause in newsrooms deciding whether to continue using a tool internally. Making the proactive decision to stop using a tool and being transparent with the audience about that decision allows your newsroom to better maintain its credibility with stakeholders, including the audience and staff. It also allows your newsroom to be intentional in its decisions about how it uses AI tools and mitigates associated risks, and to discontinue their use when they no longer serve their purpose. Following these steps with intention can help newsrooms navigate the implementation of AI, responsibly.