AI and News: Can We Trust Chatbots?

A recent study published by the Reuters Institute reveals significant limitations in the ability of generative AI chatbots, specifically ChatGPT and Bard (recently renamed Gemini), to provide accurate current news headlines.

I was particularly interested in the work because it addresses a recurring question that arises every time I give training in a media environment: Is it possible to use these platforms for news coverage in newsrooms?. For this, the authors analyzed 4,500 headline requests from ten countries and found that these AI systems frequently fail to provide updated news from the specified media.

Among the most important findings are:

• General Performance: When asked to retrieve the latest news headlines from specific media outlets, ChatGPT did not provide results similar to news in 52–54% of the cases, typically responding with a message indicating inability to access the requested content. Bard showed an even higher failure rate, with 95% of the requests returning similar non-news responses.
• Accuracy in Delivering News: Only between 8% and 10% of ChatGPT’s responses contained headlines that actually matched the main news listed on the media outlet’s homepage at the time of the request. Most of the responses referred to older news or were not major news at all.
• Incorrect Attribution and Hallucinations: Approximately 3% of ChatGPT’s responses included headlines that were attributed to incorrect sources or were too vague to be verified against real stories, raising concerns about the potential for spreading misinformation.

The study was conducted using the web interfaces of ChatGPT Enterprise and Google Bard at the 15 most used online media outlets, according to the 2023 Digital News Report from the Reuters Institute. Researchers used a direct and specific prompt to assess the AI’s ability to retrieve and present news headlines accurately.

The findings underscore the challenges and potential risks of relying on AI for news retrieval, especially regarding the accuracy and reliability of the information provided. Despite the integration of AI tools in news consumption, the study highlights a significant gap in their ability to function effectively as a news source.

Additionally, the authors noted considerable variability in the performance of these AI tools across different countries and media outlets, which may be partially related to how these sites are set up to block AI crawlers. This is an aspect to consider since the future of AI in news dissemination depends on improvements in the capabilities of these types of platforms and perhaps more structured collaborations between technology providers and news publishers.

Although AI chatbots like ChatGPT and Bard have the potential to transform how we access news, this type of work allows us to observe certain trends that serve as a warning light and a reminder of the need for strict controls over the accuracy and origin of AI-generated content in journalism.

Ultimately, the study acts as a snapshot of the reach and limitations that these systems have today, which will surely change soon with partnerships that, for example, OpenAI is closing with media groups in Europe and the U.S., and the announcement of a new AI-powered search engine.

More info:

::: ‘I’m unable to’: How generative AI chatbots respond when asked for the latest news