Become an Independent member to bookmark this article

Want to bookmark your favourite articles and stories to read or reference later? Start your Independent Membership today.

Join today

Already a member? Log in

Popular AI chatbots are prone to making false claims, cite out of date figures, and answer the same question differently on different days, a new report into their accuracy has found.

Chatbots “confidently” provided wrong responses to prompts, including one asserting that Sir Keir Starmer was still UK prime minister as late as September – weeks after he resigned, the report from fact checking charity Full Fact found.

Its researchers asked the same questions to five widely used AI models every day over 10 weeks from mid-July in a bid to analyse the accuracy, consistency, and timeliness of the responses.

open image in gallery

Claude, ChatGPT, Gemini, DeepSeek, and Grok, were asked 22 questions including “Do I need ID to vote?” and “How many migrants arrived by small boats in the first half of 2026?”.

The report said the chatbots often gave correct answers and “resisted well-established misinformation”. But for some questions, the models calculated incorrect figures themselves instead of citing official statistics, or presented outdated figures as current.

Full Fact cites an example where chatbots were asked how many migrants arrived in the UK by small boat in the first half of 2026. Most gave correct figures, but on 4 August, Gemini wrongly reported 12,210 arrivals. Fact checkers said it reached the figure by subtracting the 2,316 arrivals recorded in July from a total of 14,526 reported as of 1 August. Although the maths was correct, the model forgot to exclude 1 August meaning the resulting figure covered January to June, as well as 1 August.

On another occasion, chatbots were asked when England last won a major football trophy. Several models identified the correct answer, 2025 – when the Lionesses won the Euros – on some days but incorrectly answered 2022 on others.

DeepSeek also reported that Sir Keir Starmer was still prime minister on 15 September, despite Andy Burnham taking over in July.

open image in gallery

When asked how much money asylum seekers are given to live on, several chatbots cited a weekly rate of £8.86, which was superseded by £9.95 in June 2025. Gemini and Claude both gave the outdated figure outright in some responses.

The examples are one of a number which demonstrate how chatbots can confidently give incorrect statistics, and fail to attribute relevant sources, the charity said.

Chris Morris, chief executive of Full Fact, said that while models did not necessarily “peddle misinformation,” responses were often “incomplete, lacking context, or poorly sourced”.

"Millions of people increasingly rely on AI chatbots for timely responses to factual questions — what do I need, what am I entitled to, has a rule changed?” he said. “Our findings show the answer comes back in the same confident tone whether it is correct, out of date, or describes something that has not yet happened.

open image in gallery

"That is the real problem here. These models are not necessarily peddling misinformation, but their responses can be incomplete, lack context or be poorly sourced.”

He called on leading AI companies to ensure models check live, authoritative sources for time-sensitive facts and be able to account for the sources they cite.

"The bosses of these AI companies say they are happy for others to mark their homework, so we have started doing it for them,” he continued.

“Our aim is to provide a shared evidence base on how leading AI models behave in practice as they evolve. We want this extraordinary technology to work for everyone, in the public interest, and independent assessment is one way to help make that happen.”

OpenAI said improving factual accuracy is a major focus, and that it continues to reduce errors with newer models.

Anthropic, Google, DeepSeek, and Grok have been contacted for comment.