Google Pulls Some AI Health Summaries After Accuracy Concerns

Google has removed several AI-generated health summaries from its search results after an investigation found that some of the information could mislead users and potentially put them at risk of harm.

The company’s AI Overviews use generative AI to provide brief summaries at the top of search results. Google has said the feature is designed to deliver helpful and reliable snapshots of information. However, the Guardian reports that in certain health-related searches, the summaries presented inaccurate or incomplete data without sufficient medical context.

The investigation found that Google’s AI summaries on liver blood tests lacked context and could mislead patients, prompting the company to remove those results and review similar cases. Health organizations welcomed the removals but said broader concerns remain. Vanessa Hebditch, Director of Communications and Policy, British Liver Trust, says the decision addressed a specific issue but did not resolve wider risks associated with AI-generated health information. 

AI Overviews continue to appear for other health-related searches. Google says those summaries link to reputable sources and include prompts advising users to seek professional advice. The company adds that internal clinical teams review shared examples and, in many cases, have found the information to be supported by high-quality websites.

Google, which holds an estimated 91% share of the global search engine market, says AI Overviews are shown only when the company has high confidence in response quality and that it continuously reviews performance across categories. Still, technology analysts said the episode highlights the higher stakes of errors when AI-generated summaries appear prominently in health-related searches.

Addressing this issue has become increasingly important as more patients turn to AI to understand symptoms and decide how to act on health concerns. For example, health-related questions are already among the most common uses of ChatGPT, with more than 230 million people worldwide asking health and wellness questions each week, according to OpenAI’s de-identified data, reports MBN. 

Garth Graham, Director and Global Head of Healthcare and Public Health, YouTube, underscores the need for accessible and reliable health information, noting that a health journey often begins when people experience a concern, such as pain, and seek guidance online to inform their next steps. Surveys in the United States show that despite limited trust, many people rely on AI out of necessity. One in six adults uses chatbots for health information each month, while in Canada 37% turn to online sources due to limited access to doctors, reports the Canadian Breast Cancer Network. Meanwhile, in Mexico, AI has become the third most consulted source of health information in 2025, up from eighth place in 2024, according to the study The Mexican Digital Patient, presented by FUNSALUD. 

“Mexico represents a significant market opportunity to support the modernization of health care delivery through secure augmented intelligence,” says Tyler Luck, Interim CEO, BEN.

Mental health is also in the spotlight. AI-powered platforms are increasingly used as entry points for individuals who face stigma or have limited access to traditional care. Emotional chatbots, adaptive therapy tools and cognitive support applications are being adopted in Brazil and Mexico. However, specialists warn that engagement tools cannot substitute clinical evaluation. Mental health remains one of the least-funded and least-regulated areas in Latin America, despite growing demand. Experts argue that AI must be deployed with mechanisms that prioritize safety and accountability.

Concerns about accuracy, bias, and misinformation form a significant counterweight to the expansion narrative. Erick Ponce Flores, President, International Cannabis and Cannabinoids Institute (ICAN), warns that some large language models (LLMs) produce oversimplified or agreeable responses rather than evidence-based guidance. He describes the trend as “AI pandering,” emphasizing that systems optimized for user engagement risk reinforcing misconceptions rather than clarifying them. In the context of medical cannabis, he said issues are already visible. Patients often receive AI-generated recommendations for products that are not appropriate for their conditions. In some cases, chatbots have directed users to illegal or black-market options.

Many nations lack a mature regulatory framework to ensure the accuracy and legality of the information provided to the public. For example, while many countries in the European Union acknowledge AI's potential to revolutionize healthcare, only four have a dedicated national AI strategy for health, and seven more are developing one, according to the World Health Organization (WHO).

“AI relies on data to learn and make decisions. If that data is biased or incomplete, the AI’s decisions will be too, which could lead to missed diagnoses, incorrect treatments, or unequal care,” says WHO.

Some companies are already developing AI solutions designed exclusively for health-related needs. For example, OpenAI has introduced ChatGPT Health, a dedicated experience intended to bring personal health information and AI together within a secure environment, as demand for digital health guidance continues to grow. ChatGPT Health builds on existing usage by enabling users to ground conversations in their own medical and wellness information, helping them better understand their health and prepare for interactions with the healthcare system.

ChatGPT Health, says OpenAI, is positioned as a support tool rather than a substitute for medical care. It is not intended to diagnose or treat conditions, but to help users understand test results, organize information ahead of appointments, interpret data from wearables, and identify trends related to diet, exercise, or coverage options. OpenAI says the goal is to support informed conversations with clinicians, not replace them.

“As we think about how this next wave of AI will change health, it is critical to keep these three approaches in mind: develop AI responsibly, collaborate with others and remember that AI is just one of many tools to improve health,” says Karen DeSalvo, Chief Health Officer, Google.

According to DeSalvo, the long-term impact of emerging AI technologies is still unfolding. Without responsible development and deployment, these tools risk deepening existing inequalities.