CALIFORNIA (Kashmir English): OpenAI has raised new concerns about the trustworthiness of AI chatbots, warning that AI hallucinations coherent but factually incorrect responses — remain a significant and unresolved challenge despite rapid advances in artificial intelligence technology.
AI hallucination refers to when a chatbot generates text that sounds confident and accurate but is factually wrong a growing concern as tools like ChatGPT are increasingly used in education, healthcare, legal research, and journalism.
According to a recently released OpenAI research paper, large language models (LLMs) such as GPT-5 and ChatGPT continue to produce plausible but false information due to fundamental shortcomings in the pretraining process and current model testing strategies.
The Problem With Current AI Evaluation Methods
Scientists argue that existing accuracy-based AI evaluation systems are fundamentally flawed they inadvertently train models to “guess” an answer rather than acknowledge uncertainty. In other words, current benchmarks reward sounding correct over being correct.
What Researchers Propose
To address the AI hallucination problem, researchers suggest a shift in how AI models are tested and rewarded:
- Penalize overconfident errors models should be discouraged from asserting false claims with high confidence
- Reward calibrated uncertainty models that say “I don’t know” when appropriate should score higher
- New evaluation frameworks that measure honesty alongside accuracy
Large language models such as GPT-5 and ChatGPT create plausible but false information because of shortcomings in the pretraining process and testing strategies, according to a recently released research paper.
Examples where a chatbot was consistently providing false responses about a researcher’s scholarly work and personal information are given in the paper.
Scientists maintain that existing accuracy-based evaluation systems prompt models to “guess” instead of acknowledging uncertainty.
They suggest new testing approaches that penalize errors of confidence but reward good uncertainty, so that AI outputs will be more reliable.
