Large language models like ChatGPT are prone to making things up due to their design and training data. Researchers at OpenAI have found that even with perfect training data, the problem of hallucinations persists because the way these models learn is mathematically inevitable.
The issue lies in how these models generate sentences by predicting one word at a time based on probabilities. This naturally produces errors, which can accumulate over multiple predictions, leading to higher error rates for generating sentences compared to simple yes/no questions. The researchers also found that hallucinations are more likely when models see fewer facts during training.
The proposed fix involves having the AI consider its own confidence in an answer before putting it out there and scoring it based on that basis. However, this approach would require significant changes to user experience, as users expect confident answers to virtually any question. The computational economics problem also poses a challenge, as uncertainty-aware language models require more computation than current approaches.
The business incentives driving consumer AI development remain misaligned with reducing hallucinations. Until these incentives change, hallucinations will persist, affecting the reliability and trustworthiness of large language models like ChatGPT.
Source: https://theconversation.com/why-openais-solution-to-ai-hallucinations-would-kill-chatgpt-tomorrow-265107