A team from Anthropic has used brain-scanning techniques to study large language models (LLMs), shedding light on why chatbots struggle with simple math and hallucinate. The technique, called circuit tracing, allows researchers to track the decision-making processes inside a model step-by-step. By applying it to their Claude 3.5 Haiku LLM, Anthropic discovered that LLMs use an odd method for basic math, which can be unreliable. Additionally, they found that these models do not solely predict the next word but rather plan ahead in complex ways. These breakthroughs have significant implications for understanding and designing AI systems.
Source: https://www.pcgamer.com/software/ai/anthropic-has-developed-an-ai-brain-scanner-to-understand-how-llms-work-and-it-turns-out-the-reason-why-chatbots-are-terrible-at-simple-math-and-hallucinate-is-weirder-than-you-thought