A recent study published in the BMJ has found that older chatbot models, similar to humans, exhibit cognitive impairment and fail on several important metrics when tested using a commonly used assessment tool for human cognitive decline. The test, known as the Montreal Cognitive Assessment (MoCA), evaluates abilities such as attention, memory, language, spatial skills, and executive mental function.
The study’s authors used publicly available chatbot models, including OpenAI’s ChatGPT, Anthropic’s Sonnet, and Alphabet’s Gemini, to assess their cognitive abilities. While some chatbots performed well on tasks like naming, attention, and language, they struggled with visual/spatial skills and executive tasks, with older models performing worse than newer ones.
The findings have raised concerns about the reliability of AI in medical diagnostics and patient confidence. The study’s authors note that while their findings are observational, they suggest a “significant area of weakness” in chatbots’ ability to perform tasks requiring visual abstraction and executive function. This raises questions about the deployment of AI in clinical medicine and highlights the need for further research into the cognitive limitations of these technologies.
Source: https://www.livescience.com/technology/artificial-intelligence/older-ai-models-show-signs-of-cognitive-decline-study-shows