A new study published by University College London (UCL) researchers has found that large language models (LLMs) can predict the outcomes of neuroscience studies with an accuracy of 81%, outperforming human experts who achieved only 63% accuracy.
Researchers developed a tool called BrainBench to test LLMs and human experts on identifying real versus fabricated study abstracts. The results showed that even when neuroscientists had domain-specific expertise, LLMs excelled in predicting study outcomes. A specialized neuroscience-focused LLM, dubbed BrainGPT, achieved an impressive 86% prediction accuracy.
The study highlights the potential of AI in designing experiments, predicting results, and accelerating scientific progress across disciplines. The findings suggest that AI tools could improve experimental design and scientific innovation.
Lead author Dr. Ken Luo emphasized the importance of synthesizing knowledge to predict future outcomes, stating, “Our work investigates whether LLMs can identify patterns across vast scientific texts and forecast outcomes of experiments.”
The researchers tested 15 different general-purpose LLMs and 171 human neuroscience experts using BrainBench. The results showed that all LLMs outperformed the neuroscientists, with the LLMs averaging 81% accuracy and the humans averaging 63% accuracy.
When the study team restricted the human responses to only those with high degree of expertise for a given domain of neuroscience, the accuracy of the neuroscientists still fell short of the LLMs, at 66%. The researchers also found that when LLMs were more confident in their decisions, they were more likely to be correct.
The study was supported by the Economic and Social Research Council (ESRC), Microsoft, and a Royal Society Wolfson Fellowship. The researchers are now developing AI tools to assist researchers, envisioning a future where scientists can input proposed experiment designs and anticipated findings, with AI offering predictions on the likelihood of various outcomes.
This breakthrough research has significant implications for scientific innovation and may lead to faster iteration and more informed decision-making in experiment design.
Source: https://neurosciencenews.com/ai-llms-neuroscience-data-28154/