
AI models agreed with each other more than with reality in a new test
Representative photo ยท Photo: Jemimus / Wikimedia Commons, CC BY 2.0
A study of AI predictions found that models often align with each other rather than scientific reality, though consensus can improve overall accuracy.
This is news, not investment advice. NexVani Media is not a SEBI-registered investment adviser and does not recommend buying or selling any security. Do your own research or consult a registered adviser before investing.
A recent study examining how various AI models interpret scientific experiments has revealed that these systems often exhibit higher levels of agreement with one another than with actual laboratory measurements. In the research, titled Science Prediction Universe and submitted to arXiv on Nov. 11, investigators evaluated 13 agent configurations across 12 different models. These configurations, which combined specific models with reasoning settings and software, were tasked with ranking experimental outcomes in fields including chemistry, biology, medicine, and materials science. When users search for information on these complex AI models, they often look for how these systems process data compared to human logic.
Researchers tested the agents on 100 public tasks and 16 private tasks. On the public tasks, the average agreement between different configurations reached a Spearman rank correlation of 0.52. In contrast, the correlation between these AI predictions and the actual measured outcomes was only 0.33. Notably, the agents demonstrated higher agreement with each other than with empirical reality in 81 of the 100 tasks evaluated. The study focused on the ability of these systems to rank experimental results, such as reaction yields or material conductivity, rather than requiring them to predict specific numerical values.
Pooling AI predictions improves average results but can also perpetuate shared scientific errors.
The impact of consensus
The study identified a clear benefit to aggregating predictions. By averaging the rankings from different configurations, researchers created a consensus that scored 0.08 points higher than the average individual submission, yielding improved results on 73 of the 100 public tasks. However, this consensus approach is not infallible. The authors noted that while averaging can reduce errors unique to specific systems, it often preserves shared mistakes. These common errors frequently stem from simple, uniform assumptions about how experimental variables should behave. In one instance involving an oncology task, eleven configurations showed high internal agreement of 0.9, yet their consensus averaged only 0.27 when compared against measured laboratory data.
The research also highlighted variability in performance. Some models, such as Claude Opus 5.5 and Claude Fable 5.1, were limited in their task coverage due to safety refusals. Furthermore, the findings are based on specific task sets, and the study did not utilize a human prediction baseline, meaning laboratory measurements remain necessary to verify the accuracy of AI-generated rankings.
Why it matters
This study highlights that while AI consensus can be a useful signal for scientific prediction, it remains a fallible metric that can mask underlying systematic biases. For researchers and developers, this means that increasing the number of AI opinions does not automatically guarantee accuracy if the models share the same fundamental flaws.
Understanding these limitations is crucial as AI becomes more integrated into scientific workflows. The findings suggest that relying solely on model agreement could lead to misleading conclusions, emphasizing the continued necessity of empirical laboratory testing to validate AI-driven scientific insights.
Sources
More like this

Aster CEO Leonard Leung Suspects Foul Play After Token Plunges

Asian Chip Stocks Tumble Ahead of Nvidia Q3 Earnings

Luxor CEO Warns of AI Cyberthreats After Mining Platform Compromise

