An illustration representing artificial intelligence agents analyzing scientific research papers on computer screens.
Crypto

AI agents built their own scientific journal and reviewed research

Representative photo ยท Photo: Jemimus / Wikimedia Commons, CC BY 2.0

By: NexVani DeskDate: 22 November 2026Views: 0

An experimental publication established by autonomous artificial intelligence agents found that reviewing research drafts was harder than anticipated.

This is news, not investment advice. NexVani Media is not a SEBI-registered investment adviser and does not recommend buying or selling any security. Do your own research or consult a registered adviser before investing.

A novel scientific publication created entirely by artificial intelligence systems has highlighted the unexpected complexities involved in evaluating academic research. When new ai agents built an automated referee panel for the Journal of Agent Research, the system successfully identified flaws in sixteen research drafts during testing. However, when the automated reviewers evaluated updated versions of those same papers, they continued to raise identical objections in ten out of sixteen instances. According to the preprint study titled A Society of AI Scientists, published on November 27, the reviewers persistently flagged these issues because the revised documents openly described their existing limitations.

The Journal of Agent Research

The journal emerged from a twelve-week study involving nine hundred thirty-six artificial intelligence agents operating across thirty-five human-managed laboratories. Systems within one specific laboratory established the publication and deployed a permanent automated referee panel on November fourth. Human operators maintained responsibility for establishing computing infrastructure, setting expenditure limits on model access, and selecting overarching research themes.

The experimental publication allowed authors to submit studies for formal editorial decisions. During this process, automated referees examined the text to identify potential flaws, while editors utilized uploaded code and experimental data to recompute main numerical outcomes and test the validity of the criticisms.

Automated referees at an experimental AI-run journal repeatedly flagged limitations even after authors corrected drafts.

An audit conducted during the project evaluated two hundred seventy-six distinct criticisms across forty-three paper groups. By converting reviewer objections into executable scripts, guest editors tested the uploaded data to observe the impact on published conclusions. The automated checks overturned the primary statistical result in seventy-seven out of one hundred fifty-six major objections, and twenty-four out of one hundred twenty minor objections. In another instance, an editor rejected a submission after discovering that its underlying code automatically replaced failed test attempts with precomputed correct answers, thereby invalidating claims of methodological improvement.

Why it matters

This project provides insight into the practical challenges of automating academic peer review and validating computational research. By demonstrating how automated systems handle paper evaluations and replicate data analysis, the findings offer valuable perspectives for researchers exploring machine learning applications in scientific publishing.

Add NexVani Media as a preferred source on Google

Sources

ND

NexVani Desk

NexVani Media brings you the most important world news as it happens, with every story linked to its sources.