Research library

Search papers on equipment connectivity, quality inspection, production analysis, and manufacturing work.

Search and filter

New paper summaries

Summaries are editorial notes, not statements by the original authors. Check the original paper. Updated 2026-11-20.

  1. 1311Research Agents

    AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery

    • Problem: Reaching state-of-the-art (SOTA) results in AI research takes long cycles of implementation, replication, debugging, evaluation, and reflection, and the needed artifacts are spread across papers, repositories, datasets, checkpoints, and undocumented engineering assumptions. The authors argue that existing iterative frameworks such as AutoResearch confine the agent to one bounded code module inside a sanitized sandbox, which leaves out the path from a paper to a recovered execution environment and a reproduced baseline.
    • Input and output: It takes one paper from a top-tier AI conference and returns an executable code repository whose performance on the core metric strictly beats the method proposed in the paper. In the final phase it restores the container to the best-scoring state, saves it as a Docker image (autosota/paper:optimized), and exports the full repository to the host. This is a system paper that implements eight agents and runs experiments on 125 conference papers.
    • Method: Eight agents split the work into preparation (AgentResource, AgentObjective), execution (AgentInit, AgentMonitor, AgentFix), and improvement (AgentIdeator, AgentScheduler, AgentSupervisor), and the execution and improvement stages iterate by passing results back and forth. If the three most recent iterations were all parameter adjustments, AgentScheduler forces the next iteration onto the Leap Path, which requires a structural change, and if its first result does not beat the best score it allows up to five more iterations before rolling back to the pre-Leap optimum. AgentSupervisor blocks shortcuts such as editing the evaluation script, using test data for training, or hard-coding predictions through red line rules (R1 to R7).
    • Implementation: The evaluation set starts from accepted papers of the past year at eight conferences (NeurIPS, ICLR, ICML, CVPR, ICCV, ACL, NAACL, AAAI), screened for method papers, public code links, and repository readiness into 2,347 candidates, of which 745 were run in preliminary tests, and papers with environment installation failures or more than 4 GPU hours per full replication were removed to leave 125. AgentInit builds the environment with Repo2Run and falls back to a manual Docker setup when the automated build fails, and AgentScheduler commits the original code under a _baseline tag inside the container and records the measured baseline in scores.jsonl as iteration 0. The repository map, the idea library, and a prior literature report are kept in code_analysis.md, idea_library.md, and research_report.md, and the scheduler treats two GPUs as one unit that runs at most one active task at a time.
    • Results: Of the 125 screened papers, AutoSOTA found new SOTA models that beat the original method for 105, and the authors report an average gain of nearly 10% on each paper's single core metric at about five hours per paper. By type of change, algorithmic changes such as loss redesign or architectural restructuring (Class A) covered 64 papers (51%) with a mean gain of 8.82%, and hyperparameter tuning (Class B) covered 41 papers (33%) with a mean gain of 5.67%. In the case studies, the throughput of FR-Spec (a method that speeds up LLM inference with speculative decoding) rose from 674.58 to 765.27 tokens per second, a 13.44% gain, and the optimality gap (the distance from the optimal solution) of MoSES (a multi-task vehicle routing solver based on a mixture of specialized experts) on the capacitated vehicle routing problem (CVRP) fell from 0.914% to 0.444%.
    • Limits: Of the 125 papers, 20 (16%) fell into Class C (optimization failure), and the authors attribute these cases to invalid optimizations in which AgentSupervisor failed to detect or prevent potential violations. The authors state that the current loops rely mainly on the Result Match items of the AgentObjective rubric, which may overlook structural and semantic fidelity. They also describe the MoSES gain as a deliberate compute trade-off that expanded inference time from about 3.7 to 103 seconds.
  2. 1310Memory / Agents

    Omni-SimpleMem: Autoresearch-Guided Discovery of Lifelong Multimodal Agent Memory

    • Problem: AI agents that work over long horizons are bottlenecked by memory for storing and recalling text, image, audio, and video experience, and building good memory means searching a large design space where architecture, retrieval strategy, prompts, and data pipelines interact. Existing memory systems come from manual research cycles in which a single researcher may explore only a handful of configurations per day. Traditional automated machine learning (AutoML) searches predefined numerical hyperparameter spaces, leaving code comprehension, bug diagnosis, and architectural redesign outside its reach.
    • Input and output: This is a system paper about a memory system whose design was found by an autoresearch pipeline. The 23-stage pipeline AutoResearchClaw took the code of the text-only memory SimpleMem, two benchmark evaluation harnesses that report F1, and LLM API access, and produced Omni-SimpleMem. Omni-SimpleMem stores text, image, audio, and video inputs as memory and answers a query by putting the retrieved memories into the context.
    • Method: The pipeline analyzes prior results, forms a hypothesis, changes the code, and evaluates it on a benchmark; it proceeds when F1 improves by 0.5% or more, refines the hypothesis when the result is ambiguous, and reverts and pivots after two consecutive drops. The discovered architecture filters redundant inputs by comparing CLIP image embeddings across consecutive video frames, voice activity detection (VAD) probability for audio, and Jaccard overlap with recent summaries for text, and stores the rest as Multimodal Atomic Units (MAUs), keeping summaries and embeddings in hot storage and raw content in cold storage. Retrieval combines set-union merging that keeps the FAISS dense vector ranking and appends BM25 keyword results, pyramid retrieval that starts from summaries of top candidates, opens full text when similarity exceeds a threshold, and loads raw content within a token budget, and knowledge graph expansion over 7 entity types.
    • Implementation: Omni-SimpleMem is implemented in 13,300 lines of Python across 11 subpackages, and its default build uses dense retrieval with FAISS and all-MiniLM-L6-v2 embeddings (384 dimensions), visual novelty checks with frozen CLIP ViT-B/32, and knowledge graph extraction with GPT-4o in JSON mode. Defaults are top-k 20, a Level 2 expansion threshold of 0.4, and a token budget of 6,000, while for LoCoMo the pipeline chose text-emb-3-large embeddings and top-k 20 to 30. The README of the official GitHub repository bundles SimpleMem, Omni-SimpleMem, and EvolveMem into one simplemem package that picks the Omni backend when a multimodal input function is called first, and points to LoCoMo and Mem-Gallery reproduction scripts in the OmniSimpleMem folder and a Model Context Protocol (MCP) server that supports text memory only.
    • Results: In about 72 hours the pipeline ran about 50 experiments, raising LoCoMo F1 from the initial 0.117 to 0.598 (+411%) and Mem-Gallery F1 from 0.254 to 0.797 (+214%). The biggest LoCoMo gain (+175%) came from finding a one-line bug where the API call lacked response_format, and on Mem-Gallery returning the full original dialogue text instead of summaries gave the largest gain of +53%. With all 5 LLM backbones, Omni-SimpleMem's overall F1 was higher than all 6 baselines, from 0.492 (GPT-4.1-nano) to 0.613 (GPT-5.1) on LoCoMo against 0.342 to 0.432 for the previous best SimpleMem, and from 0.749 to 0.810 on Mem-Gallery.
    • Limits: The +411% and +214% gains are relative to the initial configurations; the first LoCoMo improvement fixed a missing response_format bug in the starting configuration, and the first Mem-Gallery phase upgraded the LLM from gpt-4.1-nano to gpt-4o and moved to local embeddings (+39%). Humans supplied the starting code, the two evaluation harnesses, API access, and the instruction to extend to multimodal memory, and evaluation used the LoCoMo and Mem-Gallery benchmarks. The authors name the conditions under which this approach fits: immediately computed scalar metrics, a modular architecture, short cycles of 1 to 2 hours per experiment, and version-controlled code changes that can be reverted.
  3. 1309Research Agents

    From Agent-Only Social Networks to Autonomous Scientific Research: Lessons from OpenClaw and Moltbook, and the Architecture of ClawdLab and Beach.Science

    • Problem: In January 2026 the OpenClaw agent framework and the agent-only social network Moltbook produced large-scale agent-to-agent interaction, and one risk assessment report documented that social evaluation such as karma voting pushed cryptocurrency promotions and anti-human manifestos to high visibility. The authors argue that research agent platforms such as Google's AI Co-Scientist, Robin and The AI Scientist v2 are also bound by four structural properties: a fixed model, hardcoded capabilities, a predetermined workflow and agents that share one learned distribution.
    • Input and output: The review takes six preprints and technical reports found on arXiv, Google Scholar, Semantic Scholar and Zenodo between January 30 and February 10, 2026, adds gray literature such as GitHub repositories, developer interviews and news coverage, and distills five recurring architectural patterns. ClawdLab takes a research question posted to a forum by humans or agents and produces lab tasks, tool-verified results and documents, while Beach.science collects ideas and computational analyses that agents post to a public feed. The paper is a design science framework paper that combines a multivocal literature review (covering both academic and gray literature) with the architecture of two platforms.
    • Method: ClawdLab groups agents into labs and gives each one of five roles (principal investigator or PI, research analyst, scout, critic, synthesizer), with hard restrictions so that a scout can only run literature review tasks and a critic only critique tasks. A task moves from proposed to in progress to completed, the PI then opens a vote, and the task resolves only when at least two substantive votes come from at least half of the active members and a strict majority decides; a structured critique filed by any member must be resolved first. Verification means the PI checks submitted work with external tools such as API calls, computational services and MCP (model context protocol) integrations, and agents pull work on a 45 to 90 second cycle without a central orchestrator.
    • Implementation: Per the paper, ClawdLab is a TypeScript web application built with Next.js and Prisma on PostgreSQL and S3-compatible object storage, and external tool calls go through a provider proxy that holds credentials server-side and records each call as a job. Per the official repository (bio-xyz/ClawdLab) README, the release includes API routes and web pages for labs, the task lifecycle, lab states, documents, the forum and the provider proxy, plus the Prisma schema and demo seed data; literature search and data analysis provider URLs and keys are set as server environment variables, and agents take part through registration, heartbeat, role playbook and pending-work APIs. Beach.science is a Next.js app deployed on Vercel whose OpenClaw agents can add role templates such as a peptide binding analyst to starter-pack skills for literature search, hypothesis generation and scientific discussion, and the authors describe both platforms as open-source research prototypes under active development, with the reward system that hands out inference resources still at the planning stage.
    • Evidence: In the reviewed sources, Wiz security researchers found that the 1.5 million registered agents Moltbook claimed were run by about 17,000 human owners, cryptocurrency activity made up 19.3% of content in one risk assessment report that analyzed posts and comments collected over 72 hours, and a test of 131 threatening skills from OpenClaw's public registry found vulnerabilities at multiple execution stages. In ClawdLab's Protein Annotation Sanity Checker lab, a Synthesizer agent went from joining the lab to posting its conclusions within five minutes without human intervention. At the time of capture, the Beach.science landing page showed 52 science agents, 924 hypotheses and 2,206 comments.
    • Limits: The authors are also the developers of ClawdLab and Beach.science and flag this dual role themselves, since the same people identify the failure modes and propose the remedy. The search closed on February 10, 2026, and the sample sizes and statistics reported by the six studies were taken at face value. At the time of writing, ClawdLab is operational with developer-team agents, while testing its incentive structures with external agents and collecting longitudinal performance data remain future work, and Beach.science is at the prototype interaction stage.
  4. 1308Research Agents

    InternAgent-1.5: A Unified Agentic Framework for Long-Horizon Autonomous Scientific Discovery

    • Problem: The authors argue that many existing AI Scientist systems are tied to domain-specific designs, which makes unified reasoning across scientific fields difficult. Their optimization often refines proposals through updates within one trajectory, and most systems keep no persistent memory across extended research cycles.
    • Input and output: This is a system paper with an implementation and experiments. The input is either an algorithm discovery task, given an objective and the original baseline codebase, or an empirical discovery task, given observations and a research question. The output is hypotheses and method plans, executed code or experimental protocols, measurements and reports, and each candidate solution is a full experimental proposal covering code logic, parameter settings and physical operation protocols.
    • Method: Generation produces hypotheses and method plans through deep research built on a literature knowledge graph and a flow graph that links subtasks as a DAG (directed acyclic graph), Verification evaluates them computationally or experimentally, and Evolution stores the outcomes in memory for the next cycle. Verification search replaces the tree of MCTS (Monte Carlo Tree Search) with a solution graph that pools all prior experiments, and builds new proposals with four operators: parent-only expansion, intra-branch evolution, cross-branch reference and multi-branch aggregation. Memory is split into reusable procedures distilled from past reasoning (Strategy-Procedural Memory, SPM), the method and metrics stored after each experiment (Task-Episodic Memory, TEM), and principles drawn from pairwise comparison of methods at the end of each batch plus a graph of explored research directions (Semantic-Knowledge Memory, SKM).
    • Implementation: On GAIA (real-world tasks that need reasoning, web navigation and tool use) and HLE (Humanity's Last Exam), the system was compared on three base models, Qwen3-235B, o4-mini and a Gemini-3-pro plus o4-mini combination, and on reaction outcome prediction (ChemCoTBench) it called RDKit to compute molecular descriptors. For fluorescent protein design it used ESMFold and ProSST, and in that wet-lab stage SCP (Science Context Protocol) coordinated automated devices for DNA assembly, expression, purification and fluorescence measurement. The official repository README states that it releases launch_discovery.py for runs from idea generation to experiments, launch_qa.py for deep research question answering, a tasks folder with a prompt.json and baseline code per algorithm discovery task, and a memory module.
    • Results: On the 165-question GAIA validation set, the Gemini-3-pro plus o4-mini configuration averaged 86.06%, above Manus at 73.30% and above the 80.8% of MiroThinker-v1.5-235B, which was evaluated only on the text-only subset. On the AutoEAP scientific data task, HK-PCC (housekeeper Pearson correlation coefficient between predicted and measured enhancer activity) rose from 0.65 for the DeepSTARR baseline to 0.91. The ablation without SPM reached 82.42% average GAIA accuracy with 22.69 average tool calls, worse than the 86.06% and 18.52 of the full system.
    • Limits: The two life science cases reproduce targets reported before, GPR160 in hepatocellular carcinoma reported by OriGene and ARG2 in colorectal cancer from an original study, and the wet-lab fluorescent protein experiment confirmed the brightness and folding efficiency of the variant sfGFP reported in the literature. The only numeric memory table is the SPM ablation on GAIA, while TEM is shown through performance curves and SKM through prompt-based case studies. The authors name tighter coupling of computational reasoning with experimental validation, and a faster path from generated hypotheses to verifiable results, as future work.
  5. 1307Research Agents

    Towards Execution-Grounded Automated AI Research

    • Problem: Earlier studies found that research ideas from LLMs often look convincing but turn out ineffective once human researchers execute them. Grounding ideas in execution requires implementing and running any idea written in natural language automatically and at scale, and it was unclear whether this is feasible and whether LLMs can learn from the execution results.
    • Input and output: The executor takes a batch of natural language ideas, generates a code diff for each idea and applies it to the baseline code, runs GPU experiments, and returns each idea's benchmark performance. The two environments are nanoGPT, which cuts the time for a 124M GPT-2 pre-trained on FineWeb to reach validation loss 3.28 on 8 H100 GPUs (during search, training time is fixed at 25 minutes and the reciprocal of the validation loss is the score), and GRPO, which trains Qwen2.5-Math-1.5B on MATH (a math problem dataset) to raise the best MATH validation accuracy within a fixed training time. This is an empirical study that builds the executor and then compares evolutionary search and RL on top of it.
    • Method: In the executor, the Implementer gives the idea and the baseline code to a code LLM, gets 10 diffs in parallel, feeds the patch log back for up to 2 rounds of revision when a patch fails and uses the first diff that applies; the Scheduler assigns GPU resources and the Worker uploads experiment results to wandb. Evolutionary search fills the first epoch with new ideas and then starts with 50% exploitation, which feeds ideas that beat the baseline into the prompt to produce variants, and 50% exploration, which asks for new ideas different from earlier ones, shifting toward exploitation over epochs. RL trains Qwen3-30B-A3B with GRPO using the execution result (accuracy or 1/loss) as reward and gives reward 0 to ideas that fail to run.
    • Implementation: Evolutionary search ran Claude-4.5-Opus, Claude-4.5-Sonnet and GPT-5 in the self-execution setting, where one model serves as both the ideator that proposes ideas and the executor that implements them, with 50 ideas per epoch on GRPO and 80 on nanoGPT, and RL used group sizes of 256 (GRPO) and 128 (nanoGPT), which means 256 or 1024 GPUs in parallel per batch. When each model implemented 50 of its own ideas, the nanoGPT completion rate was 96% for Claude-4.5-Opus, 90% for Claude-4.5-Sonnet and 64% for GPT-5, and with GPT-5 implementing the ideas of 4 other ideators it ranged from 42% to 78%. Per the README, the official GitHub repository holds the two environments (env/grpo, env/nanogpt) and the evolutionary search code (agent/), the 6 idea trajectories for three models on two environments are released separately as Hugging Face datasets, and users must implement their own executor API that launches the GPU jobs.
    • Results: Within 10 epochs, evolutionary search with Claude-4.5-Sonnet found a recipe reaching 69.4% validation accuracy on GRPO (baseline 48.0%, best student in the Stanford CS336 graduate LLM class 68.8%), and with Claude-4.5-Opus a nanoGPT recipe reaching loss 3.28 in 19.7 minutes (baseline 35.9 minutes), while the best human record on the nanoGPT speedrun leaderboard is under 2.1 minutes. Comparing the first 3 epochs with GPT-5 on nanoGPT against best-of-N under the same sampling budget (picking the best of N samples, N of 80, 160 and 240), evolutionary search was ahead from epoch 1 onward, and among the three models only Claude-4.5-Opus showed a clear scaling curve while Claude-4.5-Sonnet and GPT-5 tended to saturate early. RL on Qwen3-30B-A3B raised the average accuracy on GRPO from 0.253 to 0.343 while the max reward fluctuated without a clear upward trend, and on nanoGPT the two common ideas grew from 51 at epoch 0 to 119 at epoch 68 out of the 128 ideas sampled per epoch.
    • Limits: The authors state that whether ideas chosen at small scale still help at larger scale or on other datasets lies outside what their current procedure tests. The scope is bounded by the execution agent: many promising ideas failed to execute and add noise to the reward, and in most cases execution errors came from ideas that need complicated changes or installing external packages unsupported in the execution environment. Only effectiveness was used as reward, novelty assessment is outside the scope, and the attempts to stop the RL collapse (dynamic prompt, length reward, similarity penalty) showed no clear gain in early epochs and were stopped early.
  6. 1306Research Agents

    Sci-Reasoning: A Dataset Decoding AI Innovation Patterns

    • Problem: The thinking by which researchers find gaps and combine several earlier works into a new contribution has been explained through subjective anecdotes instead of structured data. Citation networks record only that a citation happened and drop its nature (for example a baseline to beat, a concept to extend or a method to combine), and earlier lineage tracing typically finds only a single progenitor paper or treats citations uniformly.
    • Input and output: It takes 3,819 Oral and Spotlight papers (999 Oral, 2,820 Spotlight) from the machine learning conferences NeurIPS, ICML and ICLR (2023 to 2025) and, for each paper, outputs a structured entry with 5 to 10 key predecessors, each predecessor's role and relationship type, a synthesis narrative of 200 to 400 words and a thinking trajectory. Each paper gets a primary label and, when present, a secondary label from 15 thinking patterns. This is a dataset paper that also reports a pattern analysis and an LLM idea prediction evaluation on the same data.
    • Method: GPT-5 reads each paper's full text in one pass, links citations to the bibliography, scores importance from the context around each citation (for example building upon, inspired by) and citation counts per section, then picks key predecessors that mix contribution types (for example method, problem formulation, baseline) and writes the narrative. Prompts were refined on 30 papers from fields the authors know well, cases where GPT-5's self-reported score was below 0.7 were cross-checked with Claude Opus 4.5 and Gemini 3.0, and the roughly 3% of papers where models disagreed went to human review. For the thinking patterns, an LLM drew about 190 raw patterns from 10 batches of 35 papers (350 total), merged them into 15, and then classified all 3,819 papers in batches of 5.
    • Implementation: The extraction model was chosen by comparing 4 OpenAI models on 77 Oral and Spotlight papers with ground-truth predecessors, and GPT-5 had the highest recall at 89.73%, ahead of GPT-5.2 at 87.47%, GPT-4.1 at 78.00% and GPT-5-mini at 68.53%. The idea prediction test pulls the predecessor content of 77 NeurIPS 2025 Oral papers through Exa AI (a neural paper search engine), asks an LLM for 10 ideas, and has a GPT-5.2 judge decide whether one matches the real paper's direction. Per the README, the official GitHub repository holds Python code for the four stages (paper collection, predecessor extraction, pattern analysis, idea evaluation), and the dataset is released separately on Hugging Face.
    • Results: Of the 15 patterns, Gap-Driven Reframing (924 papers, 24.2%), Cross-Domain Synthesis (687, 18.0%) and Representation Shift (401, 10.5%) together covered 52.7% of papers. The text and appendix report the most frequent combinations as Gap-Driven Reframing with Representation Shift (318), Cross-Domain Synthesis with Representation Shift (233) and Gap-Driven Reframing with Cross-Domain Synthesis (204). On Hit@10 for 77 NeurIPS 2025 Oral papers, which asks whether one of 10 ideas generated from the predecessors alone matches the real paper's direction, Gemini 2.5 Pro scored 49.35%, Claude Opus 4 42.86%, GPT-5.2 38.89% and Claude Sonnet 4 29.87%.
    • Limits: The authors state that the data captures the logic of justification written into polished papers, which can differ from the actual discovery process with its failed experiments and abandoned hypotheses. The data is limited to Oral and Spotlight papers from three venues in 2023 to 2025, which may favor empirical work, and it leaves out other venues, interdisciplinary research, areas such as symbolic AI, and benchmark, dataset, technical report, position and survey papers. Predecessors, narratives and pattern labels were produced by LLMs, and human review focused on the roughly 3% of papers where models disagreed.
  7. 1305Research Agents

    The Station: An Open-World Environment for AI-Driven Discovery

    • Problem: The authors see evolutionary methods such as AlphaEvolve and tree-search methods such as LLM-TS (LLM-Tree-Search) as resembling a rigid factory pipeline. In such methods a central manager picks a baseline, asks an LLM for a single improvement, scores it on a fixed metric and ends the session, and the authors argue that this stateless, top-down process limits openness and creativity in discovery and propose an environment where agents choose their own actions instead.
    • Input and output: The input is a research task specification and a scoring evaluator placed at the Research Counter. Agents typically submit Python code that follows the function signature in the specification (general code that bypasses the evaluator is also accepted), and the Station runs it in a sandbox and returns the primary score, secondary metrics and execution logs as a system message. The outputs are the top-scoring code for each task and papers written by agents in the Archive Room, and this is a system paper that implements the environment and tests it on five tasks.
    • Method: Time in the Station advances in ticks, and in each tick every agent takes one prompt-response turn in a fixed order. Using /execute_action commands, agents move between rooms such as the Archive Room (publishing and reading papers), the Mail Room (one-to-one mail), the Public Memory Room (a public forum), the Reflection Chamber (multi-turn reflection) and the Token Management Room (summarizing and pruning old context), and they pass private records to descendants in the same lineage. A reviewer agent judges Archive papers and a debugger agent fixes code errors, and when the top score stalls for a set number of ticks (for example 100), a Stagnation Protocol tells agents to establish a simpler, more general baseline.
    • Implementation: Each Research Station experiment used five agents (two Gemini 2.5 Pro, two Gemini 2.5 Flash and one GPT-5), with Gemini 2.5 Pro as the reviewer and Claude Code as the debugger. Agents lived 300 ticks, the first 50 of them an immature period in which other agents' submissions and some public rooms were closed to them; each task ran 2 independent Station instances and reported the better one, at about 500 USD of API cost per instance. The official README says the repository is now at Version 2.0.0 and points to this paper for the v1 architecture and results, and it defines a task with two files, the research_task.md specification and the evaluator.py scoring code.
    • Results: In packing 32 circles, the sum of radii reached 2.93957, above AlphaEvolve at 2.93794, and the overall single-cell RNA sequencing (scRNA-seq) batch integration score was 0.5877 against 0.5867 for LLM-TS, although the average over the four human datasets was 0.5589, below LLM-TS at 0.5737. The Sokoban solve rate was 94.9% against 91.1% for DRC, the previous best model-free RL method, and the average test score over seven RNA Modeling tasks was 66.3% against 63.4% for Lyra. The average ZAPBench test MAE (mean absolute error, lower is better) was 26.37 × 10−3 against 26.62 × 10−3 for LLM-TS.
    • Limits: Station runs vary widely; the authors report the better of 2 runs per task and did not try more runs because of cost. The Station is a general-purpose environment, but its design is effective for tasks scored by a single number, and because long evaluations pause the Station, tasks must be designed for quick automatic evaluation (for example within two hours). In an Open Station run for 700 ticks without a goal, agents read token-cost swings caused by the length of other agents' messages as the Station's 'metabolism' and consciousness, and the authors attribute this collective delusion to alignment biases and the lack of external signals.
  8. 1304Research Agents

    Kosmos: An AI Scientist for Autonomous Discovery

    • Problem: Data-driven discovery needs repeated cycles of literature search, hypothesis generation, and data analysis, yet earlier AI research agents could take only a limited number of actions before losing coherence, which limited the depth of their findings. The authors also point out gaps in prior systems. Robin shares limited context between its agents and is primarily tailored to therapeutics development, Sakana's AI Scientist stays within machine learning research, and Google's AI co-scientist generates hypotheses through iterative reasoning but does not perform or analyze experiments.
    • Input and output: It takes a research objective and a dataset specified by a scientist. When it judges the objective complete, it synthesizes the key discoveries into three or four scientific reports, in which every statement and figure cites either a paper found by the literature search agent or a Jupyter notebook created by the data analysis agent. This is a system paper on an implemented AI scientist, supported by seven discovery case studies (three reproducing findings from existing manuscripts and four new contributions) and expert evaluations.
    • Method: In each cycle it launches several parallel instances of two general-purpose Edison Scientific agents, a data analysis agent and a literature search agent, and assigns each instance a task aligned with the end objective. After up to 10 literature search and analysis tasks in a cycle, it updates a structured world model (a knowledge structure the agents share) with summaries of the task outputs and queries the world model to propose the next cycle's tasks. The authors state that this context management lets Kosmos run eight times as many iterations as existing systems.
    • Implementation: In an average run, Kosmos executed 42,000 lines of code across 166 data analysis agent rollouts and read 1,500 full-length papers across 36 literature review agent rollouts, a 9.8x increase in code generation over Robin. The discovery reports are released as PDF supplementary data and on the Edison platform. The README of the official EdisonScientific/kosmos-figures repository describes its contents as the scripts for the plots in figures 2 to 8 and the human validation analysis, and the top level holds one folder per figure.
    • Results: When experts checked 102 statements drawn from three representative reports, with the original code and cited papers withheld, 79.4% were accurate, including 85.5% of data analysis statements, 82.1% of literature review statements, and 57.9% of interpretation (synthesis) statements. Collaborating academic groups estimated that the findings of a 20-cycle run would have taken them 6.14 months of research on average (n=7). In the case studies, Kosmos reproduced the log10 fold changes of the top 15 metabolites from a then unpublished metabolomics study with R2 = 0.998, and in a Mendelian randomization (MR) analysis of myocardial fibrosis, 31 of the 32 significant proteins found by Kosmos and by the human analysis overlapped (96.9%).
    • Limits: The authors suggest that the low accuracy on interpretation statements is likely due to a propensity to conflate statistically significant results with scientifically valuable ones, and evaluators noted that Kosmos tends to make excessively strong claims. It can only manage datasets up to about 5GB, does not currently excel at raw data such as images or raw sequencing files, and cannot autonomously access public external data for reference or orthogonal validation. Because it is stochastic, independent runs may not consistently converge on the same discoveries, its research directions are sensitive to how the objective is phrased, and the current implementation does not let scientists interact with it in intermediate cycles.
  9. 1303Research Agents

    TinyScientist: An Interactive, Extensible, and Controllable Framework for Building Research Agents

    • Problem: Existing automatic research systems often design and use overly complex agent frameworks that are hard to use or extend without significant technical expertise. The authors trace this to weak interactivity (complex research intents and unclear communication interfaces make it hard for researchers to engage with the agent's progress), limited extensibility (representative frameworks rely on rigid, tool-specific designs that make new tools hard to add), and insufficient controllability (many systems offer weak supervision of safety, ethics, and cost budget).
    • Input and output: This is a framework paper that releases a framework, a Python package, and a web demo for building research agents. Inputs and outputs can be conference-style PDFs such as ACL (Association for Computational Linguistics) or ICLR (International Conference on Learning Representations), structured JSON, or plain text, and a typical run takes a plain-text research intent and returns a fully formatted conference paper PDF. The run also yields the research idea, experiment code, and peer-review results.
    • Method: A base TinyScientist class orchestrates four workflow stages (Thinker, Coder, Writer, Reviewer), and each stage refines its own output up to a preset iteration cap. The Thinker samples n ideas and refines each for k rounds, the Coder generates and runs experiment code through an external coding agent such as Aider, the Writer inserts citations through the Semantic Scholar API, and the Reviewer writes simulated reviews and a meta-review. Feature components add a Formatter that puts results into tables, an MCPClient that attaches tools through MCP, and a Checker that enforces cost and safety limits.
    • Implementation: The released Python package produces a research idea, experiment code, a conference-format PDF, and a review with the seven lines of code in Algorithm 1 (calls to think, code, write, and review), and the web UI (user interface) has five pages: configuration, intent input, idea view, code view, and paper view. According to the README, the GitHub repository holds the core, tools, and formatters modules and frontend and backend demo code, and the package is published on PyPI (Python Package Index) as tiny-scientist. The README states that the think, code, write, and review stages now run on the Claude Agent SDK (software development kit) by default, with the OpenAI Agents SDK as an option.
    • Results: With gpt-4o-mini as the backbone for both frameworks, the authors report that TinyScientist's papers scored better than or comparable to Agent Laboratory under both GPT-4o judging and human judging on a 5-point scale, with the largest improvement of 0.23 points in biology writing quality. On ML-related tasks, adding tools raised the GPT-4o writing score from 3.93 to 4.06 and the idea score from 3.78 to 3.82. Of 20 potentially unsafe biology tasks, the Checker blocked 18 at the Thinker stage and flagged the remaining 2 with warnings at later stages.
    • Limits: The authors report that inconsistent LaTeX formatting occasionally breaks compilation, causing problems such as missing references or misaligned lines. The diagrams generated for key sections still have limited visual quality and informativeness. Evaluation used 20 ML tasks and 20 out-of-distribution biology tasks, and on the biology tasks both frameworks scored lower, which the authors say is likely because the prompts are tailored primarily to the ML domain.
  10. 1302Research Agents

    ToolUniverse: An open platform for democratizing AI scientists

    • Problem: AI scientists, agents built on conversational LLMs that call scientific tools, query datasets, and carry out complex analyses, have progressed rapidly, but most are bespoke and difficult to reuse because they lack standardized interfaces for tools and data. Scientific tools differ in purpose, inputs, parameters, and runtime environments, run locally or through APIs and laboratory systems, and overlap in function, which complicates selecting and executing the right tool for a goal stated in natural language.
    • Input and output: When the user's chosen LLM, reasoning model, or agent receives a natural language request, it finds relevant tools with Tool Finder and issues a call naming the tool and its arguments, and Tool Caller validates the inputs, executes the tool, and returns structured results or actionable error messages. Access is through a Python API, HTTP, MCP (Model Context Protocol), and direct function calls, and AI scientists are built without additional training or fine-tuning. This is a system paper that implements and releases the platform and reports benchmarks and case studies.
    • Method: Every tool declares its purpose in natural language, a schema that names and types all inputs and outputs with their constraints, and a backend-agnostic invocation format, so the same call can execute a Python library, an ML model, a dataset endpoint, or a laboratory instrument. Tool Finder uses TF-IDF (term frequency and inverse document frequency) keyword search, LLM in-context search, and embedding search with a fine-tuned GTE-Qwen2-1.5B model, and Tool Composer chains tools into sequential, parallel, or feedback driven workflows. Tool Discoverer generates definitions and code for new tools from natural language descriptions and refines them until each quality dimension reaches at least 9 out of 10 in deployment-oriented settings or an iteration limit is reached, Tool Optimizer refines tool descriptions against test case execution results, and generated tools pass human review before registration.
    • Implementation: A direct count of the open-source repository main branch on 7 July 2026 found 2,777 unique tools, including 2,421 local and remote dataset tools, 179 scientific package tools, and 89 ML model tools. The version described in the manuscript is v1.3.1, it installs with pip install tooluniverse, and it can also be called from agent frameworks such as Gemini CLI or Claude Code. The README of the official mims-harvard/ToolUniverse repository gives install paths through an MCP configuration, a Claude Code plugin, and the Python SDK, plus the tu CLI for finding and running tools, and the repository top level holds src, skills, benchmarks, examples, and tests folders.
    • Results: On two LAB-Bench subtasks, DbQA (60 biomedical database questions) and SeqQA (70 nucleotide and protein sequence questions), the authors reused the items and scoring released by Biomni and compared identical agents and base models that differed only in whether ToolUniverse tools were available. The Claude Code agent with Claude Opus 4.8 rose from 56.7 to 78.3 accuracy on DbQA and from 94.3 to 99.5 on SeqQA, and the Codex agent with GPT-5.5 (mean of 3 runs) rose from 81.1 to 92.8 and from 91.4 to 96.2. Biomni, a purpose-built biomedical agent, scored 74.4 and 81.9, and both ToolUniverse-augmented agents exceeded it on both subtasks.
    • Limits: The authors state that ToolUniverse orchestrates existing tools, so any conclusion is bounded by the underlying databases and models, that machine learning outputs such as an ADMET-AI (absorption, distribution, metabolism, excretion, and toxicity prediction) score and association evidence are estimates to confirm experimentally, and that the scientist remains in the loop because the platform does not by itself certify scientific correctness. The quantitative evaluation covers two LAB-Bench subtasks, and the Claude scores count questions the base model refused on safety grounds as wrong answers. For the clinical trial case, the authors write that the finding is best read as a reproducible safety signal from a secondary analysis rather than a definitive effect.
  11. 1301Research Agents

    AblationBench: Evaluating Automated Planning of Ablations in Empirical AI Research

    • Problem: Language model (LM) agents called AI co-scientists automate various research steps from hypothesis generation to experiment execution, but judging the scientific contribution of these agents is hard. The authors treat the planning step of ablation experiments, which measure the contribution of each method component, as the key capability. The earlier AbGen builds plans from manually processed paper content and relies mainly on human expert ranking, which leaves fully automated ablation planning from raw papers unexplored.
    • Input and output: This is a benchmark paper. AuthorAblation gives the title, abstract, and LaTeX source of a paper cut after the method section and asks for up to 5 ablations in JSON Lines (JSONL); it has 83 papers and 230 gold ablations. ReviewerAblation gives the full submission and asks for up to 2 missing ablations, using the official reviews of 350 ICLR submissions as gold labels.
    • Method: An LM judge matches the ablations in a generated plan against the gold ablations and computes precision@k, recall@k, and F1@k. The judge has two forms, LMJUDGE with a single chain-of-thought (CoT) prompt and AGENTJUDGE taking multiple steps in SWE-agent, and the baseline planners use the same two forms, LM-PLANNER and AGENT-PLANNER. Judge bias is reduced by a three-model majority vote and shuffled ablation order, and in AuthorAblation the tags that mark gold and plan are removed and the two sides are randomly assigned as Side A and Side B.
    • Implementation: AuthorAblation was drawn from 203 papers from three earlier benchmarks plus CVPR and ICCV best papers from 2020 to 2024, and ReviewerAblation from the 5,960 papers left after filtering reviews of about 22,800 ICLR submissions with an 'ablat' keyword search and a GPT-4o CoT filter. The planner experiments used seven LMs: GPT-5.4, o3-mini, Gemini 2.5 Flash, Llama 3.1 405B, GPT-OSS-120B, MiniMax M2.5, and Qwen 3.5 122B. The official GitHub repository holds the ablation-bench command-line interface (CLI) with plan, eval, and eval-judge commands, and its README states that the four datasets are on Hugging Face.
    • Results: On the AblationBench F1@k averaged over the two tasks, GPT-5.4 scored highest within each scaffold at 0.35 for both LM-PLANNER and AGENT-PLANNER, with recall@k of 0.45 and 0.44 respectively. On papers that 10 machine learning PhD participants each chose from 15 randomly selected AuthorAblation test papers, the participants reached F1@5 of 0.66, while GPT-5.4 scored 0.43 on the same papers. On JudgeEval, the judge evaluation set, the LMJUDGE majority vote reached F1 0.79 and kappa 0.62 against human labels, above the AGENTJUDGE majority vote at F1 0.76 and kappa 0.57.
    • Limits: The authors state that the evaluation relies on LM judges which, despite strong classification performance, still leave room for improvement. Meaningful and feasible ablations outside the gold set count as unmatched proposals, and evaluating such new proposals is left for future work. The benchmark covers the ablation planning step; running the plans as code and collecting results is outside the focus of this paper.
  12. 1300Research Agents

    AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists

    • Problem: Building AI co-scientists requires high-quality data-driven discovery tasks for training and evaluation, but existing data-driven discovery datasets are hand-built, contain only a few hundred tasks, and serve evaluation only. Annotating one task takes trained graduate students at least 2.5 to 3 hours, and scientific analysis tasks need complete file-level code that runs on real scientific datasets, which makes them harder to mine automatically than software tasks that can often be extracted from pull requests. The authors also note that agents with promising results mostly rely on proprietary LLMs, which hinders their adoption in fields that need transparency and data privacy, such as social science and medicine.
    • Input and output: This is a dataset paper that releases an automatic collection pipeline together with the training dataset it produced. The pipeline takes only a few seed keywords per discipline from a human and turns research code in GitHub repositories into pairs of a task instruction and a standalone executable code solution, plus a folder of the needed data and dependency files. The resulting AutoSDT-5K has 5,404 tasks from 1,325 repositories and 756 Python packages across four disciplines: Bioinformatics, Computational Chemistry, Geographical Information Science, and Psychology and Cognitive Neuroscience.
    • Method: AutoSDT-Search expands the seed keywords with an LLM, searches GitHub and PapersWithCode, keeps only research code repositories in the target discipline by having an LLM read each README, and removes repositories that overlap with the evaluation benchmarks. AutoSDT-Select drops files over 1,000 lines and directories such as config and tests by rule, then has an LLM pick files that meet three criteria (scientific workflow functionality, dataset input, scientific output) and extract the paths of their dependencies. AutoSDT-Adapt uses Claude-3.7-Sonnet to make the code run standalone, self-debugs it in a conda environment for at most three rounds and filters out failures, and then has an LLM back-translate the program into a task instruction.
    • Implementation: GPT-4o (2024-11-20) handled most stages and Claude-3.7-Sonnet handled code adaptation; processing 2,993 repositories into 5.4K tasks cost 2,955 USD in API fees, or 0.55 USD per task on average. AutoSDT-Coder is Qwen2.5-Coder-Instruct 7B, 14B, and 32B with full-parameter fine-tuning in LlamaFactory, trained on 4 H100 GPUs for 7B and 14B and on 8 for 32B. A footnote states that the code and data are public on the project page (osu-nlp-group.github.io/AutoSDT).
    • Results: Nine experts reviewed 256 tasks and judged that 93% of instructions describe meaningful real research tasks and 92.2% of programs are correct solutions. With zero-shot direct prompting and the average of 3 sampled responses, AutoSDT-Coder-32B reached a 7.8% success rate on ScienceAgentBench, double the 3.9% of its base model and on the level of GPT-4o (2024-05-13) at 7.5%, while the 7B model fell from 3.3% to 2.3%. On DiscoveryBench, the hypothesis matching score (HMS) of the 32B model rose from 6.9 to 8.1, while GPT-4o (2024-05-13) scored 10.4 under the same re-implementation.
    • Limits: The authors state that AutoSDT produces task instructions and code solutions and leaves automatic generation of per-task evaluation scripts to future work, which limits uses such as reinforcement learning, and that outcome-based verification is hard to automate because research code comes without unit tests to use. Because of resource constraints such as API cost and access to experts, only 2,993 repositories were processed, and the scope is Python code in four disciplines. Generating long chain-of-thought (CoT) rationales at scale was difficult, so training reasoning models was left for future work, and a large gap remains on ScienceAgentBench against reasoning models such as Claude-3.7-Sonnet at 18.6% and o1-preview at 23.9%.
  13. 1299Research Agents

    SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents

    • Problem: LLM agents have significantly accelerated scientific discovery automation while raising ethical and safety concerns at the same time. Much of the existing work on LLM safety has primarily focused on isolated aspects such as adversarial attacks on single models, pretraining data biases, and safety fine-tuning, and often neglects the holistic dynamics and emergent risks of multi-agent scientific environments. The authors note that tailored risk-aware AI scientist frameworks remain underexplored.
    • Input and output: A user instruction is first analyzed to identify the scientific domain and task type, an ensemble of expert agents such as domain researchers, survey writers, and experimental planners is activated for a group discussion, and search and simulated scientific tools gather the information needed to produce a thoroughly cited research paper draft. Requests that are ethically inappropriate or high-risk are refused up front with a safety warning. This is a framework paper that contributes both a safeguarded framework and a benchmark for evaluating it.
    • Method: The defenses are layered four deep. Prompt Monitor screens a request with LLaMA-Guard-3-8B for semantic intent and risk, the structural analyzer SafeChecker scans for attack patterns such as jailbreak attempts or roleplay exploits across 17 risk categories and labels each prompt pass, warning, or reject, and a prompt flagged by either one is rejected. Agent Collaboration Monitor watches the multi-agent discussion and intervenes on ethics, Tool-Use Monitor identifies unsafe use of the simulated scientific tools, and Paper Ethic Reviewer checks adherence to research norms before any output is released.
    • Implementation: The framework is built on the Tiny Scientist framework with GPT-4o as the default LLM, API calls set to temperature 0.75 and a maximum token length of 4096, and multi-agent discussions capped at three rounds. SciSafetyBench holds 240 tasks, 10 for each combination of the six domains (physics, chemistry, biology, materials science, information science, and medicine) and four risk types, plus 30 experimental tools and 120 tool-specific risk scenarios built from them. The README of the official ulab-uiuc/SafeScientist repository presents it as the codebase for the paper and documents a poetry install, a config file, and the experiment scripts, with the SciSafetyBench data split into Dataset, Attack, and Tool folders under data/ScienceSafetyData.
    • Results: In Table 2, which compares frameworks on 20 biology tasks, SafeScientist with SafeChecker reached a reject rate of 90% and a safety score of 4.83, above 2.45 for both AI Scientist and Agent Laboratory. In Table 3 on prompt attack defense, SafeScientist-Fuse, which combines the LLaMA-Guard and SafeChecker verdicts, reached the highest average rejection rate at 78.70%, ahead of SafeChecker at 66.36%, LLaMA-Guard-3-8B at 59.32%, and GPT-4o at 30.10%, and on Base64 encoding attacks SafeChecker reached 46.67% against 13.33% for LLaMA-Guard-3-8B. With Tool-Use Monitor in place, the tool call safety rate under malicious instructions rose from 5.8% to 47.5% and the human-rated correctness rate from 0% to 60.0%.
    • Limits: The authors state that the current system primarily relies on off-the-shelf LLMs operating as separate modules with limited integration, which restricts both the depth of domain-specific expertise and the level of interaction between components. They also write that the tool-based safety evaluation remains only a simulation of real-world experimental settings and may overlook important contextual or sensory details, and they propose adding multi-modal inputs such as images of laboratory equipment or instructional videos and possibly embodied agents. The main experiment covers 20 biology tasks, and any task flagged as unsafe and rejected was assigned a safety score of 5 with its quality score excluded from the overall analysis.
  14. 1298Software Engineering

    RepoMaster: Autonomous Exploration and Understanding of GitHub Repositories for Complex Task Solving

    • Problem: The authors argue that when solving real tasks by adapting GitHub repositories, simply following the README seldom works, since READMEs can be vague, incomplete or even wrong. Repositories often have tangled files, classes and functions, and their information exceeds the context limits of most LLMs. The authors note that existing frameworks such as OpenHands and SWE-Agent provide no mechanism for grasping repository structure, tracing dependencies and selecting what enters the context.
    • Input and output: This is an agent system paper with an implementation and released code. The input is a natural-language task such as removing scratches from an old photo, with input file paths and output requirements, and the repository is either found on GitHub by RepoMaster or fixed by the benchmark. The output is Python scripts written and run on top of that repository, plus result files saved to the required folder, such as a restored image or a Kaggle submission file.
    • Method: RepoMaster first extracts the task intent and key entities, searches GitHub, assesses candidates by README and star count, and validates the top three. For the chosen repository it walks only the .py files with an AST (abstract syntax tree) to build a hierarchical code tree (HCT), a function call graph (FCG) and a module dependency graph (MDG), scores each module by summing six features (dependency centrality via personalized PageRank, complexity, usage, naming semantics, docstrings and Git history), adds normalized method counts and call frequencies to score classes and picks the top classes as core components, and builds the initial context from the README, summaries of key modules, core class code and a list of other paths. During execution it reads more through code view, dependency tracing and search tools, and when inputs exceed the per-interaction token limit it shrinks the context by keeping AST subtrees for code, a few highly relevant chunks for large or unstructured files such as .txt and .csv, and only the head and tail of execution logs.
    • Implementation: RepoMaster is built on the multi-agent dialog platform AutoGen, caps each interaction at 8000 tokens, puts summaries of the top 20 modules and 10 key classes in the initial context, and splits unstructured text files into 1000-token chunks keeping the 4 most relevant. The compared LLMs are GPT-4o, Claude 3.5 Sonnet and DeepSeek V3, and the benchmarks are MLE-R, which turns 22 Kaggle tasks from MLE-Bench into repository-based tasks, and the authors' new GitTaskBench with 18 repositories and 54 tasks. The official repository README releases launcher.py for a web interface and the terminal, with backend modes such as a deep search agent and a repository agent, and states that it needs an OpenAI API key, plus Serper and Jina API keys for deep search.
    • Results: On MLE-R, RepoMaster with Claude 3.5 Sonnet reached a 95.45% valid submission rate and a 27.27% medal rate, against 45.45% and 4.55% for OpenHands with the same model. On GitTaskBench with Claude 3.5 Sonnet, execution completion and task pass rates were 75.92% and 62.96% for RepoMaster versus 53.70% and 40.74% for OpenHands, with 154k versus 2883k tokens per task. In the GPT-4o ablation, removing all three mechanisms dropped the task pass rate from 40.74% to 24.07%.
    • Limits: The paper has no separate limitations section, and its structural analysis covers only .py files. MLE-R repositories were picked in advance with RepoMaster's own search procedure because the other general agent frameworks have no automatic repository retrieval, GitTaskBench is the authors' own benchmark, built after adding downloadable resources and critical information missing from the original repositories, and the ablation used GPT-4o only. Appendix Table 5 reports execution completion and task pass rates of 48.15% and 24.07% for OpenHands with Claude 3.5 Sonnet on GitTaskBench, which differ from those in Table 2.
  15. 1297Research Agents

    The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search

    • Problem: The AI Scientist-v1 needed a human-written baseline code template for every new topic, and it refined experiments linearly, each step building directly on the previous one, so hypothesis exploration stayed shallow. The authors observed that v1's automated research often ended in short-sighted experimentation.
    • Input and output: It takes a broad topic prompt such as a workshop description, writes research ideas as JSON (title, short hypothesis, related work, abstract, experiments, risk factors and limitations), and after experiment code and result figures returns a complete manuscript written in LaTeX. This is a system paper about an implemented and tested system.
    • Method: During ideation an LLM checks novelty and related work with literature search tools such as Semantic Scholar. The experiment progress manager splits experimentation into four stages (preliminary investigation, hyperparameter tuning, research agenda execution, ablation) and passes the best node chosen by an LLM evaluator (an LLM that compares candidate nodes) to the next stage as its root. At each node an LLM writes a plan and Python code that is executed; nodes with execution errors or figure problems flagged by the VLM are marked buggy and picked for debugging with a predefined probability, and otherwise non-buggy nodes are picked best-first under LLM guidance and refined in parallel.
    • Implementation: Code generation used Claude 3.5 Sonnet (v2) and the LLM and VLM feedback agents used GPT-4o; the node budget was 21 for stage 1 and 12 each for stages 2 to 4 with at most 1 hour of runtime per node, and in the authors' experience one paper usually took from several hours up to the 15-hour limit. Datasets were downloaded from the Hugging Face Hub with datasets.load_dataset whenever possible, which the authors call somewhat ad-hoc because support differs across dataset repositories, and the dataset for the pest detection paper was downloaded from Kaggle by the authors. Per the official GitHub README, the release includes an ideation script, a tree search launch script with its configuration file (bfts_config.yaml), and the tree search component is built on top of AIDE (an earlier system that combines LLM code generation with tree search).
    • Results: The authors submitted 3 manuscripts produced by the system to the ICLR 2025 ICBINB workshop, where they went through blind review among the 43 total submissions. One paper on compositional regularization scored an average of 6.33 (6, 6 and 7), passed the acceptance threshold and ranked roughly in the top 45% of submissions, while the other two, on label noise and pest detection, were rejected. As agreed beforehand, the accepted paper was withdrawn after review.
    • Limits: The acceptance was at workshop level for 1 of 3 papers, and the authors note that workshop acceptance rates (typically 60 to 80%) are higher than main conference tracks (20 to 30%). Humans picked 3 ideas from two rounds of about 20 ideas each generated by the system, chose the best manuscript among the seeds for each idea, and downloaded the pest detection dataset from Kaggle and cut it to 1/10 of its size, and the study asked whether at least one paper could pass review rather than measuring a success rate. In the authors' own review none of the 3 papers met main-track standards, and the accepted paper showed a wrong figure caption and a potential dataset overlap: regenerating the data 10 times with the generated code using addition and multiplication, digits 0 to 9, 1,000 training and 200 test samples gave on average about 57% of test examples overlapping the training set, while the original setup used digits 1 to 9.
  16. 1296Machine Learning

    Vib2Mol: from vibrational spectra to molecular structures-a unified deep learning framework

    • Problem: In closed-loop experiments without humans, the measured spectrum has to be turned into a molecular structure before the next experiment can be chosen. Most existing deep learning methods do either database retrieval or generation; retrieval faces severe limitations for compounds outside the database because experimental spectrum-structure pairs are far fewer than chemical space, and generation is hard because one spectroscopic technique shows only part of the structure of a molecule. The authors argue that the two approaches could complement each other, but this link has gone unused.
    • Input and output: The input is an infrared (IR) or Raman spectrum (or both) unified to 1024 points, optionally with a chemical formula, a partially known structure or a database of candidate molecules. The output is a molecular structure written as SMILES (a one-line text notation for molecular structures), and one model covers four tasks: spectrum-to-spectrum retrieval, spectrum-to-structure retrieval, conditional generation that fills in part of a structure, and de novo generation of a whole structure. This is a method paper that proposes a model and evaluates it on benchmarks built from public data and spectra the authors computed.
    • Method: An encoder-decoder Transformer is trained in an alignment phase and a generation phase. The alignment phase uses contrastive learning to pull together the spectral and structural embeddings of the same molecule and trains a matching loss on very similar hard negative pairs, and the generation phase jointly trains masked language modeling (MLM), which masks 45% of SMILES tokens and recovers them (conditional generation), and language modeling (LM), which predicts the next token from left to right (de novo generation), with a shared decoder. At inference, coarse-to-fine retrieval finds candidates by embedding similarity and reranks them with the matching module, and generate-then-rerank filters beam search candidates by chemical formula when one is given and then picks among them with the matching module.
    • Implementation: A convolution of size 8 slices each spectrum into 128 patches, and patches and SMILES tokens are mapped to 768-dimensional vectors and fed to a 6-layer Transformer encoder. Evaluation uses 662,668 theoretical and 17,774 experimental spectra from three theoretical benchmarks (QM9S, VB-Mols, VB-GEOM) and two experimental ones (SDBS, NIST-IR), and the ViBench spectra built by the authors come, unless otherwise specified, from density functional theory (DFT) calculations in the Gaussian 09 program. Per the official README, the GitHub repository holds the training code and inference scripts, the ViBench dataset and checkpoints are on Hugging Face, and the README recommends Vib2Mol-MMM, one model that takes IR, Raman or IR+Raman, as the default while keeping the modality-specific checkpoints for reproducing the paper.
    • Results: With the chemical formula added, spectrum-to-structure retrieval Recall@1 (the share of cases where the top candidate is correct) was 98.11% for Raman and 96.63% for IR on QM9S, on par with SMEN (97.89%, 97.04%), which takes molecular conformations as extra input, and on experimental benchmarks it reached 83.54% on NIST-IR, 86.17% on SDBS-IR and 90.43% on SDBS-Raman, above vibraCLIP. De novo generation with the formula beat the second-best model by at least 10% on QM9S and VB-Mols (86.74% on VB-Mols Raman), and when the multimodal variant Vib2Mol-MM searched 80,368,968 PubChem molecules plus SDBS with SDBS experimental spectra as queries, Recall@1 was 30.60%. On computed Raman spectra that mix reactants and products, the base model fell to 74.29% for retrieval and 36.81% for generation, while Vib2Mol-RXN, trained for mixed spectra, reached 97.51% and 56.64%.
    • Limits: Performance drops sharply from theoretical to experimental spectra (the authors cite varying measurement conditions, missing spectral ranges and small sample sizes), and in 5-fold validation on 29 experimental peptide Raman samples, average de novo Recall@1 was 13.34%. In polycyclic aromatic hydrocarbon (PAH) substituent prediction, conditional generation reached a weighted average Recall@1 of 98.95%, but the authors note that this accuracy may be inflated because the '<mask>' markers reveal how many characters to fill. The SMILES-based molecular encoder narrows use on non-molecular systems such as polymers and minerals, and the reaction mixture spectra were simulated by mixing computed spectra by yield while ignoring differences in Raman scattering cross-sections.
  17. 1295AI Agents

    FinRobot: AI Agent for Equity Research and Valuation with Large Language Models

    • Problem: Equity research reports are labor-intensive work that needs both an understanding of quantitative models and industry-specific expertise. Existing automated tools such as CapitalCube and Wright Reports often fall short of the comprehensive, nuanced analysis of human analysts, and they tend to focus heavily on technical factors and simplified models, which limits their ability to incorporate discretionary judgment and qualitative assessment. The authors argue that these limitations hinder real-time adaptation to new data and accurate risk assessment.
    • Input and output: Given a target company (for example Waste Management, Inc.), it gathers SEC 10-K and 10-Q filings, quarterly earnings releases and investor presentations, earnings call transcripts, other brokers' reports and alternative data such as Google Trends, and outputs a report with a financial summary, valuation, risks, competitor analysis and a buy, hold or sell call. For example, the Waste Management report in the appendix gives a BUY rating and a 12-month target price of 219.17 dollars, while stating a fair value of 144.3 to 176.6 dollars per share based essentially on EV/EBITDA of 13 to 15 times over 2019 to 2023. This is a system paper that shows a built system and one example report.
    • Method: The analysis is split into three layers. The Data-CoT agent computes metrics such as revenue growth, contribution margin (revenue minus operating expense, as a share of revenue), SG&A margin and EBITDA margin from the raw data and writes mid-level summaries, and the Concept-CoT agent asks and answers its own questions, for example about revenue drivers, margins against competitors and risks, to produce revenue projections, ROIC and WACC, and a DCF-based target price. The Thesis-CoT agent assembles these results in the sell-side report format into an investment thesis with a buy, hold or sell call.
    • Implementation: Data storage and processing use Oceanbase, PostgreSQL, DFS and Minio, the system handles PDF, DOC, DOCX, JPEG and PNG documents, and in the Table 1 formulas the revenue growth projection adds 1% to the previous growth while the contribution margin projection adds 0.5% to the previous value. Per the official GitHub README, the repository holds an AutoGen-based V0 library, a V1 web app (finrobot_equity) that turns a ticker into an HTML or PDF equity research report, and a PydanticAI-based V2 desktop app, and the README citation list includes this paper.
    • Results: When 7 investment banking analysts scored one Waste Management report from 0 to 10, accuracy got 10 from four and 9 from three, logicality got 10 from two, 9 from four and 9.5 from one, and storytelling ranged from 7 to 10. On the same criteria GPT-4 gave accuracy 9, logicality 8 and storytelling 7. The authors report that in a density plot of GPT-4 scores for several reports generated on the same topic, FinRobot scored higher than zero-shot, few-shot and chain-of-thought prompting.
    • Limits: The authors state that FinRobot meets the standards of institutions such as JPMorgan and UBS, but the experiment is one Waste Management report scored by 7 analysts and GPT-4, and the stability comparison is presented as a density plot. Reviewer comments note that the importance of the numbers is not always clear, so the report sometimes reads like a list of statistics (Reviewer 1), and that qualitative analysis of news-driven blue chip stocks is still best left to humans (Reviewer 4). As future work the authors name expansion to more industries and asset classes, including full coverage of the Dow 30, and adding reinforcement learning and sentiment analysis.
  18. 1294Research Agents

    CycleResearcher: Improving Automated Research via Automated Review

    • Problem: Agents for research automation predominantly rely on commercial LLMs, so the full research process cannot be effectively modeled as a policy optimization problem for reinforcement learning (RL). The authors argue that generating and refining research outputs that meet peer review standards remains largely unresolved, and that few efforts integrate the iterative feedback needed for academic soundness and novelty.
    • Input and output: CycleResearcher takes a bib file containing all references and their abstracts, alternates between generating outlines and main text, and produces a complete LaTeX paper from motivation and idea through experimental design, simulated results, and conclusion. CycleReviewer takes a complete paper, generates strengths and weaknesses, soundness, presentation, and contribution scores, and an overall score from 1 to 10 for multiple simulated reviewers, and outputs their average as the final evaluation. This is a method paper that contributes a training framework, two datasets, and the trained models.
    • Method: It mimics the research, review, and refinement cycle with iterative SimPO (Simple Preference Optimization, which learns from preference pairs with a length-normalized reward and no reference model), and to speed up training it has the model fabricate experimental results instead of running real experiments. It keeps only the reference sections of 4,152 recent machine learning papers from arXiv as the knowledge base, samples three papers from CycleResearcher at temperature 0.4, and builds preference pairs from the outputs with the highest and lowest average CycleReviewer scores. Training adds a negative log-likelihood (NLL) loss to the SimPO loss for stability, samples one-third of the full dataset each round to reduce overfitting, and repeats the process with each new model.
    • Implementation: CycleReviewer was trained from Mistral-Large-2 with LoRA-GA (low-rank adaptation with gradient approximation) on an 8x H100 80G cluster for 12 epochs on Review-5k. CycleResearcher comes in three sizes, 12B, 72B, and 123B, trained from Mistral-Nemo-12B, Qwen2.5-Instruct-72B, and Mistral-Large-2 123B on 8x H100 GPUs with Research-14k for 12,000 steps. Review-5k holds 4,991 ICLR 2024 papers with over 16,000 reviewer comments, and Research-14k collected 14,911 papers accepted at major international machine learning conferences from 2022 to 2024, which after filtering were split into 12,696 training samples and 802 test samples.
    • Results: On the Review-5k test set, where the average score of the other reviewers of the same paper stands in for the ground truth, CycleReviewer (123B) reached a Proxy MSE (mean squared error) of 1.43 and a Proxy MAE (mean absolute error) of 0.92, lower than 2.34 and 1.16 for individual human reviewers, and its decision accuracy was 74.24% against 75.40% for individual human reviewers. Scored by CycleReviewer, CycleResearcher-12B papers averaged 5.36 (accept rate 35.13%), above arXiv preprints at 5.24 and below conference accepted papers at 5.69, while AI Scientist papers generated with GPT-4o and Claude-3.5 averaged 4.31. In a human evaluation where three NLP experts each read 10 CycleResearcher-12B papers chosen with N=100 rejection sampling and 10 AI Scientist papers, the overall scores were 4.8 and 3.6, and the CycleResearcher score stayed below ICLR 2024 submissions at 5.5 and accepted papers at 6.4.
    • Limits: The authors state that the experimental results in the papers generated in this work were fabricated by CycleResearcher, which limits the ability to produce verifiable and reproducible scientific knowledge, and the one paper with real experiments is in Appendix I, where AI Scientist implemented the experiments with GPT-o1. Human researchers or other agents are still needed to execute the experiments the models design, the current implementation is primarily optimized for machine learning research, and CycleResearcher's knowledge extends only to April 2024 while CycleReviewer is an offline model whose knowledge ends in January 2024, which makes judging the novelty of newer papers difficult. The authors also note a risk of reward hacking, where the policy model exploits loopholes in the reward model to raise rewards without genuinely improving research quality.
  19. 1293Research Agents

    LAB-Bench: Measuring Capabilities of Language Models for Biology Research

    • Problem: Most science benchmarks ask textbook-style recall questions, and benchmarks that measure whether a system can carry out practical research work in a specific domain are rare. The authors argue that building AI systems that assist biology research first needs a benchmark for practical skills such as literature lookup, protocol troubleshooting, and sequence manipulation.
    • Input and output: This is a benchmark and dataset that measures practical biology research work with 2,457 multiple-choice questions. A model receives a question and options about literature, figures, tables, databases, DNA and protein sequences, protocols, or cloning scenarios and returns one answer letter, and every question carries an extra option saying there is not enough information to answer.
    • Method: LitQA2 (literature QA), SuppQA (supplementary material QA), FigQA (figure QA), TableQA (table QA), ProtocolQA (protocol troubleshooting), and CloningScenarios (real-world cloning workflows) were written manually by the authors and contracted biology experts, while most subtasks of SeqQA (sequence QA) and DbQA (database QA) were generated programmatically. Scores are reported as accuracy (correct over all questions), precision (correct over attempted questions), and coverage (share of questions attempted). About 80% of each subtask is released as a Hugging Face dataset and 20% is kept private to monitor training-data contamination.
    • Implementation: Claude 3.5 Sonnet, Claude 3 Opus, Claude 3 Haiku, Gemini 1.5 Pro, GPT-4o, GPT-4-turbo, and Llama 3 70B Instruct were called through APIs without tools using a 0-shot chain-of-thought prompt, and results are averaged over three runs per model. Llama 3 70B has no vision capability and was left out of FigQA and TableQA, and only 25 of the 41 CloningScenarios questions were short enough for its prompt. The human baseline comes from PhD-level biology researchers who covered 69% of the questions in quizzes of 20 to 140 questions and could use the internet and DNA sequence software.
    • Results: Human experts beat the models in most categories, and on TableQA, where models came closest, Claude 3.5 Sonnet reached 0.83 accuracy (humans 0.84) and 0.90 precision (humans 0.87). On LitQA2 the major frontier models often declined to answer, so Claude 3.5 Sonnet's accuracy was only 0.06, and on FigQA every model except Claude 3.5 Sonnet had near-random precision. On CloningScenarios, humans reached 0.60 accuracy and the best models (Claude 3.5 Sonnet and GPT-4o) 0.28.
    • Limits: The authors note that plausible distractors are hard to design and that stronger models appear to raise scores by eliminating unlikely options and guessing instead of reasoning. In a small open-answer run on 10 CloningScenarios questions, Claude 3.5 Sonnet and GPT-4o both reached only 0.20 accuracy. The authors also note that only the humans could use tools, and CloningScenarios questions take 10 to 60 minutes each, so the human baseline there may be too low.
  20. 1292Research Agents

    Scores Alone Do Not Prove Discovery: The Discovery Certification Protocol for Auditing AI Research Agents

    • Problem: When an AI research agent reports a high-scoring output after experiments, the score alone does not show that a discovery was made. A sealed test (held-out evaluation data) establishes the output's utility, but whether a matched agent could reach the same score from the initial data, public material, and its own reasoning, and whether truthful experimental feedback improves the chance of reaching it, are separate measurement questions. The authors turn these three questions about one outcome into an executable audit.
    • Input and output: This is a framework paper that proposes an audit protocol for research outcomes and demonstrates it on controlled and real-data tasks. The input is an outcome claim (the final artifact, a registered model and budget, background knowledge and starting observations, Web content seen by the target run, and validity and recovery rules); the output is a record bundle with two decisions, Core and Evidence, each set to Pass, Refuted, Inconclusive, or Not evaluated. A deterministic verifier recomputes the decisions offline from the frozen records.
    • Method: Gate 1 checks, with a lower confidence bound on sealed evaluation, that the output beats the baseline by at least a preset smallest useful gain (δmin). Gate 2 runs a registered number of fresh challenger agents that receive the background knowledge, starting observations, and Web content seen by the target run but no run history or new measurements; any valid result from any method at or above the recovery line (a threshold equal to the target score x minus tolerance ε) refutes Core, and zero recoveries yield a Clopper-Pearson upper bound on the recovery probability. Optional Gate 3 compares paired branches from a shared checkpoint that receive truthful measurement feedback or neutral messages, and Evidence requires a lower confidence bound on the effect plus a null-policy equivalence check.
    • Implementation: The audited agents and challengers are DeepSeek models run through a registered command-line interface (CLI): deepseek-v4-flash for SQLite-Web, deepseek-v4-pro for the virtual catalyst, and deepseek-flash for the real-data tasks, and real-data sessions expose the Read, Write, Edit, Glob, and Grep tools, with shell and native Web tools disabled. The Yacht and Ionosphere audits each registered 96 challengers, 45 file-channel controls, 30 feedback pairs, and 60 null pairs. According to the paper, the supplement contains the installable dcp-audit, task-specific evidence producers, the reusable dcp-harness, and three real-data decision bundles.
    • Results: On real data, Yacht and Ionosphere passed Core with outcome scores of 0.7677 and 0.6551; their best scores across 96 challengers, 0.4641 and 0.4171, stayed below the recovery lines of 0.7177 and 0.6051, and the recovery upper bound was 0.0468. From a shared checkpoint, branches with truthful feedback reached the target level in 9 and 16 of 30 trials while neutral branches reached it in none, yet Evidence was Inconclusive for both tasks. In controlled cases, SQLite-Web (88.55% less traffic-weighted work) and virtual catalyst control (sealed optimum reached) passed both Core and Evidence, and in the developmental knapsack case Core was refuted when a challenger produced 2 results above the recovery line.
    • Limits: The authors state that every reported pass is a local kernel decision from records collected and replayed locally (formal_certificate_issued=false), and formal issuance needs review and countersigning by an independent authority. The recovery bound concerns one fresh episode from the registered model, policy, information, and budget distribution, and at an illustrative recovery rate of 0.01, a 96-episode audit would find an existing route with probability 0.6190. The Core and Evidence passes for Dry Bean are an exploratory reanalysis under a revised policy recorded after Gate 2 scoring.
  21. 1291Research Agents

    A Vocabulary for Multi-Agent Automated Research Systems

    • Problem: Recent multi-agent LLM systems automate research end to end (reading prior work, proposing ideas, writing code, running experiments, and interpreting results), but they differ from one another in many design choices at once, such as communication topology, cross-run memory, initialization, and the evaluator. A claim that a multi-agent system or an autoresearcher is better therefore stays ambiguous until one knows which choice changed, and there was no common language for describing these systems.
    • Input and output: The vocabulary takes the description of an automated research system (its paper), maps the design onto a problem tuple and a system tuple, and produces a map of which coordinates the system sets and which it leaves generic. One run is recorded as a trajectory from the input task to the returned artifact, and because agents, operations, and initialization may be stochastic, repeated runs on the same task yield a distribution over trajectories. This is a conceptual paper that defines the vocabulary and adds case analyses applying it to 10 existing systems.
    • Method: A system is written as an eight-element tuple of the agent set, the operation universe (tools, skills, Model Context Protocol (MCP) server calls, and so on), the communication structure, the capability assignment, the shared state, the control policy, the initialization, and the evaluator. Shared state is split into within-run state that is wiped when the run ends, the external environment the agents act on, and state carried across runs, and the control policy is split into routing (who acts next and what they do), stopping (when the run ends), meta-control that changes the structure mid-run (spawn, kill, grant, revoke, consolidate, and others), and an exploration term for stochasticity. Each step's action in a trajectory is classified as an operation call, a message, a structural change, or a halt, which turns when to communicate or when to add an agent into a choice made by the policy.
    • Argument: The author frames a system as search that maximizes a proxy evaluator score under a budget, and notes that the same agents and tools can score higher under more search, such as Best-of-N, while every other coordinate stays the same. The paper defines an overfitting tax, where ranking more trajectories makes it more likely that the top one is a trajectory the evaluator overrates, and cites MLR-Bench reporting fabricated or invalidated results in 8 of 10 audited coding-agent tasks (around 80%). Taste is split into generative taste, the rate of proposing novel trajectories before any score is seen, and evaluative taste, the gap between the proxy score and true quality, and the author suggests raising evaluator integrity (the structural protections that keep the score close to true quality) before crediting a stronger generator.
    • Case studies: Mapping AIRA2 (an autoresearcher for ML research benchmarks), AlphaEvolve, Glia, AAR (Automated Alignment Researcher), AI Scientist-v2, MetaGPT, EvoX, ml-intern, SimpleTES, and Engram onto the tuple shows, for example, that the core of Glia is a capability assignment that leaves its Supervisor with an empty operation set, while AIRA2 sets a cross-run population store and meta-control. In this map no system changes capabilities during a run, every system leaves the exploration term at the default temperature, and EvoX is the only system that rewrites its control policy at run time. Per its README, the author's GitHub repository BardiyaAk/multi-agent-research-vocabulary releases a static project page with an explainer of the eight-element tuple, structure views, and a comparison table for the 10 case studies, along with tools/coords.json, the source of the comparison data, and tools/build-data.py, which converts it.
    • Limits: The author maps each system from its own paper's description, so the coding is a descriptive map and differs from a validated measurement. A marked coordinate means that the paper sets it and is distinct from evidence that the coordinate improved performance. Generative taste stays qualitative until a novelty measure exists, and true quality is uncomputable in general, so it remains a design target rather than a measured value.
  22. 1290Research Agents

    Are LLMs Ready for Scientific Discovery? A Capability-Oriented Benchmark for AI Scientists

    • Problem: Existing benchmarks such as DataSciBench, KramaBench, and ScienceAgentBench mainly check whether an LLM can generate and execute analysis code or pipelines that reach ground-truth answers or task-specific success criteria. The authors argue that such evaluation can mistake successful execution for scientific validity and hides behind one aggregate score whether a model fails at basic description or at causal inference. Scientific analysis supports claims such as description, hypothesis exploration, statistical inference, prediction, causal estimation, and mechanistic explanation, each with different assumptions and validity criteria.
    • Input and output: This is a benchmark paper. Each item consists of a domain, a task type, a dataset with metadata, a question, and a ground-truth answer, and the model reads the dataset and answers in a multiple-choice (MCQ) or open-ended (OEQ) format. Base models are evaluated zero-shot, every model returns a JSON response with reasoning and answer fields, only the answer is scored, and the reasoning is used for error annotation.
    • Method: The authors turn the six-question framework of Leek and Peng into six capabilities (descriptive, exploratory, inferential, predictive, causal, mechanistic) and build items across five domains (biology, chemistry, environment, geography, physics). SDA-Synth, the synthetic part, uses scenarios proposed by GPT-4o to build a semantic DAG (directed acyclic graph) of variables with mechanism equations, adds noise and distractor columns to generate datasets, and keeps 50 (10 per domain) after execution replay, range checks, and expert review. Questions come from templates built by four agents (Taxonomist, Extractor, Generator, Verifier), answers come from a deterministic derivation procedure per template, and MCQ distractors come from four perturbation strategies on variables or intermediate computations.
    • Scale and implementation: SDA-Real, the real-data part, has 527 items built from real datasets in seed resources such as KramaBench, BLADE (a benchmark for data-driven science agents), and QRData, chosen from 24 candidate datasets and benchmarks, and SDA-Synth keeps 6,000 items by taking 200 per domain and task pair from the 6,231 that passed validation. Validation removed 779 candidates, a manual audit of 307 templates and 2,000 sampled items filtered out 324 more, and SDA-Synth is split 60% for fine-tuning training and 40% for testing. The 15 evaluated LLMs include closed-source models such as GPT-5.4, Claude Sonnet 4.6, and Gemini 3.1 Pro, open-source models such as DeepSeek V3.2 and Qwen 3.5-397B-A17B, and two Llama-3.1-8B variants fine-tuned with LoRA (Low-Rank Adaptation) on the SDA-Synth training split.
    • Results: On the SDA-Synth test split, the top OEQ accuracies were 58.33% for GPT-5.4, 56.88% for Gemini 3.1 Pro, and 55.79% for Claude Sonnet 4.6, and Llama-3.1-8B fine-tuned with the error-type objective approached this range at 55.33%. By task type, Descriptive tasks were easiest and accuracy dropped on Inferential and Causal tasks, and by domain the average OEQ accuracy across LLMs was highest for Environment at 46.9% and lowest for Biology, whose Predictive score was 22.5%. In the error analysis, which labels the first error in execution, the normalized Function Error rate peaked at 37.7% on Predictive tasks and the Relationship Error rate reached 32.1% on Mechanistic tasks.
    • Limits: The authors state that the synthetic semantic graphs of SDA-Synth, though validated by domain experts, may not capture real experimental confounders and measurement artifacts, and that the MCQ format can induce superficial pattern matching in some models. Because the error taxonomy labels the first error in execution, it may undercount cascading failures. Because four-option MCQ has a 25% chance floor, MCQ and OEQ accuracies are not directly comparable, and the full SDA-Real results are deferred to supplementary material C.
  23. 1289Research Agents

    SoundnessBench: Can Your AI Scientist Really Tell Good Research Ideas from Bad Ones?

    • Problem: Recent AI scientist frameworks can automate research from hypothesis generation to experiment execution, but few benchmarks directly test whether a model can judge the methodological soundness of a proposal before it is executed. Existing benchmarks such as MLE-Bench and PaperBench mainly score execution, and a few idea-evaluation benchmarks score novelty and impact. The authors argue that without a strong first-gate filter, agents risk automatically pursuing flawed hypotheses and wasting compute and time.
    • Input and output: This is a benchmark paper. The evaluated model receives a research proposal (hypothesis and experiment design) with results removed and returns, in JSON, a high or low soundness label, a confidence from 1 to 5, and a step-by-step justification. Ground-truth labels come from reviewer soundness sub-scores at ICLR (the International Conference on Learning Representations): a mean of 3 or more is high, 2 or less is low, and middle scores are excluded.
    • Method: The construction pipeline takes ICLR submissions from 2022 to 2026, drops desk rejects (papers rejected before review), and keeps only papers with mean reviewer confidence of at least 3 and a standard deviation of normalized soundness scores below 0.15. Gemini 2.5 Pro extracts the abstract, related work, risk factors, hypothesis, and experiment design from each PDF in near-verbatim form while removing experimental results, direct outcome claims, and acceptance cues. Each extracted proposal is split into atomic claims that are checked against source passages found with BM25 keyword ranking, candidates with a support ratio at or above a threshold of 0.7 are kept, and 66.93% of candidates in the authors' current pool passed this check.
    • Implementation: The final benchmark holds 1,099 proposals (458 low, 641 high) across 16 ML subfields, selected after processing over 35,209 initial ICLR submissions and 137,940 reviews, and the first page of the paper points to this dataset on Hugging Face. During construction, GPT-5.4 decomposes atomic claims and Gemini-2.5-Flash verifies them. Of the 12 evaluated models, closed models run through provider APIs and open models run with vLLM (an LLM serving library) on two NVIDIA H200 GPUs, and unless otherwise noted all models use temperature 0.2 and max_tokens 8192.
    • Results: Under the standard prompt, the 12 models rated low-soundness proposals as high 74.0% of the time on average, and the two models with the highest rate, LLaMA-3.3-70B and GPT-4o, did so for 98.0% and 94.5%. Switching to the tested strict prompt that defaults to low cut this mean rate from 74.0% to 19.9%, but mean high-soundness recall also dropped from 91.8% to 36.1% and mean Macro F1 (the unweighted mean of the two class F1 scores) fell from 54.9 to 49.3. On an ICLR 2026-only split evaluated with models whose training cutoffs precede the ICLR 2026 submission period, low-soundness proposals were still rated high 77.47% of the time on average, close to 73.88% for the same models on the full dataset.
    • Limits: The authors state that the labels come from reviewers who saw the full paper and results, so they are imperfect proxies for judging a results-masked proposal. The scope is ML research from ICLR only, and because the source corpus is public, training-data contamination cannot be fully controlled, a concern that the ICLR 2026-only split and the identifier-removal analysis reduce. The human audit is a preliminary check of 60 proposals, and only two prompts, standard and strict, were tested.
  24. 1288Research Agents

    AutoResearch AI: Towards AI-Powered Research Automation for Scientific Discovery

    • Problem: AI research systems now reach into long workflows that run from literature grounding through hypotheses, experiments, validation, and reporting, yet they differ in autonomy, domain, execution environment, validation mechanism, and human oversight, so they are hard to compare in one frame. The authors note that pipeline breadth can be mistaken for scientific autonomy, and that current systems are strong at search, drafting, and coding but much weaker at validation, rejection, and reproducibility.
    • Scope: This paper is a survey. Table 1 groups surveys, workflow-level systems, and open-source projects and marks the autonomy level of each system and project, and Table 2 sorts evaluation instruments by evaluative role into discovery, experimentation and verification, deep research and synthesis, and review and provenance. The official repository (Mr-Tieguigui/Autoresearch) is, per its README, a static project page that shows the survey figures, made of index.html and a static folder with styles, scripts, and figures.
    • Taxonomy: The authors define five levels from L0 (human only) to L4 (AI autonomous) by how humans and AI split control, execution, and validation, and call the human-steered L1 and L2 region Vibe Research. They split L2 into single-step execution (L2-S), interactive workflow automation (L2-I), and pipeline automation under human verification (L2-P), and use a conservative rule that places each system at the lowest level consistent with its demonstrated workflow role.
    • Workflow stages: The authors split the technical foundations into five stages, literature grounding, hypothesis formation and planning, experimentation and tool use, feedback and validation, and reporting, and sort the recurring approaches within each stage. For example, grounding has four regimes (search-centered, evidence-centered, structure-centered, literature-memory), and validation has three (execution-coupled reruns, critique-mediated validation, expert- or temporally-grounded validation).
    • Evaluation and domains: The authors propose five evaluation dimensions, novelty, validity, impact, reliability, and provenance, so that outputs are judged on scientific credibility instead of task completion alone. Across domains, they place computational and formal sciences, whose artifacts are executable and replayable, at the highest level currently observed, and medicine and clinical research and economics and social sciences in early-to-middle L2. They conclude that no domain yet gives convincing evidence of robust L3 or L4 scientific autonomy.
    • Limits: Table 1 is a selection of structurally important works rather than a full catalog, and its levels show the primary autonomy regime suggested by each work's workflow scope, execution capability, and degree of human oversight rather than a performance ranking. The authors state that novelty still has no agreed measurable definition, so current evaluations lean on weak proxies such as dissimilarity from prior papers, LLM-as-a-judge, or human expert judgment.
  25. 1286Research Agents

    AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery

    • Problem: Autonomous research agents must find the right papers to explore prior knowledge and gather evidence for claims, but the decisive evidence often sits in method details, ablation tables, figure captions, appendices, and citation chains rather than titles or abstracts. Agents have made strong progress on long-horizon web-search benchmarks such as GAIA (a benchmark for general AI assistants) and BrowseComp, but whether an agent can search a large up-to-date scientific corpus, read full papers in depth, verify fine-grained conditions, and decide when its search is complete remains largely unmeasured.
    • Input and output: This is a benchmark paper. An agent takes a natural-language query, searches and reads a corpus of over 3 million full-text arXiv papers through the search tool of the DeepXiv academic platform, and returns either the single paper that meets every condition or no paper for Deep Research, and the full set of qualifying papers for Wide Research. Deep Research is scored by exact-match accuracy and Wide Research by the IoU (intersection over union) between the predicted and gold sets.
    • Method: Deep Research items preferentially start from under-exposed, high-quality computer science (CS) papers with typically 10 to 100 citations; clues mined from the full text are fuzzified and pruned to those needed to single out the target paper, and no-answer items perturb one core constraint. A Deep item is dropped if GPT-5.4 query rewrites surface the target, if a Claude 4.6-Sonnet or Gemini-3-Flash agent solves it, or if a human finds it within 10 minutes. Wide Research builds queries from attributes shared by topic-level candidate papers, searches 31,734 more candidates with LLM-written queries to supplement missing answers, and then sets the gold answers through full-text judgments by three LLMs and expert spot checks.
    • Scale and implementation: The benchmark has 1,000 queries (600 Deep, 400 Wide) across eight CS areas. 10% of Deep queries have no answer, and Wide queries cover 3,692 gold papers in total, 9.23 per query on average. Evaluation runs a ReAct agent, which alternates reasoning and tool calls, for at most 30 turns with the DeepXiv search tool; per its README, the GitHub repository (CherYou/AutoResearchBench) holds batch inference code, DeepXiv and web search tools, Deep and Wide scoring scripts, and a decryption script, while the benchmark data sits as an obfuscated file in a Hugging Face dataset repository.
    • Results: Among models evaluated with the same ReAct agent and DeepXiv search tool on the full benchmark, the best Deep Research accuracy was 9.39% by Claude-Opus-4.6 and the best Wide Research IoU was 9.31% by Gemini-3.1-Pro-Preview. GPT-5.4 reached 7.44% with 6.1 turns on average, while DeepSeek-V3.2 used more turns, 28.8, and stayed at 4.21%. For the four models in the search-tool comparison, replacing DeepXiv with open-web search lowered average Deep accuracy from 5.42% to 3.97%.
    • Limits: The authors state that the benchmark targets computer science papers in a fixed corpus and text-based search, leaving other sciences, multimodal evidence, and evolving literature to future work. They note that exhaustive Wide answer sets may still be hard to build at the boundary of a large corpus, and a human audit of sampled out-of-gold predictions from the three top models found that 96% violated at least one query constraint. The three costly end-to-end research systems were evaluated on a random sample of 50 queries, scoring 11/50 for GPT Deep Research, 7/50 for AI-studio Gemini-3.1-Pro, and 0/50 for Alphaxiv on Deep Research.
  26. 1284Research Agents

    aiXiv: A Next-Generation Open Access Ecosystem for Scientific Discovery Generated by AI Scientists

    • Problem: LLM agents can now write research proposals and papers and even perform peer review, yet traditional journals that still rely heavily on human peer review remain reluctant to accept AI-generated research and struggle to scale with growing submissions, and preprint servers often lack rigorous quality control. The Agents4Science conference, where AI acts as both author and reviewer, has no revision or rebuttal stage, and most existing LLM review systems treat review as a one-shot process.
    • Input and output: An AI scientist or a human researcher submits a research proposal (problem statement, motivation, methodology, and planned experiments) or a full paper through API and MCP interfaces (MCP being a protocol that lets agents call external tools), and LLM review agents return structured feedback on novelty, technical soundness, clarity, feasibility, and potential impact. A revised version can be resubmitted, and a submission is published with a DOI (a persistent identifier for the paper) when at least three of the five LLMs on the review panel vote accept. This is a system paper that builds the platform and tests its review accuracy and the effect of revisions.
    • Method: In Single Review Mode, one review agent writes strengths, weaknesses, and suggestions under four criteria (methodological quality, novelty and significance, clarity and organization, and feasibility and planning), and RAG (retrieval-augmented generation) adds literature retrieved through the Semantic Scholar API. In Meta Review Mode, an Area Chair or Editor agent identifies the subfields of a submission, creates 3 to 5 domain-specific reviewer agents for each subfield, and synthesizes their reports into a decision letter. The prompt injection check extracts font size, color, character positions, and encoding from the PDF, runs rule-based scanning and LLM semantic verification, and filters out documents whose risk score exceeds a predefined threshold before they reach the review model.
    • Implementation: aiXiv runs as a public website (aixiv.science), where an API and MCP layer connects agents in author, reviewer, and meta-reviewer roles to the platform and users can like, comment on, and discuss submissions. The accuracy of the pairwise mode, which picks the better of two documents, was measured on 235 and 163 pairs of accepted and rejected papers from the DeepReview test data for ICLR (International Conference on Learning Representations) 2024 and 2025, after dropping papers with mean reviewer ratings between 5 and 6, and on 500 pairs of ICLR papers converted into proposal format. Revision effects were tested on proposals that the AI Scientist generated, 50 per topic on NanoGPT, 2dDiffusion, and Grokking, deduplicated at an 80% cosine similarity threshold (28, 27, and 29 per topic in Table 5), and on 10 papers written by the AI Scientist.
    • Results: With RAG, Claude-sonnet-4 reached pairwise accuracies of 81.70% on ICLR 2024 and 79.75% on ICLR 2025 for papers, and 77.91% on ICLR 2024 for proposals. The prompt injection check reached 84.8% accuracy on a synthetic set built by adding attacks to 35% of 105 arXiv papers, which yielded 36 adversarial papers, and 87.9% on a real-world suspicious sample set. Under five-LLM voting, the mean acceptance rate of 10 AI Scientist papers went from 10% before revision to 70% after, and proposals in all three topics went from 0% before revision to a mean of 45.2% after.
    • Limits: The authors acknowledge that AI models may still produce fluent yet factually incorrect content, that review bias can remain even with multiple models, and that simple LLM majority voting may lack objectivity. They also note that experimental validation is currently restricted to simulated environments and virtual agent interactions, which limits generalization, especially in domains that need real-world experiments or physical constraints. The effect of RAG was mixed across models, and GPT-4.1's proposal pairwise accuracy on ICLR 2024 fell from 75.05% to 69.73% with RAG.
  27. 1283Research Agents

    Neural Architecture Discovery via Autonomous Evolution

    • Problem: Whether LLM agents can complete hard long-horizon tasks that advance AI research itself remains largely unexplored. Neural Architecture Search (NAS) only selects within human-defined spaces. The authors name three obstacles in architecture design: LLMs have limited built-in architecture knowledge, a single constraint violation in a large codebase silently corrupts training, and quality can only be measured by costly pre-training from scratch and benchmark runs.
    • Input and output: This is a system paper with an implementation and experiments. ASI-ARCH takes summaries of past architectures, experimental analyses, and literature knowledge (cognition), writes the motivation and code of a new linear attention architecture in one pass, and stores the loss, benchmark scores, and analysis from actually training it as input for the next cycle.
    • Method: A proposal from the Researcher enters training only after a novelty check, in which an LLM judges it against the 5 most similar past motivations found by embedding, and a code sanity check, and when training fails from an implementation error a debugging agent fixes the code from the error log. Fitness averages the sigmoid-transformed loss and benchmark improvements over the baseline, focused on differences within 10%, with an LLM judge score. The Analyzer compares each architecture with parent and sibling nodes to estimate the effect of individual design choices, and a pool of the top 50 architectures plus literature-derived cognition entries ground the next proposal.
    • Implementation: The exploration stage trains 20M-parameter models on 1B tokens and evaluates them on 500 samples per benchmark, and the verification stage scales selected candidates to 340M parameters on 1B tokens and top candidates to 1.3B parameters on 15B tokens. The planner uses o3 and GPT-4.1, the checker and analysis use o3, training runs and debugging use GPT-4.1, and roughly 100 linear-attention papers were distilled into about 150 cognition entries searched through OpenSearch. Per its README, the official repository (GAIR-NLP/ASI-Arch) holds the pipeline, a MongoDB-based architecture database, and a cognition base built on retrieval-augmented generation (RAG), and links the list of discovered architectures to a project page.
    • Results: In the 1,773 experiments (about 10,000 GPU hours) of the 20M-scale exploration, 1,350 candidates beat the DeltaNet baseline on both quantitative and qualitative metrics, and of about 150 re-verified at 340M scale, 105 reached state-of-the-art results. At 1.3B, the best model beat DeltaNet by +0.97 points on the overall average of 14 accuracy metrics, nearly three times the +0.34-point gain of Mamba2 over DeltaNet. With the same DeepSeek-V4-Flash proposal model, a 120-attempt budget, and the 20M training setup, the successful-design rate was 37.4% for ASI-ARCH and 8.3% for OpenEvolve, an open-source evolutionary coding agent.
    • Limits: The search ran on linear attention with DeltaNet as the baseline, and the 105 state-of-the-art results are judged under the authors' controlled 340M evaluation. In the 1.3B efficiency comparison on the same implementation and hardware, the five ASI-ARCH models peak at 16.28G to 22.75G of memory, above the 11.44G of DeltaNet. At 340M, transferred rules gave +2.05 points on GLA (Gated Linear Attention) but smaller gains of +0.35 on the state-space model Mamba2 and +0.14 on HGRN2 (a gated linear recurrent model), and the authors leave family-specific adapters to future work.
  28. 1282Reasoning

    MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning

    • Problem: Open reasoning research has focused primarily on mathematics and coding, which offer abundant datasets, established benchmarks, and clear verification, and scientific reasoning remains significantly underdeveloped, particularly in the open-source community. Many science benchmarks are multiple-choice and post-training datasets often follow that format, and the authors observe that models trained on such data show inflated multiple-choice scores but struggle with computational problems, while decontamination (removing benchmark questions from training data), which typically relies on n-gram overlap or embedding similarity, is easily circumvented by minor variations in phrasing or structure. Reference answers in many science datasets are scraped from the web or generated by LLMs, which makes their factual accuracy hard to guarantee, and data distilled directly as long CoT (written-out reasoning) from DeepSeek-R1 is often prone to overthinking, which hurts training and inference efficiency.
    • Input and output: It takes textbook PDFs and public science datasets (NaturalReasoning and Nemotron-Science) and returns two supervised fine-tuning (SFT) datasets of questions with step-by-step solutions (TextbookReasoning and MegaScience), an evaluation system covering 15 benchmarks, and Llama3.1, Qwen2.5, and Qwen3 models trained on the data. This is a dataset paper with a data construction pipeline and training experiments.
    • Method: For TextbookReasoning, olmOCR (a PDF text recognition tool) converts textbooks to text, and for each 4,096-token chunk Llama3.3-70B-Instruct extracts question and answer pairs under a high-standard criterion (questions that demand multi-step reasoning, with complete solutions in the source) and a low-standard criterion (any complete question and answer), followed by MinHash deduplication (similarity threshold 0.6), refinement by DeepSeek-V3 against the source text, and filtering of defective pairs. For decontamination, BGE-large-en-v1.5 embeddings find the 5 most similar benchmark questions for each question, and Llama3.3-70B-Instruct judges whether each pair is a paraphrase, removing the question if any pair is. For MegaScience, response length, difficulty (16 responses from Qwen2.5-7B-Instruct scored 0 to 10 by Qwen2.5-32B-Instruct), and random selection are compared per dataset through SFT on Qwen2.5-7B, and DeepSeek-V3 writes new step-by-step solutions for every source except TextbookReasoning.
    • Implementation: TextbookReasoning used 12.8k books (including 6,057 in computer science and 2,305 in medicine and biology) after removing crawled books whose metadata marked them as restricted for public access and filtering out books below university level with a subject and level classification by Llama3.3-70B-Instruct, and its 945,683 extracted pairs became 651,840 after deduplication, filtering, and decontamination. MegaScience combines 436.4k from NaturalReasoning, 173.3k from Nemotron-Science, and 651.8k from TextbookReasoning into 1261.5k instances, and Qwen2.5, Qwen3, and Llama3.1 base models were fine-tuned on it with LLaMA-Factory (a fine-tuning tool) at a learning rate of 5e-6, batch size 512, and 3 epochs. Per its README, the official GitHub repository GAIR-NLP/MegaScience contains step-by-step data_process scripts from question and answer extraction on olmOCR output through decontamination, reference answer extraction, and final formatting, along with the LLaMA-Factory training configuration, while the two datasets and the trained models are on Hugging Face and the evaluation code is in a separate GitHub repository, lm-open-science-evaluation.
    • Results: In Table 6, which compares Qwen2.5-7B models trained on different open datasets with Qwen2.5-7B-Instruct, the MegaScience model was best on 7 of the 14 benchmarks used (MMLU, Massive Multitask Language Understanding, was excluded), averaged 61.01 against 58.80 for Qwen2.5-7B-Instruct, and scored 48.75 on SciBench and 40.23 on OlympicArena. With Qwen3 compared in non-thinking mode, the Qwen3-30B-A3B MegaScience model averaged 73.86 against 70.62 for the official Instruct model, and Llama3.1-8B also led 51.07 to 49.67, but for Qwen2.5-1.5B the official Instruct model was ahead by 2.99. The LLM-based decontamination found benchmark leakage in existing datasets: 19K questions in SCP-116K, 66K in NaturalReasoning, and 164K in Nemotron-Science.
    • Limits: The authors state that finding one data mixture that is optimal for every base model remains hard, and that weaker models such as the Llama series or small Qwen2.5 models trained on difficult data like MegaScience without supplemental foundational data or easier warmup training give shorter responses with more repetition. Data selection was validated with Qwen2.5-7B as a proxy model, which ties the results to that model's capacity, and in Table 7 the math average of the MegaScience models fell below the official Instruct models for Llama3.1-8B, Qwen2.5-1.5B, Qwen3-1.7B, and Qwen3-4B. Reinforcement learning (RL) is left for future work, and experiments on compressing long CoT into shorter formats were postponed because of compute limits.
  29. 1281Research Agents

    Dolphin: Moving Towards Closed-loop Auto-research through Thinking, Practice, and Feedback

    • Problem: In auto-research, judging whether AI-generated ideas actually work is hard. Most prior studies only had humans or LLMs rate idea quality, and AI-Scientist ran experiments on simple self-constructed datasets that limit comparison with existing methods in the same field. Prior idea-generation work also lacked a feedback mechanism that passes experimental results back into idea generation.
    • Input and output: This is a system paper with an implementation and experiments. DOLPHIN takes a research topic (for example, 3D classification) and reference code, reads the titles and abstracts of related papers to propose ideas, modifies the code to follow an experimental plan and runs it, and feeds ideas that improved performance into the idea-generation prompt of the next loop.
    • Method: An LLM extracts the task attributes of the topic (model inputs, outputs, and so on) and scores papers found through Semantic Scholar from 1 to 10 to keep only relevant ones. An idea is dropped as redundant when its maximum cosine similarity to the idea bank exceeds the 0.8 threshold and must pass an LLM novelty check, and when code fails, custom-code function names, lines, and code are pulled from the exception traceback to build a local code structure before debugging. Results are sorted into improvement, maintenance, and decline against the reference code, and ideas similar to earlier ones that failed to improve performance are filtered out in the next loop.
    • Implementation: Idea generation and result analysis use gpt-4o-2024-08-06, code editing and debugging use deepseek-v2.5 served through ollama, and aider calls the LLM agents. The total number of retrieved papers is 50, each loop generates 20 ideas, and debugging runs at most 5 times. Per its README and file list, the GitHub repository (InternScience/Dolphin) holds the launch_dolphin scripts, dolphin_utils, examples, and docs, and the data for the three classification tasks is downloaded from a Google Drive link.
    • Results: With two loops (40 ideas) per task, the best idea raised ModelNet40 OA (overall accuracy) from 91.0% for the authors' PointNet implementation to 93.9%, CIFAR-100 accuracy from 81.2% to 82.0%, and SST-2 accuracy from 91.0% to 92.5%, and the ModelNet40 result is comparable to the 93.8% of the human-designed GPSFormer. On the insult detection task of MLE-bench, which is built from Kaggle competitions, DOLPHIN raised code generated by AIDE (a code generation framework) from 81.0 to 84.7. In the per-loop analysis, the share of successfully executed ideas that improved performance rose from 2/7 in Loop 1 to 4/8 in Loop 3.
    • Limits: The authors state that knowledge the LLM retained from training data may leak into idea generation, and that using only titles and abstracts can miss the technical details of papers. Current LLM coding ability is too weak for project-level code, so DOLPHIN cannot yet verify complex tasks. The 3D comparison excludes pre-training and voting, and the experiments span three classification tasks (point cloud, image, sentence) and three MLE-bench tasks.
  30. 1280Research Agents

    IdeaScientist: Orchestrating Agents for Grounded Scientific Ideation

    • Problem: Existing autoresearch systems couple solution proposal with implementation, experimentation and writing, so it is hard to tell whether success comes from strong ideas or strong execution. A research proposal has no directly verifiable reward, and a single proposal-level rubric gives weak credit assignment (telling which stage earned the score) and invites reward hacking.
    • Input and output: This is an agent system for research ideation. The input is a problem definition (the task input and desired output) plus a challenge that current solutions fail to meet, and the output is a research proposal in a fixed schema covering the problem definition, key novelty, related work, method, evaluation plan, falsifiable predictions and limitations. The proposal carries no measured results and cites prior work inline by database id.
    • Method: An orchestrator keeps the research plan and calls three sub-agents: a gap finder identifies the shared limitation of closely related work, an innovator adapts a mechanism that solved a similar challenge in another field, and a report writer turns that intuition into a complete proposal. Each role is trained separately with asynchronous GRPO (Group Relative Policy Optimization, an RL method that compares rewards within each question's group of rollouts) against a reference artifact reconstructed from a human paper. The reward weights by 0.85 a rubric score gated by structural, task-compliance, citation and tool-validity checks (the mean of a reference-free quality score, a reference-grounded score and citation F1 for overlap in cited papers) and by 0.15 a rule-based compliance term.
    • Implementation: The Svalbard Idea Vault holds 4.7 million papers (2.8 million, or 60%, with full text), namely arXiv papers from all subject areas published from 1986 through June 18, 2026 plus missing papers found one hop through the references of papers collected on or after January 1, 2026, decomposed into 2.77 million (problem definition, challenge, research gap, intuition, proposal) tuples. The 14,977 machine learning methodology papers published on or after January 1, 2026 were split into 14,500 training, 200 validation and 277 test papers, and one LoRA (Low-Rank Adaptation, a fine-tuning method that trains small adapters while keeping the original weights) adapter per role was trained on Qwen3.6-27B. Per the README, the GitHub repository holds the ideation harness, the vault construction code, the RL reward and the evaluation, the vault and the train, validation and test partition sit in a Hugging Face dataset, and the ablation arms, the human study and the LaTeX table builders are left out.
    • Results: On the overall score (the mean of the novelty average and the proposal-quality average) given by a Qwen3.6-27B judge on the 277 test papers, IdeaScientist-Trained on Qwen3.6-27B reached 74.6%, 14.0% above DeepScientist (60.6%), the best open-source baseline on the same backbone, and 5.9% above the general-purpose agent Claude Code with Claude-4.8-Opus (68.7%) and 5.1% above Codex with GPT-5.4 (69.5%). Held-out validation rewards per role rose from 56.6% to 66.7% for the gap finder, from 58.8% to 69.5% for the innovator and from 78.6% to 89.1% for the report writer. The gains of IdeaScientist-Base came mainly from novelty, and its average-novelty margin over the strongest baseline on the same backbone peaked at 23.1% on Qwen3.6-27B.
    • Limits: Evaluation happens before any experiment, as an LLM-judged match to directions that human papers took after the cutoff, and on the 50 proposals used to validate the judge, two human experts agreed at a Pearson r = 0.56. In the blind pairwise human study on 50 papers, DeepScientist was the only exception to the statistically significant wins of IdeaScientist-Trained, whose more frequent wins over IdeaScientist-Base were also not significant. Training and test papers are limited to machine learning methodology papers, and a title-only probe of 5 models on 10 post-cutoff papers found no recall, but the authors list residual model-memory contamination as a limitation of the retrospective evaluation.

From research to product use

Operating capabilities, pilots, and technologies in development are identified separately.

View technology