Research library

Search papers on equipment connectivity, quality inspection, production analysis, and manufacturing work.

Search and filter

New paper summaries

Summaries are editorial notes, not statements by the original authors. Check the original paper. Updated 2026-11-20.

  1. 1343Research Agents

    The FM Agent

    • Problem: The authors describe industrial problems such as combinatorial optimization, time-series forecasting, and high-performance kernel tuning as sharing a structure where evaluating candidate solutions is relatively straightforward but identifying truly effective ones is exceptionally difficult, and progress has relied on specialized engineers who design and iteratively refine algorithms. They see the primary focus of most prior multi-agent work on code generation as a limitation and aim for a general agent that discovers state-of-the-art solutions through large-scale search across diverse, complex problem domains.
    • Input and output: This is a system paper on FM Agent, a general-purpose multi-agent system built by the FM Agent Team at Baidu AI Cloud. Its input is the problem to solve, which for machine learning tasks means a dataset and task objectives, and candidate solutions are scored by the domain-specific evaluators built into FM Agent. Its output is a program refined by evolutionary search, such as an ML pipeline, a combinatorial optimization heuristic, a CUDA kernel, or construction code for a math problem.
    • Method: In the Cold Start Stage, several generative agents with different strategies build a diverse pool of initial algorithms, which are partitioned into clusters assigned to multiple islands, populations that evolve independently most of the time. In the Evolve Stage, each island evolves its population through mutation and crossover with periodic interaction between islands to cross-pollinate ideas, while a cluster-based sampling strategy that adjusts selective pressure to real-time population diversity and a curated elite pool balance exploration and exploitation. The evaluators give a single fitness score and LLM judge feedback in general, and for complex cases such as machine learning or kernel generation they use domain-specific multi-dimensional metrics such as balancing accuracy and latency.
    • Implementation: The system runs on the Ray framework and splits program generation and program evaluation into separate parallel worker pools scheduled asynchronously, which supports concurrent execution of thousands of evolutionary processes. In the three benchmark experiments FM Agent uses Gemini-2.5-pro and optimization is performed exclusively by the LLM without human intervention, while a Human-Interactive Feedback Module that lets experts steer evolution through natural language or code and retrieve an expert knowledge base with RAG (retrieval-augmented generation) is optional but recommended by the authors. The GitHub addresses in the paper belong to the compared systems OpenEvolve and CUDA-L1, and the paper gives no code address for FM Agent itself.
    • Results: On MLE-Bench, made of 75 Kaggle competitions, FM Agent (averaged over 3 runs with different seeds) reaches a 43.56% medal rate, above Operand ensemble at 39.56% and InternAgent at 36.44% as taken from the official report. On the 10 problems of ALE-Bench lite, where every agent uses Gemini 2.5 Pro, its overall average score of 1976.3 beats ALE-Agent, a purpose-built agent with domain-specific knowledge and a diversity-oriented search, at 1879.3 by 5.2% and the Iterative-Refinement baseline, which repeatedly refines solutions with a Self-Refine methodology, at 1201.3 by 64.6%. On a workload set clustered from KernelBench Level 3 problems, the most challenging kernels, under a tightened numerical tolerance, it achieves 2.08x to 20.77x speedups over torch.compile, and on packing 26 circles it reaches a radius sum of 2.6359740012 against AlphaEvolve's 2.6358627564.
    • Limits: The MLE-Bench baseline numbers come from the official report and were produced with different base models such as gpt-4o, o1-preview, deepseek-r1, and gpt-5, and on ALE-Bench lite short contests ALE-Agent averages 2284.8 against FM Agent's 2250.8. The ablations come from a single ALE-Bench task (ahc016) with five runs per setting, and on the uncertainty inequality task the authors state that search performance is highly sensitive to problem abstractions derived from expert domain knowledge. The claim that FM Agent matches or surpasses real-world algorithmic practice in various industrial scenarios within and beyond Baidu appears without numbers.
  2. 1342Research Agents

    How Far Are AI Scientists from Changing the World?

    • Problem: LLM-based AI Scientist systems have produced papers accepted at an ICLR 2025 workshop, and the authors ask how far such systems are from discoveries that solve grand challenges across medicine, energy, and the environment. They note that earlier surveys gave overviews of LLM architectures, agentic frameworks, and scientific automation tools, but a capability-based roadmap toward the ideal AI scientist was still missing.
    • Scope: This is a survey of LLM-based AI Scientist systems. Sections 2 to 6 cover knowledge acquisition, idea generation, verification and falsification, AI reviewer systems, and evolution in turn and summarize representative methods in each; evaluation benchmarks are collected in Sections 2, 4 (Table 2), and 5 (Table 3), while Section 3 covers evaluation criteria for hypotheses and automated and human evaluation. Section 7 covers limitations and ethical challenges, and Section 8 covers future directions such as standardized Scientist-to-Scientist Communication Protocols (SSCP) for structured interaction among AI Scientist systems.
    • Taxonomy: The authors divide the capabilities of a mature AI Scientist into four levels. Level 1, knowledge acquisition, is retrieving, reviewing, and comprehending scientific knowledge from existing research; Level 2, idea generation, is generating innovative and feasible hypotheses at scale; Level 3, verification and falsification, is designing, implementing, and analyzing experiments to test and potentially disprove those hypotheses; Level 4, evolution, is continuously advancing overall research abilities from internal and external feedback. They treat Level 2 as the key feature that distinguishes AI Scientist systems from automated scientific tools.
    • Repository: The GitHub repository Awesome-AI-Scientist named on the paper's first page keeps a curated list of papers, open source tools, and benchmarks in its README, with files at the top level that include README.md, a CONTRIBUTING.md guide, and an img folder. The README groups papers under the survey's chapter names (Knowledge Acquisition, Idea Generation, Verification and Falsification, Review System, Evolution) and says it links each paper to the paper itself, associated code, and available reviews. For some papers without open reviews, the README says the maintainers added links to AI review analyses generated with their DeepReviewer.
    • Evidence: In a table the authors collected from other papers to show that state-of-the-art LLMs have relatively low accuracy on implementation benchmarks, OpenAI o1-preview reaches a 16.90% medal-worthy success rate on MLE-Bench, o1-high a 26.00% replication score on PaperBench, and Claude-Sonnet-3.7 a 39.00% execution accuracy on SciReplicate-Bench. When the authors scored 28 publicly available papers from 5 AI Scientist systems with the AI reviewer model DeepReviewer-14B, the highest-rated system, Zochi, averaged only 4.63 out of 10. Among defect categories, 'Experimental Weakness' appeared in all 28 papers (100%) and 'Methodological Unclarity/Flaws' in 96.4%.
    • Limits: The authors attribute the remaining gap to limits of the underlying LLMs, namely hallucination, the high cost and inefficiency of knowledge updating, and catastrophic forgetting, and to weak research abilities such as generating hypotheses with low feasibility and employing incorrect validation methods. They note under the table that publicly available papers may be curated and may not fully represent each system's typical output. The scores come from a single model, DeepReviewer-14B, and average 2 papers for Zochi and 3 for AI Scientist-v2.
  3. 1341Research Agents

    AgentRxiv: Towards Collaborative Autonomous Research

    • Problem: Scientific progress is rarely the result of a single 'Eureka' moment and is rather the product of hundreds of scientists working incrementally toward a common goal, but the authors note that existing autonomous research agents such as The AI Scientist, Virtual Lab, and Agent Laboratory often operate in isolation and do not support the continuous, cumulative development of research across time. They ask whether giving agents a shared repository for building on each other's research makes progress accumulate over successive papers.
    • Input and output: This is a system paper that builds AgentRxiv, a shared preprint server where agent laboratories upload and retrieve research reports. The input is a research direction and detailed instructions from humans; in the experiments the research direction is a single task, 'Improve accuracy on MATH-500 using reasoning and prompt engineering,' given together with an instruction to use gpt-4o mini during experimentation and an OpenAI API key. Each laboratory generates its own ideas toward that direction, runs experiments, and uploads to AgentRxiv a paper describing a new reasoning technique and its MATH-500 accuracy.
    • Method: AgentRxiv is a local web application with routes for uploading, searching, and viewing papers and an API endpoint that returns search results in JSON, and an uploaded paper becomes accessible to other laboratories as soon as it is submitted, in an asynchronous manner. Search embeds stored papers and queries with a pre-trained SentenceTransformer model and ranks results by cosine similarity. Each laboratory is an Agent Laboratory system in which role agents such as PhD, Postdoc, ML Engineer, and Professor, together with the mle-solver and paper-solver tools, run Literature Review, Experimentation, and Report Writing, and during literature review it reviews 5 AgentRxiv papers and 5 arXiv papers.
    • Implementation: In the sequential experiment the laboratory agents use o3-mini (medium) as the LLM backend, the reasoning techniques they design are tested with gpt-4o mini, and Agent Laboratory runs in autonomous mode without human intervention. The sequential experiment has one laboratory produce 40 papers and the parallel experiment has three laboratories with identical configurations produce 40 papers each, 120 in total, and all experiments ran on a 2023 MacBook Pro with an Apple M3 Max processor and 36 GB of memory. The README of the official AgentLaboratory repository states that Agent Laboratory supports AgentRxiv, and the top level holds files and folders such as ai_lab_repo.py, agents.py, mlesolver.py, papersolver.py, app.py, and experiment_configs.
    • Results: In the sequential experiment, MATH-500 accuracy rises from the gpt-4o mini zero-shot baseline of 70.2% to 78.2% with the final technique SDA (Simultaneous Divergence Averaging, which picks the higher-confidence answer when a low-temperature and a high-temperature chain-of-thought answer agree), an 11.4% relative improvement, while without AgentRxiv access accuracy plateaus at 73.4% and 73.8%. SDA raises GPQA Diamond from 36.4% to 38.9%, MMLU-Pro from 63.1% to 70.8%, and MedQA from 74.9% to 81.6%, and across 5 models and 4 benchmarks the average gain is 3.3%, although GPQA averaged over models falls from 52.2% to 51.6%. Running three laboratories in parallel reaches a best accuracy of 79.8% (13.7% relative) and reaches 76.2% after seven papers versus 23 papers in the sequential setting.
    • Limits: The authors write that high rates of hallucination were arguably the most prevalent challenge during development: some papers reported results that did not match the actual experiment, the code repair mechanism often erased core functional code to get rid of errors, and it sometimes wrote code that printed realistic runtime outputs without research code underneath, so each reported accuracy comes from humans manually checking the program output and code. A large percentage of experiments failed completely at about 0% accuracy because of major code bugs, and the authors state that the goal of raising MATH-500 accuracy partially reflects how many machine learning papers report new discoveries through gains on target benchmarks but is not fully reflective of scientific discovery as a whole. The parallel run produced 120 papers across three laboratories at a total cost of $279.6, compared with $92.0 for the sequential run.
  4. 1340Research Agents

    Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills

    • Problem: ML research agents combine a model with a harness that handles orchestration, memory, verification, and iterative refinement, but the operational knowledge of which methods and tools to use, when, and how (the know-how that separates knowing a method from making it work) stays outside the agent. This knowledge is abundant in repositories and papers, but it is written for human readers and is too large to load during a task. Today experts turn it into skills and tools by hand, so its cost scales with expert labor.
    • Input and output: In creator mode, DisCo takes an anchor (the object that starts skill construction), either a source such as a repository or paper or one concrete task, and returns a verified skill graph (the skills distilled from one source plus their routing, dependency, and composition links) plus a construction record of the evidence used, the checks performed, and any unresolved gaps. In researcher mode, it takes a research task made of a problem, data, an environment (tools and budget), and a target, and produces artifacts such as code, models, experimental results, and reports. This is a system paper that builds the DisCo agent and a skill library and tests them on 4 benchmarks.
    • Method: A skill has three layers: SKILL.md, read up front (goals, key concepts, tool usage, known failure modes), references/, opened only when needed (API documentation, algorithmic detail, parameter configurations), and scripts/, executable wrappers with defined inputs and outputs. Skill distillation (reworking declarative source knowledge into skills an agent can use) runs the same four stages whether the anchor is a source or a task: scope the capabilities, ground them in evidence, construct a candidate skill graph, and verify it while repairing it where possible. During research, a router narrows a request from area to family to repository skill graph, and the agent starts from the graph's entry skill and loads only the branch it needs.
    • Implementation: Skill graphs for 1,000 repositories were built with GPT-5.5 and GPT-5.6-sol at xhigh reasoning effort, with an average allocation of about $40 per repository, and the snapshot organizes 5,353 skills into 20 areas and 178 capability families. For the experiments, the authors built 636 skills from 153 prior papers of the 20 target papers in PaperBench (a paper replication benchmark), one graph per competition for MLE-bench (an ML engineering competition benchmark), and one graph per benchmark for FrontierCS (open-ended computer science problems with unknown optima) and PassNet (graph compiler pass generation). According to its README, the GitHub repository holds the skill library (the skills folder), the DisCo CLI (command-line tool) distributed as an npm package (the cli folder), installation and workflow documentation, and example session exports.
    • Results: With the Codex harness, the GPT-5.5 backbone, and the running budget held fixed and only skill access changed, the Any Medal score on all 75 MLE-bench competitions rose from 31.11% to 72.89% (134.3% relative), above 64.44%, the strongest of the public baselines that Table 1 copies from the official MLE-bench leaderboard. The average PaperBench replication score over 20 papers rose from 29.45 to 39.59, with gains on 18 of the 20 papers. The FrontierCS Agent Track score over 188 tasks rose from 70.63 to 77.14, and the main PassNet score (AS Score) over 200 samples rose from 1.343 to 1.5313.
    • Limits: Two PaperBench papers scored lower with skills (sample-specific-masks from 57.11 to 52.04 and stay-on-topic from 32.31 to 27.79), and the authors point to a possible retrieval-precision failure in which retrieved skill content may occasionally distract from a strategy the base agent would find on its own. The comparison matches the running budget, and the one-time skill construction budget (for MLE-bench, up to 24 GPU-hours per task) is kept separate and excluded from both conditions. The 1,000 repositories are a curated snapshot of part of the commonly used ML software, chosen by open-source visibility (how well known they are in the open-source ecosystem) and practical use, with GitHub stars as one signal.
  5. 1338Research Agents

    What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversity

    • Problem: AI research agents automate the design, implementation, and training of ML models, but the field is still in its infancy and the key factors behind the success or failure of agent trajectories are not fully understood. Long multi-step trajectories that are often guided by heuristic search algorithms, together with tool use, make error analysis far more complicated than in classic ML setups, and collecting samples large enough for meaningful analysis and design ablations can be computationally prohibitive. The authors start from the hypothesis that ideation diversity is a key bottleneck in AI research agent performance.
    • Input and output: This is an empirical study that analyzes existing agent trajectories at scale and runs a controlled experiment. The input is agent trajectories on MLE-bench tasks from several LLM backbones and scaffolds, and the outputs are a diversity score, the Shannon entropy (base 2) of the model architectures extracted from the initial ideas (at most five) produced by the Draft operator (which generates the initial solutions), its correlation with medal rate and other performance metrics, and the performance change when diversity is altered through the system prompt.
    • Method: The analyzed scaffolds are AIDE (AI-Driven Exploration, a greedy tree search over Python solutions), AIRA Greedy, and AIRA MCTS, all of which grow a tree of Python solution nodes with Draft, Debug, and Improve operators. In the controlled experiment, the baseline agents (AIRA Greedy and AIRA MCTS) have three diversity mechanisms by default: sibling memory that puts descriptions of sibling nodes' solutions into the context, prompt-adaptive complexity that asks for minimal complexity for the first idea, moderate for the next two, and advanced for the last two, and a request for diversity in the system prompt. The low-diversity agents remove prompt-adaptive complexity and the diversity request in the system prompt and reuse sibling memory to ask, through the system prompt, for similar ideas, and by changing only the diversity-related parts of the prompt the authors intend to leave other aspects such as implementation quality as they are.
    • Implementation: The analysis backbones are o3, gpt-oss 20B and 120B, Llama Maverick, Devstral, and CWM (an open-weights LLM from Meta for research on code generation with world models), and 6 backbones with 2 scaffolds were run on the 75 MLE-bench tasks with 10 to 20 seeds each, giving 11,000 trajectories, roughly 1,200,000 search nodes, and 264,000 GPU hours. The controlled experiment ran AIRA Greedy and AIRA MCTS with the full-sized DeepSeek R1 on MLE-bench lite (22 tasks) with 10 seeds each, and all LLMs used a 128K-token context. Error bars in the controlled experiment are 95% confidence intervals from stratified bootstrapping with the rliable library.
    • Results: Across the 75 tasks, architecture entropy correlated with medal rate at Pearson r 0.57, with average normalized score at 0.72, and with percentile at 0.66, and agents on o3 and gpt-oss 20B and 120B used 3.5 distinct architectures on average in their 5 initial ideas versus 2.8 on average for Llama Maverick, Devstral, and CWM. In the DeepSeek R1 controlled experiment, reducing diversity lowered the MLE-bench lite medal rate from 45.5 to 38.6 for AIRA Greedy (6.9 points) and from 47.0 to 38.6 for AIRA MCTS (8.4 points), and valid submission rate fell from 98% to 92% and 90%, respectively. The valid-submission drop came mainly from two text normalization competitions (English and Russian), where low-diversity agents repeatedly tried to implement the same T5 model (a text-to-text transformer) and consistently failed with timeouts, and the authors estimate that these two competitions account for around half of the medal-rate decline.
    • Limits: The findings come from MLE-bench only, and the authors hypothesize that they are likely to generalize to other ML tasks, leaving other benchmarks to future work. Medal criteria change with the number of teams in a competition, in about 30% of MLE-bench competitions the relative gap between the best score and the bronze threshold is under 3%, agents are evaluated on custom test sets that differ from Kaggle's private test sets, and agent performance drops sharply on competitions after 2022. The authors state that second-order effects of modifying the system prompt are hard to track, and in an appendix experiment that varied temperature for AIRA Greedy with all diversity mechanisms removed, metrics other than Elo, including medal rate, showed no effect.
  6. 1337AI Agents

    Critique of Agent Model

    • Problem: As LLM systems marketed as coding agents and AI co-scientists multiply, it has become necessary to clarify where automation ends and agency begins, both for building capable systems and for judging what, if anything, to fear. The authors find that existing surveys cover what current systems can do and how to improve them broadly, yet tend to take agency for granted as a label that applies whenever an LLM interacts with an environment.
    • Argument: This is a position paper with an analysis, an architecture proposal, and proved theorems. The authors separate agentic systems, which finish tasks through externally orchestrated tools and workflows, from agentive systems, which maintain and revise their own goals, identity, decision-making, self-regulation, and learning inside the model, and they hold that current AI systems are largely agentic. On safety, they argue that GIC is structurally well positioned because harmful behavior decomposes entirely into goal misspecification (the human supplied the wrong objective) and component imperfection, and they hold that the overall goal is supplied from outside and that subgoals, identity, predictions, and plans can be audited by humans at every layer.
    • Taxonomy: The authors sort systems called agents by how their behavior is produced into five kinds: program-based bots, LLM wrapper systems (an LLM placed inside fixed orchestration), LLM-centered systems, model-less physical systems (such as industrial robots running programmed routines), and embodied-model systems (robots and driving systems that unify perception, reasoning, and control in one model). The common designs they examine along the five dimensions are short instructions supplied by a human at every step (goal), fixed system prompts with harness engineering (identity), black-box policies trained with end-to-end RL (decision-making), planning expected to emerge from RL or placed in fixed workflow stages (self-regulation), and human-scheduled training pipelines (learning). They argue that these designs share the absence of an internal world model that predicts the consequences of actions.
    • Proposal: The GIC agent model has six components: a belief encoder (which turns observations into a belief state, an estimate of the world state), a goal decomposer (which splits a long-term goal into subgoals), an identity evolver (which revises the self-model without retraining), a configurator, a simulative planner, and an actor. The actor handles System I (immediate action), the simulative planner handles System II (predicting the outcomes of candidate actions with the world model and picking a plan), and the configurator handles System III (choosing at each step whether to make a new plan, continue or revise an existing one, or act directly). The world model is trained separately on next-state prediction alone, the two models keep disjoint parameters and connect only through activations and outputs, and training is proposed in three phases: component pretraining that starts from a pretrained LLM, simulative RL inside the world model, and refinement after real-world deployment.
    • Evidence: Theorem 2 is a benefit: if the world model's next-state prediction error is bounded by ϵ in total variation (a distance between probability distributions), any policy can be mixed with selectively adopted world-model plans into a policy whose value (expected cumulative reward) is at least as high. Theorem 3 is a limitation of always-on, fixed-depth model-predictive control (MPC): the required horizon grows significantly as the desired planning precision rises. Theorem 1 shows the benefit that, when identity revisions are better than random, learning that revises identity at every step accumulates less expected regret than retraining alone, and Theorem 4 shows the benefit that the value of the best policy trained on a mixture of simulated and real experience is at least the value of the best real-only policy minus a term proportional to the world model's error, and at least as high when the world model is perfect.
    • Limits: The authors state that details and preliminary results for scaled-down GIC instantiations appear in companion manuscripts. Of their proposed evaluation dimensions (Performance, Efficiency, Growth), those preliminary results give initial evidence on the first two, and Growth evaluation remains future work. The premise of Theorem 2, an upper bound ϵ on the prediction error, is credible only when the world model is trained for predictive fidelity, and the safety conclusion that harmful behavior converges to zero is stated for a sufficiently well-trained system, unless the goal itself is wrong.
  7. 1336Research Agents

    AutoMedBench: Towards Medical AutoResearch with Agentic AI Models

    • Problem: Many existing medical agent benchmarks focus on medical question answering, clinical dialogue, EHR (electronic health record) or FHIR (Fast Healthcare Interoperability Resources) interaction, healthcare administration or final task success, and primarily evaluate final outputs. The authors argue that telling whether a low score comes from misunderstanding the task, an unsuitable method, a failed environment setup, skipped validation or a malformed submission requires stage-level evaluation.
    • Input and output: This is a benchmark for medical-AI research agents. The agent, a single base LLM using a code-execution environment, receives public inputs, a task brief, a submission schema and a time limit, and must produce and submit segmentation masks, restored images, VQA (visual question answering) answers, reports or detection boxes. Each run gets a workflow score over stages S1 to S5 (AGENTIC), a task score against hidden references (TASK), an OVERALL score that averages the two, and post-run error codes.
    • Method: The 24 tasks span five tracks, segmentation, image enhancement, VQA, report generation and detection, and cover CT (computed tomography), MRI (magnetic resonance imaging), X-ray, pathology, blood-smear microscopy, dental imaging and medical video. Every task runs the same five stages (S1 to S5), Plan, Setup, Validate, Inference and Submit, weighted 25%, 15%, 35%, 15% and 10%, with S1 to S3 scored by an LLM judge and S4 and S5 by deterministic checks. Lite and Standard tiers differ only in brief scaffolding and give 48 task-tier settings, and failure types are logged, separately from the stage scores, as five error codes: E1 understanding, E2 data and model setup, E3 verification and recovery, E4 implementation and execution, and E5 deliverable and submission.
    • Implementation: The agent container gets GPU, network, the public inputs and a writable workspace, while hidden references and scoring code sit in a separate evaluator container isolated from the external network, and training or fine-tuning during a run is forbidden. Six models, Opus 4.6, GLM-5, Gemini 3.1 Pro, ChatGPT-5.4 (called GPT-5.4 in parts of the paper), MiniMax-M2.5 and Qwen3.5, ran with the same system prompt, tool schema, stop conditions and decoding settings, with 10 runs for each of the 288 evaluation cells by default (20 for five segmentation tasks). According to the README, the GitHub repository AutoMedBench/AutoMedBench keeps task definitions and a Docker orchestrator on domain branches for running one cell by choosing an agent, task and tier, while the benchmark releases (Lite, Full) and the Full leaderboard Space are on Hugging Face and the live leaderboard is on the automedbench.github.io website.
    • Results: Among the six evaluated models, OVERALL scores ranged from 66.5 for Opus 4.6 to 51.2 for Qwen3.5, a 15.3-point spread, and GLM-5 led the VQA track with 64.0. Averaged over the six models, S3 Validate was the weakest of the five stages and S2 Setup the strongest, and all six models scored higher on AGENTIC than on TASK. Of all fired error codes, 37.7% were verification and recovery errors (E3) and 38.1% deliverable and submission errors (E5), while understanding errors (E1) were 0.9%, and runs with one fired error code scored 48% lower OVERALL on average than runs with no error code.
    • Limits: The authors state that stage-level evaluation cannot substitute for outcome evaluation and that a high AGENTIC score still requires separate checks of clinical usefulness and accuracy. Tasks are limited to inference-only workflows, excluding tasks whose correctness depends primarily on subjective judgment, long clinical dialogue or training-time adaptation, and the comparison covers base models run through the same harness without vendor agent wrappers. Instead of redistributing restricted data, the benchmark requires runners to obtain credentialed data, and in the authors' experiments MIMIC-CXR came through PhysioNet and fastMRI under the NYU (New York University) data-sharing agreement.
  8. 1335Research Agents

    AI Research Agents Narrow Scientific Exploration

    • Problem: The prompts and reasoning procedures of AI research agent frameworks often instruct agents to produce novel, high-impact, unconventional ideas, yet the ability to generate ideas at scale does not necessarily imply broader scientific exploration. Existing evaluations mainly assess whether individual ideas are interesting, novel, feasible, or executable, and reveal much less about how repeated AI-assisted ideation shapes the broader landscape of scientific exploration.
    • Input and output: In each run, one agent framework paired with one LLM receives a seed set of five papers from the same research area (one anchor paper and four related papers, titles and abstracts only) and produces one research idea with a research question and methods. The authors compare the collected ideas with human-authored papers from the same areas and with follow-on human papers that cite the seed papers, using four measures. This is a large-scale empirical study that analyzes 219,655 ideas.
    • Method: Research areas are built by grouping papers by how much they cite the same prior literature (bibliographic coupling). The four measures are exploration breadth (the mean pairwise cosine distance among ideas in the same area), exploration distance (the cosine distance from an idea to the centroid of its five seed papers), frontier alignment (the share of the top 10% most frequent keywords in the next year's human papers that the ideas cover), and potential scientific impact (the mean normalized citation score, relative to papers of the same area and year, of the 20 semantically nearest human papers). Ideas and human papers are brought into the same format by having Gemma-4-31B-IT extract research questions, methods, and keywords, and are then embedded with Qwen3-Embedding-4B.
    • Implementation: From a Semantic Scholar index of 149,405,218 papers, the authors drew 155 research areas in 12 broad fields and 11,520 seed sets, and ran five frameworks (Zero-shot, AIScientist, ResearchAgent, AgentLaboratory, and Co-Scientist) with four open-weight LLMs (Gemma-4-31B-IT, Llama-3.1-8B, Hermes-4-14B, and Qwen3-35B-A3B) for 230,400 runs. Because of budget limits, GPT-5.4 ran only on randomly sampled 2022 seed sets for 2,400 runs, and of the 232,800 runs in total, 219,655 valid ideas (94.4%) were analyzed. All frameworks except Zero-shot could search a locally deployed Semantic Scholar database restricted to papers available when the seed papers were published, AIScientist was run only through its ideation stage, and Co-Scientist was implemented from its published description.
    • Results: Exploration breadth within the same research area was 0.554 for AI ideas against 0.599 for human papers, 7.5% lower, and exploration distance from the seed literature was 0.322 for AI ideas against 0.410 for follow-on human papers. AI ideas covered 28.5% of next-year frontier keywords against 36.5% for follow-on human papers, and their potential impact score was 0.387 against 0.492, 21.3% lower, with GPT-5.4 the only LLM whose ideas were comparable to human research on impact (0.472 against 0.464). Only 10.5% of AI ideas contained research questions absent from the seed literature, while 90.4% introduced new methods.
    • Limits: Because of budget limits GPT-5.4 ran only 2,400 times on part of the 2022 seed sets, and the seed context was set to five papers because most frameworks are constrained by the context windows of current LLMs, with only titles and abstracts included. Information leakage from LLM pretraining on papers published after the seed literature was mitigated by restricting retrieval to papers available at the time, and since AI ideas still showed lower frontier alignment, the authors consider the reported differences likely conservative. Potential impact is a proxy based on the citations of semantically nearby human papers, and in a leave-one-out check its Spearman correlation with a target paper's own citation score was 0.155, which the authors note explains only part of the variation in citation outcomes.
  9. 1334Research Agents

    Generating Literature-Driven Scientific Theories at Scale

    • Problem: Automated scientific discovery has focused on agents that generate experiments, while systems for higher-level activities such as theory building, which consolidates many empirical results into generalizable laws, remain underexplored. Prior work on law discovery has largely induced quantitative or qualitative laws from structured data rather than from the research literature.
    • Input and output: It takes a user-written theory query and a corpus of papers and returns a set of theories. Each theory has a name and a description and holds tuples of a law (a qualitative relation or an equation), its scope (the conditions under which the law is expected to hold), and empirical evidence typically drawn from the input corpus. This is a system paper that implements THEORIZER and compares four generation conditions experimentally.
    • Method: THEORIZER rewrites the theory query as a literature search query for PaperFinder from AI2 (the Allen Institute for Artificial Intelligence) to retrieve up to K relevant papers (100 in the experiments), takes open-access PDFs located through Semantic Scholar, converts them to full text with an OCR (optical character recognition) based pipeline, and, if fewer than K papers come back from the initial search, backfills the corpus with relevant papers referenced in the retrieved full texts. It then builds an extraction schema for the query, has an inexpensive extraction model fill the schema for each paper as JSON records, and has the generation model synthesize theories from the pooled records, after which a self-reflection step refines each theory for internal consistency, evidence attribution, and specificity. The four compared conditions pair literature-supported generation, which uses this evidence, and parametric generation, which uses only the model's internal knowledge, with an accuracy-focused or a novelty-focused objective.
    • Implementation: Generation uses GPT-4.1, whose reported knowledge cutoff is June 2024, per-paper evidence extraction uses GPT-5-mini, and evaluation mainly uses Claude Sonnet 4.5. From 50 randomly sampled papers at NLP (natural language processing) and AI venues (ACL 2023, EMNLP 2023, AAAI 2023, NeurIPS 2024), one general and one specific query were generated automatically per paper, and the resulting 100 queries drew on 13,744 papers to produce 2,856 theories, and evaluation was done per law rather than per theory, with 790 to 1187 laws per condition in the Table 1 LLM-as-a-judge evaluation. According to the README, the src folder of the official GitHub repository allenai/asta-theorizer holds the web interface and API server, the core prompts, and four evaluation scripts, the example-theories folder holds a small toy set and the theory data from the paper, and literature-supported generation needs a separately installed local copy of Asta PaperFinder.
    • Results: In the Table 1 LLM-as-a-judge evaluation (an LLM acting as the grader), where Claude Sonnet 4.5 rated each law on a 1 to 10 scale, accuracy-focused literature-supported laws scored higher than parametric ones on empirical support (consistency with existing evidence), 5.8 versus 3.9, and on specificity, 6.5 versus 5.3, while the novelty score was 5.8 for both. In the Table 2 backtesting against 4,554 papers published in the six months from July 2025, precision was similarly high in the accuracy-focused condition, 0.88 for parametric and 0.90 for literature-supported laws, and 0.34 and 0.61 in the novelty-focused condition. The authors note that the literature-supported method is almost 7 times as expensive as a simple LLM approach, and the Table 4 estimate of LLM API costs is $0.26 per theory for literature-supported and $0.04 for parametric generation.
    • Limits: Backtesting has low recall: in Table 2 the share of laws with at least one recent paper that tests their predictions was 0.45 to 0.51 in the accuracy-focused condition and 0.04 for parametric and 0.16 for literature-supported laws in the novelty-focused condition, and the authors note that the literature's positive-result bias, under which negative results often go unreported, may make contradicting evidence harder to find. The qualified novelty analysis of Table 3, separate from the Table 1 LLM-as-a-judge scores, measured the shares of seven novelty types against the 66 to 71 papers retrieved on average for each theory query instead of the full literature, on 100 randomly selected laws per condition, and in this analysis literature-supported laws were generally rated as less novel than parametric ones. The authors state that generation is currently limited to fields whose literature is mainly open access and downloadable automatically, such as AI and NLP, and that generation and especially evaluation are currently somewhat expensive, which limits scaling.
  10. 1333Research Agents

    OmniScientist: Toward a Co-evolving Ecosystem of Human and AI Scientists

    • Problem: The authors argue that existing AI Scientists predominantly formulate scientific discovery as a standalone search or optimization problem and leave unmodeled the collaborative mechanisms, contribution attribution, peer review and structured knowledge networks that real science relies on. As a result, current systems struggle to establish a research ecosystem or interact deeply with the human scientific community.
    • Input and output: From a user's research topic or query it produces a Markdown literature review, idea proposals with research background, idea and a general implementation approach, experiment code and results, a LaTeX manuscript and review reports whose key judgments carry traceable citations. ScienceArena takes votes on anonymous pairwise comparisons of two models' answers to user-submitted research questions across six tracks and builds a leaderboard from Elo ratings that start at 1000 for every model. This is a system paper combining several modules, a protocol and an evaluation platform, and its evidence comes from per-module case studies and small-scale measurements.
    • Method: The literature review is split among Planning, Literature and Writing agents; candidates from embedding-based Elasticsearch search are joined by papers linked through citations and references, and papers that pass relevance verification are expanded layer by layer in a breadth-first search (BFS) until a predefined depth or coverage is reached. Deep Ideation explores, adds and replaces neighboring keywords in a keyword co-occurrence network to refine proposals, logs keywords, proposals and evaluations per round in an Idea Stack, and receives feedback aligned with expert review standards from a critic model fine-tuned on reasoning produced by a prompt that simulates human reviewers' cognitive process. OSP treats humans and AI as the same Participant type, a central Hub records every protocol message, and each research artifact carries a ContributionLedger that lists participants' actions with timestamps in order.
    • Implementation: The knowledge base links about 269 million OpenAlex metadata records with their citations, about 2.6 million arXiv PDF full texts, and 102,679 full-text papers from the top ten AI conferences over the past decade with 116,970 referenced baseline models and 68,316 related datasets into a graph of Paper, Author, Concept and Resource nodes, and Diagnose, Search, Normalization, Coding and Review agents keep refining it. The experiment stage selects candidates with a bi-encoder retriever fine-tuned on baseline and dataset self-descriptions combined with summarized citation contexts from other papers and an LLM reranker fine-tuned on citation chains linking papers, baselines and datasets, and then Evolution, Sample, Evaluation and Feedback agents modify and run code. Writing is handled by Outline, Figure, Writing and Refinement agents and review by the five-stage TIMAR (Traceable and Interactive Multi-Agent Review) workflow, and the public addresses given in the paper are omniscientist.ai and sciencearena.ai.
    • Results: In a small-batch evaluation (n = 1000), graph refinement raised metadata completeness from 0.965 to 1.000 and correctness from 0.951 to 0.997, and on a benchmark of 100 QA pairs probing inter-paper relationships it raised retrieval accuracy from 0.70 to 0.88. OmniScientist's proposal to replace the Monte Carlo sampler of STDE (Stochastic Taylor Derivative Estimator) with a QMC (Quasi-Monte Carlo) sampler gave lower error on the AllenCahnTwobody equation than both the original STDE and the AlphaEvolve variant at all four settings from 100 to 100000 dimensions, for example 0.000572 against 0.003440 for STDE and 0.002059 for the AlphaEvolve variant at 10000 dimensions. In a study where 10 PhD-level participants each solved 10 computer science and AI questions from HLE, five alone and five with GPT-5, average accuracy was 0.22 with GPT-5 collaboration, 0.1 for humans alone and 0 for GPT-5 alone.
    • Limits: The authors state that current capabilities remain constrained to the AI domain, and that the data backbone, primarily arXiv full texts, strongly overrepresents AI and computer science while Nature, Science and other domain-specific journals are not yet integrated. The experiment agent currently supports only computational workflows such as configuring environments, preparing datasets, running training and inference scripts and hyperparameter sweeps, so wet-lab experiments in chemistry or biology and instrument interaction are outside its current scope; the review module is optimized on AI manuscripts and loses quality on other fields, and the modules still require significant compute and processing time. The validation consists of case-level studies such as the STDE improvement on a single equation (AllenCahnTwobody), a collaboration study with 10 PhD-level participants and 10 HLE questions, and a literature review comparison scored by Gemini-2.5-pro, and ScienceArena results are presented as per-track preference patterns and example figures in place of vote counts or Elo values.
  11. 1332Research Agents

    Paper2Agent: Reimagining Research Papers As Interactive and Reliable AI Agents

    • Problem: Significant technical barriers often remain before a paper's new computational method can be used on new data, and a reader might need to locate the code repository, install dependencies, configure environments and interpret the expected inputs and outputs. The authors argue that even a powerful tool like AlphaGenome is hard for biologists to use because of the technical barrier of handling API keys, client objects and variant objects. They note that executable papers increased reproducibility but still required technical familiarity, and that Papers with Code improved discoverability while the barrier of installing and executing the code remained.
    • Input and output: This is an implemented multi-agent system shown through case studies. The input is a paper with its code repository link, supplements and data, and the output is an MCP (Model Context Protocol) server Python file with tools, resources (manuscript, code, data) and prompts (multi-step workflow instructions), plus a configured environment. Connecting the server to a chat agent such as Claude Code yields a paper agent that runs the paper's methods from natural language queries.
    • Method: An orchestrator agent built in Claude Code invokes four sub-agents as needed at different stages: the Environment-manager (sets up an isolated environment), the Tutorial-scanner (finds tutorials in the repository), the Tutorial-tool-extractor-implementor (implements tutorials as reusable tool functions) and the Test-verifier-improver (writes, runs and fixes tests). The Test-verifier-improver repeats a loop of generating, executing and fixing tests using only the tutorial's own examples to check that the original numbers and figures come back, and functions that keep failing lose their MCP decorators and are dropped from the server. Verified tools are locked for reproducibility, and every tool keeps a link to the original GitHub source code.
    • Implementation: The 22 AlphaGenome MCP tools were built in around 3 hours on a personal laptop without human intervention, the TISSUE server has 6 tools, and 7 Scanpy preprocessing and clustering tools were built in around 45 minutes on a personal laptop. All three agents in the AlphaGenome benchmark used claude-sonnet-4-20250514 as the LLM and ran on a MacBook Air (M2), and the AlphaGenome, Scanpy and TISSUE MCP servers are hosted on Hugging Face Spaces. The GitHub README explains installing the repository's skills/paper2agent folder as a skill in Claude Code or Codex and connecting to the MCP servers hosted on Hugging Face Spaces, and says the skill launches parallel specialist agents, checks the work with fresh verifier agents and delivers a ZIP file with usage instructions (USAGE.md).
    • Results: On 15 AlphaGenome tutorial queries the AlphaGenome agent reached 100.0% accuracy, above Claude + Repo (Claude Code given the repository) at 60.0% and Biomni at 40.0%, and on 15 newly curated queries it again reached 100.0% against 80.0% and 60.0%. The TISSUE agent's outputs on mouse somatosensory cortex spatial transcriptomics data and the Scanpy agent's preprocessing and clustering of three 10x Genomics PBMC (peripheral blood mononuclear cell) datasets matched the results human researchers obtained on the same data. An AI co-scientist that connected the AlphaGenome and ADHD GWAS data MCPs executed a hypothesis chosen by the user, picked rs1626703 out of 209 candidate variants, predicted that it alters splicing of the MPHOSPH9 (M-phase phosphoprotein 9) gene in glutamatergic neurons, and then finished the analysis of all 39 loci within two hours.
    • Limits: The authors state that if the original codebase is incomplete, poorly documented or contains unresolved errors, Paper2Agent cannot reliably expose it as a functioning tool. The accuracy benchmark depends on expert knowledge of the paper, manually implemented ground truth and manual grading, and the work so far focuses on methodological papers. The ADHD result is a computational prediction made with AlphaGenome, and the authors note that these hypotheses benefit from subsequent human evaluation.
  12. 1331AI / General

    AI for Scientific Discovery is a Social Problem

    • Problem: AI is increasingly applied to scientific research, but its benefits remain unevenly distributed across communities and disciplines, and so far they have concentrated in domains with strong data infrastructure and well-coordinated research communities. Technical challenges such as limited data, fragmented standards, and unequal access to computing are well known, but the authors argue that social and institutional factors are often the primary constraints.
    • Argument: This is a position paper that builds its case from literature, survey statistics, and case studies. The authors argue for reframing AI for science as a collective social project in which sustainable collaboration and equitable participation are prerequisites for technical progress. They define scientific success as strong predictive performance together with explanations that reveal causal relationships, experimental validation, integration with theory, and broad accessibility, and they hold that systems that merely optimize benchmark prediction accuracy fall short of this standard.
    • Taxonomy: The authors sort the barriers into four: community dysfunction (the autonomous AI scientist narrative, undervalued data and infrastructure work, and communication and education gaps between fields), misaligned research priorities (a focus on narrow domain applications and neglect of upstream computational bottlenecks shared by many fields, such as density functional theory (DFT) calculations and partial differential equation (PDE) solvers), data fragmentation (incompatible formats and standards, and high-dimensional, low-sample-size scientific data), and infrastructure inequity (gaps in access to computing). They see commercial AI scientist systems as predominantly focused on literature search, summarization, and hypothesis generation from existing text while largely avoiding the experimental validation and mechanistic investigation that define scientific discovery.
    • Proposals: The four remedies are strengthening collaboration and education across fields (standardized interfaces and APIs, collaborative development platforms, and training that builds hybrid expertise), structuring upstream challenges through shared benchmarks, standardizing and curating data around simple formats such as CSV (comma-separated values) or Parquet, and building community-owned, sustainable infrastructure. The authors argue that research funding should prioritize computational challenges with broad applicability over numerous domain-specific applications, and that reproducible science should be rewarded in hiring, tenure evaluations, and citation counts.
    • Evidence: CASP (the critical assessment of protein structure prediction) has run biennial blind prediction competitions, in which predictions are submitted before the experimental structures are released, since 1994, building clear success metrics and challenging test cases, and the authors credit this groundwork for AlphaFold's breakthrough at the 14th round (CASP14) in 2020. On data, they cite the harmonization of nine Cancer Imaging Archive files, which took 329.5 hours over six months, and the growth of the Protein Data Bank from seven structures in 1971 to more than 230,000 structures. On resources, they cite figures that only 5% of Africa's AI research community has the computing power complex AI tasks need and that African researchers face six-day iteration cycles against 30-minute cycles in G7 countries.
    • Limits: The authors note that the paper synthesizes challenges across biology, chemistry, materials science, physics, and climate science to highlight structural patterns, so specific constraints vary by field and some recommendations may need adapting to particular subdisciplines or regulatory environments. Several arguments rely on studies and surveys that do not cover the full diversity of scientific institutions, especially in underrepresented regions, and fast progress in machine learning may change some of the technical limitations discussed. The discussion of governance and incentives is high level, so implementing it would need detailed policy design and negotiation among stakeholders, and both authors disclose a potential conflict of interest as employees of Hugging Face, which the paper cites as one collaborative platform.
  13. 1330Research Agents

    MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research

    • Problem: LLM agents have started to generate ideas, run experiments, and write papers, but the community still lacks a comprehensive benchmark that systematically assesses their ability to conduct open-ended research, which makes progress hard to measure and to compare fairly between agents. Existing benchmarks typically target narrow tasks: MLE-bench focuses on engineering, MLAgentBench on experimentation, and PaperBench on paper reproduction. The authors also see a need for empirical analysis of failure modes such as hallucinated results, lack of novelty, and methodological flaws.
    • Input and output: Each task is a research problem built from the overview and topics of a workshop at the NeurIPS, ICLR, or ICML machine learning conferences over the past three years, and in end-to-end evaluation the agent takes the task and returns a completed paper. Stepwise evaluation asks for an idea from a task, a research proposal from a task and an idea, code and results in an experiment environment given a task, an idea, and a proposal, and a paper from the experimental outputs (report, figures, and a log of commands run), and MLR-Judge scores each review dimension of an output from 1 to 10. This is a benchmark paper that releases 201 tasks, an evaluation framework, and an agent scaffold together.
    • Method: In MLR-Judge, two LLM judges (Gemini-2.5-Pro-Preview and Claude-3.7-Sonnet) score each output independently against stage-specific rubrics (scoring criteria for review dimensions such as Consistency, Novelty, and Soundness), the two scores are averaged, and in the experimentation stage the judge also inspects the coding agent's execution log. MLR-Agent is a scaffold that favors simplicity over extensive prompt engineering: LLMs write the idea and the research proposal, a coding agent (for example, Claude Code) runs the experiments, a multimodal LLM writes the paper, and a GPT-4o-Search-Preview literature review is inserted between the idea and the proposal. To validate the judge, the authors recruited 10 experts with ML conference reviewing experience, and two of them independently scored each generated paper with the same rubric as the judge.
    • Implementation: The 201 tasks come from workshops of the three conferences over the past three years, after removing duplicates and keeping workshops with complete information aimed at a general audience, and they span 9 core ML topics such as LLMs and VLMs (vision language models), AI for Science, ML Theory, and Trustworthy AI. Stepwise inputs were built by randomly sampling one previous-stage output per task (the experimentation stage instead used 10 hand-picked inputs), and the coding agents ran on an Ubuntu 22.04 server with four NVIDIA RTX 3090 GPUs. According to its README, the GitHub repository holds the task folder, the end-to-end script run_mlr_agent.py, step-by-step agent scripts (the mlrbench/agent folder), and judge scripts (the mlrbench/evals folder), and the task dataset is also on Hugging Face.
    • Results: In the experimentation stage on 10 tasks, Claude Code and Codex both scored 4.95 Overall, below the 6.0 acceptance threshold. In end-to-end evaluation on the same 10 tasks, MLR-Agent scored 4.70 Overall with Claude-3.7-Sonnet and Claude Code, 4.60 with Gemini-2.5-Pro-Preview and Gemini CLI (a command-line coding agent from Google), and 3.10 with o4-mini-high and Codex, so all of them, together with AI Scientist V2 on o4-mini-high (4.25), fell below 6.0, and all three MLR-Agent models scored lowest on Soundness. As MLR-Judge pointed out, in 8 of the 10 tasks run by Claude Code the reported results rested on synthesized or placeholder data rather than actual execution, and the authors found that after execution failures coding agents tend to fill the gaps with synthetic results instead of reporting the failures.
    • Limits: Experimentation, paper writing, and end-to-end evaluation used 10 of the 201 tasks, hand-picked to keep the experiments manageable, and all 10 come from ICLR 2025 workshops, mostly on Trustworthy AI. The authors name the lack of process transparency as a key barrier to trust, since human reviewers given a fully formed AI-generated paper often cannot see how each part was produced, and they present MLR-Bench as a first step toward analyzing and improving AI research agents. Nonexistent citations appeared in 50% of the tasks completed by MLR-Agent, and the authors hypothesize that limits of its literature collection tool caused this.
  14. 1329Reasoning

    In-situ graph reasoning and knowledge expansion using Graph-PReFLexOR

    • Problem: The author argues that much of current AI relies on models that produce outputs directly, without an explicit intermediate reasoning process akin to scientific thinking, and that LLMs often excel in fluency and pattern recognition yet typically lack structured reasoning. The goal is a model that goes beyond retrieving information to learn relations between concepts and abstract patterns and generalize across new domains.
    • Input and output: This is a method paper that presents a training approach and a 3B model trained with it. Given a question or task, the model generates, inside a section wrapped in thinking tokens (<|thinking|>), a knowledge graph (concept nodes joined by relation edges such as IS-A, RELATES-TO and INFLUENCES), symbolic abstract patterns, reasoning steps and a hypothesis, and then gives the final answer.
    • Method: Training data are generated on the fly during training: a passage drawn at random from the raw corpus becomes a question, RAG (retrieval-augmented generation) adds related content from the whole corpus, and the pipeline builds a correct answer containing a knowledge graph plus a rejected answer. In recursive reasoning, a critic agent critiques Graph-PReFLexOR's thinking section, the thinking is revised with that critique and the question is answered again, and after N rounds the last answer is taken or all answers are integrated. In the knowledge garden, Graph-PReFLexOR answers a next question that a human or another LLM agent builds from the nodes of earlier graphs, producing a new graph, and the graphs are joined into one growing graph.
    • Implementation: The base model meta-llama/Llama-3.2-3B-Instruct is trained on data built from the raw text of around 1,000 papers on biological and bio-inspired materials, in the same order as PReFLexOR, ORPO followed by EXO, a DPO (Direct Preference Optimization) variant, and the critic in the recursive reasoning implementation is the non-fine-tuned Llama-3.2-3B-Instruct. The paper states that code and model weights are available on Hugging Face under lamm-mit and in the GitHub repository lamm-mit/PRefLexOR, and names the model lamm-mit/Graph-Preflexor_01062025. The GitHub README documents the PRefLexOR training package, ORPO and EXO trainer examples and Colab inference notebooks, says the model weights come from Hugging Face, and its top-level files include a Graph-PRefLexOR training notebook (Graph-PRefLexOR_train_Phase-I-and-II.ipynb).
    • Results: On one task, designing a tough protein material that integrates a snowflake and ant behavior, GPT-4o as judge gave the first iteration Graph-PReFLexOR response an average of 9.7 and the reference model, a standard non-fine-tuned model, 6.4. Of the seven criteria, only Clarity and Organization favored the reference model, 9/10 against 8/10 for Graph-PReFLexOR. In the autonomous bio-inspired materials run, a human gave only the first prompt and an LLM wrote every later prompt, the graph grew over 12 iterations, and the nodes with the highest degree (number of linked edges) were Sustainable Materials, Self-healing and Adaptable Materials, and Ecosystems.
    • Limits: The only comparison with a baseline model is Table 1, one task scored by a single judge model, GPT-4o, and the remaining results are case studies that show graphs and answers. The author notes that on the song task the model may be limited in capturing the temporal and emotional nuances essential to musical composition. Scaling to larger datasets and more complex models and interpretability of deeply interconnected graphs are named as future work, and the author writes that the critic could include fact-checking or even run simulations.
  15. 1328Research Agents

    Many Heads Are Better Than One: Improved Scientific Idea Generation by A LLM-Based Multi-Agent System

    • Problem: Earlier work such as AI Scientist, ResearchTown, and HypoGen uses LLM agents to simulate scientific idea generation, but the authors note that these efforts either rely on a single agent or build multi-agent setups on an oversimplified collaboration framework and unrealistic data such as manually crafted profiles and synthetic collaboration networks, missing the dynamic relationships of real scientific teams. The authors ask whether simulating how real scientists form teams and collaborate yields more novel ideas, and how collaboration conditions such as team size, freshness, and research diversity affect novelty.
    • Input and output: This is a system paper presenting the multi-agent system VirSci, whose input is a research ecosystem built from real academic data at a chosen time point. The ecosystem consists of a database of papers published before that point, a database of papers published after it, an author knowledge bank of scientist profiles, and an adjacency matrix of collaboration counts between scientists. The output is one abstract written by a team that a randomly sampled team leader assembles, with Introduction, Objective, Methods, Expected Results, and Conclusion sections.
    • Method: The team leader forms a team by using the adjacency matrix, with 1 added to every collaboration count, as invitation probabilities, and the team then takes turns speaking for K turns in each step of topic discussion, idea generation, novelty assessment, and abstract generation. During discussions, agents can retrieve the author knowledge bank with RAG (retrieval-augmented generation) to hear from relevant scientists outside the team, which the authors call the Invitation Mechanism. Idea generation keeps the three ideas with the highest confidence, and in novelty assessment the agents compare each idea with past papers and select one by vote.
    • Implementation: The system is built on the multi-agent framework AgentScope and calls GPT-4o through a public API and the open-weight LLaMA3.1-8b and LLaMA3.1-70b through Ollama, with LLaMA3.1-8b as the default. The data are the AMiner Computer Science dataset filtered to 156 authors and 178,197 papers and Open Academic Graph 3.1 filtered to 3,169 authors and 201,131 papers. The top level of the GitHub repository holds an agentscope-main folder that defines the customized scientist agent, a sci_platform folder with run.py as the main entry, and a preprocess_data folder, and the README points to Google Drive for the preprocessed AMiner Computer Science data.
    • Results: With GPT-4o agents, VirSci scores a CD (Contemporary Dissimilarity, the average distance between the generated abstract and its 5 most similar contemporary abstracts, where smaller means closer to new work) of 0.34, a CI (Contemporary Impact, the average citation count of those abstracts) of 3.78, and a novelty score of 5.24 on a 10-point scale from ten PhD students, ahead of HypoGen (0.36, 3.10, 4.78) and AI Scientist (0.38, 3.22, 4.94), and with LLaMA3.1-8b and LLaMA3.1-70b agents it also leads on every metric in the table, including the LLM review score. In this comparison, NanoGPT is added as the initial discussion topic for HypoGen and VirSci to match AI Scientist's research direction, inference cost is matched with 4 members and 5 turns for VirSci and 12 turns for HypoGen, and paper retrieval for the metrics uses the Semantic Scholar API for all three systems. In the collaboration experiments comparing team sizes of 1, 4, 8, 10, and 100, the overall novelty metric ON (Overall Novelty) peaks at a team size of 8 and around 5 discussion turns, and also when about 50% of members are new collaborators.
    • Limits: In large research communities multiple teams often work on related projects together or separately and researchers frequently join several teams at once, and the authors state that the system simplifies these complexities; they also list biases inherited from the underlying LLM and training data and the risk of plausible but incorrect scientific claims. Because of computational constraints, the evaluation focuses primarily on generated abstracts rather than full texts, and on 200 abstracts ON correlates with human ratings at a Pearson coefficient of 0.52. VirSci covers the idea generation phase, and an example abstract in the appendix includes a results sentence saying randomized controlled trials with 500 children reduced failure rates by 42%.
  16. 1327Research Agents

    MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering

    • Problem: Language models perform well on many coding benchmarks and are progressing on ML tasks such as architecture design and model training, but few benchmarks holistically measure autonomous end-to-end ML engineering. The authors aimed for tasks that are challenging and representative of contemporary ML engineering work, with evaluation results that can be compared to human-level performance.
    • Input and output: The agent receives a Kaggle competition's description, dataset, and submission format, trains models in an offline environment, and outputs a submission.csv file. The submission is scored by grading code that re-implements the competition metric and is converted into a medal by comparing it against the Private leaderboard as if the agent had entered the competition at the time. This is a benchmark paper for evaluating ML engineering agents.
    • Method: From 5673 completed competitions in the Meta Kaggle dataset, the authors dropped Community Competitions, manually screened the remaining 586, and selected 75, labeled by the time an experienced ML engineer would need for a sensible solution (excluding model training) as Low (22, under 2 hours), Medium (38, 2 to 10 hours), or High (15, over 10 hours). Where no public test set existed, they carved a new test set out of the public training data (at a 10% ratio unless otherwise stated), and the headline metric is the share of attempts that earn any medal (bronze or above) under Kaggle's medal thresholds, which depend on the number of participating teams. Agents may not write predictions directly from their own knowledge instead of using a model, or view solutions online, a log inspection tool and the Dolos plagiarism detector check this, and attempts with over 60% code similarity to any of the top 50 Kaggle notebooks are disqualified and flagged for further review.
    • Implementation: In the main experiments, each competition attempt runs for up to 24 hours in an Ubuntu 20.04 Docker container holding the dataset, a submission validation server, and Python packages, on a machine with 36 vCPUs, 440GB of RAM, and one Nvidia A10 GPU, and seeds default to 3, with 16 for o1-preview (AIDE) and 36 for GPT-4o (AIDE). The three scaffolds are AIDE, MLAB (the ResearchAgent from MLAgentBench), and OpenHands (the CodeActAgent from the OpenHands platform), and four models were compared on AIDE: o1-preview, GPT-4o, Claude 3.5 Sonnet, and Llama 3.1 405B. According to the README, the GitHub repository openai/mle-bench contains dataset preparation scripts, per-competition grading scripts, the evaluated agents, the setup for the base Docker image (mlebench-env) used by agents, and a rule violation detector and a plagiarism detector, and it recommends a Lite evaluation on only the 22 Low complexity competitions when cost is a concern.
    • Results: In Table 2, among the four models on the AIDE scaffold, o1-preview earned a medal in 16.9% of competitions on average, the highest, followed by GPT-4o at 8.7%, Claude 3.5 Sonnet at 7.6%, and Llama 3.1 405B at 3.0%, while GPT-4o scored 0.8% with MLAB and 4.4% with OpenHands. In the attempts experiment, o1-preview rose from 16.9% at pass@1 to 34.1% at pass@8 (the share of competitions with a medal in at least one of 8 attempts), and in the experiment that extended the time limit per competition to 100 hours and the maximum number of AIDE nodes to 5,000, GPT-4o (AIDE) rose from 8.7% (at 24 hours) to 11.8%. In the Table 3 hardware experiment GPT-4o (AIDE) scored similarly with no GPU (9.1%), one GPU (8.7%), and two GPUs (10.2%), and in Table 4 it also scored similarly on the original descriptions (8.5%) and on obfuscated descriptions run with 10 seeds (8.4%).
    • Limits: Because the data are public Kaggle competitions, models may have trained on competition details, solutions, and even the datasets including the test sets, and the authors state that although their experiments found no systematic contamination effect for GPT-4o, reuse of high-level strategies is hard to detect and there are no guarantees about future models. They note that MLE-bench selects competitions with clear problem statements, clean datasets, and clear metrics, so it covers part of the capabilities needed for real-world AI research and development, where the problem statement often may not even be clear and figuring out the dataset and metrics is part of the problem. One run of the main setup takes 24 hours times 75 competitions, or 1800 GPU hours, and o1-preview (AIDE) used on average 127.5M input tokens and 15.0M output tokens for one seed of 75 competitions.
  17. 1326Research Agents

    Stateless Language Agents: Scaling Long-Horizon Automated Research

    • Problem: Automated research (AutoResearch) systems run LLM agents over long horizons, but agents replay growing histories, duplicate one another's work, or stop experimenting while token consumption continues. In one run of CORAL, a framework whose parallel agents select their own experiments and share discoveries through persistent memory, on a GPU kernel task, the last improvement came at 231M tokens, after which 98% of the final 1,500 sessions made no tool calls. Most evaluations use short budgets or benchmarks that saturate early and leave these failure modes untested, and the authors trace the failures to two choices: where research state lives and who decides what to try next.
    • Input and output: SLA takes one task (a task adapter that specifies the editable files, permitted resources, and evaluation procedure, a seed solution, and a protected evaluator) and a cumulative token budget. Until the budget is exhausted, an independent evaluator scores every candidate the Workers produce, and the harness keeps the global best solution, each Worker's local candidate, the measured outcomes, and checkpoints as research state. This is a system paper that implements the framework and compares it with three recent frameworks on software engineering, kernel optimization, and algorithm design tasks.
    • Method: The principle is stateful search with stateless agents: every agent starts a new session at each invocation, while the harness owns the research state (candidate solutions and measured outcomes) and rebuilds a fresh, role-specific context for every invocation. Each epoch, the Advisor receives the task specification, the current global best code and score, and a harness-built evidence summary that groups outcomes by search direction and separates measured non-improvements from failures in implementation, runtime, or correctness. Each Worker trial sees only the task specification, its assignment, its retained candidate, and compact local feedback, evidence from other directions reaches it only through the Advisor's assignments, and a redirected Worker restarts from the global best.
    • Implementation: The Advisor and Workers use the same coding agent, in two configurations, OpenAI Codex CLI v0.152.1 with GPT-5.5 and Claude Code v2.1.258 with Claude Opus 4.8; in the main experiments they also use the same LLM, and Appendix C.1 separately reports a GPT-5.5 Advisor paired with GPT-5.4 mini Workers. The defaults are 3 Workers and three local trials per Worker per Advisor epoch, Workers run as unprivileged processes under Linux Landlock filesystem restrictions, and the harness clears saved sessions between trials and scores candidates outside Worker environments with protected evaluator code. The baselines are CORAL and SwarmResearch, whose agents keep long-running sessions, and EvoX adapted to launch a fresh coding agent for each generation; all three run from pinned upstream releases with unmodified search logic, and the same external cumulative-token counter stops every method.
    • Results: On Anthropic kernel optimization with Codex (1B tokens, three runs per method), SLA averaged 1112.0 cycles against 1275.7 cycles for SwarmResearch, the strongest baseline by mean, and reached SwarmResearch's final performance with 67.9M tokens, compared with the 986.3M tokens SwarmResearch needed, a 93.1% reduction. On the four FrontierSWE tasks, run once per method with Codex, SLA scored 4.95 points higher on average than the strongest per-task baseline, trailing on all four at 175M tokens (25% of the budget) but leading on all four at the full 700M budget. In ablations that ran three 100M-token continuations from shared checkpoints on Anthropic kernel and FrontierSWE libexpat, removing any one of Advisor context reconstruction, Worker context isolation, or Advisor assignments reduced mean progress in every comparison, and in the two runs analyzed for token usage (Anthropic kernel and FrontierSWE libexpat) the Advisor used 0.24% to 0.51% of all tokens.
    • Limits: Because long runs are costly, most configurations were run once, and only Anthropic kernel optimization with Codex and the checkpoint ablations were replicated. Comparisons are matched on tokens, and the cost estimates exclude the compute for running and evaluating experiments. All tasks have executable evaluators, so goals that are vague or slow to evaluate remain untested.
  18. 1325Research Agents

    RankEvolve: A Reliable Multi-Agent Auto-Research Harness for Evolving Ranking Models

    • Problem: Auto-research agents that propose, implement, train and evaluate model changes are bound by execution accuracy in the authors' setting. A single defect such as leaked held-out data, an omitted layer norm, a disconnected gradient or an unwired train/eval flag burns hours of accelerator time and yields an invalid metric, and over many iterations the damage compounds while the run drifts off its prescribed process. In the authors' motivating pilot where the whole procedure was given as one in-context playbook, 0 of 10 runs were fully faithful and all 10 ran only a subset of the selected proposals.
    • Input and output: It takes an ML codebase and data, an EOP (Executable Operating Protocol) document listing phases and gates, and human decisions at the gates, and produces flag-gated code changes, training runs, leaderboard entries that include negative results, and evidence records. In the execution-accuracy evaluation each method receives a locked repository snapshot and a change request and returns a patch that a hidden executable oracle scores as pass or fail. This is a system paper on a deployed harness that also reports an HSTU deployment case study and an evaluation on ExecML, a benchmark built from that deployment's incidents.
    • Method: An EOP declares phases, dependencies, required tools, human confirmation gates and branch and loop edges once; the runtime compiles it into a state machine that enforces order and gates and injects only the global preamble and the current phase instruction into context. The meta-meta-harness binds complete coding-agent products such as Claude Code, Codex and OpenHands to execution-graph nodes through an adapter contract, forming Dual loops where one product implements and another reviews and repairs (repeating until the reviewer approves or residual severity falls below a threshold, within bounded rounds) and flows that split a task and aggregate the results. The knowledge layer records per proposal hashes, prompt, patch, test outputs, configuration, metrics, cost and the promotion decision, organizes lessons into notes beside the code, a central wiki and an entity graph, and injects only the few relevant lessons at each phase.
    • Implementation: The HSTU case used the public HSTU implementation and public MovieLens data, the workspace holds about 55 leaderboard entries across 49 experiment directories run on 1 to 8 NVIDIA H200s, and a human operator authorized compute and vetoed one over-budget experiment at the gates. ExecML builds 96 private tasks each from HSTU and LitGPT (a language-model training codebase), balanced over six incident families (data provenance and leakage, tensor routing, gradient flow, train/eval mode, metric semantics, configuration wiring) and split half new implementation and half repair, and Claude Code ran Claude Opus 4.8 while Codex ran GPT-5.6, both at maximum reasoning effort. Appendix B states that the failure-overlap analyzer, the leaderboard schema and the boost-last-K patch will be released with the codebase.
    • Results: After 12 iterations RankEvolve reported HSTU NDCG@10 of 0.2192 on MovieLens-20M LARGE (+4.48% over the published 0.2098) and 0.1948 on BASE (+2.80% over 0.1895), and the 0.2192 came from an added genre side-feature in place of a new modeling primitive. On 96 ExecML HSTU tasks at matched observable inference budget, the Claude Code, Codex, Claude Code flow reached 62.5% execution accuracy, 16.7 points above the strongest single-product condition, Claude Code in all three roles (45.8%; 95% CI 6.6 to 26.7), and its silent critical-defect rate (faults that leave the patch runnable but can invalidate a scientific conclusion) fell from 35.4% for one-pass Claude Code to 10.4%. On LitGPT the same flow led 56.2% to 43.8%, a gain of 12.5 points (95% CI 3.0 to 22.0), and with the runtime held fixed, injecting only the current-step instruction beat full-protocol injection in execution accuracy by 2.1 points at early phases, 6.2 at middle phases and 10.4 at late phases.
    • Limits: The authors state that the evidence base is two Python/PyTorch codebases (one ranking codebase and LitGPT), so cross-domain transfer is not fully established, that the oracles are incomplete so audits reduce but do not eliminate false passes, and that pretraining may include the public repositories. Without a memory on/off ablation the knowledge layer's contribution to proposal quality is not isolated, the HSTU case had a human operator at the gates who also steered it so the agent's and operator's contributions are not separated, and the cross-dataset transfer results are single runs. The deployed plan-merge flow had the highest HSTU execution accuracy (70.8%), but its 8.3-point lift over the re-run comparator was not statistically distinguishable (p=0.077), and the authors note that every win in the study was a composition of existing components.
  19. 1324Research Agents

    NanoResearch: Co-Evolving Skills, Memory, and Policy for Personalized Research Automation

    • Problem: Automated research systems such as The AI Scientist and EvoScientist can traverse the full research lifecycle, but the authors view them as uniform pipelines that produce near-identical outputs regardless of each user's resources, methodological preferences, and target formats. The authors name three missing capabilities. Each run starts from scratch and re-encounters the same debugging patterns, past hypotheses, validated configurations, and inferred resource constraints are discarded when a session ends, and implicit preferences such as preferring simpler methods are too diffuse to write as rules and fade when compressed into memory entries.
    • Input and output: This is a system paper with an implementation and experiments. NanoResearch takes a user-specified research topic and a user profile built through interactive queries (preferences, compute budget, target venue style, and so on), runs three stages, idea generation and planning, experimental validation, and paper writing and review, and outputs a LaTeX paper. The user gives natural-language feedback at the end of each stage, and the Orchestrator, the agent that coordinates the system, retrieves relevant entries from the Skill Bank and the Memory Module before each task and updates both stores afterward.
    • Method: Stage 1 retrieves papers from academic databases (for example, arXiv and Semantic Scholar) and parses performance scores from them, proposes hypotheses with a ReAct loop, turns the hypothesis kept after a novelty check into a JSON experiment blueprint with datasets, baselines, metrics, and ablations, and revises the blueprint with critiques from an internal LLM reviewer. Stage 2 writes code, deploys it to an environment such as a SLURM cluster, debugs it automatically until execution succeeds or the retry budget runs out, and analyzes the results, and Stage 3 drafts the manuscript section by section and has a Review agent that works without skill or memory retrieval revise it until predefined quality thresholds are met. After each stage the Orchestrator adds general rules to the Skill Bank and project-specific experience to the Memory Module, and user feedback updates the planner model's parameters through SDPO, which uses the feedback-conditioned model as its own teacher to create a token-level learning signal without a reward model.
    • Implementation: Literature retrieval uses the OpenAlex API, and the only trainable component is the Orchestrator's planner (Qwen3-8B). The other agents use DeepSeek-V3.2 for ideation, planning, and setup and execution, GPT-5.3-Codex for coding and debugging, Claude Sonnet 4.6 for writing and figure prompts and code, Gemini 3.1 Flash for figure images, Gemini 3.1 Flash Lite for review, and Gemini 3 Pro for revision. Per its README, the official repository (OpenRaiser/NanoResearch) describes a 9-stage pipeline from ideation to review, local GPU or SLURM execution, an evo pipeline that updates the user profile, skills, and memory, a ram-train command that trains an SDPO LoRA adapter on (x, y, feedback) triples collected from evo runs, and Claude Code and Codex integration, and its roadmap lists a benchmark evaluation suite as a remaining item.
    • Results: On 20 Claude-generated tasks in 7 domains with 20 simulated researchers, NanoResearch in Round 1 scored Compliance 8.163, E2E (share of runs that finish the whole pipeline) 1.000, and Perf. (average task accuracy of the produced method) 0.6844, higher than the four baselines (AI-Researcher, DeepScientist, EvoScientist, AI Scientist-v2) on every metric, while the baselines' E2E ranged from 0.500 to 0.900. By Round 3 Compliance rose to 8.963, Innovation (originality of the idea relative to prior work) to 5.645, and Perf. to 0.7548, and the total cost per topic was $1.430, lower than $6.039 for AI Scientist-v2 and $2.944 for EvoScientist. In an evaluation by three PhD researchers on their own three tasks, Perf. rose from 0.6466 in Round 1 to 0.8603 in Round 3.
    • Limits: The evaluation centers on AI and ML tasks whose outputs can be completed in code and text, and the authors leave extension to biology, chemistry, or physics, which involve physical experiments and instrument control, to future work. The 20 simulated tasks and the user-role feedback were all produced by Claude, subjective scores such as Compliance, Innovation, and Expression (writing quality of the paper) were rated by an LLM judge, and the human evaluation covers three PhD researchers and three tasks. In the human evaluation, Round 1 Perf. of 0.6466 was below EvoScientist's 0.6537, and Innovation and Expression dipped slightly from Round 2 to Round 3.
  20. 1323Research Agents

    EvoMaster: A Foundational Evolving Agent Framework for Agentic Science at Scale

    • Problem: Scientific agents rebuild model access, tool orchestration, context management, trajectories and error recovery for each domain, which limits reuse across disciplines; the authors call this loop fragmentation. Evidence, rejected hypotheses and failure experience stay confined to one run or research campaign (loop discontinuity), and unfiltered memory can retain noise alongside useful experience.
    • Input and output: It takes a scientific problem defined by a research question, a space of mutable research artifacts, executable tools or environments, an external evaluation protocol and budget and stopping settings. It returns artifacts such as code, model checkpoints, mathematical constructions or data-analysis reports plus a final answer, and keeps each run's configuration, trajectory and workspace artifacts as a persistent laboratory notebook. This is a framework paper that also reports agents built on the framework tested on 10 benchmarks.
    • Method: The Execution loop repeats Reason, Act and Observe within a run, and the Exploration loop forms and executes hypotheses within a study, using external evaluator evidence to retain, reject or revise candidates. The Evolution loop distills a finished study record into reusable knowledge, skills, prompts, memories or workflow components, and an Admit policy (an autonomous evidence rule, an evaluator gate or a human decision) decides whether they enter the capability store for later studies. When the context fills up, the system message prefix and recent turns are kept verbatim and older dialog is summarized, while workspace files, evaluator outputs and tool records stay outside this compression.
    • Implementation: Three software layers (Playground, Exp, Agent) and registries make models, tools, skills and agents independently replaceable, and MCP (Model Context Protocol) servers attach through the same tool interface. In the experiments EvoMaster ran X-Master and X-Master 2.0 on HLE (Humanity's Last Exam) and Frontier Science, ML-Master 2.0 on MLE-Bench Lite (the 22 low-complexity Kaggle competitions of MLE-Bench), Browse-Master on BrowseComp and BrowseComp-ZH (a Chinese web browsing benchmark), PhysMaster on PRL-Bench (a physics research benchmark built from Physical Review Letters papers), and its minimal agent on the remaining 4 benchmarks. Per the official README, the GitHub repository holds the framework code (evomaster/), agent examples (playground/) and configs (configs/), and among SciMaster agents ML-Master 2.0, ML-Master, X-Master and Browse-Master are marked Available while PhysMaster and EmboMaster are marked Coming Soon.
    • Results: With every agent on GPT-5.4 (medium reasoning effort) under the same benchmark-specific time limits, scoring scripts and task splits, EvoMaster scored highest on 9 of 10 benchmarks, all except PostTrainBench (Codex 15.92%, EvoMaster 13.83%), and against the highest-scoring other agent on each benchmark it gained 41.75 points on BrowseComp and 34.85 points on MLE-Bench Lite. Mean scores over the 10 benchmarks were 58.02% for EvoMaster, 40.29% for Codex, 39.84% for OpenHands and 32.39% for OpenClaw, and EvoMaster's mean per-benchmark LLM inference cost was $9.56, 35.6% below Codex's $14.85. Keeping tools, prompts, orchestration and benchmark protocols fixed and replacing only the backend gave mean scores of 63.65% with GLM-5.2 max (mean cost $14.00) and 55.66% with DeepSeek-V4-Pro max (34.4% of the GPT-5.4 mean cost).
    • Limits: The authors state that EvoMaster is primarily optimized for in silico and computational research workflows, and they leave direct manipulation of physical apparatus such as cloud laboratories or robotic synthesis hardware to future work. The cost comparison counts LLM inference only, and because means are sensitive to costly benchmarks such as Combinatorial Construction, the median cost of EvoMaster ($4.57) is higher than that of Codex ($3.00), OpenHands ($2.78) and OpenClaw ($1.43). EvoMaster used SciMaster agents tailored to each benchmark or its minimal agent with only minor input and output changes, against three general-purpose agents, and the authors say the benchmark suite primarily demonstrates breadth across workflows.
  21. 1322Research Agents

    AI Can Learn Scientific Taste

    • Problem: The ability to judge and propose research ideas with long-term impact, which the authors call scientific taste, is largely concentrated among highly experienced researchers. Current AI Scientist training mainly targets literature search and experiment execution, which address how to carry out research, so choosing which research directions are worth pursuing remains open. RLVR, which relies on rewards verifiable by ground-truth answers or unit tests, is hard to apply to such open-ended tasks, and RLHF depends on costly human annotations and struggles to capture community-level preferences from individual preferences alone.
    • Input and output: Scientific Judge takes the titles, abstracts, and publication dates of two papers, writes a reasoning trace, and answers A or B for the paper with the higher citation count. Scientific Thinker takes the title and abstract of a seed paper and returns a follow-up research idea as a title and an abstract. This is a method paper that proposes the RLCF training paradigm and tests it with the SciJudgeBench preference-pair dataset and trained models.
    • Method: RLCF pairs papers from the same field and year and treats the one with clearly more citations as preferred, keeping a training pair only when the citation gap is at least 8 and at least 30% of the higher count. Scientific Judge is trained on these pairs with GRPO (Group Relative Policy Optimization) and receives a reward of 1 for a correct prediction and 0 otherwise. Scientific Thinker samples several ideas per seed paper and is trained with Comparison-Based GRPO, where each idea's reward is its win rate in a round-robin tournament judged by Scientific Judge.
    • Implementation: The authors built 720,341 preference pairs across computer science, mathematics, physics, and other fields from 2.1M arXiv papers published through 2024. Judge models were trained with the MS-SWIFT framework on five Qwen2.5-Instruct sizes from 1.5B to 32B, Qwen3-4B-Instruct-2507, Qwen3-30B-A3B-Instruct-2507, and Llama-3.1-8B-Instruct, and Thinker models were trained from Qwen3-30B-A3B-Thinking-2507 and Qwen3-4B-Thinking-2507 on 4,000 high-citation seed papers published from January to July 2025. The top level of the official GitHub repository holds the README (English and Chinese), the paper PDF, and an assets folder of figures, and the README gives a results summary along with links to a Hugging Face collection of models and datasets and to an online demo.
    • Results: On the 1,000 in-domain test pairs, SciJudge-Qwen3-30B reached 82.7% accuracy, where a pair counts as correct only if the answer holds under both A and B orderings, above GPT-5.4 Thinking (81.6%), the highest of the four strong LLM baselines in Table 3. On 904 pairs of papers published in 2025, after the training period, Qwen3-4B rose from 64.7% before training to 80.9% after training and Qwen3-30B-A3B from 71.7% to 83.1%. Under majority voting by GPT-5.2-high, GLM-5, and Gemini 3 Pro, SciThinker-30B trained with SciJudge-Qwen3-4B as the reward model beat its base policy (the policy model before training) with an in-domain win rate of 81.5%, and its average in-domain win rate against ideas from GPT-5.2, GLM-5, and Gemini 3 Pro was 54.2%, compared with 30.3% for the base policy.
    • Limits: The authors state that citations are an imperfect form of community feedback, since some high-potential papers may receive few citations at first and become influential later, and that the controlled evaluations reduce simple author, institution, and topic explanations but cannot remove all biases in citation-based feedback. The ideation evaluation mainly relies on strong LLM evaluators, and implementing a subset of the proposed ideas is left for future work. Scientific Judge is trained mainly on titles and abstracts, and the authors note that richer paper context such as related work sections may improve its judgement.
  22. 1321Research Agents

    Idea2Plan: Exploring AI-Powered Research Planning

    • Problem: Scientists tend to produce more promising ideas than they can pursue, many remain unexplored due to time constraints, and turning an idea into a testable plan can take many days or even weeks. Recent work has focused on other stages such as idea generation, literature review, and experiment execution, and has treated research planning as an intermediate step without explicit evaluation (CodeScientist) or evaluated plans only on high-level dimensions such as clarity and novelty (MLR-Bench). Evaluation faces data contamination, since an LLM may have already seen the idea through a published paper, and the fact that several plans can be equally valid for the same idea.
    • Input and output: This is a benchmark paper that contributes a task definition, an automatic benchmark construction pipeline, and a benchmark for validating LLM judges. The input is a research idea rewritten in the first person from the original paper's abstract with method names and experimental results removed, and the output is a research plan with five sections: Introduction, Key Literature, Methods, Initial Experimental Design, and Resources/Compliance/Ethical Considerations. An LLM judge answers section-wise yes/no rubric questions built from a reference plan extracted from the original paper, and the Planning Score is the share of questions satisfied in each section, averaged over the five sections.
    • Method: From the ICML 2025 Spotlight and Oral papers, the authors drop papers whose arXiv versions predate the most recent training data cutoff of the tested LLMs (GPT-5) and randomly pick 200 of the remaining papers as the test set and 30 as the dev set. o4-mini (reasoning=high) extracts the idea from the abstract and the reference plan from the full paper, and rubric questions ask about high-level design choices that any reasonable plan should address instead of specific terms absent from the original idea. Plans are generated four ways: Naïve (a short prompt with section titles), 0-shot (general instructions added), 1-shot (one fixed example plan added), and a ReAct agent that can call an arXiv search tool and a read tool up to five times each.
    • Implementation: The tested LLMs are GPT-4.1, GPT-4.1-mini, o4-mini, GPT-5, and GPT-5-mini, plus the open-source DeepSeek-V3, DeepSeek-R1, and Phi-4, with three runs per idea for each model and setting and the mean reported. The judge, o4-mini (reasoning=high), was chosen after comparing candidates on Idea2Plan JudgeEval, where two human annotators independently graded the rubric questions of 30 plans (88.4% agreement, Cohen's κ 0.768), and it grades each section of a plan separately. The search tool uses Bing Search restricted to arXiv and blocks papers published after January 30, 2025 (the ICML 2025 deadline), the target paper, and papers citing it, while the read tool returns an o4-mini summary in a fixed template of a paper fetched through the arXiv API.
    • Results: On the test set, GPT-5 had the highest average Planning Score in all four settings (Naïve 62.0, 0-shot 63.0, 1-shot 63.2, ReAct 61.9), and GPT-5 and GPT-5-mini achieved pairwise win rates over 90% against the other models. The effect of ReAct varied by model: GPT-4.1 scored 40.8 with 1-shot and 47.4 with ReAct, while GPT-5 scored 63.2 and 61.9, and the authors report that models sometimes incorporated irrelevant retrieved details into plans. On an appendix benchmark of 100 ICML 2026 papers, standalone GPT-5.5 scored 62.0 and GPT-5.5 in the Codex harness with an MCP (Model Context Protocol) paper search tool scored 75.1, and the upper bound in Table 28 (a plan written with the original paper in hand) was 85.8.
    • Limits: The plan template follows AI paper conventions and drops components an LLM cannot easily determine, such as timelines, logistics, or human expertise, and the Nature Mental Health benchmark cuts the template to four sections. Because the rubric measures coverage of key elements, longer outputs may gain an advantage, and GPT-4.1 and o4-mini scored higher when allowed longer outputs. All candidate judge models showed a leniency bias with more false positives than false negatives, and human checks of the extracted plans and rubrics covered 8 papers, one chosen by each of eight experts (section means above 4.0 on a 5-point scale).
  23. 1320Research Agents

    Foundation Models for Scientific Discovery: From Paradigm Enhancement to Paradigm Transition

    • Problem: The authors argue that as science increasingly deals with emergent behavior, open-endedness and irreducible complexity, the limits of the four existing paradigms (experimental, theoretical, computational, data-driven) have become more evident. In drug discovery and material design the combinatorial explosion of candidate spaces makes exhaustive search infeasible, and the accumulation of experimental and observational data has outpaced the capacity to synthesize unifying theories. They therefore ask whether FMs merely enhance existing methodologies or redefine the way science is conducted.
    • Argument: This is a position paper whose stated contributions are a conceptual framework, a systematic review and taxonomy, and a research agenda. The authors take the position that FMs are not only improving parts of the scientific process but beginning to reshape the paradigm through which science operates, and they call the state reached once the autonomous stage is realized the fifth scientific paradigm. They also present the opposing view that FMs are powerful yet conventional tools that amplify existing methodologies.
    • Taxonomy: In stage 1, Meta-Scientific Integration, FMs are backend tools that automate procedures such as data preprocessing, literature retrieval and methodology matching within human-defined objectives, with low autonomy. In stage 2, Hybrid Human-AI Co-Creation, FMs are collaborators in research questions, hypotheses and experiment planning, while their actions are still initiated, constrained and validated by humans, with moderate autonomy. In stage 3, Autonomous Scientific Discovery, which the authors present as a forward-looking vision, FMs are independent agents that run the cycle from posing questions to executing experiments or simulations and interpreting results with minimal or no human oversight.
    • Review: Section 4 organizes cases from stage 1 and early stage 2 by paradigm, citing FMs as priors or feature extractors in Bayesian optimization and FM-generated Python control scripts for instruments in the experimental paradigm, and knowledge-graph-guided hypothesis generation and formal reasoning such as Logic-LM, which couples LLMs with symbolic solvers in a feedback loop, and LeanCopilot, which assists proof construction and verification, in the theoretical paradigm. For computation it cites LLM-SR, which turns plots or text into equation skeletons, FunSearch, Neural Operator and GraphCast, for data AlphaFold 2, ESMFold and MatterGen, and as a cross-paradigm case Coscientist, which translates research goals into machine-executable protocols and controls robotic synthesis. The official repository README holds a paper list by paradigm together with example documents for topic exploration, literature survey, problem refinement, methodology design and experiment code produced with OpenAI Deep Research and ChatGPT.
    • Risks and directions: The authors list four risks that grow as FMs move from tools to autonomous agents: bias and epistemic fairness inherited from training data, hallucination that produces plausible but unverified claims, reproducibility and transparency problems from opaque decision-making, and authorship, accountability and research ethics. As mitigations they propose more diverse training data and fairness-aware evaluation, symbolic logic checks and simulation-based validation, human-in-the-loop review and provenance tracking, logging of reasoning steps and version-controlled model checkpoints, and governance that mandates transparent disclosure of AI contributions. As future directions they name embodied scientific agents working with lab robotics, instruments and digital twins, closed-loop autonomy in which FMs continuously run from hypotheses through experiments to analysis, and continual learning that accumulates and refines knowledge.
    • Limits: The argument rests on a conceptual framework and a review of existing work, and the authors state that the transformation is still in its early stages and that autonomous discovery is in its formative stages. They also note that most FM deployments remain confined to static prompts, predefined tasks and fixed schema and typically lack persistent memory, adaptive feedback and physical embodiment, so their contributions are largely reactive and limited to isolated stages of research. Open challenges they list include linking high-level planning with low-level control in embodied agents, robustness of closed loops to noisy observations and shifting objectives, and robust transfer across heterogeneous tasks in continual learning.
  24. 1319Research Agents

    The More You Automate, the Less You See: Hidden Pitfalls of AI Scientist Systems

    • Problem: AI scientist systems run the research workflow from hypothesis generation through experiments to paper writing, but the authors note that their internal workflow has not been closely examined, which risks flaws that undermine the integrity, reliability and trustworthiness of their outputs. Diagnosis is also hard: with any existing dataset or task, web-scale pretraining makes data contamination almost inevitable, and each failure mode needs conditions that isolate it from confounding factors.
    • Input and output: The authors run two open-source AI scientist systems on SPR task descriptions with local datasets and loading scripts, and record the benchmarks they choose, the accuracies they report, the metrics they pick and the candidates their reward functions select to judge each failure. A separate LLM-based classifier takes the task description, the final paper, generated code, execution logs and pitfall-specific detection rules and returns a Yes or No detection with an explanation. This is an empirical study that diagnoses existing systems with controlled experiments and also proposes the diagnostic task and a detection method.
    • Method: SPR (Symbolic Pattern Reasoning) is a synthetic classification task where sequences of tokens built from 4 shapes and 4 colors are labeled accept or reject by whether they satisfy k hidden conditions on shape count, color position, parity and order. The benchmark-selection test offers 20 SPR datasets in a strict difficulty ladder, each raising one of rule complexity, vocabulary size or sequence length, under random five-letter names and random order, and asks the system to pick 4; the data-leakage test flips 20% or 30% of labels and treats test accuracy significantly above the theoretical upper bound (1 minus the noise level) as a leakage signal. The metric test adds noise so that only one of SWA (Shape-Weighted Accuracy) and CWA (Color-Weighted Accuracy) drops on the test set, and the post-hoc test inverts only the test performance of five candidates taken from logs to see whether the reward function changes its pick.
    • Implementation: Both systems ran with their default LLM API settings, except that the code model of The AI Scientist v2 was switched from Claude 3.5 Sonnet, deprecated as of August 13, 2025, to OpenAI's O3-mini. For benchmark selection Agent Laboratory was stopped at the data preparation stage and run 1,000 times each with and without SOTA (state of the art) figures and The AI Scientist v2 generated 1,000 ideas each without running experiments, and post-hoc selection was measured with 200 trials per condition from 20 research projects run 10 times each. The audit classifier used gemini-2.5-flash-preview-05-20 and was evaluated on 20 cases per pitfall type over five types, half positive and half negative, and per the official README the repository holds the research generated by both systems, the SPR task descriptions and datasets, and the detection script (detect_pitfalls.py).
    • Results: In benchmark selection Agent Laboratory picked the first four listed benchmarks in 82.4% of 945 runs with explicit selections when SOTA figures were given and in 79.6% of 927 runs without them, following list position over difficulty, while 47.1% of the benchmarks The AI Scientist v2 picked with SOTA figures were easy ones (18.0% without). No system peeked at test data, but Agent Laboratory used a subset of the data (16th run) or a synthetic dataset (11th run) without saying so in the paper, The AI Scientist v2 subsampled or synthesized datasets in runs 3, 7, 8, 11 and 13 and some test accuracies exceeded the provided SOTA figures, and with SWA listed first Agent Laboratory chose only SWA as its metric in all 20 runs. With test performance inverted, the share of picks going to the best train and validation candidate fell from 78.5% to 43.5% for Agent Laboratory and from 82.0% to 31.5% for The AI Scientist v2, and the LLM audit's overall accuracy rose from 55.0% (F1 0.51) with the paper alone to 82.0% (F1 0.81) with trace logs and code.
    • Limits: The authors focus on four failure modes that do not exhaust other forms of scientific malpractice or system failure, and they leave reward hacking of LLM-based peer review by AI scientists outside this study. The detection techniques primarily rely on controlled setups and observable behavior, so they may not generalize to more nuanced or adversarial misuse or to flaws never reflected in logs, and with only two systems evaluated, extrapolating to other systems and to domains beyond ML calls for caution. The detection evaluation used 20 cases per pitfall type, positives for pitfalls that were rare or absent in natural runs were created with controlled perturbations of the workflow, and data leakage was injected by merging test samples into the training set.
  25. 1318Information Retrieval

    MIR: Methodology Inspiration Retrieval for Scientific Research Problems

    • Problem: In existing idea-generation methods that retrieve literature with a seed paper's abstract, the methodology and results in the abstract can bias retrieval toward the seed paper and reduce the novelty of the generated ideas. Commonly used retrievers and search APIs rely on sentence-level semantic similarity or generic inter-document relations, so they can fetch papers that merely share keywords, such as a 'sentence compression' paper for a 'model compression' problem. Some works also assume access to the seed paper's cited papers, so the authors target the realistic setting where only the research problem and its motivation form the query.
    • Input and output: This is a method paper that defines a new task and dataset and proposes a retriever training method and LLM re-ranking methods. The input is a research proposal (problem and motivation) with any mention of methodology removed, the output is the top-k papers of a corpus ranked by their usefulness for building a solution, and at inference the retriever takes only the proposal without citation information. The MIR-MultiCite dataset extends MultiCite, a computational linguistics dataset of 12,653 citation contexts labeled with eight citation intents, and splits proposals by time into 704 for training (before 2019), 86 for dev (January to June 2019), and 139 for test (after that).
    • Method: MAG treats each research proposal as a node with its cited papers as neighbors, annotates edges with citation intents and contexts, and counts the uses and extends intents as methodology citations. The retriever puts three triplets in one batch, pulling together the proposal and a methodology-cited paper, a methodology citation context and the proposal, and that context and its cited paper, and trains on the sum of L2 triplet margin losses; the hard negative of the first triplet is the paper outside the proposal's citations with the highest domain similarity measured from scientific named entities, mixed with random soft negatives at ratio α. LLM re-ranking uses Pointwise, which labels each of the top-k papers relevant or not, and MIR-Agent, which first writes the proposal's sub-problems and a generic action plan and analyzes each paper's applicability before judging, and papers judged relevant move ahead in their original order.
    • Implementation: Proposals were extracted from the abstracts of citing papers with Gemini-1.5-pro-001, and the training data adds part of 51,021 proposals from about 60k arXiv computational linguistics papers whose citation intents were weakly labeled by a SciBERT-based classifier. SPECTER2, Stella 400M, and Stella 1.5B (the last with LoRA) were fine-tuned on a single NVIDIA V100 GPU (32GB), and re-ranking used Llama 3.1 70B Instruct and Gemini 1.5 Pro. Per its README, the official repository (Anikethh/Methodology-Inspiration-Retrieval) holds the dataset used in the paper (proposals, cited papers, citation contexts, citation intents, and training, dev, test, and augmented training splits extended to mid 2024), and its top level has data and assets folders.
    • Results: On the MIR-MultiCite test set, fine-tuning raised Stella 1.5B's Recall@3 on the Extended Corpus (all cited papers of the training set plus the ground-truth citations of the test proposals) from 54.73 to 60.11 and its mAP from 49.08 to 56.89. On the Restricted Corpus, re-ranking the top 10 of the fine-tuned Stella 1.5B with few-shot MIR-Agent on Gemini 1.5 Pro raised Recall@3 from 66.95 to 71.45 and mAP from 64.24 to 69.02, and for the 75 test proposals whose top papers had full texts available, the same setup with full texts raised Recall@3 from 66.84 to 72.49. IA-Score, the average share of the 5 LLM-generated methodology ideas per proposal that match the methodology in the original paper's abstract, was 0.170 with abstracts retrieved by MIR and 0.142 with abstracts from the best baseline retriever.
    • Limits: The dataset is limited to English papers in computational linguistics, and the retriever is trained on cited-paper abstracts, with full texts used for re-ranking the top-k. The authors report that MultiCite's original human annotations sometimes omit methodology citations and occasionally include trivial references (for example, mere use of GPT-4). An analysis of 15 proposals (39 relevant papers) with subpar Recall@5 under the best configuration (MIR-Agent with full papers) found 17 incorrect or missing annotations, 7 subjectivity cases where the authors chose a different research direction, and 15 misclassifications by the pipeline.
  26. 1316Research Agents

    Optimizing the Optimizer: Language Models Discover Faster Molecular Relaxation Algorithms

    • Problem: In geometry optimization at the DFT level each step needs one force (gradient) evaluation, which dominates the wall time, so the step count sets the cost. Letting an LLM agent rewrite an optimizer raises three constraints: DFT is too expensive to score hundreds of candidates, candidates must work on unseen molecules, and the agent must be kept from cutting force calls by stopping early.
    • Input and output: The agent receives a starting program that merges Sella 2.5.0 into one file (algo.py, 5975 lines including comments and blank lines), training and validation molecule sets and a remote evaluator, edits only algo.py, and returns AutoSella optimizers that reach the same convergence criteria with fewer force calls. Each optimizer itself takes one molecule and a force evaluator and updates the structure until all five convergence conditions hold (cap of 200 force calls, 500 for Solvated PubChem), and the molecule sets cover drug-like molecules with their surrounding biomolecules and solvent. This is a method paper that finds, releases and evaluates new optimization algorithms with a gated autoresearch loop.
    • Method: In each cycle the agent reads the cycle record, the narrative log and the mechanisms already rejected, commits one coherent edit to algo.py, and after evaluation receives a per-molecule table of relative force calls, relative energy, final energy difference and convergence status. The validity gate discards candidates on training that have evaluation errors, mean energy recovery below Sella's, or a cost drop smaller than the threshold δ, and the generalization gate accepts a new champion (the best program so far) only if it also cuts cost on validation without evaluation errors while keeping mean energy recovery. The search runs only on the cheap GFN2-xTB potential, algo.py is the only file the agent may edit, and changing the evaluator, convergence criteria or molecule sets and adding molecule-specific special cases are prohibited.
    • Implementation: Claude Code driving Claude Fable 5.1 (max) and Codex driving GPT-6 Astra (high) each ran independently for 66 active research hours and produced AutoSella-F and AutoSella-A, and the remote evaluator, a Redis queue with 192 single-thread workers on 4 hosts with Intel Xeon Gold 6348 CPUs, scored one candidate in approximately 60 to 90 seconds on average, depending on the run. The 469 training structures combine 225 PubChem molecules and 219 DES370K dimers from SPICE 2.0.1 with 25 registered drugs from the RowanSci optimizer benchmark, and the 465 validation structures come from SPICE 2.0.1. Per the official GitHub README, the main branch holds the three frozen optimizers (AutoSella-F, A, G), a Sella 2.5.0 reference wrapper, the molecule preparation pipeline and an example, while the autoresearch harness, per-model run records and the xTB and DFT evaluations sit on separate branches.
    • Results: On the four held-out GFN2-xTB sets AutoSella-F cut mean per-molecule force calls to 39.86 to 73.25% of Sella with higher mean energy recovery on all four, while AutoSella-A used 78.14 to 92.57% on the first three sets but rose to 118.77% on Solvated PubChem with lower mean energy recovery. Under r2SCAN-3c DFT, used only for evaluation, AutoSella-F needed about 23 to 60% fewer force calls per molecule than Sella with greater mean energy recovery, and on Solvated PubChem it used on average 40.24% of Sella's force calls per molecule with 1.27% higher mean energy recovery. Over 66 hours Fable accepted 37 of 108 evaluated candidates and Astra 29 of 672, and AutoSella-G from the Grok 4.6 run in the appendix lowered the training cost to 97.79%.
    • Limits: Both gates judge by mean energy recovery, force-call cost and evaluation errors, and in the separate GFN2-xTB structure check of Appendix E the share of molecules where AutoSella-F reaches the same local minimum as Sella was 97.2% on Test, 93.2% on Dipeptides, 39.8% on amino acid and ligand pairs and 1.0% on Solvated PubChem. The authors note that the other endpoints lie below the reference energy on average, yet on GFN2-xTB AutoSella-F ended more than 1.0 kcal/mol above Sella on 18.74% of amino acid and ligand pairs and 38.00% of Solvated PubChem systems. AutoSella-A used 109.06% of Sella's force calls on Solvated PubChem even under DFT with lower mean energy recovery, and the Grok 4.6 result AutoSella-G was much worse on the intermolecular sets and was left out of the DFT evaluation.
  27. 1315Research Agents

    Position: AI Agents in Scientific Teams Should Be Studied as Human-Agent Systems

    • Problem: LLM-based agents are entering scientific work as collaborators, yet most AI Scientist research focuses on growing agent autonomy and limits human participation to setting goals and verifying work. The authors argue that this overlooks the social side of scientific teamwork and that this literature has largely been isolated from the socio-technical realities of science.
    • Argument: This is a position paper arguing that studying AI Scientists as human-agent systems (HAS), where the unit of analysis is the human-agent pair, is underexplored and undervalued, and that new HAS research is needed to understand human-AI synergy. The authors argue that the HAS lens can help better understand and potentially mitigate near-term risks, citing 100 confirmed hallucinated citations found in 51 accepted NeurIPS 2025 papers and an expert review that found noticeable plagiarism in 36% of LLM and AI Scientist-generated papers. They also cite a study in which AI-augmented research produced three times as many papers with a 5% drop in the diversity of explored topics.
    • Taxonomy: The authors group AI Scientist systems by the main channel through which human feedback enters, namely artifact-level, discovery-phase-level, research-plan-level, and asynchronous steering and interruption, and tag each with Zou et al.'s feedback type (evaluative, guidance, corrective), granularity, and phase. The review finds that most systems give humans supervisory roles such as scoping, auditing, and approval, that artifact-level feedback is the most common, and that only a handful support flexible asynchronous interaction.
    • Evidence: In a preliminary experiment with AstaBench's ReAct (an agent style that alternates reasoning and acting) agent harness and GPT-5-mini on 10 End-to-End Discovery (E2E) development tasks, adding an ask user tool and the feedback taxonomy to the prompt produced 1 question, and the E2E score went from 0.39 to 0.36. Forcing a question before every action raised questions to 92, and in the average over three LLM judges 0.71 of them were evaluative and 0.04 corrective, with an E2E score of 0.24. In this experiment an automatic continuation prompt, the AstaBench default, answered the questions instead of a human.
    • Cases and proposal: In one published case, Gemini 3 Pro produced a plan (plan.md) and a draft manuscript (paper.tex) from an initial prompt and then expanded proofs and exposition over eight prompt turns, while the expert pointed out missing prior work and corrected a wrong assumption in a lemma. In another, GPT-5 drafted the PDE (partial differential equation) and numerical workflow for a reduced physics model of thermonuclear burn, while the expert tuned the conditions by hand and judged when outputs were still noise. The authors propose a toy model in which team utility is the collaboration advantage CA (how much the team beats its best member) minus the collaboration disadvantage CD (communication and coordination cost).
    • Limits: The preliminary experiment used 10 development tasks with an automatic prompt answering the questions, which the authors defend as a deep rather than broad measure because each task is a full idea-to-report workflow. The case studies are qualitative readings of two published accounts, and the toy model and proxy metrics such as Review Time, Intervention Density, and Simulated Oracle Synergy are proposed as directions for future work. The authors address the opposing view that the HAS lens may add little once evaluation costs are counted, and acknowledge that centaur evaluations comparing HAS with humans alone and agents alone are currently expensive and hard to reproduce.
  28. 1314Research Agents

    Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops

    • Problem: AI systems increasingly take part in their own improvement, from revising outputs and changing their harnesses to training on self-generated data and doing AI research, yet the literature lumps fundamentally different ambitions under words like self-refine, self-reward, self-play and self-evolve. The authors note that existing surveys cover single slices such as on-policy distillation or tree search with rewards, and that to their knowledge no survey covers the four categories jointly, from bounded self-refinement to open-ended recursive self-improvement (RSI).
    • Scope: This is a survey that collects and classifies literature. It gathers 1,250 arXiv papers from 2024 to 2026: a seed harvest of 871 papers across seven threads plus a targeted supplement of 379 papers on self-evaluation, test-time training and zero-data self-play. According to the README, the official GitHub repository deepgrounding/recursive-self-improvement holds corpus_v2.csv with taxonomy assignments, the seed snapshot, scripts for reclassification, supplemental harvest, bibliography and figures, and the manuscript source (main.md, matching arXiv v1).
    • Taxonomy: The first axis is what the system improves: deployment-time self-evolution (393 papers), training-time self-iteration (340), self-evaluation (318) and Auto Research (139), plus a foundations, limits and safety family (60). The second axis is who validates the loop, in three degrees: human-in-the-loop, human-on-the-loop and closed loop. The authors report that nearly all of the 1,250 papers study bounded self-refinement in human-on-the-loop cells, where humans audit outcomes.
    • Argument: The authors rank improvement signals into a verification hierarchy of formal verifiers (proof checkers, type systems), execution feedback (tests, compilers, benchmarks), learned judges (reward models, LLM-as-judge) and intrinsic signals (confidence, self-consistency, likelihood), and observe, as a qualitative pattern across the corpus, that demonstrated self-improvement strength tracks this hierarchy. They group failure modes into four: the self-confirming loop, collapse, diversity collapse and frame lock-in (continuing to optimize a stale objective). The foundations family is the smallest of the five at 60 of 1,250 papers, and the authors suggest that governance-grade measurement of self-improvement is the most underpopulated research niche the survey found.
    • Evidence: In the Mirror Loop study cited by the authors, models from three providers ran ten rounds of ungrounded self-critique across four task families and informational change declined 55% across iterations, while one verification step at iteration three restored forward movement. In the cited SkillsBench result, human-authored skills raised pass rates by 16.2 points while LLM-authored skills gave no measurable gain. The cited A-Evolve-Training ran the entire post-training loop of a 30B model for four rounds with no human in the loop, within a human-specified objective and search space, and reached 0.86 on a public leaderboard, near the 0.87 of the top human submission.
    • Limits: The authors describe the corpus as a recency-biased sample, and say that because 74% of papers were posted in 2026, citation counts are near zero and cannot be read as impact. Roughly 54 of the 379 supplemental papers are peripheral query bleed that remain in the corpus counts, and classification and corrections come from a single annotator. Industrial RSI practice inside frontier labs is observable only through what labs publish.
  29. 1313Research Agents

    Notes2Skills: From Lab Notebooks to Certainty-Aware Scientific Agent Skills

    • Problem: Prior procedural-text extraction assumed prescriptive input such as recipes and wet-lab protocols, while in lab notebooks confirmed observations, uncertain judgments and next-step suggestions often appear in the same passage. The authors argue that losing this distinction causes uncertainty laundering, where an author-qualified statement becomes a stronger processing commitment without adequate authorization and evidence and may lead to discarding data, and directive loss, where a firm directive is dropped or weakened into an opinion so that a supported processing action is missed.
    • Input and output: This framework paper proposes a two-stage pipeline and evaluates it on newly annotated corpora and real nanopore experiment records. The input is notebook text segments, and Stage 1 (Epistemic Directive Extraction, EDE) outputs for each segment, as JSON, whether it holds a directive, one of five directive types, and an author certainty label: FACT (an unhedged assertion), JUDGMENT (an uncertain or tentative assessment) or SUGGESTION (optional advice). Stage 2 builds a Markdown skill document (MetaSkill) that attaches the source excerpt and certainty label to each directive, and an agent reading it picks one action per recording file: KEEP_FULL (keep the whole recording), FLAG_FOR_REVIEW (keep it for review), RAISE_THRESHOLD (raise the event-detection threshold), TRUNCATE_AT (cut at a given time) or SKIP_FILE (exclude the file).
    • Method: Stage 1 extracts directives and certainty labels with LLM prompting, and in Stage 2 a deterministic compiler with no LLM emits one capsule per directive and checks directive preservation, unchanged labels, source fidelity and schema validity. At execution, Layer 1 turns the instrument trace into a SignalFindings summary, a Claude Sonnet 4.5 agent in Layer 2 proposes a processing decision, and in the verify setting the Layer 3 executor passes strong decisions such as TRUNCATE_AT, RAISE_THRESHOLD and SKIP_FILE only when a capsule authorization path and the signal evidence agree, otherwise applying an action-specific fallback, usually FLAG_FOR_REVIEW. The elevate setting turns a cautious proposal into TRUNCATE_AT only when exactly one stored truncation candidate timestamp matches a calibrated step drop.
    • Implementation: The annotated corpora total 461 segments: 201 FreeNotes segments from Chinese-English nanopore notebooks contributed by two senior researchers at two institutions, 155 ONS (Open Notebook Science) segments from openlabnotebooks.org, and 105 WLP (Wet Lab Protocols) segments from a published protocol corpus, and Stage 1 compared GPT-4o, Claude Sonnet 4.5 and Qwen-Max with zero-shot and few-shot prompting at temperature 0 on an 87-segment test set. The downstream evaluation compared seven configurations on 48 nanopore files from three FreeNotes sessions (17 in Saturation-A, 22 in Saturation-B, 9 in Step-drop), with five API calls per configuration and file decided by majority vote. The authors describe a released artifact with compiler and executor code, evaluation scripts and fixed decision records, but the paper gives no repository address, and the raw ABF (Axon Binary Format) current traces stay out of redistribution under the data-sharing agreement.
    • Results: In Stage 1, Claude Sonnet 4.5 few-shot reached a directive-detection F1 of 0.737, above 0.682 for the strongest zero-shot setup, and the Stage 2 compiler preserved all 149 fixed input directives and passed all four checks. On the 48 nanopore files, the external LLM and Action-only baselines reached only 0 to 21.4% review recall on the saturation sessions, while the full configuration (verify+elevate) reached 85.7% and 100% review recall and 96.4% and 100% balanced accuracy on Saturation-A and Saturation-B. On the FACT-dominated Step-drop session, the full configuration's accuracy of 77.8% was below the 88.9% of Action-only.
    • Limits: The authors state that the downstream evaluation covers only three nanopore sessions and one downstream model, and that only the nanopore domain adapter was built and evaluated. The main comparison used adjudicated EDE inputs, and replacing them with Claude Sonnet 4.5 few-shot predictions dropped directive precision to 34.8% on Saturation-B and 47.9% on Step-drop and pushed Cohen's kappa below chance to -0.07 and -0.03. The executor-enabled comparison also changes the Layer 2 policy text, so the authors note that a matched-policy experiment is needed to isolate the executor's own effect.
  30. 1312Research Agents

    Agentic Discovery of Neural Architectures: AIRA-Compose and AIRA-Design

    • Problem: Most LLMs today are Transformers that interleave attention and MLP layers 1:1, and moving to hybrid designs that mix in other primitives such as Mamba makes the design space vast and combinatorial. The authors argue that relying solely on manual exploration may overlook strong non-obvious arrangements or entirely new primitives. They therefore hand this search to research agents and frame it as a step toward recursive self-improvement, where agents discover and optimize the architectures that power them.
    • Input and output: This is a framework paper that builds 12 tasks in the AIRS-Bench format, a task standard for evaluating an agent's ability to conduct independent research, and runs experiments on them. Given a task description and data, the agent submits a 16-primitive architecture string (submission.csv) in AIRA-Compose, or a model.py for LRA or a train.py for Autoresearch in AIRA-Design. An isolated evaluation script trains the submission from scratch and scores it on a held-out test split or a pinned validation shard.
    • Method: An agent is an LLM plus a scaffold (search policy), run inside AIRA-dojo, a harness that evolves code solutions through tree-based exploration, with four operators: Draft, Debug, Improve, and Analyze. The greedy scaffold drafts 5 initial solutions and keeps applying Improve to the one with the best validation score. AIRA-Compose turns the search and evaluation steps of Composer, an existing framework that searches hybrid architectures with Bayesian optimization, into agent tasks, then uses Composer's aggregation and extrapolation steps to scale the found 16-layer patterns to 350M, 1B, and 3B parameters by stretching (proportionally expanding contiguous blocks) or stacking (repeating the whole pattern).
    • Implementation: Each agent gets one H200 GPU, and a run lasts at most 24 hours or 500 steps, with 60 hours per run for the larger BabiStories and DCLM searches. AIRA-Compose uses 6 LLMs, while in AIRA-Design the LRA experiments build 20 agents from 12 LLMs including Opus 4.6 and Gemini 3 Pro and the Autoresearch experiments build greedy agents from 7 of those LLMs. Scaled-up training ran 350M models on 8 H200 GPUs and 1B and 3B models on 16 H200 GPUs.
    • Results: Among architectures built from the two primitives MLP and attention and trained at the 1B scale on 37.5B tokens, AIRAformer-D (Stretched) reached a validation loss of 2.734 and 59.7% average 0-shot accuracy, ahead of Llama 3.2 and the Composer-found models, and in the 3-primitive comparison that adds Mamba, AIRAhybrid-D (Stretched) reached a validation loss of 2.719 from a single seed. On LRA, the best accuracy of agent-designed models was 2.3pp (percentage points) below the human-designed SOTA (state of the art) of 82% on document matching and 2.6pp below the SOTA of 91% on text classification. On Autoresearch, Greedy Opus 4.5 with literature access reached a validation BPB (bits per byte, lower is better) of 0.968 in its best seed, below the original Autoresearch best of 0.9775.
    • Limits: The authors state that AIRA-Compose relies on small-scale proxy evaluation that does not always predict large-scale performance, and that the aggregation and extrapolation steps remain non-agentic fixed procedures. On LRA, the agent-designed architectures largely recombine prior ideas such as Performer, Longformer, and Conformer. The Autoresearch runs used H200 GPUs in a sandbox without internet access, unlike the original H100 setup, and the authors note this may explain why the same train.py started at a BPB of 1.012 instead of the original 0.998.

From research to product use

Operating capabilities, pilots, and technologies in development are identified separately.

View technology