Research library

Search papers on equipment connectivity, quality inspection, production analysis, and manufacturing work.

Search and filter

New paper summaries

Summaries are editorial notes, not statements by the original authors. Check the original paper. Updated 2026-11-20.

  1. 1279Research Agents

    AIM: Agentic Idea Management for Automated Research

    • Problem: In automated research, implementation and evaluation are expensive, so a limited experimental budget has to be allocated across possible approaches. Idea-driven search, which selects ideas first and delegates implementation to a solver, must organize a growing pool of ideas and evidence, select which ideas to implement, and keep the solver's code faithful to the selected idea. Existing methods typically rely on fixed rules such as UCB or MCTS (Monte Carlo Tree Search) or on LLM judgments without explicit estimates, and when the code does not realize the idea, its score and lessons are attributed to the wrong idea.
    • Input and output: AIM takes one AutoLab task (a natural-language instruction, a containerized environment, a deliberately weak baseline codebase, and an evaluation script) together with a research brief generated from the task description using Claude Code. When the experimental budget is spent, it returns the highest-scoring executable solution among those evaluated as the final output. This is a system paper that implements the framework, tests it on 10 benchmark tasks, and adds a theoretical analysis of when idea-level search pays off.
    • Method: Agentic Surrogate follows the surrogate role in Bayesian optimization, regrouping the whole idea pool into 2 to 5 semantic clusters at every iteration and producing ordinal estimates of promise for clusters and ideas. Agentic Acquisition uses two LLM calls to assign each branch an explore or exploit action at both the cluster and the idea level, sends the chosen ideas to parallel Solvers, and then generates new ideas from extracted lessons with one of four modes: score-guided refinement, cross-pollination, error-guided repair, or novel idea generation. Solution Auditor removes results flagged as trivial, task_mismatch, or reward_hacking from the history and lessons, and when idea_mismatch is the only issue it rewrites the idea to match the implementation and attaches the score to that idea.
    • Implementation: All methods use Gemini-3.1-Pro-Preview as the backbone LLM, implementation is mainly done by the Gemini Deep Solver from ScientistOne, and the initial pool of about 10 ideas comes from the ScientistOne Ideator. The budget is at most 300 executions and 6 hours for System Optimization, 60 executions and 24 hours for Model Development, and 60 executions and 12 hours for CUDA, and the Resource Planner launches 5 Solver branches in the first iteration and then sets between 3 and 10 branches per iteration. The appendix gives pseudocode for the full pipeline and the prompt templates for every LLM-driven operator.
    • Results: On the five AutoLab System Optimization tasks, AIM averaged 67.0%, 1.6 points above ScientistOne (65.4%), the strongest idea-driven baseline, and 3.7 points above AdaEvolve (63.3%), the strongest solution-driven baseline. On the five Model Development and CUDA tasks, AIM averaged 55.8%, 4.9 points above ScientistOne (50.9%), and the authors report that AIM reached ScientistOne's best score up to 3.1 times faster in wall-clock time. In the Flash Attention ablation, removing any single component scored below the full AIM's 90.5%, and removing the Agentic Surrogate caused the largest drop, to 85.9%.
    • Limits: The authors state that the assumption behind the theoretical analysis, that the implementation faithfully realizes the selected idea, could be slightly idealized and that real solver agents may produce incomplete or incorrect implementations. AIM tends to need more executions to reach its best score and generally uses a larger token budget, and on the data selection task Data Select IFEval it scored 61.8, below AdaEvolve (82.7). The authors write that isolated containerized environments and the Solution Auditor reduce risk, that some risk remains, and that human review is needed before applying autonomous research systems to consequential scientific or engineering settings.
  2. 1278Research Agents

    Dr. Claw: An AI Scientist Workspace for Vibe Research

    • Problem: Command-line (CLI) coding agents such as Claude Code and Gemini CLI read and write files and sustain long sessions, yet end-to-end research still splits across chat tools, IDEs (integrated development environments), terminals and writing environments. These agents optimize execution rather than control, so the plan, intermediate decisions and artifacts that make a research process reviewable get scattered or lost, and humans have few explicit takeover points.
    • Input and output: Given goals, constraints and acceptance criteria in natural language, the system produces a structured research brief and a dependency-aware task plan, writes execution outputs into the Artifact Store, updates task states in the Task Graph and keeps all interventions in the Decision Log and Execution Trace. This is a system paper: the workspace keeps an existing command-line coding agent as the executor and adds state, control and audit layers on top. The authors call this split, where humans own research direction, evaluation criteria and final acceptance while AI handles retrieval, coding, running, summarizing and drafting, Vibe Research.
    • Method: Four state objects (Task Graph, Artifact Store, Decision Log, Execution Trace) form the workflow state, and each interaction is formalized as a state transition from the current state, an action and the observed feedback. The system has three layers (Interaction, Orchestration, Execution) so the executor can be switched without changing workflow semantics or the audit view, and each task goes through Plan, Execute, Verify and Write-back. Skills reach a task through a stage-skill map keyed on the task's stage and type, keyword detection over user instructions, or manual invocation, and every action must belong to the action set allowed by the current permission policy.
    • Implementation: The deployed skill catalogue holds 171 skills, of which 58 are mapped to five research stages (survey, ideation, experiment, publication, promotion). Moving to a new domain means editing one JSON map instead of code, backends attach through Claude and Codex SDK (software development kit) adapters, and pipeline mutations are written to persistent state first and then broadcast to the interface over WebSocket. Per the official README, the release includes a React and Vite frontend, a Node.js and Express backend, executor integrations for Claude Code, Gemini CLI, Codex and OpenRouter, a Chat flow that generates the research brief and task list with a skill, a Research Lab to review them, and Auto Research for one-click sequential task execution.
    • Results: Against a bare agent using the same executor (codex provider, gpt-5.4) on three medical tasks (melanoma and nevus classification on Derm7pt and a clinical-note risk baseline), pooled completeness, the share of 21 research best-practice elements present in the output files, was 0.873 for the bare agent and 0.952 for Dr. Claw (two wins, one tie), and the whole gap came from research hygiene elements (basic parts of a research report such as a limitations section, subgroup analysis and real literature citations), whose pooled pass rates rose from 0.33 to 1.00, 0.33 to 1.00 and 0.00 to 0.67. In a demonstration that injected a wrong-path error into one Derm7pt mini-project, the project recovered to a result with 0.892 accuracy while keeping all 5 earlier files. In a retrospective study of seven AI PhD researchers, Dr. Claw was associated with shorter completion-time bands and the highest output-quality ratings of the three conditions compared with working without AI or with general web and desktop AI assistants, and the experience score difference was significant in every pairwise comparison.
    • Limits: The authors call the evaluation a small exploratory pilot with few tasks and participants: with one run per task the pooled completeness gain of +0.079 has a 95% bootstrap CI (confidence interval) of [-0.00, +0.14] that includes zero, and the human study collected the control conditions as retrospective reports in a hybrid design. Holding the executor fixed adds the task graph, state objects and skill library as one bundle, so no single component's effect can be isolated; the comparison target is the bare executor Dr. Claw wraps instead of a state-of-the-art orchestration framework; and completeness counts which of 21 elements are present, so correctness needs separate review. Tasks come from one domain (medical), Dr. Claw was slower than the bare agent, and skill invocation is guaranteed only up to a suggestion in the task prompt, so the reference audit did not fire on the clinical-note task.
  3. 1277Research Agents

    How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks

    • Problem: Existing research-agent benchmarks either use simulated worlds or, when grounded in real science, restrict themselves to verifiable settings, most commonly machine learning and software engineering tasks with a well-defined target. Nearly all of them anchor scoring at the endpoint, so a single number hides where and why an agent broke down. Among prior failure diagnoses, expert case studies cannot generalize across systems or tasks, categorization efforts come from small corpora, and trace-level analyses such as MAST, TRAIL and Who&When annotate conversational or tool-call traces, so failures outside those traces stay invisible.
    • Input and output: This is a benchmark paired with a failure taxonomy. The agent receives only the Premise and Tension extracted from a published paper, works alone through six stages in a sandbox with code execution, and delivers a decision.json plus a report.md with an appended Peer Review section. On the evaluation side, a judge reads the whole trajectory and outputs a stage-by-stage analysis.md with ARFT pattern labels.
    • Method: From 5,878 candidate papers the authors built 100 runnable tasks: 70 open-ended discovery tasks with no metric to aim at, and 30 target-anchored optimization tasks with a human SOTA (state of the art) or a computable metric. Three experts induced ARFT bottom-up from full trajectories, and it sorts 45 patterns by the stage where a failure surfaces (A to F, plus a cross-stage layer X) and by four root causes.
    • Implementation: The eight agents are Claude Code driving opus-4.8, claude-sonnet-5, qwen3.7-max, glm-5.2, minimax-m3 and deepseek-v4-pro, Codex driving gpt-5-mini, and Gemini CLI driving gemini-3.5-flash; the 800 trajectories hold 73k tool calls at 92.3 steps per episode on average. The judge is a Claude Code agent running claude-opus-5 that opens the code, execution logs, final reports and data files itself. Per the README, the GitHub repository holds the task-construction pipelines and the agent-as-a-judge code and leaves out the rollout infrastructure, while the paper states that tasks, trajectories and annotations are released and lists a Hugging Face dataset as the data address.
    • Results: On 50 trajectories with human expert labels, the agent judge reached a Cohen's kappa (an agreement measure) of 0.75 at the pattern level and 0.83 at the taxonomy level, against 0.53 and 0.62 for a single-call LLM judge that reads only the transcript. Across 800 analyses the taxonomy recorded 12,712 hits; Grounding & Faithfulness (31.0%), Scientific Integrity & Alignment (33.5%) and Cognitive Depth & Adaptability (27.6%) together made up 92.1%, and Engineering Robustness 7.9%. The most frequent pattern, F.4, appeared in 660 of 800 analyses (82.5%), and total hits across the eight models ranged from 1,396 for opus-4.8 to 1,818 for qwen3.7-max.
    • Limits: ARFT comes from trajectories of eight combinations on this task suite, and the authors present the failure frequencies as indicative. All runs had fixed wall-clock and token budgets, so patterns such as premature termination and shallow search may correlate with resource pressure, and data contamination (the source papers appearing in model training data) cannot be fully excluded. Judge validation is reported only in aggregate over the 50-trajectory sample, and whether orchestration-level interventions can close the metacognitive gap is left as an open question.
  4. 1276Research Agents

    Scaling Automatic Research Agents via World Models

    • Problem: Training AutoResearch agents (LLM agents that take a research question, form an idea, implement the experiment, analyze the outcome, and iterate) with RL needs many trajectories. Generation shares compute through batching, but every candidate solution needs its own isolated sandbox that loads data and trains models on real GPUs, so execution cost grows linearly with the number of trajectories and becomes the dominant bottleneck for scaling RL.
    • Input and output: This is an RL training method paper with convergence theory and experiments. In the AutoResearch tasks, the world model takes the task description and the agent's solution code, predicts the execution outcome, and aggregates strict JSON verdicts from eight checks such as imports and names, API calls, file and column references, and the compute budget into a score. The final output is the post-trained agent policy, and AutoResearch evaluation uses the percentile on real Kaggle competition leaderboards.
    • Method: About 10% of groups are anchor groups graded by both the world model and real execution, and these score pairs form the thin stream of ground truth kept during training (the anchor signal). Online Debiasing fits a monotone map to the score pairs with isotonic regression (regression constrained to a non-decreasing function) to remove the world model's bias, and Inverse-Variance Denoising combines the policy gradients of the two group types weighted by inverse variance to suppress noise. The authors prove that bias and noise each add an error term to the convergence bound and that the two corrections shrink these terms, strictly improving the convergence guarantee.
    • Implementation: In the AutoResearch experiments, Qwen3.5-4B and Qwen3.5-9B agents are trained with GRPO (group-relative policy optimization), and the world model is the same backbone used through prompting only and kept fixed during training. Training uses 45 MLE-Dojo competitions (Kaggle machine learning competitions wrapped as interactive environments), evaluation uses 14 held-out competitions and 60 tasks from DSBench (a data science benchmark built on Kaggle modeling tasks), and the vision-language-action (VLA) experiments use the MiniVLA-1B policy with the progress model Robometer as the world model. According to the official repository README, the release contains the method in numpy (wmrl/), the MLE-Dojo and DSBench pipeline (ml_research/), the VLA track (embodied/), and tests; it is a clean reimplementation of the production stack used for the paper and implements only WMRL itself.
    • Results: WMRL cut training compute relative to real-execution GRPO from 883 to 286 GPU-hours at 4B and from 1174 to 349 GPU-hours at 9B, a 3.1x and 3.4x reduction. Average leaderboard percentile at 9B rose from 18.8 to 21.6 on MLE-Dojo and from 31.2 to 32.8 on DSBench, and on both benchmark averages the 9B WMRL agent scored above Nemotron-120B-A12B (20.5, 31.7) while the 4B WMRL agent scored above Kimi-48B-A3B. On LIBERO-Long (a simulated robot-arm manipulation benchmark), the overall success rate rose from 37.4% after SFT to 41.2% with WMRL, a 3.8 point gain.
    • Limits: The authors assume tasks whose outcomes are predictable from the agent's solution and where a small stream of real execution stays available as an anchor, and they note that when outcomes hinge on randomness that no reading of the solution reveals, WMRL degrades toward standard GRPO. The convergence guarantee relies on the assumption that the bias acts as a monotone distortion of the score, the AutoResearch experiments cover two Qwen3.5 scales (4B and 9B), and the MLE-Dojo tasks were chosen from the 20 smallest competitions by dataset size per category to fit the sandbox.
  5. 1275Research Agents

    CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists

    • Problem: Existing LLM causal reasoning benchmarks generally turn causal graphs, datasets or narratives into question answering and classification tasks and usually provide the variables, evidence, graph or query up front, leaving the 'causal parrot' concern that models succeed through memorized causal facts or linguistic cues. The interactive Auto-Bench primarily asks whether an agent discovers a hidden DAG (directed acyclic graph), and the authors argue that evaluation should also measure whether the discovered mechanism transfers to a new instance.
    • Input and output: This is a benchmark environment for interactive causal discovery by LLM agents. In each episode the agent receives prior measurement records generated by a randomly sampled hidden SCM, intervenes on one controllable property of a manipulator crystal at a time within a fixed budget and observes the results, and then predicts the resonance frequency of a separate reactor crystal governed by the same SCM. At every step it emits a DSL record with five fields, memory, thought, past data, hypothesis and the next planned experiment, and the hypothesis is parsed into a graph, a frequency equation and coefficients for scoring.
    • Method: A DAG is sampled over the observable properties and frequency, root nodes take exogenous source values and the other variables are computed in topological order with linear or quadratic structural equations; the graph, equations and coefficients are shared by the records, the manipulator crystal and the reactor crystal, while controllable-property base values differ. Interventions are shift-style, replacing the variable's base term with v and keeping parent contributions, in place of hard do(X=v), and some graph families add a hidden disturbance H that is resampled after each intervention and added to a subset of variables. Scoring reports reactor frequency accuracy separately from graph precision, recall and F1, SHD (structural Hamming distance, the count of missing, extra and reversed edges), frequency-coefficient F1 and root-node identification.
    • Implementation: The authors built a new SCM generator, intervention and reactor mechanics, DSL logging and scoring code on top of the 2D game-style DiscoveryWorld environment, and a DSL record that fails the schema check is re-prompted up to two times with the parser error. The main suite ran four models, GPT-5-mini, GPT-5.2-high, Qwen3.5-Thinking and Qwen3.5-Non-thinking, on graphs with 3 to 7 nodes, with up to 50 topologies per (graph size, model) cell and one run per task, at temperature 0.1 and a fixed budget of 2 observations and 4(k-1) interventions for a k-node graph. Observation-intervention scaling used GPT-5-mini and GPT-5.2-high on 4-node and 6-node graphs, and the other follow-ups primarily used GPT-5-mini.
    • Results: With the 50 four-node topologies fixed, replacing the linear mechanism with a hard-quadratic one cut GPT-5-mini accuracy from 48% to 24% and frequency-coefficient F1 from 0.589 to 0.251, while root-node F1 rose from 0.559 to 0.829. For GPT-5.2-high on 6-node graphs, 20 observations with no intervention gave 92% accuracy and 0.471 all-edge F1, while 5 observations plus 20 interventions gave 80% and 0.799, and a Golden intervention chain that leaves a small set of candidate graphs explaining the data (Markov equivalence class) raised GPT-5-mini 4-node accuracy from 48% to 90% but lowered all-edge F1 from 0.793 to 0.728. Among the four models GPT-5.2-high had the best accuracy and lowest SHD at every graph size, yet it reached 64% accuracy and SHD 4.761 at 7 nodes, and adding one verification step before answering raised GPT-5-mini 4-node accuracy from 48% to 60%.
    • Limits: The authors frame CausaLab as a controlled synthetic benchmark whose 3 to 7 node SCMs with mostly linear mechanisms and one hard-quadratic extension cover part of the causal structures, functional forms, latent variables and measurement processes found in real scientific domains. Experiments cover a finite set of LLM agents, prompts and interaction budgets, performance may differ with other models, tool interfaces, decoding policies or longer exploration budgets, and the main analyses focus on final predictions and final hypotheses. Interventions are shift-style in place of perfect hard-do operations, the main suite ran each task once, and the authors name overgeneralizing success on templated SCM tasks to real scientific, medical or policy settings as the main risk.
  6. 1274Research Agents

    SciAtlas: A Computable Atlas of Science for Knowledge-Grounded AI Research

    • Problem: Most AI scientists gather scientific knowledge through retrieval procedures designed around individual workflows, and even when agents iterate, these procedures usually leave a transient collection of documents used for one task. The authors see this as a lack of breadth, depth and standardization, and note that existing scientific knowledge graphs are often bounded by particular disciplines, information objects or schemas, which limits their shared use across research workflows.
    • Input and output: This is a system paper that builds a scholarly knowledge graph, a shared retrieval mechanism and three research workflows. It takes a research object such as a topic, research question, seed paper or idea, grounds it in seed nodes of the graph, and returns ranked paper and author pools. Research Development Trajectory (RDT) and Research Opportunity Discovery (ROD) mainly use the paper pool and Innovation Assessment (IA) also uses the author pool, producing respectively a review outline organized by period and research thread with section-level evidence packs, a motivation, mechanism, falsifiable hypothesis and phased experimental plan, and reviewer-specific assessments, a meta-review and revision advice.
    • Method: Under one schema, the graph centres on papers and holds five layers: citation and publication relations, normalized keywords with co-occurrence relations, a topic-to-domain disciplinary hierarchy, authorship, affiliation and co-authorship relations, and peer-review and evaluation signals attached where available. For retrieval, an LLM parser splits the input into typed query units such as concepts, papers, authors and topics, symbolic indexes over names and identifiers and approximate nearest-neighbour indexes over dense embeddings pick seed nodes, and relevance propagates over a query-centred local graph expanded for a fixed number of hops by mixing a restart term back to the seeds with diffusion from neighbours until scores converge or reach the iteration limit. On top of this, RDT diagnoses missing periods and research threads to repeat retrieval rounds, ROD condenses field bottlenecks into a Research Structure Signature to find same-field and cross-domain mechanisms, and IA splits an idea into motivation, method, experiment and impact claims to build claim-level evidence, reviewer profiles and rubrics.
    • Implementation: The primary source is OpenAlex, and the normative layer was filled by an automated search agent that located public peer-review pages and attached them to papers. The resulting graph spans 26 disciplines and links more than 157M entities through 3B relational triples, including over 43M papers, and configurable LLM components in the three workflows and the foundation evaluations used GPT-5.5. Per the README, the GitHub repository holds a pip-installable client and CLI for the hosted SciAtlas API, workflow code for literature review, idea evaluation and idea generation, and an Agent Skill pack for coding agents such as Codex and Claude Code, while graph retrieval runs through the hosted API with a personal API token.
    • Results: On SurveyLens, made of 1,000 human-written surveys in 10 disciplines, SciAtlas-assisted RDT reached a macro-average Rubric Score of 4.82 against 4.67 for the strongest baseline, SurveyForge, and on LiveIdeaBench, made of 1,180 keywords across 22 domains, ROD scored a mean of 7.38 against 7.12 for the strongest baseline, AI-Scientist. On SciAtlasEval-Idea with 349 ideas, where three LLM judges compared reports pairwise, IA had mean win rates of 99.6% against CoT (chain-of-thought prompting), 96.6% against RAG (retrieval-augmented generation), 74.5% against DeepReviewer, 78.5% against NoveltyAgent and 68.2% against ScholarEval. On 200 SciAtlasEval-HQR (Heterogeneous Query-Object Retrieval) cases with direct query-target edges masked, nDCG@20 (ranking quality of the top 20 results) was 0.51, nearly three times the 0.18 of the strongest baseline, and TaxoBench paper-retrieval F1 (the harmonic mean of precision and recall) was 32.8 against 21.8 for the strongest baseline.
    • Limits: The authors note that the graph is built by an automated pipeline but updated periodically, so emerging work may enter late, coverage depends on the underlying scholarly records, and the availability of expertise and evaluation signals can vary across disciplines and venues. The study focuses on knowledge-intensive stages and leaves closed-loop experimental validation out of scope, and representing procedures, conditions, measurements and outcomes as explicit graph objects remains future work, so the claims concern scientific-context construction and knowledge-grounded decision support. The graph's scale demands substantial storage, indexing and retrieval infrastructure, so propagation is restricted to query-centred local neighbourhoods; SciAtlasEval was curated by the authors, and correctness gains on the single-paper PubMedQA and SciFact tasks of ScholarQABench were comparatively small.
  7. 1273Research Agents

    Intern-Atlas: A Methodological Evolution Graph as Research Infrastructure for AI Scientists

    • Problem: Scholarly platforms such as Google Scholar, Semantic Scholar, and OpenAlex treat the paper as the unit and link papers by citations, so whether a citation extends a method, resolves its bottleneck, or is mere background stays buried in text. The authors argue that humans reconstruct this lineage in their heads, while AI research agents cannot reliably reconstruct method evolution from unstructured text.
    • Input and output: This is a dataset paper that also proposes three operators running on top of the graph. The input is 1,030,314 AI papers from 1965 to 2025, and the output is a graph with 8,155 method nodes, 9,545 aliases, and 9,410,201 typed edges, where every non-background causal edge carries verbatim quotes as evidence. Given a method name, an idea description, or a research question, the operators return evolution chains, five-dimension scores, or a new research proposal with evidence.
    • Method: An LLM classifier reads the text around each citation and labels the edge with one of seven types (extends, improves, replaces, adapts, uses_component, compares, background), and the first four form the strong-causal subgraph used for lineage search. Each causal edge gets bottleneck, mechanism, and trade-off spans quoted verbatim from the citing paper plus a confidence, and a deterministic checker discards edges whose quotes fail substring match or whose endpoints violate publication-year order. Lineage search uses SGT-MCTS (Self-Guided Temporal Monte Carlo tree search), which adds an edge-confidence and year-gap prior to UCT (the MCTS selection rule that balances high-scoring and under-visited branches), idea evaluation computes five dimension scores from graph statistics instead of LLM judgment, and idea generation requires every proposal to cite the stored bottleneck quote of a real edge.
    • Implementation: PDFs are parsed with the Nougat v1.0 parser, with the GROBID v0.8.0 parser as the fallback for the 3.7% of documents where Nougat fails, and extraction reads only the Introduction, Method, and Related Work sections of each citing paper. Both extraction phases use Qwen3.6-35B-A3B at temperature 0, with a separate audit model used only in the reliability benchmarks. The authors state that they release the graph and pipeline as open infrastructure.
    • Results: On a benchmark built from 30 surveys (2,268 nodes, 1,462 edges, 133 chains), Intern-Atlas reached a node match ratio of 91.0%, an edge reachable ratio of 89.7%, and a path semantic correctness of 92.0%, and SGT-MCTS reached a node recall of 84.8 against 44.9 for the strongest baseline, Beam@10 (beam search with width 10). On 1,200 papers, mean idea scores followed the strata order of 8.48 for top-tier conferences, 7.83 for core conferences, 6.85 for workshops, and 5.84 for rejected submissions, and on 100 ideas rated by 10 experts the overall Spearman correlation was 0.81 against 0.58 for a pure LLM judge. Ideas generated for 100 research questions scored 7.20 overall, 1.02 points above the best of four baselines (Semantic Scholar, 6.18), and won 88.0%, 82.0%, and 81.0% of blind expert pairwise comparisons against No-KB (generation without external knowledge), OpenAlex, and BM25 RAG (generation grounded in papers found by BM25 keyword retrieval).
    • Limits: The authors report Phase-1 edge-type accuracy of 70.4% for the production model and 93.0% for the audit model, reflecting how hard fine-grained relations such as extends versus improves are to separate. The 14-axis bottleneck taxonomy is fixed at release, the temporal coherence function (the weight given to publication-year gaps) is calibrated on post-2015 AI literature and needs recalibration for fields with different research cadences, and the generated ideas were scored with the same Intern-Atlas evaluation pipeline.
  8. 1272Information Retrieval

    DiagramBank: A Quality-Audited Dataset of Scientific Schematic Diagrams with Multi-Level Document Context

    • Problem: Schematic diagrams in papers convey methods, workflows, and system structure, yet existing scientific-figure datasets often mix them with plots, screenshots, and photos and often omit the body text that explains each figure. The authors argue for a resource that identifies schematic figures and keeps paper and figure context, provenance, and filtering uncertainty together.
    • Input and output: This is a dataset paper, and the authors limit the contribution to the resource itself. The input is PDFs and metadata of AI/ML venue papers hosted on OpenReview, and the output is records for 57,100 schematic diagrams. Each record carries the figure image, paper title, abstract, caption, body paragraphs citing the figure, venue and year, provenance, and filtering labels.
    • Method: PDFs and metadata come from the OpenReview Python API, figures and captions are extracted with PDFFigures 2.0, and body paragraphs citing the figure number are extracted with PyMuPDF. OpenCLIP ViT-B/32 first sorts figures into diagram, plot, photo, and other, and a VLM cascade selects the final diagrams through votes from two smaller VLM judges and confirmation by a larger VLM. Candidates that CLIP labels as diagrams enter a majority-vote path, and non-diagram candidates with CLIP confidence below 0.70 enter a rescue path.
    • Implementation: Relational data are managed in DuckDB, and records are denormalized so that each figure directly holds its title, abstract, caption, context, quality labels, and provenance. The release includes an all-CLIP view and a view with CLIP confidence of at least 0.85 alongside the cascade-filtered release. The official repository README describes a download script that fetches data.jsonl, FAISS indices, and DuckDB from Hugging Face and a demo notebook that finds similar diagrams from a title, abstract, and caption, and it states that the reproduce scripts cover OpenReview collection through first-stage CLIP classification.
    • Results: A blind human check of a stratified sample of 500 records from the cascade-filtered release estimates precision at 93.67% (95% confidence interval 90.11 to 97.22), with screenshots or photo-like figures as the main error. In the CLIP view check, precision is 94.4% at a threshold of 0.85, and raising the threshold from 0.50 to 0.90 lifts precision from 64.0% to 96.0% while retention falls from 100% to 55.7%. In a retrieval check with 20 queries, Precision@10 (the share of the top 10 retrieved diagrams that VLM judges rate as relevant) is 78.0% for hierarchical retrieval and 81.0% for caption-only retrieval.
    • Limits: The authors state that relying on automated PDF parsing, figure classification, and context extraction can leave records with imperfect captions, missing body-reference paragraphs, or diagram versus plot classification errors. The release favors precision and may drop useful lower-confidence diagrams, and coverage is limited to accessible OpenReview venues and years, which may tilt the resource toward recent AI/ML practice. The retrieval check is a sanity check at the scale of 20 queries, the authoring case is qualitative, and the authors say the VLM judge model names are kept in the release documentation.
  9. 1271Research Agents

    AI scientists produce results without reasoning scientifically

    • Problem: LLM-based scientific agents (AI scientists) increasingly run research on their own, yet benchmarks for them mainly score task completion, such as answering domain questions or finishing a workflow. Such scores show whether an agent succeeded but not how it got there or how it will behave on new problems, so the authors measure the reasoning process itself.
    • Input and output: This is an empirical study. The inputs are tasks in 8 scientific domains: workflow tasks (molecular simulation, adsorption surface construction, ML-based property prediction, atomic force microscope experiments), a strategic task (retrosynthetic planning), and hypothesis-driven tasks (spectroscopic structure elucidation, inorganic qualitative analysis, circuit inference). The outputs are scores from more than 25,000 agent runs, diagnostic question scores, and graphs of reasoning traces made of nodes and edges.
    • Method: The authors model an agent as a language decision process, a partially observable Markov decision process with text observations and actions, whose policy combines a base-model part and a scaffold part, and they cross 3 models (GPT-4o, Claude Sonnet 4.5, GPT-OSS-120B) with 2 scaffolds (ReAct and structured tool calling) on the same environments and tools. They fit item response theory (IRT) to per-domain knowledge and reasoning questions to estimate abilities, then split the variance in success with M7, chosen by PSIS-LOO (an approximation of leave-one-out cross-validation) among Bayesian models M1 to M8. Each trace step is labeled as hypothesis, evidence, test, judgment, update, or commitment, and the share of traces showing productive motifs and reasoning breakdowns is counted on the resulting graphs.
    • Implementation: The authors built Corral, an evaluation framework that separates environments from agents through an HTTP interface, and its environments rest on LAMMPS simulations, a live atomic force microscope, a custom thermodynamic equilibrium engine, and manually curated spectra, covering more than 15 environment scopes and more than 90 tools. The paper states that tasks, run reports, traces, and question data are public in a Hugging Face collection. The official repository README describes a corral command-line interface (CLI) with run and bench commands and a Python API, the afm, corral_md, resistor_network, retrosynthesis, spectra_elucidation, and wetlab environments plus a lightweight math environment called samplemath, and agents such as Reflexion, LLMPlanner, and AI Scientist in addition to the ReAct and tool calling agents used in the paper.
    • Results: In the selected M7 model, reasoning ability accounts for 41.4% of the explained variance and environment with scope for 30.1%, while the scaffold accounts for 1.5% and tool description verbosity for 0.1%. Gathered evidence goes unused in 68% of traces, untested claims appear in 53%, and refutation-driven belief revision appears in 26%. In spectroscopic structure elucidation, inorganic qualitative analysis, and retrosynthetic planning, injecting early successful steps does not raise performance, and gains appear only with near-complete successful trajectories.
    • Limits: The authors treat each task as an independent episode, test two simple scaffolds (ReAct and tool calling), and leave advanced orchestration such as plan-then-execute pipelines, hierarchical planners, and multi-agent setups out of scope. They use 3 models, and the main evaluation fixes a budget of 20 to 40 LLM calls per task and temperature 0.0, and the log-probability analysis covers only GPT-OSS-120B. GPT-OSS-120B produced malformed responses in 41% of traces, and the authors measured them without fixes such as constrained decoding.
  10. 1270Research Agents

    An Axiomatic Benchmark for Evaluation of Scientific Novelty Metrics

    • Problem: Automatic metrics for a paper's novelty help AI scientists avoid spending compute on already explored ideas, yet existing metrics are generally validated against confounded signals such as citation counts or review scores. When the authors take the 100 highest- and 100 lowest-rated papers with strong reviewer consensus from ICLR 2022 and 2023, these novelty buckets match the accept/reject decision for 93.5% (technical novelty) and 97.0% (empirical novelty) of papers, so reviewer novelty scores are entangled with quality.
    • Input and output: This is a benchmark that compares novelty metrics without ground-truth novelty labels. A metric under test receives a focal paper and a reference pool of prior papers, the pool is changed in controlled ways, and each focal paper counts as a pass or fail depending on whether the score moves in the expected direction, giving a pass rate per probe. The benchmark consists of 10 tasks in three domains (NLP, computer vision, and biomedical AI), with 100 focal papers randomly sampled per task.
    • Method: Three axioms are tested through probes. The Redundancy axiom says the score must fall as the pool covers more of the paper, probed with an exact copy, a paraphrase, and planting 25% to 100% of the paper's atomic claims into separate pool papers. The Relevance axiom says the score must rise when the pool is swapped for another field or the paper's cited works are removed, and the Temporality axiom says the score must fall as the pool's time window moves later.
    • Implementation: The data comes from the PapersWithCode archive on Hugging Face (dated July 29, 2025), using tasks with roughly 1,500 tagged papers each, with references fetched through the Semantic Scholar API. The 10 evaluated systems are the embedding metrics Yin, RND (relative neighborhood density), SemNovel, and FastTextLOF, plus NovaScore, the AI Scientist judge, RAG-Novelty, ARC (the lexical novelty gate of AutoResearchClaw), and the Si pairwise tournament and OpenNovelty, which give rankings or text instead of scores; systems that retrieve through tools are sandboxed so retrieval reaches only the constructed pools. The authors state that they release arXiv IDs, derived files, and scripts that fetch paper texts, and API usage totaled just under $400.
    • Results: Embedding metrics such as Yin and RND catch nearly all exact copies (1.00, 0.99) and paraphrases (0.99, 0.97) but reach only 0.66 and 0.53 when 25% of the claims are planted. RAG-Novelty, where an LLM reads the paper with retrieved neighbors, passes only 0.28 even on the exact-copy probe and fails every redundancy probe. On the monotone temporal probe the best of the eight scoring systems reaches 0.24, and the authors conclude that surface redundancy is largely solved while conceptual redundancy remains open.
    • Limits: The authors note that pools hold only one to two thousand papers per task, so performance at the scale of the real literature cannot be tested, and that all tasks are AI research tasks. Sandboxing controls what an LLM judge retrieves but leaves the model's parametric knowledge in place, and since the axioms are necessary conditions, passing every probe does not guarantee a good novelty judge. The AI Scientist judge and the Si tournament were run on three tasks and OpenNovelty on two, and the Si tournament and OpenNovelty were tested only on the Redundancy axiom.
  11. 1269Research Agents

    Synthetic Sandbox for Training Machine Learning Engineering Agents

    • Problem: An MLE (machine learning engineering) agent must run a full ML pipeline of data preprocessing, model training, and metric evaluation at every rollout step, and a single code execution averages nearly 200 seconds on standard problems. This latency makes trajectory-level on-policy RL far too slow, so prior work retreated to SFT or step-wise RL with offline proxy rewards.
    • Input and output: This is a training method paper with an implementation and experiments. The input is 60 seed tasks taken from MLE-bench, and the output is a set of synthetic MLE tasks whose training data hold only 50 to 200 samples (data, hidden rules, evaluation scripts, task descriptions), together with Qwen3 policy models trained with RL on these tasks.
    • Method: A Data Strategist extracts a Structural DNA (the mathematical structure of the task stripped of domain context) from a seed task, adds a new domain and noise, and defines a hidden rule linking features to labels, while an ML Developer generates the data and trains and evaluates baseline methods to fix milestone thresholds. An MLOps (machine learning operations) Engineer writes an evaluator that scores against a hidden test set, a Technical Writer writes the task description, and only tasks whose thresholds keep the required order stay in training. Training is trajectory-level GRPO with ReAct rollouts, and the dense reward splits weights across format compliance, successful execution, and passing the median, bronze, silver, and gold thresholds.
    • Implementation: Starting from 1,200 synthetic tasks built from the 60 seeds, execution checks and threshold checks leave 912, of which 848 are used for training and 64 for validation. Qwen3-8B, Qwen3-14B, and Qwen3-30B-A3B-2507 are trained with the RLLM framework, with at most 20 turns and a 90-second execution limit per step during training and one NVIDIA H200 GPU per task. The reward weights are 0.1 for format, 0.3 for execution, 0.1 for median, 0.2 for bronze, 0.2 for silver, and 0.1 for gold.
    • Results: With code written by Gemini-2.5-Flash, the average execution time falls from 196.17 seconds on the original MLE-bench tasks to 14.31 seconds on the synthetic tasks. Measured with ReAct on MLE-Bench-Lite, the Any Medal rate rises from 13.6% (Base, and Seed-SFT trained by SFT on Claude-4.5-Sonnet trajectories) to 22.7% for 8B and from 13.6% for Base to 27.3% for 30B, while Claude-4.5-Sonnet reaches 31.8%. On MLE-Dojo, the 30B SandMLE model with the MLE-Agent scaffold reaches a HumanRank score (based on rank among human competitors on the leaderboard) of 38.56, above 29.12 for Base.
    • Limits: The authors report that in a test-time experiment with a growing turn limit, performance peaks at 30 turns and then drops because the context window overflows frequently. The 8B SandMLE model has a valid submission rate of 63.6%, below 68.2% for Base, and on MLE-Bench-Lite with the AIDE scaffold the 30B model's Any Medal rate stays at 13.6%, the same as Base. Evaluation covers MLE-Bench-Lite with 22 questions and MLE-Dojo with 62 tasks.
  12. 1268Research Agents

    RoboPhD: Evolving Diverse Complex Agents Under Tight Evaluation Budgets

    • Problem: Systems such as GEPA and Autoresearch let LLMs iteratively improve prompts, code, and agent architectures from evaluation feedback, but the authors state that no systematic comparison under identical tasks, budgets, and evaluation infrastructure exists. Evaluations may be expensive (for ARC-AGI, even at a cap of $0.25 per problem, 1,500 evaluations can cost up to $375) or the pool of examples may be small, so how the evaluation budget is allocated determines which agents can be found, and existing methods split it between training and validation, where, per the authors, validation evaluations inform selection but do not help improve the next candidate.
    • Input and output: This is a method paper with an implementation and experiments. RoboPhD takes an objective, task background, a seed agent (code and prompts), a pool of training examples, an evaluator that returns a score and diagnostics, and an evaluation budget, and searches for the agent with the highest expected score on unseen test examples. The output is an evolved agent program; on ARC-AGI, for example, a 22-line seed agent became a 1,013-line multi-strategy system.
    • Method: Each iteration, 3 agents compete on 20 freshly sampled training examples and pairwise comparisons update Elo ratings (K = 32), so every evaluation goes to training-time competition with no validation split. The evolution AI, which designs the new agents (here Claude Code with Opus 4.6), receives the Elo rankings, comparative error reports that point to problems where agents diverge, and per-problem diagnostics that the agents print() themselves, creates a new agent, and refines it in the same session against data from a prior iteration (Deep Focus). Six diversity mechanisms, such as a 200-point Elo penalty for clones whose predictions match a competitor, are included, and a 1,500-evaluation budget yields about 21 iterations.
    • Implementation: With a budget of 1,500 evaluations, the experiments cover four tasks: ARC-AGI (the solver, the model that solves the task, is Gemini 3.1 Flash Lite; 400 training and 400 test problems), Can't Be Late (a Python strategy class with no LLM that decides cloud spot instance use), Text2SQL on BIRD (solver Claude Haiku 4.5, co-evolving a database analysis script and an agent), and DocFinQA (solver GPT-4.1-mini with text-embedding-3-small, over 10-K annual reports filed with the SEC that average 123K words). The baselines, which receive the same diagnostics, are GEPA v0.1.1 with Opus 4.6 through the Claude API, an Autoresearch adaptation generalized to arbitrary artifacts with Opus 4.6 through Claude Code, and a King-of-the-Hill variant that pits the current champion against one new challenger each iteration. Per its README, the official andborth/RoboPhD repository releases the optimize_anything() API, per-domain examples that include the four tasks of the paper, and an engine option to choose GEPA or Autoresearch, and evolution requires the Claude Code command-line interface (CLI).
    • Results: On the test sets, default RoboPhD beat both GEPA and Autoresearch on three of four tasks: ARC-AGI 65.8% versus 58.5% and 54.2%, Text2SQL 64.5% versus 60.4% and 60.7%, and DocFinQA 50.4% versus 40.0% and 48.2%. On Can't Be Late, Autoresearch scored best at -87.6 against -90.7 for RoboPhD, and the King-of-the-Hill variant beat the default on ARC-AGI with 67.0%. Turning off Deep Focus lowered all four scores (DocFinQA from 50.4% to 41.2%), and shrinking the validation set from 200 to 100 raised test scores in all eight GEPA and Autoresearch comparisons.
    • Limits: On the simplest task, Can't Be Late (no LLM calls, solvable in under 90 lines), Autoresearch won, and default RoboPhD and the King-of-the-Hill variant were very close, which the authors say warrants further study. In the simulation by the authors, when three agents one accuracy point apart are compared on 20 problems, the best one ranks first only 45.0% of the time against a 33.3% random baseline, so selection relies on Elo accumulating weak signals over iterations, and non-determinism in LLM responses may cause variation across runs. During development, the Autoresearch adaptation found an exploit that read future spot availability from the internal state of the simulator and scored -85.4, so the authors redesigned the task API and stress careful sandboxing between the evaluator and the agent.
  13. 1267Research Agents

    FlowPIE: Test-Time Scientific Idea Evolution with Flow-Guided Literature Exploration

    • Problem: In scientific idea generation (SIG), most existing methods use a two-stage structure that retrieves literature and then generates ideas, and this pipeline typically retrieves literature once by keywords and semantic similarity. As a result, they receive contexts that are only topically similar and frequently produce similar ideas. The authors argue that such designs risk trapping the LLM-based generator in an information cocoon bounded by its internal knowledge and fixed retrieval results.
    • Input and output: This is a method paper with an implementation and experiments. The input is a research topic or query, and the output is a set of structured ideas made of Motivation, Method, and Experimental Plan. It uses patents from the USPTO (United States Patent and Trademark Office) instead of papers as the literature source and builds a literature graph from citations, shared core technical features, and semantic similarity between patents.
    • Method: Stage 1 treats the query as the root, explores the patent graph with a flow-guided MCTS inspired by GFlowNets, generates an idea for each path, and feeds the score from an LLM-based generative reward model (GRM) back into the flow probabilities of that path. Stage 2 repeatedly applies crossover that mixes the core technical features of two parent ideas, isolation island mutation that pulls literature from distant subgraphs, and tournament selection based on GRM fitness to the initial idea population.
    • Implementation: All methods use GPT-4o-mini as the idea generator, DeepSeek-V3.2 serves as the GRM and fitness evaluator, and patent embeddings come from BGE-M3. The literature data are 16,050 USPTO patents published in the last three months of 2025, and evolution runs for at most 20 iterations and stops when the standard deviation of fitness scores falls below 0.05. The official repository README presents the repository as the paper's evaluation code and lists run commands for Phase 1 (flow-guided MCTS) and Phase 2 (test-time evolution), while the literature database and a software demo are marked as coming soon.
    • Results: On AI Idea Bench 2025 IMCQ, FlowPIE scores 0.780 for motivation and 0.635 for experiment plans, ahead of the strongest baseline VirSci (0.560 and 0.580), and its I2I motivation score (agreement with the target paper's idea) is 4.44. The average reward from the DeepSeek-V3.2 GRM is 0.76 for FlowPIE against at most 0.63 for baselines, and FlowPIE also has the highest average human evaluation score among the compared methods at 0.39. The reward shows Spearman correlations of 0.60 for novelty and 0.87 for feasibility with average human scores.
    • Limits: The authors state that absolute performance stays bounded by the capability of the base model and that differing idea formats across studies make evaluation unstable. The reward evaluation uses the same GRM (DeepSeek-V3.2) as the fitness computation during evolution, and idea quality is measured by LLM judge scores, similarity metrics, rewards, and human ratings. On IdeaBench, Chain-of-Ideas scores higher on Novelty Insight (0.926) and Research Agent scores higher on Feasibility Insight (0.138) than FlowPIE.
  14. 1265Research Agents

    Towards a Medical AI Scientist

    • Problem: The authors argue that existing AI Scientists are largely domain-agnostic and have shown promise in fields with relatively standardized problems and evaluation, such as mathematics, chemistry and general machine learning, but are hard to apply to clinical research that needs medical evidence and specialized data. They name three gaps: ideas that ignore medical priors such as diagnostic workflows and disease-specific pathology, unreliable experiments on heterogeneous medical data such as 3D structures, and manuscripts that largely overlook data provenance and ethics statements.
    • Input and output: This is a system paper that presents Medical AI Scientist, a multi-agent system automating clinical medical AI research, together with the Med-AI Bench benchmark. In Reproduction and Innovation modes the system takes task instructions, dataset information and reference papers, while Exploration mode takes only task instructions and dataset information. It outputs a research idea, executable experiment code with results, and a typeset manuscript.
    • Method: In the Idea Proposer, a Preparer and Surveyor collect reference papers and code and break them into formula and code components, and a Generator forms hypotheses through clinician-engineer co-reasoning, while in Exploration mode an Analyzer first lays out the core clinical and technical challenges and an Explorer finds fitting recent paradigms with code. An Assessor checks scientific quality and ethics, returning hypotheses below the quality threshold for refinement and rejecting unethical ones, and in the Experimental Executor an Investigator gathers code and medical toolboxes, a Planner writes the execution protocol, an Executor runs training and evaluation in a Docker environment, and a Judger reviews the results and returns corrective feedback. The Manuscript Composer follows the structure of reference papers, an Ethics Reviewer reports dataset origin and ethical approval, and LaTeX compile errors are fixed automatically.
    • Implementation: All agents use a general-purpose LLM such as GPT-5 as the base model with role-specific prompting, and the Analyzer retrieves literature through the OpenAlex academic search engine. Med-AI Bench covers 19 tasks across six modalities (medical images, medical video, electronic health records (EHR), clinical text, physiological signals and multimodal data); for each task three papers of different difficulty were taken from Google Scholar as 57 ground-truth papers, and each paper yields 3 cases with different input modes, giving 171 cases. To speed up automated experiments the datasets were randomly subsampled, and the public address given in the paper is a project homepage.
    • Results: In the expert evaluation of ideas from the Innovation and Exploration modes, three experts on a 5-point scale gave the system 4.40 and 4.32 for technical innovation and 4.65 and 4.68 for maturity across the two modes, above GPT-5 and Gemini-2.5-Pro. First-run experiment success over 57 research instances was 0.91, 0.93 and 0.86 in the Reproduction, Innovation and Exploration modes, against 0.72, 0.60 and 0.75 for GPT-5 and 0.40, 0.49 and 0.53 for Gemini-2.5-Pro under the same conditions. On the same medical task (diabetic retinopathy classification), the Stanford Agentic Reviewer scored the generated manuscripts 4.60 on its 0 to 10 scale against 4.86, 4.06 and 3.74 for human-written papers selected from MICCAI (Medical Image Computing and Computer Assisted Intervention), BIBM (Bioinformatics and Biomedicine) and ISBI (IEEE International Symposium on Biomedical Imaging), and separately one generated manuscript was accepted at ICAIS 2025 (International Conference on AI Scientists), which had 114 submissions and a 36.8% acceptance rate.
    • Limits: The authors report that generated method designs can at times become overly intricate, in which case implementation may simplify or degrade components and drift from the original design. They also state that experiments are confined to predefined datasets and that the generated methods still perform below the state of the art. The expert manuscript study used a single task, diabetic retinopathy classification, with 20 papers (5 generated and 15 human-written), and the generated manuscripts scored 3.44 on Coverage (breadth of experimental design) against the 3.68 the paper compares them with.
  15. 1264Research Agents

    DOVA: Deliberation-First Multi-Agent Orchestration for Autonomous Research Automation

    • Problem: For research queries that need academic database search, code repository analysis, cross-checking of model registries, and synthesis with citations, the authors see four limits of single-agent systems. They process information linearly and miss cross-domain links, accept initial findings without adversarial challenge, call tools on keyword patterns, and spend the same reasoning depth on trivial and complex queries, wasting tokens on the former and starving the latter.
    • Input and output: This is a system paper proposing the architecture and algorithms of DOVA (Deep Orchestrated Versatile Agent), a multi-agent platform that answers research queries. It takes a user query, a persistent user model, and conversation context, and returns an answer with source attribution drawn from ArXiv, GitHub, HuggingFace, and the Web, plus a confidence score. The same orchestration backend is exposed through a REST (Representational State Transfer) API, a CLI (command-line interface), a browser UI (user interface), and an MCP (Model Context Protocol) server, which the authors say offers five tools to Claude Code.
    • Method: Before any tool call, the orchestrator weighs the user's expertise, mentioned entities, the six most recent turns, and the available tools to decide whether external information is needed, and temporal words such as latest or recent, or years from 2025 on, always force tool use. Collaborative reasoning chains an ensemble in which agents solve the problem independently, a blackboard where they post evidence and votes, and iterative refinement through rounds of critique. Thinking budgets have six levels from OFF to XHIGH (65,536 tokens), chosen by task type, a complexity hint, and query length.
    • Implementation: The agent pool has five agents: ResearchAgent for MCP search, ProfilingAgent for the user model, ValidationAgent for code analysis and sandboxed execution, SynthesisAgent for writing with source attribution, and DebateAgent, where Bull and Bear agents argue for 2 rounds by default. The experiments use Claude Haiku 4.5 (Basic), Sonnet 4.6 (Standard), and Opus 4.6 (Advanced) by task tier. The official repository alfredcs/dova holds src, tests, frontend, infra, and dova-claude-plugin folders plus Docker configuration files, and its README describes the system as built on the AWS Strands Agents SDK and Amazon Bedrock AgentCore and lists biomedical sources such as PubMed, ClinicalTrials.gov, and PubChem in addition to the four sources in the paper.
    • Results: The authors state that the values in their ablation table (Table 5), which compares seven configurations, are expected relative performance based on architectural analysis. In that table the full DOVA has confidence 0.82 and source coverage 0.90, removing collaboration gives 0.68 and 0.65, and the single-LLM baseline gives 0.52 and 0.00. For token efficiency, based on the thinking budgets, classification drops from 16K to 1K tokens (94%) and summarization by 75% versus a fixed MEDIUM budget, and the authors put aggregate savings at 40 to 60% for workloads where 40 to 60% of queries are simple.
    • Limits: The authors state that the ablation rests on architectural analysis and that evaluation on standard benchmarks such as HotpotQA and MMLU is future work. Confidence is scored by the same LLM that produced the answer, and all agents share one LLM backbone. In-memory MMR (Maximal Marginal Relevance) search cost grows with the number of stored items and results, so very large memory stores need indexing.
  16. 1263Research Agents

    AI-Assisted Scientific Assessment: A Case Study on Climate Change

    • Problem: Current AI co-scientists are pipelines of parallel LLM calls that search hypothesis spaces at scale, and they have worked well on tasks such as drug discovery or code optimization, where repeatable experiments give immediate ground truth. Some questions in macroeconomics, epidemiology, and climate science allow few controlled experiments, and their ground truth is a state of knowledge reached through scientific consensus instead of measurement, so the authors ask whether AI can help with such verification-poor tasks.
    • Input and output: This is a single case study paper. The AI Assistant takes as input the current document, the papers in its evidence panel (uploaded directly or found through Google Search or a pre-collected corpus of about 1,660 AMOC papers), the chat history, and instructions that scientists attach to the whole document or to a highlighted text span. It outputs an outline draft, section drafts with in-text citations and references, revisions that follow the instructions, reviews of a revision based on rhetorical and argumentative structure, and a summary of changes shown at login.
    • Method: The work followed three phases modeled on assessments by the IPCC (Intergovernmental Panel on Climate Change). In Phase 1 the scientists edited an outline produced by the Assistant; in Phase 2 the Assistant expanded the outline into a full draft and, for each section, went through the whole AMOC corpus in batches to pick relevant sources and add citations; in Phase 3 it merged the sections into one report with a single reference list. AI contribution is measured with a sentence alignment computed by Gemini: Retention is the share of AI draft sentences whose theme survives in the final version, and Intervention is the share of final sentences with no counterpart in the AI draft.
    • Implementation: The system calls Gemini through Vertex AI, using Gemini 2.5 Pro in Phase 1 and Gemini 3 Pro Preview in Phases 2 and 3. About 1,660 AMOC papers were converted to Markdown by an LLM and split into 200-word chunks with 20 words of overlap, and retrieval combines 3072-dimensional gemini-embedding-001 embeddings with BM25, a classic keyword-matching retrieval method. The participants were 13 climate scientists with extensive experience as leads on IPCC and other institutional reports, and the authors describe the tool as a proprietary prototype.
    • Results: The 13 scientists finished the report through 104 versions in 46 hours and 33 minutes of total logged-in time, while participants estimated that a comparable task typically takes 100 to 200 person-hours. For the report, Retention of the AI draft was 93.9% and Intervention was 58.3% (98.8% and 37.2% for the outline), and the scientists overturned the AI claim of 'full scientific consensus' on AMOC weakening and lowered the confidence in anthropogenic attribution from Medium to Low. In the post-study survey, all respondents self-reported a speedup of at least 2x, and 31% reported more than 10x.
    • Limits: The authors state that the findings come from a single experiment in one domain (climate science), on one topic (AMOC), with one group of 13 experts, so there is no formal control condition such as a human-only baseline. The model switch between Phases 1 and 2 is a confound that cannot be resolved analytically, work time is approximated by logged-in time, and the revision attribution is an exploratory Gemini self-evaluation. The AI made a few errors where it picked a single number from a single source when asked to synthesize number ranges, and when pushed adversarially toward a conflicting study it gave that study disproportionate space, which the authors call obsequious behavior.
  17. 1262Research Agents

    Aster: Autonomous Scientific Discovery over 20x Faster Than Existing Methods

    • Problem: The author argues that existing autonomous discovery frameworks, which use LLMs to improve code iteratively, often need hundreds to thousands of iterations per discovery, which restricts them to problems with short evaluations. For tasks with long evaluations, such as multi-hour model training, running thousands of iterations becomes prohibitive.
    • Input and output: This is a system paper that introduces Aster, an AI agent available through a web interface and API at asterlab.ai (the author is at Aster AI Labs), and reports its results on several tasks. The user provides a task prompt, an initial program and an evaluation script, and Aster returns an improved program. The appendix lists the programs Aster produced (the overlap optimizer, the single-cell denoiser, the TriMul (Triangular Matrix Multiplication) kernel, the NanoGPT diff and the ZAPBench (Zebrafish Activity Prediction Benchmark) model).
    • Method: The loop described in the paper has an LLM write a new program from the existing program and the prompt, an evaluator score it, and a database store the result, then repeat. Aster takes the same inputs as existing frameworks (initial program, evaluator, prompt) and aims to cut the number of iterations needed so that evaluations lasting hours become tractable. The paper describes the method only at the overview level of Figure 2.
    • Implementation: The paper states model settings for two circle packing runs: the speed comparison with OpenEvolve used the same model mix (80% Gemini 2.0 Flash, 20% Claude 3.7 Sonnet), and the run that reached the known best used Aster's default model configuration, whose models are not named. Many evaluation scripts and prompts were adapted from the repository of TTT-Discover (the work behind the previous AI records in Table 1), TriMul ran on an NVIDIA H100, and the ZAPBench evaluation ran on an NVIDIA T4 GPU. The access route the paper gives is the web interface and API at asterlab.ai.
    • Results: On packing 26 circles, Aster with the same model mix passed the 2.634 that OpenEvolve reached after 115 iterations with 2.6353 in 5 iterations, and Aster with its default configuration hit the known best of 2.635983 in 6 iterations. It reached 0.380874 on the Erdős minimum overlap problem in 40 iterations (TTT-Discover 0.380876), 0.711 on single-cell denoising in 30 iterations (TTT-Discover 0.709) and 1114 µs on the TriMul kernel in 70 iterations (TTT-Discover 1161 µs), and it cut the NanoGPT Speedrun record from 96.8 to 95.2 seconds in 8 iterations. On ZAPBench it matched the best human model's MAE (mean average error) of 0.0182 after 34 iterations, which the author reports as 190 times less compute.
    • Limits: The speed comparison covers one task, circle packing, against one framework, OpenEvolve, whose 460 attempts generated 4 at a time were counted as 115 iterations. On ZAPBench Aster only matched the best human result, and the paper notes that the tree search system of Aygün et al. reached an MAE of 0.0176 under similar runtime constraints. The 70 TriMul iterations started from the best program of an earlier 94-iteration run on a different GPU image.
  18. 1261Reasoning / Survey

    A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents

    • Problem: LLMs reason well in closed settings such as math and code benchmarks. Techniques that make intermediate reasoning explicit, such as Chain-of-Thought prompting, decomposition and program-aided solving, often assume static contexts and short-horizon reasoning, and conventional LLMs lack mechanisms to act, adapt or improve in open environments where information changes. Earlier surveys mostly cover reasoning inside the model or agent architecture, so the authors reorganize the field around reasoning as the principle that links planning, action and learning.
    • Scope: This paper is a survey. It covers work up to 2025 on planning, tool use, search, feedback, memory and self-improvement in single-agent and multi-agent settings, and gives separate chapters to applications and benchmarks in math and coding, scientific discovery, embodied agents, healthcare, and web and research agents. The official repository (Awesome-Agentic-Reasoning) publishes a README with paper tables in the same taxonomy, a figs folder with figures and a materials folder with talk slides.
    • Taxonomy: To characterize environmental dynamics, the survey uses three layers: a foundational layer (single-agent planning, tool use and search), a self-evolving layer (feedback, memory and adaptation) and a collective layer (role division, collaboration and co-evolution among multiple agents). Each layer is split into in-context methods, which keep parameters frozen and scale inference-time compute and orchestration, and post-training methods, which move successful reasoning patterns into the weights with RL and fine-tuning. The formalization factors the policy on a POMDP (partially observable Markov decision process) into internal reasoning and external action.
    • Synthesis: Self-evolution is split by what changes: verbal (textual reflections and guidelines, Reflexion), procedural (a library of executable tools and skills, Voyager) or structural (the agent's own source code and architecture, AlphaEvolve). Feedback is split into reflective feedback (self-critique during inference), parametric adaptation (training feedback into parameters) and validator-driven feedback (resampling until a validator such as a unit test passes); the survey notes that the last kind is non-diagnostic, so it corrects individual outputs and leaves the reasoning unchanged.
    • Applications and benchmarks: For research agents, the survey covers systems such as The AI Scientist, which chains idea generation, code writing, experiment execution and result visualization, and CycleResearcher, where a reviewer agent's automated peer-review feedback and an iterative preference-training loop help the research agent refine its next drafts. Benchmarks are grouped into mechanism-level suites (tool use, search, memory and planning, multi-agent) and application-level suites, with examples such as ToolLLM-ToolBench, which collects 16,464 real-world APIs across 49 categories.
    • Limits: The authors list open problems in personalization, credit assignment across tokens, tool calls, skills and memory updates over long interactions, training and evaluating world models in non-stationary environments, multi-agent collaborative training, auditable latent reasoning, and governance for real deployment. They state that existing benchmarks and guardrails mostly test short-horizon behavior and that post-training of multi-agent memory remains largely unexplored.
  19. 1260Research Agents

    Why LLMs Aren't Scientists Yet: Lessons from Four Autonomous Research Attempts

    • Problem: The authors ask whether state-of-the-art reasoning LLMs can go from a research idea to a paper with a high degree of autonomy, minimal code scaffolding and the most basic tools. Various AI Scientist systems predefine domain-specific workflows or frame the problem in a system-specific way, Sakana's tree-search approach would have required complex meta-orchestration, and AlphaEvolve requires a verification metric set in advance by a human expert. The scope was limited to machine learning research because its experiments can be done entirely digitally.
    • Input and output: The system takes two papers from one subdomain, generates an idea, and aims to move through hypotheses, an experiment plan, code and experiment outputs to an ML paper manuscript. The paper is a case study of four attempts with the authors' own system: starting from more than 135 papers in three domains (world models, multi-agent RL, AI safety and alignment), four ideas were selected for end-to-end attempts.
    • Method: Six agents split the research stages: Idea Generation, Hypotheses Generation, Experiments Planning, Experimental Output Evaluation, Revision and Paper Outlining. Claude Code takes the plan.md and agent.md files from the Experiments Planning Agent, writes the code and runs it on Modal (a cloud execution platform); every artifact stays as a file in a per-idea repository, and each agent receives that repository state in its prompt and works with four tools: read file, write file, list files and llm search. The Experimental Output Evaluation Agent primarily checks hypothesis implementation fidelity and statistical validity, and when it signals failure the Revision Agent chooses between revising the idea, revising the hypotheses suite or requesting LLM mentor feedback.
    • Implementation: All six agents used Gemini 2.5 Pro, which the authors chose primarily for its long context, and Claude Code handled code implementation and paper writing with Claude Opus 4.1 and Claude Sonnet 4. Each domain started from about 45 to 50 papers and produced up to 15 ideas, which were shortlisted with four zero-shot reviewer prompts run on Claude Opus 4.1; the first three ideas tried were chosen with feedback from the seed papers' authors, and the last one through an internal review informed by the earlier attempts. Per the official README, the repository holds four idea folders (three failures, one success) and prompts for idea generation, hypotheses generation, experiment planning and paper creation, and the authors state they released only prompts and selected outputs instead of the full system architecture.
    • Results: Three of the four attempts failed at the execution or evaluation stage, and only AS-1 (semantic entropy for jailbreak detection) was accepted at Agents4Science 2025, a venue requiring AI first authors that accepted 48 of 254 valid submissions. The six failure modes the authors list are falling back to training data defaults such as outdated libraries, implementation drift, memory and context loss over long tasks, overexcitement, insufficient domain intelligence and weak scientific taste; for example, in WM-1, which tried to add differentiable tree search to a stochastic world model, Claude Code hit time limits and swapped the tree search for an actor-critic approach, and the experiment ran with one seed. From these they propose four design principles: Start Abstract, Ground Later; Verify Everything; Plan For Failure and Recovery; and Log Everything.
    • Limits: The authors state that the scope is limited to ML research run entirely in software, that four ideas across three subdomains were each pursued with a single implementation instead of a best-of-N approach, and that because conference submission was the main goal, design changes were not recorded as systematic ablations, so the changes behind one success and three failures are hard to isolate. The failure modes come from qualitative observation, and the release covers prompts and selected outputs instead of the full system architecture. AS-1's experimental design, execution and writing were rated at least 95% AI, yet humans intervened in idea review, paper writing and meta-prompting during execution, and most failed experiments were stopped by human decision.
  20. 1259Research Agents

    FML-bench: Benchmarking Machine Learning Agents for Scientific Research

    • Problem: Existing ML agent benchmarks mostly use Kaggle-style, application-oriented tasks that stress engineering execution such as feature engineering and standard model training, and they primarily assess final performance and computational cost. They pay limited attention to the ability to tackle fundamental ML research problems such as representation learning and generalization, and they limit analysis of how an iterative research process relates to its outcomes.
    • Input and output: This is a benchmark paper. The agent receives a task description, the complete code of a real research repository, files suggested for editing, protected evaluation code it cannot modify, a command list for running experiments, the baseline performance, and the target metrics, and through repeated cycles of proposing a hypothesis, editing code, and running the commands it is expected to propose a new or improved method with stronger empirical results than the baseline. A post-processing module converts each repository output into a standard format, and the Performance score is the best held-out test result across iterations.
    • Method: The datasets and baselines of the eight tasks are ColoredMNIST and ERM (Empirical Risk Minimization) for generalization, Mini-ImageNet and Prototypical Networks for data efficiency, CIFAR-10 and MoCo for representation learning, splitMNIST and Synaptic Intelligence for continual learning, IHDP (Infant Health and Development Program) and DragonNet for causality, MNIST poisoned with backdoor attacks and dp-instahide for robustness, CIFAR-10 and Wide-ResNet-28-2 for privacy, and COMPAS (a recidivism prediction dataset) and Adversarial Debiasing for fairness, all built on real research repositories such as DomainBed, Lightly, and AIF360. Besides performance and cost, the paper proposes Exploration Diversity, the spread of the GraphCodeBERT embeddings of the code from each iteration around their centroid, Step Success Rate, the share of executed iterations that yield valid results without errors, and Step Completion Rate, the share of assigned iterations actually executed.
    • Implementation: The paper compares three agents with different search strategies: TheAIScientist (broad parallel exploration), AIDE (tree search, run through its commercial variant Weco), and Claude Code (linear refinement instructed by a prompt). TheAIScientist and AIDE used GPT-5 and Gemini-2.5-Pro, Claude Code used Opus-4.1, and each agent ran three independent rounds with a budget of 100 steps, keeping the best test-set result. The README of the official qrzou/FML-bench repository states that the code for the version in this paper lives on the legacy branch, while the default branch README describes a newer version with 18 tasks and 7 agents.
    • Results: TheAIScientist with Gemini-2.5-Pro performed best, ranking first in 4 of the 8 tasks, and AIDE with Gemini-2.5-Pro ranked first in 2. Mean Exploration Diversity was 24.66 for TheAIScientist, 23.50 for AIDE, and 8.75 for Claude Code, and diversity correlated positively with performance in five tasks, of which data efficiency (0.6287) and continual learning (0.8374) had p < 0.05. Step Completion Rate was 1.00 for TheAIScientist and 0.07 for Claude Code, and of the two agents with token records, Claude Code used more tokens than TheAIScientist (9.06M in total) despite its lower performance.
    • Limits: The authors state that when the agent is fixed and the LLM changes, other LLM abilities such as reasoning and code writing also drive performance besides exploration diversity, and the correlation between diversity and performance was negative for causality, robustness, and privacy. AIDE sometimes misread the structure of the target codebase and created classes never wired into execution, and its scores on generalization and data efficiency equaled the baseline, which the authors suggest may stem from AIDE supporting edits to only a single file. The study is limited to 8 tasks, 3 agents, and the best of three rounds per agent, and AIDE ran through the commercial Weco service, which offered limited control over its internals, lacked token usage records, and sometimes ended runs early because of cloud failures.
  21. 1258Computer Vision

    SciVideoBench: Benchmarking Scientific Video Reasoning in Large Multimodal Models

    • Problem: Most existing video benchmarks cover general scenes such as movies, daily activities, and instructional videos, and Gemini 2.5 Pro already saturates them with over 85% accuracy on VideoMME and Neptune. Recent benchmarks that add scientific and educational videos mostly stay at college level, and some MMVU questions can be answered without the video, so a test was needed that jointly measures perception, expert knowledge, and reasoning on research-level experiment videos.
    • Input and output: This is a benchmark paper. A model receives one research experiment video, which JoVE publishes alongside a journal article, and a multiple-choice question, and picks one of 10 options. The 1,000 questions come from 241 videos covering more than 25 subjects across physics, chemistry, biology, and medicine, split into 245 quantitative, 385 hypothetical, and 370 conceptual reasoning questions.
    • Method: Exemplars written by four PhD students, one per discipline, are expanded with GPT-4o into 12 examples per question type, and a QA Generator (the agent that writes questions and answers) based on Gemini 2.5 Pro drafts questions from the video, a Whisper transcript of the narration, and the paper text. The Evaluator answers each question to confirm it is answerable, the Visual Comparer checks that the reasoning is grounded in the video and adds timestamps, and the Refiner rewrites questions to point at specific video segments so they can only be solved by watching the video. Finally, human verifiers check the timestamps and answers, and key details such as mass or temperature that appear only in the transcript are overlaid onto the relevant video frames by a script.
    • Implementation: For conceptual and hypothetical questions, Gemini 2.5 Pro writes nine plausible but wrong distractors, while quantitative distractors are made by adding Gaussian noise to the correct answer. Evaluation runs with the LMM-Eval toolkit on eight NVIDIA H100 GPUs at temperature 0. The official repository dengandong/SciVideoBench contains a README, a figs folder, and a scivideobench task folder for lmms-eval, and the README states that the benchmark is integrated into lmms-eval and that the data is hosted on Hugging Face.
    • Results: In the main evaluation (Table 2), Gemini-2.5-Pro has the highest overall accuracy at 64.30%, and the best open-source model, InternVL-3-78B-Instruct, reaches 38.80%. GPT-4o given only text without the video gets 15.80%, and graduate students, three per discipline answering closed-book, get 17.4%. Chain-of-thought prompting raises Gemini-1.5-Pro's overall accuracy by 21.10 points and GPT-4o's quantitative accuracy from 11.84% to 34.29%.
    • Limits: All videos come from a single source, JoVE, and every question is multiple-choice with 10 options. The only human baseline is three graduate students per discipline answering closed-book. The error-cause analysis compares the reasoning of only Gemini-1.5-Pro-Thinking and Gemini-2.0-Flash-Thinking against expert-verified rationales.
  22. 1257RAG / Agents

    IoDResearch: Deep Research on Private Heterogeneous Data via the Internet of Data

    • Problem: Scientific data are often multi-source, heterogeneous and multimodal, and lacking unified representation standards and interoperability mechanisms lowers how efficiently they are used (materials science research, for example, typically handles heterogeneous data from experiments, simulations and literature records). Existing Deep Research agents rely mostly on web search with limited exploration of local heterogeneous and multimodal data, so they retrieve private data inefficiently and often fall short of the FAIR principles (findable, accessible, interoperable, reusable data).
    • Input and output: It takes a user's private heterogeneous data (papers, datasets, images, tables and more) and a question or report topic, and returns relevant digital objects, a factual answer or a structured scientific report. This is a system paper that implements and evaluates a data representation and a multi-agent Deep Research system, and it also proposes the authors' own IoD (Internet of Data) Deep Research Benchmark.
    • Method: Raw data are wrapped as digital objects (DOs) with a DOI (Digital Object Identifier) and metadata, long documents are split into Level-2 DOs per chunk, an LLM adds attributes such as summaries, hypothetical questions, classification labels and keywords to the metadata, and domain IoDs are joined into one IoD network. A knowledge refinement layer builds vectors, atomic knowledge (minimal factual units such as a typical aspirin dosage of 300 mg) and LLM-extracted knowledge graphs, and MCP (Model Context Protocol) tools offer retrieval at three granularities (DOs, chunks, facts and subgraphs) through strategies including keyword, vector, graph-reasoning and hybrid search. IoDAgents splits into a Planner that drafts a plan for user confirmation, a Worker Team that filters results with an LLM, refines and reissues queries when context is insufficient and uses a code tool for analysis and visualization, and a Reporter Team whose Writer drafts the output and whose Checker corrects inconsistencies with the retrieved context.
    • Implementation: Documents are parsed with open-source tools such as MinerU, and every compared method runs on the same raw data with the same base LLM, Qwen-Turbo. The benchmark wraps over 500 documents and papers (about 6 million tokens) from four domains (Chinese law, geophysical exploration, computer science, molecular dynamics) as DOs, and has 200 digital object retrieval questions, 800 question answering (QA) questions for retrieval-augmented generation (RAG) (400 single-domain, 400 cross-domain multi-hop) and 60 report-writing questions (30 single-domain, 30 cross-domain). Reports are scored by Qwen-Turbo at temperature 0, averaged over three scorings on five criteria, and by human experts blind to the method who apply the same criteria plus factual accuracy.
    • Results: In Task 1 (digital object retrieval), IoDResearch reached F1 82.64 (precision 76.26, recall 90.18), above LightRAG at 78.93, DO-RAG at 76.21 and Naive RAG at 62.05. In Task 2 (question answering), answer accuracy was 79.98 single-domain and 59.40 cross-domain, above LightRAG at 75.55 and 56.67, the best of the three baselines for this task (Naive RAG, DO-RAG, LightRAG). In Task 3 (report writing), human expert scores were 7.01 single-domain and 6.45 cross-domain, above DeepSearcher at 6.77 and 6.02, the best of the compared methods, and the ablation without the multi-agent component dropped to 6.56 and 5.94.
    • Limits: The authors note that the pipeline is mainly text-based, so multimodal data can lose information during preprocessing and context construction, and they leave an end-to-end multimodal pipeline and post-training strategies to future work. Experiments use the authors' own benchmark across four domains with a single base LLM (Qwen-Turbo), and the LLM judge is the same Qwen-Turbo. The comparison with DeepSearcher covers only the report-writing task, and the conclusion limits the claim to the selected baselines and evaluated task settings.
  23. 1256Research Agents

    The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements

    • Problem: Scientific findings are trusted when they can be reproduced, so automated science needs automated reproducibility: reimplementing an experiment from its description so that it reproduces the reported outcome. Prior reproducibility benchmarks evaluate on a wide set of often unrelated papers, so the authors build a benchmark that measures reproduction of a chain of cumulative research findings within one task, speeding up LLM training.
    • Input and output: This is a benchmark paper. In each of the 19 tasks the agent receives the previous record's GPT-2 training script (train_gpt2.py), optionally with hints at three levels (pseudocode, a text description, or a paper-like mini-paper), and returns a new training script that reaches a FineWeb validation loss of 3.28 faster. The score is FSR (fraction of speedup recovered), the share of the human next record's training-time reduction that the agent recovers, and a record optimization setting without hints asks the agent to find its own improvements.
    • Method: The authors also release a search scaffold, a program that calls an LLM iteratively to find a solution, that generalizes AIDE. Each node in the search tree is a solution holding the training script and a results file, and each step has the coding assistant Aider make diff-based code edits (implementation), runs the code on an 8xH100 node (execution), and summarizes the output in natural language (analysis). Varying the branching factors and debug probability gives five search variants, Flat (best-of-M), Tree, Forest, AIDE, and Multi-AIDE, which are compared under the same budget of search steps.
    • Implementation: For each of the 19 records, DeepSeek-R1, o3-mini, Gemini-2.5-Pro, and Claude-3.7-Sonnet were run with five search variants, six hint regimes, and three seeds, for 6,840 agent runs in total; each solution had a runtime cap of 60 minutes, and an agent run took about 10 hours on average. The hints were drafted by R1, manually verified, and edited where necessary, and the search used N0 = 3, N = 3, pdebug = 0.5, Dmax = 5, and a budget of M = 20 nodes. Per its README, the official facebookresearch/llm-speedrunner repository releases the scaffold code (core), starter workspaces for the tasks, the hint data for each level, Slurm launch scripts for the paper's experiments, and the analysis notebooks.
    • Results: Without hints, the speedup recovered by every agent averaged at most 20% of the human solutions. The best model, o3-mini, reached FSR 0.40 to 0.43 across the five search variants with pseudocode hints, and with all three hints combined it reached 0.46 on Multi-AIDE but only 0.24 to 0.39 on the other four variants, below its pseudocode-only scores. In a cumulative run where o3-mini built each record on its own earlier code, it recovered about 60% on the first task, dropped to about 20% on the next, and reproduced no speedup by the third record.
    • Limits: The authors note that the hints are short, manually defined, and handed directly to the agent, leaving a more realistic setup where the agent retrieves external knowledge through function calling and memory for future work. Many records may have been published before the tested models' knowledge cut-off, so the models may have seen the solutions in training, and the authors note that separating memorization from generalization may become more necessary as models begin to saturate the benchmark. The benchmark mainly tests reproduction rather than innovation, and the authors argue that agents which generalize to future advances in machine learning as a whole will need harder tasks spanning multi-file codebases, metrics beyond training time, and distributed training beyond a single node.
  24. 1255Machine Learning

    ALINE: Joint Amortization for Bayesian Inference and Active Data Acquisition

    • Problem: Fields with limited budgets and data, such as clinical trials and materials science, need systems that strategically acquire informative data and perform inference on it right away. Amortized Bayesian inference is fast but typically assumes passive data collection, while amortized data acquisition selects informative points well, but the inference update on new data often sits outside the amortization and may need separate, costly procedures such as MCMC (Markov chain Monte Carlo). Existing amortized acquisition methods usually optimize information gain about all parameters or one fixed predictive target, which makes it hard to switch goals when only some parameters matter.
    • Input and output: ALINE takes the observations gathered so far (context set), the candidate points that could be queried next (query set), and the current inference goal (target set). In a single forward pass it returns the posterior over the targeted parameters or the predictive distribution and selects the next point to query from the candidates. This is a method paper that unifies amortized Bayesian inference and active data acquisition in one model, with experiments on simulated tasks and on the HPO-B benchmark of real hyperparameter performance data.
    • Method: One transformer processes the three inputs together, and an added query-target cross-attention lets candidate points attend directly to the target set so that selection fits the current goal. The policy is trained with policy-gradient RL using a per-step reward equal to the increase in the log-probability of the targets under its own inference head, and the authors show this objective is a variational lower bound on the total information gain. Training starts with a warm-up phase in which only the inference part is trained on randomly chosen points, after which both parts are trained jointly, and gradients from the policy loss update only the policy parameters.
    • Implementation: The two heads share 3 transformer layers (4 attention heads each, internal embedding dimension 32), the inference head represents posteriors and predictive distributions with a 10-component Gaussian mixture, and the policy head is a 2-layer MLP that outputs a probability for each candidate point. The active learning model is trained only on synthetic functions drawn from Gaussian Process (GP) priors, the whole study used about 5000 GPU hours on AMD MI250X GPUs, and training one model takes about 20 hours. The official repository README lists the training script train_aline.py and folders for hydra configuration (config), model, tasks, loss, distributions, utils, and evaluation notebooks, and gives training commands for GP active learning in 1D and 2D, Location Finding, CES, and the psychometric model.
    • Results: On Location Finding, ALINE reached a total EIG lower bound of 8.91, above every baseline including the amortized BED methods RL-BOED (7.70), DAD (Deep Adaptive Design, 7.33), and vsOED (7.30), and on CES it scored 13.50, second to RL-BOED (14.60). In active learning it performed comparably to the best GP-based methods on in-distribution GP functions and had lower RMSE (root mean squared error) than the baselines on 3 of 4 benchmark functions unseen in training. On the psychometric model, when the guess and lapse rates were the targets, ALINE and Psi-marginal, an existing method that can focus on a parameter subset, inferred them significantly better than QUEST+, which targets all parameters together, and ALINE took 0.002 s per step to propose the next stimulus, against 0.07 s for QUEST+ and 0.02 s for Psi-marginal.
    • Limits: ALINE is trained with predefined, fixed priors, so a different prior requires retraining, and it estimates marginal posteriors per parameter, leaving joint posterior estimation to future work. Performance can degrade when deployment observations differ substantially from the training data, and on the Three Hump Camel function, whose output range extends beyond that of the pre-training GP functions, ALINE and ACE-US (the amortized inference model ACE paired with uncertainty sampling) both showed reduced accuracy. The architecture is tailored to fixed input dimensionalities and discrete design spaces, and the authors list dimension-agnostic models and continuous experimental designs as future work.
  25. 1254Research Agents

    AI-Researcher: Autonomous Scientific Innovation

    • Problem: Systems exist for literature analysis or experiment design, but the authors argue they do not connect the full research workflow from hypothesis generation to publication-quality reporting. They also note that no standardized benchmark measures autonomous research across domains, which makes progress hard to track.
    • Input and output: This is a system paper that presents AI-Researcher, a multi-agent system automating the full research workflow, together with Scientist-Bench, a benchmark for it. Scientist-Bench uses 22 papers from 2022 to 2024 as ground truth and gives the system 15 to 20 reference papers, a research instruction and datasets (Level-2 tasks omit the instruction). The system returns implementation code and a technical report covering background, method, experiments and results.
    • Method: A Knowledge Acquisition Agent selects at least 5 GitHub repositories and gathers the matching arXiv LaTeX sources, then a Resource Analyst splits the idea into atomic concepts and maps each formula to its code. An Idea Generator proposes five directions and develops the most promising one, while a Code Agent and an Advisor Agent iterate implementation and feedback like a student and an advisor. A Documentation Agent writes the paper in three stages: outline, detailed writing, and checklist-based revision.
    • Implementation: All analysis and execution run inside a Docker container, and prototypes are typically tested first on 1 to 2 epochs or small data subsets before full experiments. The paper review agent used for evaluation (an agent that compares two papers and scores them) rated the accepted paper higher in 65.62% to 90.62% of 32 ICLR (International Conference on Learning Representations) accept-reject pairs from 2021 to 2023, depending on the LLM judge. According to the README, the GitHub repository holds the research_agent and paper_agent code, the benchmark data and its collection process (benchmark and benchmark_collection folders), a Dockerfile, a gradio-based web GUI (graphical user interface) and example generated papers with code (examples folder), and the README explains how to pull the runtime Docker image (tjbtech1/airesearcher:v1) or build it from the Dockerfile.
    • Results: With Claude-series models over the whole benchmark, the completeness rate (executable code within the inference budget) was 93.8%, and correctness, scored from 1 to 5 by a Judge Agent using an Advisor Agent's analysis, averaged 2.65. In Level-1 pairwise comparison of generated papers against the ground-truth papers (Table 2), five LLM judges rated on a -3 to 3 scale, and the share of generated papers scoring at least -1.0 ranged from 13.64% to 81.82% depending on the judge. In Level-2 without instructions (Table 3), the same share ranged from 50.00% to 100.00%.
    • Limits: In the failure analysis, the authors report that generated research relied predominantly on conventional techniques and standard formulas with shallow theoretical analysis, and that in one Level-1 knowledge graph recommendation task it missed the Gumbel reparameterization the human researchers used. The current system carries information through the LLM context window and summaries of earlier stages in place of an external memory system, and the authors also note that LLM judges tend to overvalue presentation over substance. The experiments cover 22 papers in four AI domains (6 Level-2 tasks per Table 1), and the summary ranges in Sections 4.3 and 4.4 differ from the values in Tables 2 and 3.
  26. 1253Research Agents

    Agentic AI for Scientific Discovery: A Survey of Progress, Challenges, and Future Directions

    • Problem: LLM-driven agents have made strides in automating general research workflows such as citation management, document discovery, and academic survey generation, but the authors note that these systems often lack the domain-specific focus and compliance-driven rigor needed in fields like biomedicine, where structured assessment of literature is critical. For example, Agent Laboratory showed high success rates in data preparation, experimentation, and report writing but its performance dropped significantly in the literature review phase, and system reliability, reproducibility, and ethical governance continue to pose significant hurdles.
    • Scope: This is a survey of agentic AI for scientific discovery that covers the definition of agents and single-agent versus multi-agent designs, a taxonomy of systems, literature review agents, discovery systems in chemistry, biology, materials science, general science, and machine learning, implementation tools, datasets, evaluation metrics, and open challenges. The authors split the research lifecycle into four steps, ideation, experiment design and execution, data analysis and interpretation, and paper writing and dissemination, and describe what agents take on at each step.
    • Taxonomy: Fully autonomous systems automate end-to-end workflows with minimal human intervention, including Coscientist, a GPT-4-powered agent that plans, designs, and executes chemical experiments, ChemCrow, which integrates 18 expert-designed tools, and ProtAgents for protein design. Human-AI collaborative systems work with researchers, including Virtual Lab, which organizes interdisciplinary research through team meetings and individual tasks, BioPlanner, which designs experimental protocols and leaves their execution to researchers, CALMS (Context-Aware Language Model for Science), a lab assistant that interacts with scientists and laboratory instruments, and Agent Laboratory, which takes human-provided ideas through literature review, experimentation, and report writing. The authors summarize that autonomous systems are efficient on well-defined, repetitive tasks but may struggle with tasks that need creativity or domain intuition, while collaborative systems rely on human oversight, data quality, and interpretability and still require manual validation and execution.
    • Cases: In chemistry, the survey describes LLM-RDF, which splits literature search, experimental design, reaction optimization, and data analysis across six LLM-based agents and demonstrated end-to-end synthesis automation on aerobic alcohol oxidation, and the robotic system Organa, which achieved results comparable to human chemists in electrochemistry while reducing execution time by over 20%. In biology, it describes CellAgent, a multi-agent system whose Planner, Executor, and Evaluator roles collaborate, reaching a 92% task completion rate in single-cell RNA sequencing data analysis. These figures are the survey's summaries of the original papers of each system.
    • Tools and evaluation resources: The survey lists implementation tools such as AutoGen, a framework for managing multi-agent systems, MetaGPT, which assigns roles to agents for software development, Letta, an open-source framework for deploying persistent agents as services, and LangChain and AutoGPT. Its dataset table includes LAB-Bench for reasoning and planning in biological research, MoleculeNet for molecular property prediction, ZINC for virtual screening in drug discovery, MaScQA for materials science question answering, and ICLR 2022 OpenReview data for evaluating an automated paper reviewer. For metrics, reasoning and planning typically use accuracy, task completion rates, and response coherence, prediction and discovery use precision, recall, and prediction error, and the authors note that standardized metrics for complex tasks such as multi-agent cooperation in experimental automation and hypothesis generation are still in development.
    • Limits: The authors see literature review as a significant challenge across nearly all approaches: in Agent Laboratory the literature review phase had the highest failure rate compared with the data preparation, experimentation, and report writing phases, and ResearchAgent and The AI Scientist lack structured literature review. They note that emerging areas such as materials discovery and entire research process automation still need comprehensive benchmarks that assess real-world impact and adaptability. As risks they cite compounding errors in highly autonomous agents with little human oversight, possible deviations from established protocols or safety measures in experimental automation, and agents tied to physical systems misreading situations in ways that lead to unexpected escalations or system failures, and they stress AI governance and human oversight.
  27. 1252Research Agents

    Accelerating scientific discovery with Co-Scientist

    • Problem: As publications grow quickly, scientists struggle to combine deep subject expertise with broad knowledge that bridges disciplines. Literature summarization and deep research tools already synthesize existing information, so the authors target a system that proposes novel hypotheses for experimental validation.
    • Input and output: This is an implemented system paper. It takes a research goal written in natural language (from short statements to documents of tens of thousands of tokens and hundreds of paper PDFs) with preferences and constraints, and returns a ranked list of hypotheses and experiment plans plus a research overview written by the Meta-review agent. Scientists can refine the goal during a run and add their own hypotheses and reviews to the tournament.
    • Method: A Supervisor agent turns the goal into a research plan configuration and asynchronously assigns, from a task queue, the Generation (hypothesis generation), Reflection (peer-review style critique), Ranking, Proximity (clustering of similar hypotheses), Evolution (refinement of top hypotheses), and Meta-review (synthesis of reviews) agents. The Ranking agent starts each new hypothesis at an Elo rating of 1200 and runs a pairwise tournament, comparing top-ranked hypotheses through multi-turn scientific debates and lower-ranked ones through single-turn comparisons. New hypotheses from the Evolution agent compete again, and the Meta-review agent collects recurring issues from reviews and debates and appends them to the agents' prompts in the next iteration, so feedback propagates without fine-tuning.
    • Implementation: All agents use Gemini 2.0 models as the base model, with web search, open databases, a scientist-specified repository of papers and experimental data, and specialized models such as AlphaFold as tools. The full source code is not public; the authors publish pseudocode (Supplementary Note 8) and the exact agent prompts (Supplementary Note 9).
    • Results: In a blinded expert evaluation on 11 curated goals, Co-Scientist was the most preferred with an average preference rank of 2.36 (1 most preferred, 4 least), and its novelty and impact ratings of 3.64 and 3.09 out of 5 were higher than those of three baselines, Gemini 2.0 Flash Thinking Experimental, Gemini 2.0 Pro Experimental, and OpenAI o1. In the ablation, the Reflection agent without search gave already published ideas a novelty score of 6.14 out of 10, which fell to 2.38 with search. In the wet lab, the new candidate drug KIRA6 had an IC50 (the drug concentration that halves cell viability) of 10 nM in KG-1a leukemia cells and 180 nM in the non-leukemic control line TK6, an 18-fold separation.
    • Limits: The authors state that Co-Scientist relies on open-access literature, so it can miss paywalled prior art and negative results, and that it inherits the hallucination risk of its base models. They also note that Elo is a self-assessed score without independent ground truth, the expert evaluations are small in scale, and the cell line experiments are an initial biological validation step.
  28. 1251Research Agents

    Evaluating Sakana's AI Scientist: Bold Claims, Mixed Results, and a Promising Future?

    • Problem: Sakana promoted the AI Scientist as automating the entire research lifecycle, from idea generation through experiment design and execution, analysis and paper writing to review, and it drew wide attention, but to the authors' knowledge a comprehensive independent assessment based on direct experimentation had yet to be conducted. The authors point out that the most positive reviewers relied solely on Sakana's information in place of testing the system themselves.
    • Input and output: This is an independent evaluation case study that installs and runs the AI Scientist, followed by recommendations (a position part) for the IR (Information Retrieval) community. The input is a template built by the authors: a prompt on Green Recommender Systems, experiment.py and plot.py that train FunkSVD on an 80/10/10 split of MovieLens-100k and evaluate it with RMSE (root mean squared error), two seed ideas (adaptive learning rates for SGD (stochastic gradient descent) and e-fold cross-validation), and Sakana's default LaTeX templates. The AI Scientist produced 10 ideas with novelty decisions, experiment code edited by the LLM-based coding assistant Aider, manuscripts and its own reviews, which the authors counted stage by stage and checked against the literature.
    • Method: The authors measured each stage separately. For ideas they checked whether the novelty check, which queries the Semantic Scholar API for up to 10 results per query over up to 10 iterations, filters out known techniques; for experiments they counted the characters changed and whether runs succeeded as Aider edited experiment.py up to five times. For manuscripts they examined length, number and age of references, structural errors and numerical errors, they applied the review function to their 7 generated manuscripts and to 10 human-written OpenReview papers (5 accepted, 5 rejected) and compared it with human decisions, and they recorded cost and human effort.
    • Implementation: Setup, coding and experiments were done by the third author, a third-year computer science undergraduate with basic ML knowledge and strong Python skills, and the foundation LLM was gpt-4o-2024-05-13. The main experiments ran FunkSVD on CPU on a university computing cluster, and the authors state that they released their own codebase for replicating the experiments with a Singularity container definition file. The linked GitHub repository SakanaAI/AI-Scientist is the code of the evaluated AI Scientist; per its README it holds NanoGPT, 2D Diffusion and Grokking templates, the paper-generation script launch_scientist.py and an LLM paper-review function, and it assumes Linux with NVIDIA GPUs using CUDA (NVIDIA's GPU computing platform) and PyTorch.
    • Results: The AI Scientist judged all 10 generated ideas and both seed ideas novel, including known techniques such as micro-batching for SGD, 5 of the 12 ideas (42%) failed to run because of unresolved coding errors, and on a 6,260-character template Aider added on average 529 characters (+8%) in the first iteration and then 118, 83, 66 and 21. The 7 manuscripts had a median of 5 references, only 5 (14.7%) of all 34 references were from 2020 or later, 4 (57%) had structural problems such as missing or misplaced figures, incomplete sections like 'Conclusions Here' and duplicates, and 4 (57%) contained incorrect or hallucinated numbers. The review function rejected 9 of 10 human-written OpenReview papers, including 4 that human reviewers had accepted, and counting only OpenAI API costs the main experiments cost 42 dollars (6 dollars per manuscript on average) with about 3.5 hours of human effort per manuscript excluding setup.
    • Limits: The authors note that they used one dataset (MovieLens-100k), one research domain (Green Recommender Systems), fixed seed ideas and one template, which may affect generalizability and reproducibility. Setup and experiments were done by one undergraduate, the only foundation LLM was gpt-4o-2024-05-13, and the system, whose pipeline is optimized for GPU deep learning, was tested with a simple CPU recommendation algorithm. The human-paper review test covered 10 papers, which the authors call indicative, and the claim of being about 3 to 11 times faster than human researchers rests on the authors' estimates of at least 20 to 40 hours for an undergraduate and 10 to 20 hours for an experienced researcher.
  29. 1250Research Agents

    Recursive self-improvement of AI research agents

    • Problem: AI agents have improved the efficiency of the artifacts they produce, such as training and inference, but the efficiency of the research process producing them stays fixed. Research and development (R&D) has long shown diminishing returns as cumulative spending grows, and recent self-improving systems show promise in coding and other domains, but how effective their loops are at raising frontier AI research efficiency was unclear.
    • Input and output: This is a system paper. The input is the code of a starting research agent, AIDE0, a pared-down version of AIDE (AI-Driven Exploration), and a selection benchmark of three task families, ML engineering, heuristic algorithm engineering and harness engineering, where each task has a public score, private held-out scoring and a fixed cost budget. The output is the code of the current best research agent (the incumbent), which changes whenever the grade improves, and the edited target is the harness, the code around the model that controls search, context and verification.
    • Method: In the inner loop, a research agent edits task code while seeing public scores, and when its budget is spent the solution it picks is scored on hidden data; the average of hidden scores across tasks becomes the agent's grade. The outer-loop agent reads earlier agents and their grades, rewrites the code of the current best agent (the incumbent), and accepts the rewrite as the new incumbent only if the grade improves. Separating public scores from the hidden selection score blocks the inner-loop agent from directly optimizing the selection criterion, and giving every agent the same budget makes gains come from a better algorithm rather than extra compute.
    • Implementation: The starting agent AIDE0 is a tree-search agent made by removing the ML-specific parts of AIDE, with draft, debug and improve operators and greedy selection of the highest-scoring solution. The 8-day recursive self-improvement run produced a 100-node trajectory of the initial agent and 99 rewrite proposals, and during it the outer-loop agent, Weco's production agent AIDEhuman, ran on Claude Opus 4.7 while every inner-loop agent was evaluated with Gemini 3 Flash. External evaluations used Gemini 3 Flash for ALE-Bench (AtCoder combinatorial optimization), MLE-Bench (Kaggle ML engineering) and KernelBench (GPU kernel optimization), GPT-5.4 for FML-Bench (research codebase tasks) and Gemini 3.1 Pro for WeatherBench 2 (physics-based weather forecasting), and the paper text gives no code repository for AIDE².
    • Results: In the 8-day run, seven accepted rewrites raised the grade from 0.703 to 0.778, while AIDEhuman scored 0.749 on the same grade. On external benchmarks separated from selection, the final agent AIDE85 versus AIDEhuman scored 1790 vs 1511 on ALE-Bench, 0.722 vs 0.708 on MLE-Bench, 19.9% vs 19.6% on FML-Bench and 0.793 vs 0.404 on the out-of-distribution WeatherBench 2. On 38 KernelBench pairs, a separate task family, the reward hacking rate was 55% for AIDE0, 39% for AIDE47 and 32% for AIDE85, against 39% for AIDEhuman.
    • Limits: Noise compounds across both loops, and if it is high enough one falsely accepted rewrite can derail the later outer-loop search. In the ignition test, which checks whether a discovered agent drives the loop better than the agent that produced it, the arm with AIDE47 as the outer-loop agent ran 3 seeds of 50 steps and ended at a seed-average grade of 0.780 against 0.782 for the AIDEhuman arm, and the authors call the result inconclusive. The discovered agents are complex and hard to interpret, and it is unclear which components drive their performance.
  30. 1249Research Agents

    Autonomous Scientific Discovery via Iterative Meta-Reflection

    • Problem: Most existing LLM discovery systems treat each hypothesis-generation step independently, so they lack a step that surveys what has already been found and reasons about what remains unknown. Classical structure learning can only express edges between variable pairs, and full-pipeline AI scientist systems need initial research objectives or seed hypotheses to guide their search.
    • Input and output: By default the system receives only tabular observation data (optionally with images) and no research question, and outputs a list of validated discoveries, each with a natural-language statement, test code and statistical evidence. This is a method paper that also builds a new benchmark, iNatDisco, from iNaturalist observations. iNatDisco-800 has 800 observations of eight species with nine literature patterns, and iNatDisco-50K has 50,000 observations of 9,776 species with 12 patterns.
    • Method: In PROPOSE, an LLM agent reads a data summary, the current claims and the latest guidance, and outputs a hypothesis plus Python code that tests it. EVALUATE uses seven statistical tools, including correlation, group difference, prediction, clustering, stratified re-tests and VLM (vision-language model) visual attribute tests, tunes the code on a training split, runs it once on a held-out split, and accepts a claim only if both splits show an effect size of at least 0.2 and p of at most 0.05. REFLECT analyzes accepted and rejected claims together and returns guidance on gaps, recurring confounds and compound hypotheses; in the experiments it ran every five iterations.
    • Implementation: The main text names Claude Sonnet 4.6 as the default LLM, while the appendix model comparison and baseline description name Claude Sonnet 4.5; the same base LLM also serves as the VLM for reading images, except the DeepSeek V4 Pro run in the model comparison, which was text only. The default setting runs 100 iterations per experiment three times for mean and standard deviation, the counterfactual experiment used 50 iterations, and the iteration-scaling experiment ranged from 10 to 100 iterations. All experiments took about 86 completed runs, about 8,250 API calls, an estimated cost of about 220 US dollars at Sonnet pricing and about 72 hours of wall-clock time.
    • Results: On iNatDisco-800, DiscoPER recovered 8 of 9 literature patterns with a 72.7% support rate, while HeurekaBench-like and ExperiGen-like baselines re-implemented in the same framework found 3 and LLM-assisted classical causal discovery found at most 1. On iNatDisco-50K it found 8 of 12 with a 74.2% support rate, against 6 for the version without REFLECT. On the counterfactual iNatDisco-800-CF data, where five known ecological relationships were reversed in the tabular data, most proposed hypotheses were rejected by held-out validation and the accepted discoveries followed the modified data instead of the LLM's prior knowledge.
    • Limits: The authors state that DiscoPER can only find what the input data supports and that citizen-science data carries known biases, so human verification of any proposed discovery is essential. Valid patterns outside the iNatDisco answer list can exist; recall is judged by an LLM judge (Claude Sonnet 4.5) that counts a partial match (score 1) or better as a discovery, and when the authors manually inspected all exact-match (score 2) judgments in the primary experiments (iNatDisco-800, iNatDisco-50K), 95% agreed. On a synthetic image benchmark it found 3 of 8 patterns, and the authors identify statistical validation of visual features as the bottleneck.

From research to product use

Operating capabilities, pilots, and technologies in development are identified separately.

View technology