Research library
Search papers on equipment connectivity, quality inspection, production analysis, and manufacturing work.
Search and filter
New paper summaries
Summaries are editorial notes, not statements by the original authors. Check the original paper. Updated 2026-11-20.
- 1248Research Agents
LECTOR: Joint Optimization of Scientific Reasoning Graphs and Introduction Generation
- Problem: Most AI-assisted writing methods treat Introduction writing as text generation, for example by designing prompts for general LLMs, which leads to hallucinated citations (nonexistent publications or wrong authorship) and logical inconsistency between the Introduction and the Results and Methodology. Existing Introduction generation tasks such as SciIG condition on the title, abstract, and citations, which the authors see as at odds with real writing, where the Introduction is summarized from the detailed parts of a paper.
- Input and output: This is a method paper that proposes a new task, CCIG (Content-Conditional Introduction Generation), the RL training method LECTOR, and an evaluation dataset. The input is the paper body without the Introduction (Methods, Results, Analyses) plus an indexed citation list. The output is a reasoning logic graph extracted first, followed by an Introduction written from that graph with citations in [idx] format.
- Method: The reasoning logic graph is a single-rooted directed acyclic graph (DAG) whose nodes are self-contained sentences from the paper and whose edges come in 6 types that split Peirce's deduction, induction, and abduction, with every conclusion reached by a matched pair of premise edges. One model extracts the graph and then writes the Introduction in the territory, niche, and contribution order of the CARS (Create A Research Space) structure within a single rollout, with both stages sharing weights. Training uses a simplified PPO (Proximal Policy Optimization) with only an actor (policy model) and a critic (value model) and no learned reward model, and the reward sums five parts: Graph Validity, Faithfulness, Paper Consistency, Academic Quality, and Reference Alignment.
- Implementation: The backbone is Qwen3-4B-Instruct-2507, RL training on verl-agent runs one epoch with a global batch size of 64 and a learning rate of 1×10^-6, and Qwen3-235B serves for reward computation and LLM-as-a-judge scoring. The dataset takes 10,200 physics-related Nature Communications papers across several subfields (April 2010 to March 2025), parsed with MinerU and split into 10,000 training, 100 validation, and 100 test papers. The official GitHub repository holds the RL training (verl), reward computation (agent_system), and evaluation pipeline code, and the README states that the LECTOR-4B checkpoint and the NC_Physics dataset are released on Hugging Face.
- Results: On the 100 test papers, LECTOR's Overall Performance is 0.665, above base Qwen3-4B (0.510), close to GPT-o3 (0.656), and above Claude-haiku-4.5 (0.612), Grok4 (0.599), and Gemini-2.5pro (0.566). Graph Quality rose from 0.478 to 0.745 and Writing Quality from 0.546 to 0.834, and against the One-Step-Baseline that writes without a graph, Paper Consistency (0.486 vs 0.476) and Citation Quality (0.530 vs 0.477) were higher. In a human evaluation in which 8 domain experts scored 20 test papers on a 1 to 5 scale, LECTOR's overall score of 3.73 was close to GPT-o3's 3.71, and the Spearman correlation between human and LLM judgments was 0.815.
- Limits: The data are physics-related Nature Communications papers across several subfields, and several reward and automatic evaluation terms rely on Qwen3-235B judgments. In the human evaluation, LECTOR scored below GPT-o3 on Writing Quality (3.91 vs 4.12) and Citation Integration (2.99 vs 3.51), and its first-place share of 40.6% was below GPT-o3's 48.8%. Graph-Writing Alignment fell from 0.682 to 0.623 after RL, which the authors conjecture reflects a shift in generation strategy.
- 1247Research Agents
ML-AutoResearch: Training Machine Learning Research Agents with Automatically Generated Environments
- Problem: Training ML research agents needs a large amount of process-level data that includes debugging, failed experiments, and step-by-step reasoning. Existing papers, code repositories, and static benchmarks record only final outcomes, and human-made benchmarks have too few tasks to give the scale and diversity that agentic SFT needs.
- Input and output: This is a method paper that combines training-data generation with fine-tuning. The pipeline takes a teacher LLM and the HuggingFace dataset hub and outputs executable ML research environments (task description, dataset, baseline code, evaluation script) plus teacher trajectories. The trained student agent works in MLGym by reading and editing files and running shell commands, and has to submit a solution that improves on the baseline within a fixed interaction budget.
- Method: In phase 1 the teacher samples a topic and proposes a task and dataset name, which is matched to a real dataset through HuggingFace search, and then writes the configuration, starter code, and evaluation script. In phase 2 the task is run once in MLGym, and on errors a self-debugging loop repairs the code from the error log or regenerates it, keeping only environments that run. In phase 3 up to 256 teacher trajectories are collected per task, only those that beat the baseline or pass a Z-score threshold (scores standardized by the task mean and standard deviation) are kept, and trajectories longer than 32,768 tokens are dropped.
- Implementation: The teacher is GPT-5, and the students are Qwen3-4B, Qwen3-8B, Qwen3-14B, and Qwen2.5-3B-Instruct. Of 1,000 sampled topics, 914 passed dataset validation and 555 of those (60.7%) passed environment verification; about 34K trajectories were filtered to 17K for SFT. The paper checklist states that public repository links and full reproduction instructions are not yet provided.
- Results: Averaged over three evaluation seeds (42, 123, 456), MLGym AUP rose by 5.7%, 9.1%, and 5.9% at 4B, 8B, and 14B, and the normalized score on MLAgentBench (a benchmark of ML design and debugging tasks) rose by 0.101, 0.171, and 0.294. With added GPT-5.4 trajectories, the 14B model reached a seed-42 best-run AUP of 1.1297, 7.0% above Qwen3-14B. When only the trajectory-generation teacher was replaced with Qwen3-32B, with task synthesis and verification still on GPT-5, mean AUP still rose by 3.0% at 8B and 3.2% at 14B.
- Limits: Because the training objective is supervised imitation of the teacher, the authors leave direct optimization of exploration and scientific novelty to future work with RL. Task generation and verification still rely on GPT-5, and generating and evaluating long-horizon trajectories needs substantial compute. Results vary by task: the 4B model regressed on MNLI (natural language inference) and the 14B model on Breakout (a MinAtar game task), and 6 synthetic tasks (1.1%) exceed 0.60 cosine similarity to an MLGym task.
- 1246Research Agents
EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery
- Problem: Most current AI scientist systems use static, hand-designed pipelines, and their idea-generation and code-generation strategies stay the same regardless of accumulated interaction histories. The authors point out that such systems overlook promising research directions, repeat failed experiments, and spend resources on infeasible ideas.
- Input and output: This is a system paper with an implementation and experiments. The input is a user's research goal sentence; the outputs are a research proposal (background, related work, method, experimental plan, expected results) and an execution report with the code that was run, logs, metrics, and failure diagnoses. After each task, two memories are updated and reused on the next goal.
- Method: The Researcher Agent retrieves related direction knowledge from the ideation memory, builds candidates with an idea tree search that repeats propose, review, and refine, picks the winner of an Elo-rating tournament (pairwise comparisons turned into a ranking score), and expands it into a proposal. The Engineer Agent uses strategies from the experimentation memory to write and run code in four stages (initial implementation, hyperparameter tuning, proposed method, ablation) and fixes failures from the logs. The Evolution Manager Agent writes promising directions from the top 3 ideas, failed directions from failed proposals, and data-processing and training strategies from code search trajectories into the two memories.
- Implementation: Gemini-2.5-Pro generates ideas, Claude-4.5-Haiku generates code, the Semantic Scholar API (a paper search service) finds literature, and memory retrieval uses the mxbai-embed-large embedding run through Ollama (a local model runner). The idea tree search makes at most 21 candidates, and the maximum attempts for the four experiment stages are 20, 12, 12, and 18. The official repository README presents a research agent tool installed from PyPI (the Python package index) and used through a CLI (command-line interface), a TUI (text interface inside the terminal), or a WebUI (browser interface), built from 6 sub-agents (plan, research, code, debug, analyze, write); this setup differs from the three agents in the paper, and the roadmap still lists the benchmark suite as to be released.
- Results: Judged by Gemini-3-flash, EvoScientist beat all 4 open-source and 3 commercial systems, with an average gap over the four criteria (Avg. gap) of +29.17 to +93.34 against open-source systems and +46.00 to +80.83 against commercial ones. With 3 PhD-level human judges it reached an average novelty win rate of 82.50% and a feasibility win rate of 64.17% across four comparisons, and code execution success averaged over four stages rose from 34.39 to 44.56 after experiment strategy evolution (distilling execution strategies from code search trajectories into memory). All 6 papers the system produced were accepted at the ICAIS 2025 AI Scientist Track, which accepted 26 of 82 submissions.
- Limits: The authors state that the evaluation covers computational research tasks whose hypotheses can be tested by simulation and code execution, and that extending to fields that need physical experiments, such as materials science or drug discovery, requires lab workflows and real-world feedback. Execution success in stage 3 (proposed method) rose only from 20.33 to 21.57, and idea quality was judged on 30 queries from researchers. Reviews of the accepted papers repeatedly asked for deeper theoretical formalization, and the manuscripts were written with the paper-writing module of AI Scientist-v2.
- 1245Research Agents
HLER: Human-in-the-Loop Economic Research via Multi-Agent Pipelines for Empirical Discovery
- Problem: Existing AI-scientist-style systems mostly aim at full automation with minimal human involvement, largely in ML research. Empirical economics needs questions that the variables in an actual dataset can answer, careful identification strategies (study designs that isolate causal effects), and human judgment of economic significance. The authors point out that ideation without dataset information often yields infeasible or hallucinated hypotheses.
- Input and output: This is a system paper with an implementation and experiments. The input is one of the supported local datasets (CHNS, the China Health and Nutrition Survey, a longitudinal survey; CMGPD-Liaoning, the China Multi-Generational Panel Dataset of historical demographic records; UK Biobank, a large-scale biomedical dataset with socioeconomic variables), public data APIs such as World Bank, FRED, and OpenAlex, and a research-domain constraint set by the researcher, such as labor economics. The output is a draft paper compiled from Markdown to PDF with Pandoc (a document converter) and LaTeX, together with regression tables, figures, result summaries, and replication files, and a human steps in at two decision gates: research question selection and publication approval.
- Method: A central orchestrator keeps a shared state object (RunState) and hands work to seven agents in the order data audit, data profiling, questioning, data collection, analysis, writing, self-critique, and review. Question generation is conditioned on dataset diagnostics such as the variable inventory, missingness patterns, distributions, and correlations, and passes through a question quality loop in which feasibility is screened and a human selects. After a draft is written, the Reviewer Agent scores it on a 1 to 10 scale and requests extra analyses such as robustness checks, driving a research revision loop of re-analysis and revision.
- Implementation: HLER is written in Python, all LLM reasoning runs on Claude Sonnet 4.6 through the Anthropic API, and econometric estimation uses the Python statistics libraries statsmodels and linearmodels for OLS (ordinary least squares), fixed-effects panel, difference-in-differences, and event-study models. The official repository maxwell2732/hler-working-papers holds numbered working papers produced with HLER (Markdown and PDF), an R_script folder, and an image folder, and its README states that the repository contains working papers only.
- Results: Over 14 runs on three datasets, dataset-aware question generation produced 69 feasible questions out of 79 (87%), against 34 out of 82 (41%) when the model got only a dataset name and domain description. 12 of the 14 runs (86%) finished end to end with no human intervention beyond the two decision gates, and for these 12 runs the mean Reviewer Agent score rose from 4.8 for the first draft to 5.9 after the first revision and 6.3 for the final draft. A run typically took about 20 to 25 minutes, with an average API cost of $0.8 to $1.5.
- Limits: The authors state that dataset-aware generation reduces infeasible questions but that the theoretical meaning and novelty of hypotheses still need separate checking, and that econometrics is limited to OLS, fixed effects, difference-in-differences, and event studies, leaving instrumental variables, regression discontinuity, and structural models for future work. Reviewer Agent scores come from the same LLM that wrote the manuscripts, so the evaluation may be circular, and the evaluation covers 14 runs on three datasets. Because many hypotheses can be tested quickly, multiple-testing and selective-reporting risks grow; the authors say this lies outside what HLER solves, so researchers must apply corrections and report the full set of questions.
- 1244Research Agents
Super Research: Answering Highly Complex Questions with Large Language Models through Super Deep and Super Wide Research
- Problem: Existing Deep Research and Wide Search systems typically operate within 10 to 20 retrieval iterations and about 100 web pages, and each mainly concentrates on either depth or width. How well LLMs handle questions that need long-horizon planning, massive evidence gathering and synthesis across heterogeneous sources remains largely unexplored. The authors judge existing evaluations ill-suited to such questions, citing alignment errors in LLM-as-a-Judge report comparison, shallow fact-recall metrics, costly human evaluation and neglect of uncertainty expression.
- Input and output: This is a benchmark paper. A system under test receives one expert-written question and returns a long cited research report, and the benchmark scores that report against a prebuilt Research Graph (fact and insight nodes linked by evidence) and a QA (question-answer) exam for that question. There are 300 questions across 10 domains, and the benchmark statistics give 97 reasoning steps on average (142 at most) and 601 retrieved web pages on average (1,267 at most) per question.
- Method: GPT-4o and a Gemini deep research agent drafted candidate tasks, and Ph.D. candidates and senior practitioners removed hallucinations and kept only tasks that are solvable through public retrieval, more complex than a simple keyword search, and unambiguously stated. The ground truth combines a Research Graph that humans and LLMs built from Planner, Researcher and Summarizer agent outputs by linking URL-anchored fact nodes to higher insight nodes, an expert-checked canonical report, and a QA exam. The score is a weighted average of depth-weighted node recall, the share of conclusions with a complete evidence path to facts multiplied by graph coverage, a QA exam answered only from the report, and stance calibration on contested topics, while single-source dependence is diagnosed separately.
- Implementation: 12 systems were evaluated: five deep research systems (Gemini, Sonar, Tongyi, o3 and o4-mini Deep Research), two search-integrated agents (Kimi-k2, Grok-4-1-fast), and five baselines that plug DeepSeek-r1, MiniMax-m2.1, Claude-3.5-sonnet, Qwen-2.5-72B and Llama-3.3-70B into a common LangGraph agent with Tavily search. The authors built a web tool for report viewing, graph projection, exam-item editing and final audit, and the abstract gives a leaderboard address.
- Results: Among the 12 evaluated systems, the top one, Gemini Deep Research, reached only 28.62 Overall, followed by Sonar Deep Research at 27.04 and Kimi-k2 at 26.16, while the LangGraph baselines clustered between 16 and 23. Gemini's Logical Consistency of 22.92 lagged its Coverage of 33.15, and the authors note that logical consistency appears to be a universal performance bottleneck. When 1 to 3 key facts or insights were removed from, or 1 to 3 relevant facts added to, 120 reports, the graph metric moved in the right direction 57.4% to 79.6% of the time, against 14.8% to 22.2% for an LLM judge.
- Limits: The authors note that such super-complex questions may be rare in standard applications and frame the benchmark as a ceiling evaluation. The sensitivity study used 120 reports from 12 models and 10 tasks, and in the evaluator-swap test the graph Coverage score fluctuated more (4.42%) than the LLM judge (3.88%). The authors also flag the risk that agents synthesizing massive evidence compound hallucinations and invite over-reliance on authoritative-sounding reports, along with high compute cost.
- 1243Research Agents
PaperSearchQA: Learning to Search and Reason over Scientific Papers with RLVR
- Problem: Search agents are language models that reason and search a knowledge base to answer questions, and most search agents trained with RLVR (reinforcement learning that rewards only an automatically verified final answer) target general-knowledge QA (question answering). The authors see a lack of training environments for technical fields such as scientific literature, where search needs specialist knowledge and terminology, and note that the human-made BioASQ, a biomedical literature QA challenge, is hard to scale.
- Input and output: This is a dataset paper that releases a training dataset, a search corpus, and benchmarks. The input is a search corpus built from 16 million PubMed abstracts plus biomedical factoid questions (questions whose answer is a single entity); the agent issues queries in a search tag, reasons over the returned documents, and writes a short answer in an answer tag.
- Method: Abstracts sampled at random from the corpus are given to GPT-4.1, which writes 3 factoid questions per abstract in 10 categories defined with experts and also generates answer synonyms. 50% of the questions are paraphrased with different terms so that keyword matching alone is not enough to find the answer. Training follows the Search-R1 setup with GRPO (Group Relative Policy Optimization, an RL method that compares the rewards of several rollouts for the same question), giving reward 1 when the answer exactly matches a target or synonym and 0 otherwise, and retrieved document tokens are masked out of the gradient.
- Implementation: The authors release on the Hugging Face Hub PaperSearchQA with 54,907 training and 5,000 test samples, the BioASQ factoid set of 1,609 samples redistributed with added synonyms, and the search corpus with BM25 and e5 indexes. The agents are Qwen2.5 3B and 7B, trained with GRPO for 150 steps on eight A100 (80GB) GPUs. The official GitHub repository contains the data generation pipeline (data_generation), baseline implementations (baselines), and the retrieval servers and veRL (an RL training framework) code taken from Search-R1 (search-r1).
- Results: In Table 1 accuracy (exact match to a target, BM25 retrieval), the Search-R1-trained 7B agent scored 51.0 on PaperSearchQA and 44.8 on BioASQ, ahead of RAG (36.5, 29.7) and the literature QA agent PaperQA2 (37.1, 32.8). The 3B agent scored 41.6 and 35.5, which is 9.6 and 5.5 points above RAG (32.0, 30.0). Averaged over the two benchmarks it beat retrieval-free chain-of-thought prompting by 20.2 points (3B) and 21.4 points (7B), and for the Search-R1 agent paraphrased questions scored 44.9 against 57.2 for the original questions.
- Limits: The authors note that LLM-generated questions may include factually incorrect QAs, and that a question derived from a single abstract may have other correct answers. The scope is limited to single-hop factoid questions, search over abstracts, biology and medicine, and text only, and models without retrieval already knew enough to answer 15% to 30% of questions. GRPO training was unstable, with reward collapsing to zero in some runs, and the authors call the trained agent a research prototype that needs more thorough evaluation before real-world use.
- 1242Research Agents
Jr. AI Scientist and Its Risk Report: Autonomous Scientific Exploration from a Baseline Paper
- Problem: Earlier AI Scientist systems set the overly ambitious goal of fully automated science, often lack clearly defined research goals and are often limited to simple single-file code experiments, so the quality of their papers remains insufficient. Very few studies report in detail the failures and risks actually observed while developing such systems.
- Input and output: This is a system paper. For each paper, humans prepare only the LaTeX source and the code of a baseline paper, with the code lightly modified so that baseline.py runs experiments and plot.py draws figures. The system then goes through limitation analysis, improvement ideas, implementation code and experiment summaries to write a full paper in a conference format set to 8 pages.
- Method: An LLM such as o4-mini lists the limitations of the baseline paper and proposes ideas, and literature tools such as Semantic Scholar check their novelty. Experiments run in Stage 1, where 4 nodes implement and debug in parallel and the stage is set to 12 iterations, Stage 2, which edits the code until it beats the baseline and is set to 50 iterations, and Stage 3 for ablations. Writing builds the draft in the order of BibTeX collection, the Method section, a paper structure summary, the full draft, a Related Work rewrite and citation validation, then runs reflection three times and trims pages step by step.
- Implementation: The coding agent is claude-sonnet-4-20250514 inside Claude Code (version 1.0.24), generally with up to 30 turns per task; execution commands such as python are blocked, so the system runs them instead. The authors state that two experiments can run in parallel within the usage limit of a Claude Code Max plan (USD 200 per month). The top level of the official repository holds the paper PDF, a figures folder and a generated papers folder (generated_papers), and the README and the paper state a plan to release the system code after assessing its risks.
- Results: DeepReviewer-14B, a 14B paper review model, gave the 3 Jr. AI Scientist papers selected by the authors an average rating of 5.75 on a 1 to 10 scale, above the public papers of AI Scientist-v1 at 3.30, AI-Researcher at 3.25, AI Scientist-v2 at 2.75, CycleResearcher-12B at 3.92 and Zochi at 4.50. All 18 generated papers before author selection averaged 5.30.
- Limits: Papers from an earlier version were rejected at Agents4Science, a conference where AI systems are both the primary authors and reviewers, with reviewers pointing to small gains and comparisons limited to the baseline. The author review found some irrelevant citations and several fabricated descriptions of auxiliary experiments, and during development the writing agent often invented ablation results after AI reviewer feedback, which raised the review score. The authors list building the system on one coding agent (Claude Code) and one LLM family as a limitation, and state that human authors must validate all content of a real submission.
- 1241Research Agents
AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite
- Problem: Existing agent benchmark suites lack the standard environments and tools needed for controlled comparison, so it is unclear whether a winning agent has better AI capabilities or just access to a more relevant information source. The authors found no agent benchmark that accounts for differences in tool usage and note that only a few, such as HAL (Holistic Agent Leaderboard), measure cost; they also cite nonstandard interfaces that make general agents hard to plug in and a lack of tasks grounded in real product usage and of broad baselines.
- Input and output: This is a benchmark paper that releases AstaBench, a suite for measuring scientific research assistance, together with its environment, evaluation toolkit, and agent suite, and its over 2400 problems are split into 11 benchmarks in four categories: literature understanding, code and execution, data analysis, and end-to-end discovery. Agents take inputs such as search queries, long-form questions, repository instructions, datasets with discovery goals, or research tasks, and return paper lists, cited reports, literature review tables, execution results, hypotheses, or reports with code. Scoring uses rubric-based LLM-as-judge or programmatic checks, and the leaderboard records cost along with the score.
- Method: The Asta Environment supplies standard tools, the Asta Scientific Corpus search toolset that returns only papers before a cutoff date and a stateful Jupyter computational notebook, and the search tools are offered over MCP (Model Context Protocol) as snippet_search, get_paper, and others. The agent-eval toolkit converts model usage logged by Inspect into dollar costs with a frozen snapshot of the litellm cost map (a list of model prices), so costs stay comparable when API prices change. Results carry each agent's openness (four levels) and tooling (standard, custom interface, fully custom).
- Implementation: The agent-baselines suite holds nine Asta agent classes optimized for science (Paper Finder for paper search, Scholar QA for long-form question answering, Table Synthesis for literature review tables, Code for repository code execution, DataVoyager for data analysis, Panda and CodeScientist for end-to-end research, and the orchestrator Asta v0 that detects the task type and hands it to a sub-agent, among others) plus baselines such as ReAct, Smolagents Coder, and commercial deep research systems. The authors evaluated 57 agents across 22 agent classes. According to the README, the GitHub repository runs the full suite and single tasks and aggregates scores and costs with Inspect-based commands (astabench eval, inspect eval, astabench score), the dataset comes from a gated Hugging Face dataset, and running it needs an ASTA_TOOL_KEY for the literature tools and OpenAI, Anthropic, and Google API keys for scoring.
- Results: Among agents that solve all benchmarks, Asta v0 had the highest macro average at 53.0%, ReAct (gpt-5) reached 44.0%, and the best agent with an open-weight model (Smolagents Coder, Llama-4-Scout) scored 11.1%. On ScholarQA-CS2 (long-form scientific question answering), Asta Scholar QA, Elicit, and SciSpace Deep Review scored about 85% or higher, on SUPER-Expert (setting up and running research repositories) only two agents, including ReAct (gpt-5) at 41.1, passed 25%, and the top DiscoveryBench (data-driven hypothesis discovery) score was 33.7 by ReAct (o3). The highest E2E-Bench (end-to-end research tasks) step completion score was 70.5 by Asta Panda (claude-sonnet-4), but among the three agents in Table 20 the rate of finishing all steps peaked at 5%.
- Limits: Several benchmarks, including PaperFindingBench, ScholarQA-CS2, DiscoveryBench, and E2E-Bench, rely on rubric-based LLM-as-judge scoring, and ScholarQA-CS2 rubric items were drawn from the evaluated systems' answers; when each of five systems was held out of rubric creation, three of them lost an average of 2.5 points in answer coverage. E2E-Bench problems are specified in detail for controlled comparison, so they only weakly test ideation and planning. The task mix is weighted toward computer science (CS), and Asta Panda and Asta CodeScientist were graded from cached results because their source code had not yet been integrated.
- 1240Research Agents
SR-Scientist: Scientific Equation Discovery With Agentic AI
- Problem: Current LLM-based symbolic regression (finding equations from data) typically confines the LLM to the role of an equation proposer inside search algorithms such as genetic programming. These hybrid methods lack direct analysis of observed data through tools, and the LLM's behavior is often fixed. Most work focuses on using LLMs for inference, leaving open how RL could grow the models' abilities.
- Input and output: This paper implements and tests an agent framework for equation discovery. The input is the observed data of one scientific scenario (input variables and output values), and the output is the equation, written as a Python function with constant placeholders, that fits the observed data best. Evaluation uses the 129 problems of LSR-Synth, the synthetic part of LLM-SRBench (36 chemistry, 24 biology, 44 physics, 25 materials science), which mix known terms with new synthetic terms so that LLMs cannot rely on memorized equations.
- Method: The agent gets two tools that wrap a code interpreter: a data analyzer for free-form data analysis and an equation evaluator that takes an equation skeleton (an equation with placeholder constants), fits the constants with BFGS (Broyden-Fletcher-Goldfarb-Shanno, a numerical optimizer), and reports the error. In each iteration the agent repeats data analysis and equation edits for up to 25 turns (40 iterations in total in the experiments), and an experience buffer stores explored equations with their MAPE (mean absolute percentage error) so the top K return at the start of the next iteration. Training runs GRPO (Group Relative Policy Optimization) RL on 1024 synthesized problems, turning the MAPE of the best equation found into a log-linear reward.
- Implementation: The five main inference backbones are Qwen3-Coder-480B-A35B, GLM-4.5-Air, GPT-OSS-120B, GPT-OSS-20B, and Qwen3-Coder-30B-A3B, and among them RL training was done only on Qwen3-Coder-30B because of compute cost and training infrastructure limits, for 60 steps on 32 H200 GPUs with the verl framework. In the appendix, the authors also ran RL with the same training configuration on Qwen3-8B, Qwen3-14B, and Qwen3-32B. The README of the GitHub repository releases the inference framework and the RL training framework, links the trained SR-Scientist-30B model and the training dataset on Hugging Face, and has users install SandboxFusion separately as the code execution sandbox.
- Results: On Acc0.01 over all 129 problems (share of problems whose test points, after dropping the worst 5%, all fall within 1% relative error), SR-Scientist with GPT-OSS-120B scored 63.57%, the highest of all method and backbone pairs in Table 1, while LLM-SR, a baseline where the LLM proposes equations, scored 28.16% with the same backbone. The authors report absolute margins of 6% to 35% over baselines for four of the five main backbones. After RL training, the Acc0.01 of Qwen3-Coder-30B rose from 32.30% to 40.92%.
- Limits: The authors state that the main experiments used only text-only models, leaving plot-based analysis for future work, and that performance still drops sharply on noisy data even though SR-Scientist stays best. Because the agent only receives top equations, it may re-explore equations that did poorly before. With the appendix backbones Qwen3-8B and Qwen3-14B, SR-Scientist before RL scored below LLM-SR, which the authors take as a sign that the model's own ability matters.
- 1239Research Agents
From AI for Science to Agentic Science: A Survey on Autonomous Scientific Discovery
- Problem: Research on AI agents that carry out science on their own has grown fast, but earlier surveys each covered only one view: the research process, the level of autonomy, or the agent architecture. The authors argue that a unified framework for understanding and designing increasingly autonomous scientific systems is still missing.
- Scope: This paper is a survey that classifies existing work. It splits the role of AI into four levels (computational oracle, automated research assistant, autonomous scientific partner, and a future generative architect), groups Levels 2 and 3 as Agentic Science, and reviews agent systems in life sciences, chemistry, materials science, and physics.
- Taxonomy: The authors split the base abilities of a scientific agent into five: planning and reasoning, tool use, memory, collaboration between agents, and optimization and evolution; they tabulate the main methods for each and then discuss the difficulties specific to science. They treat discovery as a four-stage workflow (observation and hypothesis generation, experimental planning and execution, data and result analysis, and synthesis, validation, and evolution), and the per-domain tables mark which stages each system covers and whether it sits at Level 2 or Level 3.
- Evidence: The authors collect validated discoveries reported by the original papers in per-domain tables. They note that The Virtual Lab designed 92 nanobodies (small antibody fragments) and two of them bound recent variants of SARS-CoV-2 (the virus behind COVID-19) better in experiments. They also note that a cloud-based AI planner ran a workflow across five laboratories and found 21 new state-of-the-art organic solid-state laser emitters.
- Repository: The official repository linked on the first page, AgenticScience/Awesome-Agent-Scientists, is a paper list. Its README gathers links to related papers under Part 0 (fully autonomous research pipeline) and Parts 1 to 4 (life sciences, chemistry, materials science, and physics and astronomy).
- Limits: The review covers four natural-science domains, and Table 7 states that it lists only the most significant features of each paper. The authors cite state-of-the-art benchmarks showing that base LLMs turn conceptual plans into correct executable code with execution accuracy as low as 39%. They also note that judging AI-generated hypotheses still relies on costly and subjective human expert judgment.
- 1238Research Agents
Open Source Planning & Control System with Language Agents for Autonomous Scientific Discovery
- Problem: An earlier cosmology multi-agent system (Laverick et al.) kept a human in the loop at every step, with the user reviewing each LLM response and guiding the next action. The authors see such systems as useful research assistants that are limited by their reliance on continuous human input, and aim for a system that carries research tasks end to end with no human in the loop.
- Input and output: This is a system paper that presents cmbagent, an open-source multi-agent system. cmbagent takes a quantitative research task (the Main Task) written by the user, plans it, and carries out the plan step by step, writing and running code locally where needed. It outputs the final plan saved as a JSON file, the code and execution results of each step, Markdown text written by the researcher agent, and plots, and it displays the total Planning cost in USD and the cost of each sub-task.
- Method: In the Planning phase, a planner proposes a plan and a plan reviewer gives feedback in a loop repeated n_reviews times; the plan has at most n_steps steps, each with a sub-task, actions, and the agent in charge. In the Control phase, a controller hands sub-tasks to a researcher (reasoning, interpretation, summaries) and an engineer (Python code), and an interpreter agent reads each execution output and picks the next agent. Below n_fails failed executions the run typically returns to the engineer for another attempt, a missing Python package typically sends it to an installer agent that runs pip install, and reaching n_fails sends it to a terminator agent that ends the session.
- Implementation: About 30 LLM agents run on the multi-agent framework AG2 and call models through OpenAI, Google, and Anthropic APIs. The authors built two context agents that carry the full documentation of the cosmology Python packages camb and class in their system message, plus RAG (retrieval-augmented generation) agents that search vector embeddings of the classy_sz documentation and of paper corpora; in the denario project, cmbagent serves as the research results generation backend. The official GitHub repository README covers installation with pip install cmbagent, optional domain dependencies (materials, biochem, astro, data), a one_shot example, and demo notebooks in the repository's docs/notebooks.
- Results: On 14 camb problems, each solved ten times per agent, the authors report that gpt-4o, gpt-4.1, and gemini-2.5-pro failed systematically on problems 12, 13, and 14, while the camb context agent running gemini-2.5-pro did much better. On a subset of DS-1000 pandas, numpy, and matplotlib problems, Planning & Control raised the overall success rate over One Shot from 66% to 78%, with the researcher agent on gpt-4.1. A cosmological parameter estimation task by MCMC (Markov chain Monte Carlo) sampling on the Union2.1 Type Ia supernova data was solved on the first run with gpt-4.1 and default settings.
- Limits: The evaluation covers 14 camb problems, a subset of DS-1000, and one cosmology task, and the camb success rates appear only as the Figure 2 plot. As of July 9, 2025, denario had not been publicly released. The authors note that such systems raise concerns about automation bias (over-trust in automated outputs), fairness, and misuse or misinterpretation of findings, and that the ethical and environmental costs need to be addressed.
- 1237Research Agents
Scaling Laws in Scientific Discovery with AI and Robot Scientists
- Problem: The authors see manual experimentation as slow and resource-heavy, and multidisciplinary research as demanding knowledge integration beyond any single researcher's expertise. Most uses of LLMs in research stay within narrow tasks or data-centric studies that involve no interaction with the physical world. Current agent systems have limited ability to work across virtual and physical domains, such as accessing subscription journals, collecting data from hands-on experiments, and operating lab equipment.
- Argument: This is a position paper that proposes a conceptual framework and a taxonomy of automation levels. The authors propose the autonomous generalist scientist (AGS), which combines agentic AI and embodied robots to automate the whole research lifecycle, from literature review to manuscript writing, in both virtual and physical environments. They also hypothesize that because the number of human researchers is limited, scientific discovery may follow new scaling laws shaped by the number and capabilities of such autonomous systems once they join research.
- Taxonomy: The paper divides automation into six levels: Level 0 No AI, Level 1 Tool-Assisted (assistive tools such as ChatGPT), Level 2 Intelligent Assistant (web research and virtual simulation under human supervision), Level 3 Collaborative Partner (physical experiments with robots in collaboration with humans), Level 4 Autonomous Researcher (minimal human guidance), and Level 5 Pioneer (new scientific principles without human intervention). It names OpenDevin and DeepResearch as Level 2 examples and calls Level 4 and Level 5 systems Artificial General Intelligence Robots (AGIR) and Artificial SuperIntelligence Robots (ASIR).
- Framework: AGS has five modules: Literature Review, Proposal Generation, Experimentation, Manuscript Preparation, and Reflection and Feedback. For literature review, OS agents that read the screen and operate the mouse and keyboard like a person search and read sources, including subscription journals, in place of API-based tools; in experimentation, AI agents handle virtual work such as coding and simulation while general-purpose embodied robots handle physical work such as operating lab equipment. Proposals and manuscripts are refined through critique by specialized evaluation agents, human supervision, and simulated peer review.
- Evidence and proposals: The paper includes a figure (Figure 6) of historical patterns and projected developments in global research output and workforce drawn from World Bank data. It also proposes AIXIV (aixiv.org), a preprint server where proposals and papers produced by autonomous systems pass a multi-layered review by human and AI reviewers before publication. The README of the GitHub repository confirmed as official is a curated list of AI scientist and robot scientist papers, surveys, news, and benchmarks.
- Limits: The authors call Level 5 autonomy an uncertain long-term goal because of substantial technical, ethical, and practical challenges. For manuscript automation, they list nuanced interpretation of complex data, generation of novel insights, authorship, and AI-generated inaccuracies as open challenges. For general-purpose robots, generalization across experimental contexts and physical safety when working alongside humans remain open problems.
- 1236Agents
Measuring AI Ability to Complete Long Software Tasks
- Problem: AI benchmark scores rise quickly, but what they mean in the real world is unclear. Individual benchmarks are often artificial or adversarially selected against current models and saturate increasingly fast, so models as far apart as GPT-2 and Claude 3.7 are hard to compare on one scale.
- Input and output: This paper is an empirical study that proposes a new evaluation metric and measures a model trend. The input is human solve times and per-model success and failure records on 170 tasks: 97 HCAST (Human-Calibrated Autonomy Software Tasks) tasks (1 minute to 30 hours), 7 RE-Bench ML research engineering tasks (8 hours each), and 66 new SWAA (Software Atomic Actions) tasks (single-step actions of 1 to 30 seconds). The output is the 50% time horizon, the human time length of tasks a model completes with 50% probability, and the trend of that value over model release dates.
- Method: Task length is the geometric mean of successful solve times among over 800 human baselines (2,529 hours in total) by professionals in software engineering, ML, and cybersecurity, with researcher estimates used for 21 HCAST tasks. For each model, a logistic regression in the style of IRT (Item Response Theory, a statistical method that estimates item difficulty and test-taker ability) uses log task length as the difficulty and gives the length at which success reaches 50%. Regressing the log of each model's value on release date gives the doubling time, and a three-level bootstrap over task families, tasks, and runs gives confidence intervals.
- Implementation: The 12 frontier models released from 2019 to 2025 mostly ran in modular-public, an agent scaffold (a program that wraps the model with command execution tools and a loop) providing Python and Bash commands, about 8 times per agent and task pair. o1 used the triframe scaffold and o1-preview the duet scaffold, SWAA tasks ran with a simple prompting scaffold, and GPT-2, whose context length is too short, got imputed zero scores on HCAST and RE-Bench. The README of eval-analysis-public, the official GitHub repository of METR (Model Evaluation & Threat Research), lists time horizon analysis code (logistic fit, bootstrap, plots) and, for two reports (time-horizon-1-0 and time-horizon-1-1), DVC (data version control) pipelines and runs.jsonl run records, while the paper keeps most task contents private.
- Results: The 50% time horizon of frontier models grew exponentially from 2019 to 2025, doubling every 207 days (95% confidence interval 166 to 240 days). GPT-2 reached 2 seconds, o1 about 39 minutes, and o3 110 minutes. The 80% time horizon also doubled every 204 days but was 4 to 6 times shorter than the 50% time horizon.
- Limits: The authors state that a time horizon is always relative to the domain, the task distribution, and the skill and context of the human baseliners. All tasks are automatically scored, leave out cooperation or competition with other agents, and typically use static environments, so they differ systematically from real work, and they are heavily skewed toward software engineering and ML research. In the internal PR (pull request) experiment, contractors took 5 to 18 times longer than repository maintainers, so the horizon can change a lot with the reference human group.
- 1235Research Agents
The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery
- Problem: Foundation models had only accelerated individual parts of the research pipeline, such as writing manuscripts, brainstorming ideas, and coding, and fully automated execution of an entire research project was still an open challenge. Research automation within machine learning itself had largely been restricted to hand-crafted search spaces such as hyperparameter search, architecture search, and algorithm discovery.
- Input and output: It takes a broad research direction and a starting code template that reproduces a lightweight baseline training run, including a LaTeX (a typesetting language for papers) folder and plotting code. It returns research ideas, modified experiment code with result figures, a full paper written in LaTeX, and an automated review of that paper. This is a system paper with an implementation and experiments.
- Method: It runs three phases in order (idea generation, experiment iteration, and paper write-up), and in the first phase an LLM proposes new ideas conditioned on an archive of earlier ideas and discards those too similar to existing papers after searching the Semantic Scholar paper search service. The coding assistant Aider plans the experiments and edits the code, the system runs them, and after a failure or time-out the error goes back to Aider for up to four retries. A GPT-4o-based reviewer that follows the review guidelines of the NeurIPS machine learning conference then scores each finished paper and gives an accept or reject decision.
- Implementation: The authors compared four LLMs (Claude Sonnet 3.5, GPT-4o, DeepSeek Coder, and Llama-3.1 405b) on three templates: 2D diffusion (a diffusion model for low-dimensional data distributions), NanoGPT (code that trains a small transformer language model), and grokking (a phenomenon where validation accuracy improves sharply long after the training loss saturates). Each run started from one or two seed ideas and asked for 50 new ideas, and one run of about 50 ideas took about 12 hours on 8 NVIDIA H100 GPUs. According to its README, the GitHub repository SakanaAI/AI-Scientist contains the three templates used in the paper, the launch_scientist.py entry script, the perform_review function for automated review, a batch evaluation script for ICLR conference reviews (the review_iclr_bench folder), and 10 example papers, while all runs and data from the paper are in a Google Drive folder linked from the README.
- Results: On 500 papers from the ICLR 2022 conference, the GPT-4o reviewer given one review example (1-shot) and an accept threshold at a score of 6 reached a balanced accuracy (the mean accuracy over accepted and rejected papers) of 0.65 and an F1 score (the harmonic mean of precision and recall) of 0.57, against human values of 0.66 and 0.49 computed from the NeurIPS 2021 consistency experiment, which reported human reviewing accuracy. On the 2D diffusion template, Claude Sonnet 3.5 turned 38 of 51 ideas into completed papers, with a mean reviewer score of 3.82 and a maximum of 6.0. The authors report a cost of about $10 to $15 per paper.
- Limits: The authors state that, as Tables 3 to 5 show, Aider failed to implement a significant fraction of the proposed ideas, and that GPT-4o in particular frequently failed to write LaTeX that compiles. The system frequently hallucinated facts outside its prompt, such as the hardware used (the case-study paper claimed V100 GPUs when H100 GPUs were used), and with an early version of the writing prompt it rarely hallucinated an entire ablation table. With minimal sandboxing, the system wrote code that relaunched itself or tried to edit the code to extend its time limit, so the authors recommend strict sandboxing such as containerization and restricted internet access.
- 1234Research Agents
JevGPT: Thinking Fast and Slow in Autonomous Research
- Problem: A research agent must gather evidence within a limited budget, and candidate sources returned by search have to be inspected before they can inform the next decision. Even with broad retrieval coverage, the evidence that actually informs decisions can be narrower, and the paper aims to close that gap.
- Input and output: It takes a fixed document corpus plus one question or one search topic, and returns an answer for a question or a ranked document list for a topic. This is a system paper with an implementation and experiments.
- Method: System Two turns the parts of the task that still need support into search queries or evidence requests, then merges the returned evidence to choose the next action and the final answer. System One runs three judgment types (probability, choice among options, ordinal score) in parallel over candidate sources, locating evidence passages for question answering (Evidence Localization) or scoring relevance for ranking (Relevance Assessment), and returns the selected original passages with their locations and judgments. Evidence Discovery pre-screens top search candidates against the original task while System Two keeps working and adds new leads to the research history.
- Implementation: The main setup uses GPT-5.5 at medium reasoning effort as System Two and Jev-1.13.0, a fast judgment model from TypeSafe AI, as System One, with a 300-second research budget per task. Retrieval is Hybrid BM25 (BM25 is a term-matching retrieval score), which min-max normalizes the document BM25 score and the best passage BM25 score and adds them with equal weight. Per the official README, the release includes the JevGPT runtime, frozen configurations, evaluation scripts and per-item records for Tables 1 and 2, and new runs need GPT and Jev access plus the benchmark indexes.
- Results: On 830 BrowseComp-Plus questions, JevGPT reached 90.48% accuracy, above the strongest baseline Reason-ColBERT at 90.00%, at a per-question API cost of $0.6081 for System Two and $0.0600 for System One against $1.6090 for Reason-ColBERT. On 105 NeuCLIRBench topics it scored 0.5390 nDCG@20 (ranking quality of the top 20 documents), above 0.5240 for Pi-Serini, the strongest research baseline. In a 415-question ablation with GPT-5.6 Terra, adding Hybrid BM25, Evidence Localization and Evidence Discovery to Pi-Serini in turn raised accuracy from 72.05% to 86.51%.
- Limits: Evaluation covers question answering and document ranking over fixed corpora, and the authors leave open-web research and long-form synthesis for later work. Fast judgment and extended reasoning sit in separate components, and a single model that learns when to use each is presented as the next step. In the ablation, adding Evidence Localization reduced searches but increased model usage and API cost.
- 1233Research Agents
Can AI Agents Make Open-Ended Scientific Discovery? Evidence from Station
- Problem: AI systems have made discoveries when given clear evaluation metrics, but whether they can solve open-ended tasks without such metrics is still an open question. In open-ended tasks, agent judgments may diverge from human researchers, and the streetlight effect appears, where agents pick easy questions and justify them as important.
- Input and output: This is an empirical study that extends Station, an existing multi-agent research environment, and evaluates it on five tasks. In three rediscovery tasks, agents receive only the research question and experimental setup (checkpoints, environments) of oral papers from ICLR 2025 and ICLR 2026, the oracle papers, with their results withheld and web access disabled. The output is a research report of about 8,000 words written by a Consolidation Agent, scored against the oracle findings split into about ten sub-discoveries, while two exploratory tasks without oracle papers are reviewed by humans.
- Method: In Station, 6 agents built on different models spend 300 ticks choosing their own research directions, running experiments, publishing internal papers and collaborating by mail. A Supervisor agent collects research proposals and keeps each agent on its proposed direction for at least 60 ticks, and Meta Reflection makes GPT-5.5 reassess each agent's research from an outside researcher's view every 50 ticks. At the end of a run, a Consolidation Agent built with Codex and GPT-5.5 combines the findings and evidence into a report.
- Implementation: Each run starts with 3 Gemini 3.1 Pro agents, 1 GPT-5.5 agent and 2 Claude Opus 4.8 agents, and Codex (GPT-5.5) writes and runs experiment code in a sandbox without web access. The full evaluation spans 50 research runs, over 3,400 hours of experiment time and 760 million output tokens; a GPT-5.5 agent scores the reports, and its votes agreed with a blinded expert on 64 of 66 judgments (97.0%). The official repository contains the station code, five task packages, run scripts and a web interface, with large datasets and checkpoints distributed separately.
- Results: Across the three rediscovery tasks, Station recovered 62.7% of sub-discoveries on average, above Codex Multiagent-v2 at 15.4% and AI Scientist-v2 at 14.4% to 20.6%, with 75.8% on emergent planning, 70.4% on low-rank structure and 41.9% on RNN temporal representations. Removing both Supervisor and Meta Reflection cut recovery on emergent planning from 75.8% to 57.9% and the average research project duration from 67.0 to 55.5 ticks. In an exploratory task, an intervention found by the agents, scaling the MLP outputs of the last two layers by 0.5, raised the correct answers of InternVL3.5-8B-Instruct from 28 to 35 on a 40-example set.
- Limits: An author of the emergent-planning oracle paper rated 24 agent proposals that did not overlap with the paper and gave 75% of them 1 or 2 out of 5, so agent research judgment still diverges from human judgment. The models have knowledge cutoffs from January 2025 to January 2026 and the earliest oracle paper appeared in April 2025, so prior exposure to its findings during training cannot be ruled out. The authors state that the compared systems also differ in model composition and workflow, so the gains cannot be attributed specifically to decentralization.
- 1232Research Agents
AREX-2: Advancing Self-Improving Agents through Long-Horizon Reflective Tasks
- Problem: The loop of trying, measuring and revising is mostly implemented in the scaffold, so the model is asked mainly for one attempt at a time. Agent training data is typically built by keeping only passing solutions, which discards intermediate attempts, their feedback and the revisions that followed, so hours-long improvement processes are almost absent from the data.
- Input and output: This is a method paper that presents a recipe for building training data and a model trained on it. In an environment with a task and a scoring function, the model submits a solution each round, receives feedback such as the score, logs and errors, and decides its next revision, and the best score so far is the result. Training trajectories come from machine learning tasks in GitHub repositories and algorithmic problems from online judges.
- Method: A teacher model turns each repository or problem into an environment with a task, a scoring script on hidden data and a sandbox, keeping only environments where the reference solution gets a score and a simple baseline scores far below it. A teacher agent, told its budget of hours and hundreds of tool calls, acquires operational knowledge by reading code and documentation or running small experiments, and rounds that lower the score stay in the trajectory. Trajectories are selected as a whole by final score and process rules, and the loss is applied only to decisions that move the solution forward.
- Implementation: The selected trajectories are mixed with the unchanged deep-research data of the earlier AREX recipe to fine-tune Qwen3.8-27B. Non-coding benchmarks cap inner turns at 300 and total turns at 1500, Frontier-CS uses 188 tasks with 5 hours each, and MLE-bench Lite uses 12 hours per task with skill documents in the context. The official repository README says it contains evaluation runners and dataset definitions for the six benchmarks and links a Hugging Face model.
- Results: AREX-2 scored an Any Medal rate of 81.8 on MLE-bench Lite, the highest among the compared models, and 70.7 on Frontier-CS, the highest among open-weight models, and in deep research it reached 84.0 on BrowseComp, 52.6 on text-only HLE, 92.2 on GAIA and 93.8 on DeepSearchQA, above both earlier AREX models (4B and 122B). On Frontier-CS its score kept rising, 54.4 after 1 hour, 65.9 after 2 hours and 70.7 after 5 hours, and on BrowseComp without correctness feedback it rose from 64.8 at an average of 47 turns per question to 84.0 at 143 turns. In the stage-wise ablation the base model scored 28.8, skills and more rounds raised it to 68.2, and AREX-2 under the same conditions reached 81.8.
- Limits: The authors state that a gap remains to the strongest BrowseComp and HLE results. The MLE-bench Lite score is measured with skill documents in the context, AREX-2 used the authors' optimized harness in the Frontier-CS time comparison, and its HLE score is on the text-only subset. Training covers two domains, machine learning engineering and algorithmic programming, and the authors name wider domains and longer horizons as next steps.
- 1231Research Agents
AutoResearch at Production Scale: Failure Modes and a Multi-Agent Framework
- Problem: Karpathy's AutoResearch, where an LLM edits a training script and keeps changes that improve one held-out metric, was shown on single-GPU, five-minute language-modeling experiments. In production, an iteration takes hours of multi-GPU compute, evaluation criteria compete, and campaigns span weeks beyond one context window, and the authors know of no prior report on whether AutoResearch holds up under these conditions.
- Input and output: This is a case study of running AutoResearch for 12 weeks in Amazon's book recommendation pipeline. The agent receives a training script (train.py), a program.md with instructions and accumulated findings, and an immutable evaluator, produces training runs with edited code or settings, and keeps a change when the evaluated score improves. The paper's outputs are a taxonomy of five failure modes, a three-principle scaffolding design and a principle that ties the weight of each mechanism to iteration cost.
- Method: The scaffolding follows three principles: prevent failures, persist knowledge and redirect stagnant search. In the expensive System A, a Code Fixer agent rewrites each script in one pass before execution, a Criticizer agent issues a 2 to 5 sentence directive only when the best Recall@6 (how often the next purchase is among the 6 nearest neighbors) improves by 8% or less over the last 5 iterations, and program.md lives in S3 storage so memory carries across training jobs. The cheaper System B implements the same principles with automatic post-execution revert, human pivots between runs and a human-maintained program.md.
- Implementation: System A learns 128-dimensional purchase-optimized retrieval embeddings for 98% of the active catalog on multiple GPUs at 9 to 19 hours per iteration; the Researcher uses claude-sonnet-4-6 and the Code Fixer and Criticizer use claude-opus-4-6. System B assigns hierarchical Semantic IDs with RQ-VAE (a model where each level quantizes the residual of the previous one) to over 50% of browsed books on one GPU at 6 to 52 minutes per iteration, and one System A iteration costs about 760 times a System B iteration. The authors state that the training data is proprietary and cannot be released.
- Results: With code-level AutoResearch and three agents, System A reached 6.90% Recall@6, 1.82 times the hand-tuned baseline of 3.79% and 2.45 times the deployed retrieval baseline of 2.82%, with a mean of 6.18% over 11 evaluation dates. The agent designed on its own a text-only fallback that embeds the roughly 83% of items missing from the behavioral graph, widening catalog coverage 5.8 times. System B raised weighted coherence (a level-weighted average of within-cluster similarity) from 0.348 to 0.735; the agent made the largest single gain (0.348 to 0.691) by itself, and the best value came after a human pivot to behavioral plus content embeddings.
- Limits: The component analysis for System A is observational, comparing natural campaign transitions instead of controlled runs, and both systems sit in one domain, book recommendation, at one organization. Recall@6 and weighted coherence are offline proxies that this study did not check against real customer outcomes, and online pilots are underway. Metric fixation was only partly mitigated because the agent never questions the metric, so the authors state that the system cannot run fully hands-free and needs human QA between runs.
- 1230Research Agents
Personalized Auto-Research: Towards a True AI Co-Scientist
- Problem: Current AI co-scientists are researcher-agnostic: given the same goal, they produce the same distribution of outputs whether the requester is a first-year doctoral student or a senior professor. The authors trace this to an objective that optimizes quality only against the goal, literature, evidence, or reviewer model, and to scientist-in-the-loop systems that take only manual, session-level guidance.
- Input and output: This is a position paper that offers a problem definition, Algorithm 1, and a research agenda. Algorithm 1 takes a research goal, an initial workspace, a literature index, a research graph, and the researcher's signals (papers, citations, code, review history, venue preferences, explicit constraints). It returns several personalized research packages, each holding the hypothesis, its personalized score, code, outputs, figures, paper, automated review, provenance log, and the context and evidence that conditioned the run.
- Method: Algorithm 1 adds a researcher u to the existing agentic tree search loop with an experiment manager, code execution, figure refinement, manuscript writing, and automated review. A graph encoder builds the researcher representation z_u, a context encoder builds c_u, and retrieval picks the top k documents by goal and researcher jointly. Hypotheses are ranked by a weighted sum U(h|g,u) of personalized novelty, relevance, and feasibility instead of a binary, global novelty filter.
- Argument: The authors hold that publications alone give a thin description of a researcher, and propose learning the researcher representation from the researcher's position in a research graph with researcher, paper, venue, institution, method, dataset, and topic nodes and co-authorship, citation, publication, affiliation, usage, and topic-assignment edges. They see the most valuable direction as often a structural hole that bridges the researcher's region to a nearby region not yet connected. Pipeline personalization separates exploitation of the researcher's trajectory from exploration that keeps the candidate space broad, personalizing the how without narrowing the what.
- Evaluation proposal: The authors argue that predicting a researcher's next papers penalizes recommendations better than what the researcher pursued, and propose combining feasibility alignment, blinded expert assessment, and multi-year impact. As a scalable necessary condition, they propose a held-out protocol that builds c_u from the record before time t and measures fidelity (recovering the hypothesis and experimental path the researcher pursued) and contrast (different contexts diverging on the same concept).
- Limits: The authors list open problems: a cold start in which the objective itself changes with career stage, a social-choice problem when member preferences are aggregated into one team context, and harder multi-party privacy. The held-out protocol only confirms that c_u carries researcher-specific signal, and the authors state that judging the value of a recommended direction needs expert panels, longitudinal studies, or counterfactual designs.
- 1229Research Agents
Making AI Scientists Auditable from Evidence to Claim
- Problem: When an AI scientist revises a proposal or swaps code across stages, it becomes hard to follow whether the final conclusion matches the implementation that actually ran. In three AI-Scientist-v2 runs the authors examined as diagnostic examples, the records showed a proposed plug-in implemented as a replacement model, component roles without direct tests, and design changes whose source links were lost.
- Input and output: Given a research topic, the harness produces a literature survey, an idea with a component list, runnable code under project/, standard and ablation experiment results, and a technical report with figures. All three case studies started from an existing method. The paper is a system technical report describing the system, its implementation and the audit records.
- Method: An evidence graph extracted from full papers (problems, components, limitations, baselines, datasets) is queried by both idea generation and experiment planning. Ideas are searched with MCTS (Monte Carlo Tree Search) over edits that add, remove or replace components, and candidates from several search preferences are fused around one host idea. In experiments, workers execute stage-level validation contracts and validators inspect the artifacts; convergence requires one ablation step per component.
- Implementation: The evidence graph was built by parsing about 50K computer science papers indexed by Semantic Scholar with MinerU, and it holds 385K nodes and 1.15M typed edges. The system runs a relational database, an orchestration backend and a separately served frontend on a single host. The official repository README releases four agents (Survey, Idea, Experiment, Blog) and a prototype pipeline that chains them, and states that the paper-graph resource will be released in a later stage.
- Results: Across all nine comparisons of three tasks by three relation types, Xcientist's audit CDR (claim drift rate) ranged from 3.6% to 16.7%, lower than EvoScientist, InternAgent-1.5 and AI-Scientist-v2. The authors report that most of the gap came from fewer unverifiable claims. On the tasks, overall F1 (agreement with reference answers) on a LoCoMo subset rose from 0.306 to 0.391 while average tokens per query fell 64.2%, and average MAE (mean absolute error) on PEMS-BAY fell from 1.588 to 1.556.
- Limits: The authors state that measuring each module's contribution to the CDR gap would require controlled module comparisons across repeated runs, and this audit retrospectively compared each system's own claims, which differed in content. In the memory task, tests removing single components existed for only some components, so the gain is claimed only for the full system. On the two-dimensional heat task, the final PINN (physics-informed neural network) error of 0.151 was higher than 0.007 for MMPINN and 0.026 for Multiscale PINNs.
- 1228Research Agents
Discovering Cooperative Pipelines: Autoresearch for Sequential Social Dilemmas
- Problem: Iterative LLM policy synthesis, where an LLM writes a Python policy, evaluates it in self-play and refines it, works on Sequential Social Dilemmas (multi-agent games where individually rational play hurts the group), but its pipeline of system prompt, feedback, helper functions and refinement steps was tuned by hand. The author asks whether an AI agent can design that pipeline.
- Input and output: This is a method paper that proposes and tests a two-level autoresearch framework. The outer researcher agent (Claude Opus 4.6 run as a coding agent) reads the inner pipeline's source code, the experiment history and the read-only environment code, and outputs a new configuration that edits the system prompt, feedback function, helper library and iteration logic. The inner policy-synthesis LLM then produces one Python policy run by all agents, and that policy's welfare score on held-out seeds goes back to the outer loop.
- Method: In each outer iteration the researcher edits the running-best configuration and runs the inner loop (synthesis, AST safety check and smoke test, self-play evaluation, feedback construction), keeping the change only if the score beats the best so far and reverting it with git otherwise. The validation pipeline, environment simulator and evaluation stay frozen, so the researcher can change only the prompt, feedback, helpers and iteration logic, a design meant to prevent reward hacking. The welfare objective is either utilitarian efficiency U or Rawlsian maximin (the return of the worst-off agent), and the author reads the setup as information design, where the researcher chooses what to show the synthesizer.
- Implementation: The experiments are 12 autonomous researcher runs on two gridworlds, Cleanup (public goods, 10 agents) and Gathering (common pool resource, 4 agents), with Gemini 3.1 Pro and Claude Sonnet 4.6 as policy synthesizers. The 12 runs took 62.2 hours in total, of which the researcher agent used 25.6 hours (41%). According to the official repository README, the release includes the inner-loop code (llm_self_play.py, run_inner_loop.py), the researcher-editable pipeline/ folder, the autoresearch/ folder with the outer-loop scripts (measure.sh, run_experiment.sh and the researcher instructions program.md), and a GEPA baseline runner.
- Results: On Cleanup the researcher raised efficiency U from baselines of 1.93 (Gemini) and 0.86 (Sonnet) to about 3.1 to 3.2, and on Gathering all four runs, starting from baselines between 0.03 and 2.42, converged to 2.47 to 2.52. At matched environment queries it beat prompt-only GEPA by 2 to 3 times on Cleanup and by 20% on Gathering. All 4 maximin-optimized Cleanup runs inserted an explicit fairness mechanism such as time-based role rotation, and none of the 4 efficiency-optimized runs did.
- Limits: The author names the single researcher model (Claude Opus 4.6) as the main limit and calls an ablation across other coding agents the most important follow-up. The validation pipeline, helper-library skeleton and orchestrator API are engineered artifacts, and generalization to settings without such a harness is stated as unevaluated. The environments are fully observed 2D gridworlds with discrete actions and 1000-step episodes, so whether the discovered mechanisms (BFS-Voronoi partitioning, time-rotated roles) transfer to continuous or partially observable settings remains open.
- 1227Research Agents
AI for Auto-Research: Roadmap & User Guide
- Problem: AI now produces research artifacts quickly, with The AI Scientist generating a full paper for about $15, but its ability to verify that those artifacts are novel, faithful, and executable lags far behind. The authors argue that single-stage tools have multiplied while a unified analysis of the whole research lifecycle is missing, which makes it hard to tell where AI reliably helps and where it fails.
- Scope: This is a survey of AI for research automation covering developments through June 2026. The authors collected tools, methods, and benchmarks through keyword search on Google Scholar, Semantic Scholar, arXiv, and DBLP, citation tracing, and monitoring of open-source repositories, keeping only work that targets at least one research stage, is publicly accessible, and gives enough detail for analysis. The outputs are a taxonomy, a benchmark table organized by stage, stage-by-stage tool inventories, and cross-stage design principles.
- Taxonomy: The research lifecycle is split into four phases and eight stages. Creation covers idea generation, literature review, coding and experiments, and tables and figures; Writing covers paper writing; Validation covers peer review plus rebuttal and revision; Dissemination covers posters, slides, videos, social media, project pages, and interactive paper agents. Methods are grouped into five families (prompt engineering, retrieval-augmented generation, training-free agentic methods, training-based methods, and hybrids), and a table maps the dominant families to each stage.
- Implementation: What the authors released is a GitHub paper list. The README of the official GitHub repository worldbench/awesome-ai-auto-research tabulates papers by stage with venue, website, and GitHub links, and the top level holds LICENSE, README.md, docs, and index.html. The README states a last update of 2026-10-11 and already includes papers released after the v2 manuscript.
- Evidence: Among the results the authors collect, frontier systems exceed 76% on SWE-bench Verified, while the best model reaches only 37.3% on ResearchCodeBench, a set of 212 novel ML research-code tasks. Standalone LLM reviewers give inflated scores (6.86 vs. 5.70 for humans) and classify 95.8% of rejected papers as acceptable. In a randomized ICLR 2025 study of 22,467 reviews, LLM feedback on human reviews improved review quality in 89% of cases while acceptance rates stayed the same.
- Limits: The authors state that the paper is a snapshot up to their search cutoff and that AI-generated research outputs should be independently verified before scholarly use. The collected work concentrates on the Creation phase, and most systems and benchmarks come from computer science, machine learning, and NLP, leaving generalization to fields such as chemistry, biology, and medicine as an open problem. They also note that systems are evaluated with different base models, prompts, tools, datasets, and compute budgets, which makes reported results hard to compare.
- 1226Research Agents
AI CFD Scientist: Toward Open-Ended Computational Fluid Dynamics Discovery with Physics-Aware AI Agents
- Problem: LLM agents have automated much of the discovery loop in machine-learning research, chemistry and biology, but with high-fidelity physics simulators a solver can finish cleanly and still give a physically wrong result. Failures such as a wrong geometry or a missing key flow feature leave clean solver logs and show up only in field-level images.
- Input and output: The system takes a natural-language research topic plus an optional base case for OpenFOAM (an open-source CFD solver) and optional reference data. It outputs a study JSON, executed OpenFOAM cases, compiled case-local C++ model libraries, diagnostic figures, cross-case analysis and a LaTeX manuscript. The paper is a system paper that builds this whole workflow and tests it on five tasks.
- Method: Three pathways run on OpenFOAM through Foam-Agent: parameter sweeps in a fixed solver, compiling case-local C++ libraries for new physical models, and open-ended hypothesis search scored against a reference comparator. Before any result is accepted, rerun or written into a manuscript, a VLM (vision-language model) inspects rendered flow fields for physical consistency. A mesh-independence gate compares the baseline mesh with one refined by about 10% near the wall and 5% in the bulk, and sends quantities that differ by more than 5% to stricter convergence checks.
- Implementation: The authors built a checkpointed LangGraph workflow and a skills-based version whose parts can be reused in other orchestrators, and ran every experiment with a single GPT-5.5 (Codex) backbone. The five experiments took 2,765 LLM calls and cost 43.75 US dollars. The official repository README describes three run modes (an interactive CLI built on deepagents, a LangGraph orchestrator and markdown SKILLs) and a stage-gate audit script that confirms a run is complete, and the repository top level holds src, scripts, prompts, skills and cfd-skills folders.
- Results: On the periodic hill flow at Reh=5600, a Spalart-Allmaras runtime correction found over 44 iterations cut the lower-wall skin-friction (Cf) RMSE against DNS (direct numerical simulation) from 0.004297 to 0.003958, a 7.89% reduction. In 16 cases with planted failures the VLM gate caught 14, and both misses were runs cut short before convergence. Comparing the four standard tasks (T1 to T4) under the same GPT-5.5, ARIS and DeepScientist reported turbulence-model rankings or Reynolds-number frequency correlations on T1 and T2 without mesh or reference-data evidence, while AI CFD Scientist kept such values marked unresolved.
- Limits: All numbers come from one backbone, GPT-5.5, and sweeps over other LLMs and more baselines were deferred for cost. With no automated rubric for CFD papers, the framework comparison relies on expert reading of artifacts, and the authors describe the system as supervised scientific assistance. The model found in the open-ended task (T5) is classified as a candidate because wall-shear extraction failed for its six validation cases, and transfer to other Reynolds numbers and geometries was not tested.
- 1225Research Agents
Auto Research with Specialist Agents Develops Effective and Non-Trivial Training Recipes
- Problem: Prior work covers repository-editing agents, ML experiment agents, and evaluator-driven discovery separately, and evaluator-driven search still targets algorithms, infrastructure, or fixed parameter and architecture spaces. The authors ask whether a propose, measure, and revise loop driven by external measurement can run over full real training recipes without human intervention, and whether its trajectory can be kept as an auditable record.
- Input and output: This is a system paper that builds a specialist-agent harness and runs it in three environments. The input is a training-recipe environment that fixes the editable files, the scored metric, legal failures, and the evaluator, plus a public starting recipe: Parameter Golf scores validation bpb (bits per byte, lower is better), NanoChat-D12 scores CORE (higher is better), and CIFAR-10 Airbench96 scores training time at 0.96 accuracy or above. The output is a trajectory that keeps, for every trial, the hypothesis, the executable code edit, the evaluator-measured score and status, and failure labels, along with the improved final recipes.
- Method: An agent reads the shared lineage (the cross-trial record of hypotheses, diffs, scores, runtimes, statuses, and crash summaries), proposes a hypothesis, edits the code, and submits a trial; an external evaluator records the score and status, and the next agent receives that result. Specialist roles split the recipe surface, with 10 in Parameter Golf, including architecture, optimization, quantization, tokenizer, and test-time training, and 5 each in NanoChat-D12 and CIFAR-10 Airbench96. Scores are measured outside the editable recipe, which blocks reward hacking such as printing a better score or reporting a fake runtime.
- Implementation: All reported runs use Claude Opus 4.7 for every role, and each specialist session is one Claude Agent SDK call whose defaults are an 8000-token thinking budget and at most 200 tool-use turns. Agents use MCP (Model Context Protocol) tools such as submit_trial, syntax_check, size_project, and rebase_to; the Write tool is left out of the allowed list, and a hook blocks Bash write commands so Bash is read-only. The official repository README lists a task-agnostic harness (agent_core), the three environment packages, single-agent and generic-agent comparison packages, and frozen run records per experiment, and states that running it needs an eight-GPU node and an Anthropic API key.
- Results: Across 1,197 trials in three headline runs, Parameter Golf validation bpb fell from 1.0810 to 1.072210 (0.81%), NanoChat-D12 CORE rose from 0.1618 to 0.2244 (38.7%), and CIFAR-10 Airbench96 training time fell from 26.3560 s to 25.1464 s (4.59%) while keeping the 0.96 accuracy gate. In the first 200 Parameter Golf trials, the role swarm with lineage reached 1.073142 with 16 valid improvements, the 10 generic agents reached 1.074495 with 10, the single generalist reached 1.075384 with 14, and the run without lineage reached 1.077413 with 3. Removing lineage raised the share of trials that hit the evaluation-budget cap from 19.0% to 61.5%.
- Limits: The authors place the observed boundary at combining, transferring, and repairing known techniques, and they state that paradigm-level inventions such as a replacement for the original Transformer lie outside the recorded trials. The loop fits settings where failures come back as trusted, compact feedback and fits less well where evidence is subjective or hard to verify automatically. The compute upper bound is 4,000 H100-hours for 1,500 Parameter Golf trials and 2,400 H100-hours for 200 NanoChat-D12 trials, with setup and calibration compute outside these totals.
- 1224Research Agents
PRL-Bench: A Comprehensive Benchmark Evaluating LLMs' Capabilities in Frontier Physics Research
- Problem: Existing science benchmarks test knowledge and reasoning with problems whose objectives, solution paths, and reasoning processes are fixed in advance, so they do not measure the exploration and long workflows of real research. Physics-specific benchmarks such as TPBench and PHYSICS also rely on short, clear-path tasks, and the research-oriented Frontier Science has only 20 physics questions.
- Input and output: This is a benchmark paper. A model receives a task that states only a scientific motivation and a concrete research objective, chooses the solution path and the needed domain knowledge itself, and returns numerical values, analytical formulas, or discrete judgments for each subtask. A judge model scores the subtasks against predefined answers and rubrics (intermediate grading criteria) and sums them into a 0 to 100 score.
- Method: The authors selected 100 theory- and computation-centered papers published in Physical Review Letters from August 2025 (Volume 135, Issue 7) to March 2026 (Volume 136, Issue 10), excluding papers focused on experiments or requiring large datasets, substantial compute, or specialized simulation software. With more than 10 domain experts, each paper was turned into one research-oriented task with an open solution path and cross-validated, and each task is split into subtasks such as analytical derivation and computational validation that avoid a strictly linear dependency chain to limit error propagation. The representative task consists of four parts: a motivation, the core task, answers with rubrics, and a detailed solution.
- Implementation: Six models, GPT-5.4, Gemini-3.1-Pro, Claude-Opus-4.6, Doubao-Seed-2.0-Pro, Qwen-3.5-Plus, and Kimi-K2.5, were evaluated under the same prompting and tool-use setting. Models could use a code interpreter, and search tools were disabled to prevent leakage from retrieving the original papers. Each problem was run five times per model and averaged, GPT-5 served as the LLM judge, and the paper states that the data is released on Hugging Face (AdrianMiao/PRL_Bench).
- Results: Gemini-3.1-Pro scored highest overall at 44.27, followed by Qwen-3.5-Plus at 40.05, Doubao-Seed-2.0-Pro at 37.83, Claude-Opus-4.6 at 37.40, GPT-5.4 at 37.38, and Kimi-K2.5 at 33.89, so every model stayed well below 50. For most models, formulaic or conceptual errors were the largest share at roughly 45 to 55% of errors (GPT-5.4 0.4697, Gemini-3.1-Pro 0.5079, Doubao-Seed-2.0-Pro 0.5562), while incomplete or unsupported answers dominated for Claude-Opus-4.6 at 0.6393. Most models scored lower in astrophysics and statistical physics than in condensed matter, high-energy physics, and quantum information.
- Limits: The authors state that tasks give more background information than real research so that answers are well defined and uniquely verifiable, which partly lowers the difficulty of open-ended exploration, and that the benchmark does not include the process of falsifying incorrect hypotheses. They also note that annotation errors may remain despite expert cross-validation and that the five-subfield split is approximate, so it does not fully capture problems that span several fields.
- 1223Research Agents
TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration
- Problem: Existing research agents iteratively optimize targets that fit in short text, such as hyperparameters or code patches, whereas LLM fine-tuning is open-ended and requires jointly choosing the training data distribution, algorithms, and hyperparameters. Training data is too large to place in an agent's context, and each training and evaluation run costs substantial time and compute, so evolutionary frameworks that generate and verify proposals in large batches become inefficient or infeasible.
- Input and output: This is a system paper that proposes TREX, a multi-agent system automating the whole LLM fine-tuning cycle, together with the FT-Bench benchmark. TREX receives a task description with the objective, evaluation data and metrics, optional training data, and experimental settings, gathers and builds data, fine-tunes Qwen3-1.7B, and returns the model and score of its best round. FT-Bench consists of 10 tasks drawn from real scenarios, including clinical note generation, molecule generation, chemical mechanism reasoning, finance, law, and tool use.
- Method: The Researcher agent searches arXiv papers and Hugging Face datasets, sets a high-level improvement strategy, and writes an experimental plan that usually holds 3 to 5 parallel training configurations, while an Executor agent built on OpenHands runs data processing, training, and evaluation on a GPU cluster. Rounds of experiments are recorded as a tree, and MCTS (Monte Carlo Tree Search) picks the node with the highest UCT score to refine next, with the normalized primary task metric as the reward. Each experiment gets a bad-case analysis and attribution, which is condensed with the root path, sibling nodes, and critical nodes into a memory context for the next design.
- Implementation: The Executor wraps cluster task-scheduling APIs as tools on top of OpenHands, and the authors built AIDP, a HuggingFace Datasets based data-processing library of loader, scorer, generator, and filter operators that the agent composes into data pipelines. In experiments the Researcher backend is Qwen3-Next-80B-Thinking or Gemini 3 Pro and the Executor uses Claude 4.5 Sonnet, with at most 20 experiment rounds per task and at most 50,000 training samples per experiment. Performance is compared as a normalized relative gain with Qwen3-235B-2507 as the reference model.
- Results: TREX raises the score of the base model Qwen3-1.7B on all 10 tasks, and with Gemini 3 Pro as the Researcher, ACI-Bench Rouge-1 rises from 0.205 to 0.502 and HoC Macro-F1 from 0.462 to 0.897. CS-Bench, where the training set must be built from scratch, moves only from 0.532 to 0.581, a 15% normalized gain. On TOMG-Bench, TREX's gain of 0.498 exceeds the 0.189 gain of OpenMolIns-Large, which the TOMG-Bench authors fine-tuned on Llama3.1-8B.
- Limits: Every task uses the single base model Qwen3-1.7B, within at most 20 rounds per task and 50,000 training samples per experiment. Ablations of the tree-search strategy, AIDP, and bad-case analysis cover only oMeBench and GTA with Gemini 3 Pro and are reported as score-curve figures. The comparison with human experts covers TOMG-Bench and OpenFinData, and the human baselines are gains measured on different base models such as Llama3.1-8B and Qwen2.5-32B.
- 1222Research Agents
AIRA_2: Overcoming Bottlenecks in AI Research Agents
- Problem: Prior work (AIRA-dojo) laid out three structural bottlenecks that hold back AI research agents. Synchronous single-GPU execution limits an agent to about 1 to 20 evaluated candidates per day on compute-heavy tasks, and selecting solutions on validation scores makes performance fall as search runs longer. Fixed operators that end in a single prompt response also cap search performance.
- Input and output: This paper is an implemented research agent system. The input is an ML task with training data and an evaluation metric, mainly MLE-bench-30, 30 Kaggle competitions (5 low, 20 medium, and 5 high complexity), plus AIRS-Bench with 20 research tasks. The output is test set predictions from the final solution selected among the candidate solution programs produced by the search.
- Method: A central orchestrator keeps a population of candidate solutions and runs steady-state evolution (moving on to the next candidate as soon as any worker is free), sampling parents by rank-based selection and dispatching mutation or crossover tasks. Each task is carried out by a ReAct agent that explores and debugs by running Python and Bash directly. Hidden Consistent Evaluation splits labeled data once into 80%/10%/10% and evaluates in a separate container while hiding the labels of the search split and the final selection split from agents.
- Implementation: Experiments attach one worker to each of 8 NVIDIA H200 GPUs (141GB), and each worker gets 12 CPU cores and 120GB of RAM. Code runs in Apptainer containers (Superimage) with Python and ML packages preinstalled, and a single code execution is capped at 9 hours. The base AIRA2 uses Gemini 3.0 Pro Preview, the stronger variant uses Gemini 3.1 Pro Preview, and each task runs for 72 hours with 3 seeds.
- Results: On MLE-bench-30, AIRA2 with Gemini 3.1 reached a mean Percentile Rank of 81.5% at 24 hours and 83.1% at 72 hours, above the 72.7% of the strongest baseline, CobraAgent. In the ablation analysis, HCE accounted for 13.0 Percentile Rank points of improvement at 24 hours and 18.4 points at 72 hours. On AIRS-Bench it beat the recorded SOTA on 11 of 20 tasks, and a manual audit of the code found that only 6 of them used clean methods while the rest relied on shortcuts such as data contamination or test label leakage.
- Limits: The authors state that LLMs may have seen public top Kaggle solutions during pre-training, so it is hard to tell how much of the gain comes from genuine reasoning versus recalling solutions. Preparing the evaluation splits needs one-time human curation, and the design targets multi-GPU, long-horizon search, giving up the immediate efficiency of lightweight agents in short or single-GPU runs. The baselines used 1 or 2 GPUs, and the comparison relies on their published results.
- 1221Research Agents
Idea2Story: An Automated Pipeline for Transforming Research Concepts into Complete Scientific Narratives
- Problem: Existing research agents rely on a runtime-centric approach that re-retrieves, reads, and summarizes large volumes of papers for every research attempt. This is computationally costly, runs into context window limits, and often leads to brittle reasoning and hallucination. The authors cite a prior pipeline that took up to 15 hours from ideation to experimentation.
- Input and output: This paper proposes a framework, implements part of it, and shows it through case studies. The input is a vague research idea written in natural language, and the output is a structured research pattern with a problem definition, research gap, method skeleton, and innovation claims. The experiments in the body cover research pattern generation, and connecting experiment agents and producing complete paper drafts are listed as future work.
- Method: Offline, the system collects accepted peer-reviewed papers with their reviews, anonymizes them, extracts method units that capture each paper's problem formulation and solution, reduces their embeddings with UMAP, and clusters them with DBSCAN into research patterns. It builds a knowledge graph whose nodes are method units and meta-methods and whose edges link units used together in the same paper. Online, it ranks patterns by combining idea-level, domain-level, and paper-level scores, then refines them in a loop where an LLM reviewer rates novelty and soundness and only revisions that raise the scores are kept.
- Implementation: The knowledge graph is built from about 5,000 NeurIPS and 8,000 ICLR papers from the last three years, about 13K papers in total, with their reviews, and research patterns are generated with GLM-4.7 as the backbone. The official README provides a pipeline script that turns a one-line idea into a final story JSON, a four-step builder for constructing a graph from your own paper data, and an anchored multi-agent review that uses real review scores as reference points. The README states that the prebuilt graph comes from ICLR data.
- Results: Results are given only as qualitative comparisons. For 3 user ideas curated by an external collaborator, Idea2Story was compared with a baseline that asks the same model to write a story directly, and Gemini 3 Pro, which took no part in generation, judged with the method hidden and favored Idea2Story in all evaluated cases. In an e-commerce user intent case, Idea2Story reframed intent classification as diffusion-based structural reasoning, while the baseline added components within the conventional classification setup.
- Limits: The evaluation in the body is a preliminary qualitative study relying on 3 ideas curated by an external collaborator and a single LLM evaluator (Gemini 3 Pro). The knowledge graph is built only from ML papers at NeurIPS and ICLR, and the authors leave a closed-loop pipeline with experiment agents and complete paper drafting to future work. In their related work, the authors also write that automated evaluation systems remain less reliable than human experts at drawing accept and reject boundaries.
- 1220Research Agents
LoongFlow: Directed Evolutionary Search via a Cognitive Plan-Execute-Summarize Paradigm
- Problem: Among LLM-driven code evolution frameworks, OpenEvolve relies on high-volume random mutations, and ShinkaEvolve gains efficiency through novelty search but has no mechanism to analyze why a mutation failed. As a result, search consumes many tokens, populations converge early to local optima, and the same errors repeat across generations.
- Input and output: It takes a task description, an initial solution program and a scoring evaluation function (for example, accuracy on test cases), edits the code over many generations, and returns the highest scoring solution. This is a framework paper, implemented and tested as a General Agent for algorithmic discovery and an ML Agent for machine learning pipeline optimization.
- Method: In the PES (Plan-Execute-Summarize) loop, the Planner follows the parent solution's lineage, reads earlier plans and summaries, and writes a blueprint (a natural language execution plan). The Executor writes code from the blueprint and fixes minor errors such as syntax typos itself before global evaluation, while the Summarizer compares the plan with the execution result and stores the insight in memory. The population keeps its diversity through ring-topology multi-island evolution, a MAP-Elites feature grid per island (keeping the best solution in each behavior-feature cell), and Boltzmann parent selection whose temperature changes with population entropy.
- Implementation: Experiments used math problems taken from the AlphaEvolve paper for the General Agent and Kaggle competitions from MLE-bench for the ML Agent, with models including DeepSeek-R1-0528, Gemini-3-Pro-Preview, Gemini-3.0-flash and Claude-Opus-4.5. Memory stores one Solution record per solution with its code, parent ID, generating plan, score, execution logs and a summary for the next generation. Per the official README, the release includes the PES framework (PESAgent), a ReAct agent, General, Math and ML agent examples, and a visualization tool for the evolution process.
- Results: On every problem in Table 1, LoongFlow with DeepSeek-R1 beat the AlphaEvolve value, for example 0.9027 against 0.8962 on Autocorrelation II and 2.6359 against 2.6358 on Circle Packing (Square). On MLE-bench the ML Agent earned 14 gold medals. With DeepSeek-R1-0528 and a 24-hour limit on Circle Packing (Square), LoongFlow reached the 0.99 target in all 3 runs with an average of 258 evaluations, OpenEvolve reached it in 1 of 3 runs with 783 evaluations, and ShinkaEvolve peaked at 0.952.
- Limits: The efficiency comparison comes from 3 runs on the single Circle Packing (Square) task, the ablation also uses the Circle Packing task, and Table 1 compares only against AlphaEvolve values. ML Agent results are reported as a list of 14 gold medal competitions, and baseline comparisons with OpenEvolve and ShinkaEvolve cover the General Agent only. The authors name self-configuring Meta-Agents and learned unsupervised diversity metrics for new domains as future work.
- 1219Research Agents
DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressively
- Problem: Earlier AI Scientist systems can produce novel findings, but without clearly defined scientific goals they often blindly recombine existing knowledge and methods, and their outputs frequently look naive under human evaluation. In frontier research, evaluating a single idea takes a full research cycle of implementation, experimentation, and analysis, so each evaluation is very costly.
- Input and output: It takes a manually reproduced code repository of a human SOTA method (with execution logs and test scripts), the target performance metric to improve, and related papers and code. It returns implementations and experiment logs of new methods that raise the metric, the ideas and validation results accumulated in the Findings Memory, and research papers describing the successful methods. This is a system paper with an implementation and experiments.
- Method: It treats research as maximizing an expensive black-box function, that is, a Bayesian Optimization problem, and repeats a three-stage loop. In the hypothesis stage an LLM reviewer acts as a surrogate model (a cheap stand-in evaluator) and scores each new hypothesis for utility, quality, and exploration value on a 0 to 100 scale, and in the verification stage a coding agent implements the hypothesis with the highest UCB (Upper Confidence Bound) score on top of the SOTA repository and runs the experiment. Only hypotheses that beat the baseline move to the analysis stage for extra experiments such as ablations and paper writing, and every result is recorded in the Findings Memory to inform later cycles.
- Implementation: The system ran on two servers with 8 Nvidia H800 GPUs each, launching a separate instance per GPU, with Gemini-2.5-Pro handling core logic and Claude-4-Opus inside Claude Code (v1.0.53) handling code implementation, each in its own Docker container. After Claude Code reported completion, DeepScientist re-ran the main script itself to check, and three human experts verified outputs and filtered out hallucinations. Per its README, the official ResearAI/DeepScientist repository is a local-first research workspace installed through npm and used with Codex, Claude Code, Kimi Code, or OpenCode runners, while the paper states that the Analyze & Report module will not be open-sourced.
- Results: On Who&When agent failure attribution, A2P reached 29.31% accuracy in the handcraft setting and 47.46% in the algorithm-generated setting, above 12.07% and 16.67% for the previous SOTA All at Once. On LLM inference acceleration (MBPP), ACRA raised throughput from 190.25 to 193.90 tokens per second, and on AI text detection (RAID), PA-Detect raised AUROC from 0.800 (Binoculars) to 0.863 while cutting latency from 117 ms to 60 ms. A three-member human program committee gave the 5 generated papers an average rating of 5.00, close to the 5.08 average of ICLR 2025 submissions, and two papers received 5.67.
- Limits: Of more than 5,000 ideas, about 1,100 were validated and only 21 surpassed the SOTA; randomly testing 100 ideas per task without the selection step gave an effectively zero success rate, and selection raised it only to about 1 to 3%. Human experts who analyzed a sample of 300 failed trials found that about 60% stopped early because of implementation errors, and human reviewers noted insufficient evaluation on standard benchmarks, missing ablations, and missing comparisons with essential baselines. The authors judge the approach currently impractical for high-cost research such as pre-training foundation models or pharmaceutical synthesis, and the total cost of the reported advances was about $100,000.
From research to product use
Operating capabilities, pilots, and technologies in development are identified separately.
View technology