2025-9-9

An AI system to help scientists write

expert-level empirical software
Eser Aygün1,* , Anastasiya Belyaeva2,* , Gheorghe Comanici1,* , Marc Coram2,* , Hao Cui2,* , Jake Garrison3,* ,
Renee Johnston2,* , Anton Kast2,* , Cory Y. McLean2,* , Peter Norgaard2,* , Zahra Shamsi2,* , David Smalling1,* ,
James Thompson2,* , Subhashini Venugopalan2,* , Brian P. Williams2,* , Chujun He2,4,** , Sarah Martinson2,5,** ,
Martyna Plomecka2,6,** , Lai Wei2 , Yuchen Zhou2 , Qian-Ze Zhu2,5,** , Matthew Abraham2 , Erica Brand2 , Anna
Bulanova1 , Jeffrey A. Cardille2,7 , Chris Co2 , Scott Ellsworth2 , Grace Joseph2 , Malcolm Kane2 , Ryan
Krueger2,5,** , Johan Kartiwa2 , Dan Liebling2 , Jan-Matthis Lueckmann2 , Paul Raccuglia2 , Xuefei (Julie)
Wang2,8,** , Katherine Chou2 , James Manyika2 , Yossi Matias2 , John C. Platt2 , Lizzie Dorfman2 , Shibl Mourad1,‡
and Michael P. Brenner2,5,‡
1 Google DeepMind, 2 Google Research, 3 Google Platforms and Devices, 4 Massachusetts Institute of Technology, 5 School of
arXiv:2509.06503v1 [[Link]] 8 Sep 2025

Engineering and Applied Sciences, Harvard University, 6 Google Cloud, 7 Faculty of Agricultural and Environmental Sciences, McGill
University, 8 California Institute of Technology

The cycle of scientific discovery is frequently bottlenecked by the slow, manual creation of software to
support computational experiments. To address this, we present an AI system that creates expert-level
scientific software whose goal is to maximize a quality metric. The system uses a Large Language Model
(LLM) and Tree Search (TS) to systematically improve the quality metric and intelligently navigate the
large space of possible solutions. The system achieves expert-level results when it explores and integrates
complex research ideas from external sources. The effectiveness of tree search is demonstrated across a
wide range of benchmarks. In bioinformatics, it discovered 40 novel methods for single-cell data analysis
that outperformed the top human-developed methods on a public leaderboard. In epidemiology, it
generated 14 models that outperformed the CDC ensemble and all other individual models for forecasting
COVID-19 hospitalizations. Our method also produced state-of-the-art software for geospatial analysis,
neural activity prediction in zebrafish, time series forecasting and numerical solution of integrals. By
devising and implementing novel solutions to diverse tasks, the system represents a significant step
towards accelerating scientific progress.

Keywords: Tree Search, Generative AI, Scorable Scientific Tasks, Empirical Software

Introduction
Scientists need diverse information to advance their scientific agendas. Some are simple questions for
which perfunctory answers can be fulfilled by a search engine. However, performing computational
experiments often demands deeper information. For example, one of the authors’ research involves
deforestation analyses, assessing land cover change 1 using global spatially-resolved measurements,
past and present. This is carried out using a satellite-based deforestation detector, built with code
to answer a scientific question. A deforestation detector is one of many thousands of examples of
empirical software in science. We use the term empirical software to mean software that is designed
to maximize a definable or measurable quality score, typically a fit to existing observations. If a task
can be solved with empirical software, we call this a scorable task.
We have two hypotheses about the scorable tasks and empirical software in science. First, scorable
tasks are ubiquitous in science. Almost every sub-field of science, applied mathematics, and engineering
now relies on software. In the combined experience of the authors, we have found that much of this
software is empirical software solving a scorable task. Often such empirical software is at the heart of

* Equalcontribution in alphabetical order.

**Carried out as part of a student researchership at Google Research.
‡ To whom correspondence should be addressed: shibl@[Link], mbrenner@[Link]
 An AI system to help scientists write expert-level empirical software

a scientist’s work. Empirical software has recently enabled a number of Nobel Prizes in Chemistry: in
1998 for Density Functional Theory 2,3 , in 2013 for molecular dynamics simulation 4 and in 2024 for
protein structure prediction 5,6 . Empirical software underlies our ability to create models of complex
systems, ranging from parameterizations of a vertical column of the earth’s atmosphere for weather
modeling 7 , to the parameterization of stress response in a turbulent fluid flow 8 , to the prediction of
social systems 9–11 .
Second, empirical software for science is slow and difficult to create. Domain-specific empirical
software requires tedious work, often over many years. When empirical software is used to test
complex hypotheses, it becomes ever more difficult to write purely from first principles. There usually
is no systematic search for alternative approaches. Design choices are often governed by intuition or
expediency, rather than exhaustive experimentation. Creating the software is so time-consuming that
it severely limits the possibilities that can be productively explored.
This paper presents an AI-based system that systematically and automatically creates empirical
software to solve scorable tasks. Our method is based on an LLM that rewrites software to attempt to
improve its quality score. The system creates a number of software candidate solutions, and uses Tree
Search 12,13 to decide which candidates merit further exploration (Fig. 1a). While there are many
ways of designing a code mutation system 14–18 , we developed and refined the method by designing
and competing against a benchmark of basic Kaggle competitions (Fig. 1b), described below. We
augment code mutation with research ideas, obtained from a range of sources from highly cited
papers, to specialized textbooks, to results of search engines (Fig. 1c). In practice, these ideas can be
injected either directly by the user or automatically using a search engine to access research in the
literature. The LLM uses this injected guidance in writing code.
We find that our method can be applied to a wide variety of scorable tasks from across science,
producing software that outperforms the state-of-the-art produced by scientists. This superhuman
performance arises because of the ability to exhaustively and tirelessly carry out solution searches at
an unprecedented scale, identifying needle-in-the-haystack high quality solutions.

Results

Overview of Scorable Tasks

We develop our method on a benchmark of Kaggle playground competitions, and test it by selecting
scorable tasks based on scientific or engineering problems. We selected these problems using two
criteria: first, we chose tasks which have had slow recent progress, but yet are important to a set
of scientists; second, we chose tasks which would be useful to the scientific agenda of at least one
co-author. These scorable tasks are listed below.
scRNA-seq batch integration: 19 By removing confounding factors, we can enable large-scale multi-lab
transcriptomic data integration, such as the Human Cell Atlas 20 . This is a difficult problem because it
requires distinguishing subtle biological signals from noise in high-dimensional sparse datasets.
CDC COVID Forecasting: 21 By predicting COVID cases several weeks in advance, we can inform public
health policy and resource allocations. The challenge in this task arises from predicting non-linear
disease dynamics from lagged and noisy real-time data.
DLRSD segmentation: 22 This is a problem of performing dense pixel-wise multi-label semantic seg-
mentation on complex satellite imagery. Solving this problem can lead to large improvements in
environmental monitoring and disaster response.
ZAPBench: 23 This benchmark requires modeling and predicting the activity of >70,000 neurons

2
 An AI system to help scientists write expert-level empirical software

a Tree of candidate
code solutions

Scorable
problem

Improvement
Prompt LLM

Research
ideas Code
Finish Further exploration
sandbox

b c Expert
Write

Scientific papers
Summarize

Prior ideas
Recombine
+

Gemini
Deep Research

Figure 1 | Schematic and performance of our method. a, Schematic of our method algorithm. A
scorable task, together with research ideas proposing methods to solve the task, are fed to an LLM,
which produces code to evaluate the scorable task in a sandbox. This is then embedded within a tree
search algorithm, whereby new nodes are chosen balancing exploitation and exploration, sampling
from the LLM (Methods). b, Performance of code generation methods on Kaggle Playground
benchmark. Results report the average public leaderboard percentile performance over 16 tasks.
Methods based on our method are listed in bold. Error bars indicate standard deviation. BDT,
boosted decision tree. c, Mechanisms used to create initial research ideas to solve scientific problems.

3
 An AI system to help scientists write expert-level empirical software

across an entire vertebrate brain. Performing well on this benchmark may lead to a systems-level
understanding of brain function and behavior.
GIFT-Eval time series: 24 Accurate time series forecasting is useful for climatology and healthcare.
General time series forecasting is very difficult, because of the diverse input feature semantics and
prediction time scales. An even more difficult and useful problem is zero-shot prediction, where only
an single time series is given and a prediction must be made.
Numerically solving difficult integrals: Solving integrals that defy standard numerical algorithms is
useful for modeling physical and engineering systems.

Kaggle Playground Benchmark

We designed our code mutation system to score highly on a curated set of Kaggle competitions.
Kaggle calibrates human performance with percentile rank on a leaderboard, and we score code
by submitting directly to Kaggle. Our benchmark consists of 16 playground competitions from the
2023 season, encompassing regression and classification tasks. Playground competitions are an ideal
benchmark because they offer fast iteration, simplicity, and calibration against thousands of humans.
Achieving a high score requires creating complex code without requiring solving a sophisticated
scientific task.
Our basic strategy uses a simple prompting template (Supplementary Table 1) that concatenates
the competition description with the previous trial. Fig. 1b evaluates the performance of our method
with the average public percentile rank across all 16 playground competitions: TS substantially beats
a single LLM call and best of 1000 LLM calls. During the search, the agent discovers strategies leading
to abrupt jumps in the score, with the accumulation of these jumps leading to the highest quality
solutions.
Problem-specific advice added to the prompt substantially improves performance. We illustrate
this with two examples. In TS with expert advice we give the model standard advice to win Kag-
gle competitions (Supplementary Table 2). In TS with Boosted Decision Tree (BDT) we tell the
model to implement a boosted decision tree library from scratch, without using standard packages
(Supplementary Table 3). We manually verified in both cases that resulting codes followed the advice.
We now describe evaluating our method on a series of six benchmarks in different scientific
fields, exploring distinct ways to incorporate research ideas to improve system performance (Fig. 1c,
Methods).

Genomics: Batch Integration of Single Cell RNA Sequencing Data

We first consider data analysis from single cell RNA sequencing (scRNA-seq), which has revolutionized
our ability to dissect cellular heterogeneity, discover novel cell types, infer gene regulatory networks
and developmental trajectories, and improve therapeutic target prioritization 25 , enabling hundreds of
millions of cells to be individually sequenced within thousands of datasets 20,26–28 . A major challenge
required to jointly analyze many disparate datasets is to computationally remove complex batch
effects present across samples while preserving the biological signal 29 . Nearly 300 tools exist to
perform batch integration of scRNA-seq data 30 , and multiple benchmarks have been developed for
assessing metrics of batch effect removal and conservation of biological variability 31–34 .
To assess the performance of tree search on this task, we used the OpenProblems v2.0.0 batch
integration benchmark 34 . As of July 2025, this active benchmark evaluates 15 state-of-the-art methods
and 8 control methods on 13 different metrics that quantify both the ability to remove batch effects

4
 An AI system to help scientists write expert-level empirical software

in the data and retain variability attributable to true biological differences in six CELLxGENE datasets
spanning human and mouse 27 (Fig. 2a). To avoid overfitting to the benchmark, we used a separate
dataset from CELLxGENE for hill climbing with our method (Methods, Supplementary Fig. 1). For
each tree search run, we selected the best solution based on the performance on this training set,
and report the performance on the holdout OpenProblems datasets, which contain in total 1,747,937
cells. We prompt the LLM with a description of the single cell batch integration problem, code for
reading in the dataset, code for evaluation metrics, and optional text with a particular research idea.
First, we ran tree search without guidance, and observed that its solution is conceptually similar
to ComBat 37 , yet improved over the current OpenProblems leaderboard (No advice (TS) in Fig. 2b).
We then evaluated whether our method could improve upon existing algorithms. We selected nine
methods from the OpenProblems benchmark, including the six highest-performing methods (Methods).
For each method, we obtained the paper PDF and used Gemini 2.5 Pro to add a brief summary to
the prompt (Methods). In pairwise comparisons, our method outperformed the corresponding
published result for eight of the nine methods in overall score (Fig. 2b, Supplementary Table 4).
The top-performing method was our tree search based implementation of Batch Balanced K-Nearest
Neighbors (BBKNN (TS)) 38 , yielding a 14% overall improvement over the best published method
(ComBat 37 ) and equaled or outperformed the corresponding published BBKNN in every dataset and
across 11/13 metrics (Fig. 2b). This performance highlights its capacity to effectively remove batch
effects without compromising biological signals (Supplementary Fig. 2). We note that tree search is
also able to produce performant implementations for an algorithm without publicly-available code
(TabVI 39 , Supplementary Fig. 3). Importantly, expert manual inspection of the code solutions proposed
by our method confirmed that nearly all implementations adhered to the requested algorithms
(Supplementary Table 5), with performance largely consistent across replicate runs of methods
(Supplementary Fig. 3). Additionally, tree search demonstrated improvements even when compared
to base methods with optimized hyperparameters, indicating that its contribution extends beyond
hyperparameter tuning (Methods, Supplementary Fig. 4). Supplementary Fig. 5 shows representative
examples of the tree structure and breakthrough plots (showing the evolution of the maximum score
as a function of the number of nodes in the tree) for a representative example.
For the best performing model BBKNN (TS), part of the performance boost came from combining
two existing methods, ComBat 37 and BBKNN, rather than simply implementing BBKNN (Fig. 2c). In
particular, while the original BBKNN method computes neighbors on the PCA embedding, BBKNN (TS)
computes neighbors on ComBat-corrected PCA embedding, removing global linear batch-associated
variance. Both implementations then compute 𝑘-nearest neighbors across batches and construct
a graph (with differences in exact implementation), thus removing local batch effects. Manual
modification of BBKNN (TS) and the published BBKNN implementation confirmed that the addition of
Combat-corrected PCA embedding is critical for improving both implementations (Supplementary
Fig. 6), confirming the value in idea recombination.
This motivated an exploration of systematic ways to generate more complex research ideas.
First, similar to how scientists often combine ideas to create a novel approach, we programmatically
generated 55 “recombinations” of all pairs of the 11 methods described above (No advice, nine
replications, and TabVI; hereafter: “base methods”) based on summaries of the code for each method
(Methods, Supplementary Table 6). We ran tree search, prompted with each of these “recombinations”
to assess whether it can develop new methods by combining the strengths of the existing methods.
For each base method and “recombination” group, we compared the average scores for the top nodes
over the intersection of metrics that were successfully computed for all three methods. Strikingly,
recombination implementations of tree search frequently outperformed their base counterparts, with
24 of the 55 “recombination” solutions (44%) outperforming both of their base methods and 22 of the
remaining 31 “recombination” solutions outperforming one of the two base methods (Supplementary

5
 An AI system to help scientists write expert-level empirical software

a b

Combine

✗Poor Good
✓
PC2

PC2

PC1 PC1
c

Overall score Score per dataset (mean over metrics) Score per metric (mean over datasets)

0 0.5 1 0 0.4 0.6 0.8 1 0 0.4 0.6 0.8 1

Figure 2 | Performance of tree search on scRNA-seq batch integration. a, Schematic of the batch
integration task, in which disparate datasets (teal and red) are processed to remove batch effects in
the data while retaining biological variability. b, Performance of tree search (method names bolded
and suffixed by “(TS)”) compared to the analogous published method on the OpenProblems
benchmark v2.0.0 34 . “Perfect embedding by celltype with jitter” is a positive control method that
represents the best possible performance and “Shuffle integration by batch” is a negative control that
does not perform any batch integration. Overall score is the mean over all datasets and metrics. Each
Datasets column shows the mean of all metrics computed over that dataset. Each Metrics column
shows the mean of that metric computed over all datasets. Metrics were assigned a value of 0 if they
could not be computed or if their performance was worse than the lowest negative control; these are
displayed as empty. c, Performance improvements annotated with code innovation for the
top-performing batch balanced 𝑘-nearest neighbors (BBKNN) implementation. ComBat-based
embedding generation was introduced in implementation attempt 429. d, Overall score for
OpenProblems benchmark v2.0.0 34 non-control methods, our method with and without
recombination of ideas, Gemini Deep Research 35 , and our method with AI co-scientist 36 . Y-axis lower
bound is the overall score of the “Shuffle integration by batch” negative control method. Seven
recombination, five base methods, and two AI co-scientist methods that do not match its performance
are omitted. * indicates the method is a recombination, even if not explicitly prompted for
recombination. TS, tree search; fastMNN, batchelor fastMNN; mnnCorrect, batchelor mnnCorrect.

6
 An AI system to help scientists write expert-level empirical software

Fig. 7). Second, we also used Gemini Deep Research 35 and AI co-scientist 36 to generate and implement
21 additional ideas (Methods). In total, 6/11 base methods, 29/55 recombination, 4/9 Deep Research,
and 1/12 AI co-scientist methods (40 of 87) outperform all methods currently published on the
OpenProblems leaderboard (Fig. 2d). This demonstrates the ability of our method to understand the
best features of existing approaches and effectively integrate them for superior performance.
To further understand the conceptual space explored by our method, we obtained embeddings
for each generated code using Gemini text embedding model and computed cosine similarities
(Supplementary Fig. 8). As expected, replicates exhibited significantly higher similarity to each other
compared to all other method pairs (one-sided 𝑡 -test: 𝑡 = 12.95, 𝑝 = 1.06 × 10−14 ; 𝜇 duplicate pairs = 0.95,
𝜇 other pairs = 0.91 ; 𝑛duplicate pairs = 33, 𝑛other pairs = 5853). Hierarchical clustering on the embeddings
revealed distinct clusters, generally representing linear methods, deep learning based methods,
and nonlinear non-deep learning methods, suggesting that our method is able to generate diverse
solutions.

Public Health: Prediction of U.S. COVID-19 Hospitalizations

The primary U.S. benchmark for COVID-19 forecasting is the COVID-19 Forecast Hub (CovidHub) 21 ,
a large, collaborative effort coordinated by the Centers for Disease Control and Prevention (CDC). The
hub attracts dozens of expert-led teams from leading academic institutions, industry, and government
agencies, who submit weekly forecasts generated from a wide array of methodologies. These weekly
forecasts must cover new COVID-19 related hospitalizations across 52 U.S. states and territories for the
current week and three subsequent weeks over 23 specified quantiles. Submissions are evaluated using
the Weighted Interval Score (WIS), which rewards both accuracy and well-calibrated uncertainty,
with lower scores indicating better performance.
Top-performing individual models include classic autoregressive time-series approaches (e.g.,
UMASS-ar6_pooled), gradient boosting machine learning models (e.g., UMASS-gbqr), and epidemio-
logical models based on renewal equations and Bayesian estimation of the reproductive number (e.g.,
CEPH-Rtrend_covid). The hub leverages this methodological diversity by integrating submissions into
the CovidHub Ensemble, a robust aggregate forecast that has historically provided the gold standard
for epidemiological prediction in the U.S., making it a formidable benchmark to outperform.
We designed a rigorous retrospective study to assess tree search’s performance in this competitive
environment. For every forecasting period, we ran tree search to optimize and select a model using
data from the preceding six weeks, creating a rolling validation window throughout the 2024-2025
season (Fig. 3a). The weekly performance of our resulting ‘Google Retrospective’ model is detailed in
the time-series leaderboard (Fig. 3b), which visualizes our model’s performance advantage relative to
the CovidHub-ensemble and other top-performing teams. Supplementary Fig. 9 shows the temporal
variation of WIS for each of the separate validation splits, across replicates Supplementary Fig. 10. A
direct jurisdiction-level comparison confirms our model achieved a lower (better) WIS in a majority of
states (Fig. 3c), with the geographic distribution of performance shown in Fig. 3d. Overall, our model
achieved the highest performance with an average WIS of 26, outperforming the official CovidHub
Ensemble’s average WIS of 29. A representative tree and breakthrough plot is shown in Supplementary
Fig. 11.
Beyond this retrospective performance, we investigated our method’s ability to explore the solution
space more broadly by replicating, recombining, and generating entirely new forecasting strategies
(Fig. 3e). First, we tested its ability to replicate existing methods from other teams using only their
brief public descriptions from the CovidHub (Supplementary Table 7,Supplementary Table 8). Our
tree-search-based implementations (‘Base Method (TS)’) not only adhered to the provided instructions

7
 An AI system to help scientists write expert-level empirical software

Figure 3 | Performance of tree search on COVID-19 forecasting. a, Rolling validation window

used for the forecasting experiments. Each search’s output is validated internally on a preceding
block of time (blue), and the resulting model is then used to make predictions for its corresponding
forecasting period (orange). Training data includes all dates on or after 2020-08-08 and prior to the
validation set. b, Time-series leaderboard showing weekly forecasting performance (Average WIS) for
participating teams and our ’Google Retrospective’ model, ordered by average WIS. Scores are
aggregated across all 52 jurisdictions and four forecast horizons. The number within each cell is the
model’s absolute Average WIS for that week. The cell’s background color visualizes the performance
relative to the CovidHub-ensemble, with blue indicating a lower (better) WIS and red indicating a
higher (worse) WIS. c, Direct jurisdiction-level comparison of forecasting error (Average WIS)
between our model and the ’CovidHub-ensemble’, demonstrating our model’s superior performance
in a majority of locations. d, Geographic distribution of our model’s forecasting error (Average WIS),
aggregated over the entire 2024/25 COVID-19 season. Lower error values (lighter colors) indicate
better performance. e, Comparison of aggregate forecasting performance for various modeling
strategies. This includes baseline models from the CovidHub competition, our retrospective model,
our replications of submitted models, novel hybrid models generated through recombination, deep
research 35 and AI co-scientist 36 . 14 strategies (10 recombination; two Deep Research; one AI
co-scientist and one replicated baseline) outperform the official CovidHub-ensemble for the 3-week
(3 reference dates × 4 time horizons × 52 jurisdictions) evaluation period. Models that perform
worse than CovidHub-baseline are not shown.

8
 An AI system to help scientists write expert-level empirical software

(Supplementary Table 9) but also exceeded the performance of the original submissions in six of
the eight cases tested; the two models that performed worse (replicating JHU_CSSE-CSSE_Ensemble
and OHT_JHU-nbxd) did not use external data present for the original method implementations. Next,
we explored whether solutions could be improved through recombination. For this experiment,
we prompted an LLM to analyze the core principles of two different parent models and then used
its synthesis to instruct tree search to generate a novel hybrid strategy combining their respective
strengths. As shown in Fig. 3e, 11 out of 28 generated hybrid models (‘Recombination (TS)’) achieved
a WIS score superior to both of their parent models (Supplementary Fig. 12). We manually verified
methodology of the output code for the recombined experiments–in all cases, the final methods
contained relevant aspects from both parent codes (Supplementary Table 9). Finally, we used
Gemini Deep Research 35 and AI co-scientist 36 to generate novel forecasting ideas which were then
implemented via tree search. In total, this systematic exploration yielded 14 distinct strategies that
outperformed the official CovidHub-ensemble: 10 from recombination, two from Deep Research, one
from AI co-scientist, and one of our replicated baselines. Cosine similarities between embeddings for
each generated code show clustering between different methods (Supplementary Fig. 13).
A deeper analysis of these 14 top-performing strategies reveals key patterns in how our method
achieves superior performance. The recombination models, which constitute the majority of the
winners, highlight a clear pattern of synergistic hybridization. Two base models appear most frequently
in these successful hybrids: the simple, climatology-based CMU-climate_baseline and the statistical
autoregressive model UMass-ar6_pooled. This suggests tree search consistently discovers that the most
effective strategies are built upon a robust foundation of historical averages and recent trends, which
are then enhanced by more complex methods. Indeed, the most successful recombinations consistently
fused different modeling paradigms—for instance, pairing the epidemiological CEPH-Rtrend_covid
model with the statistical UMass-ar6_pooled model created a hybrid anchored in the theory of disease
spread yet highly responsive to recent data trends, while pairing the powerful machine learning
UMass-gbqr model with the stable CMU-climate_baseline provided a robust seasonal foundation that
allowed the ML model to safely focus on learning short-term deviations—demonstrating an ability to
synthesize complementary strengths.
In contrast, the novel strategies generated via Deep Research and AI co-scientist represent sig-
nificant conceptual leaps beyond the existing Hub models. Rather than relying on conditional
uncertainty from past data, the DEEP-RESEARCH-CounterfactualSimulation model introduces un-
conditional uncertainty quantification by running thousands of Monte Carlo simulations over plau-
sible future scenarios (e.g., new variant emergence). Similarly, while some base models use deep
learning, the CO-SCIENTIST-STGNN-AgACI model implements a far more complex Spatio-Temporal
Graph Neural Network with a learnable graph structure to explicitly model inter-state dynamics.
The DEEP-RESEARCH-RegimeSwitchingDetection model introduces another novel concept: dynamic,
event-triggered adaptation, using Bayesian change-point detection to automatically initiate model
retraining in response to shifts in the underlying data generating process. Finally, the outperfor-
mance of our replicated CMU-TimeSeries (TS) model underscores that even when not inventing or
hybridizing, tree search excels at the fine-grained optimization of already-strong, expert-designed
strategies. Ultimately, this demonstrates the power of tree search as a scientific discovery engine, capa-
ble of systematically exploring a vast solution space to innovate, hybridize, and optimize expert-level
strategies.

Geospatial Analysis: Segmentation of Remote Sensing Images

We now turn to a problem in geospatial analysis: semantic segmentation of high-resolution remote

sensing images. Semantic segmentation is a computer vision task that involves assigning a specific

9
 An AI system to help scientists write expert-level empirical software

class label to every single pixel in an image. It is essential for diverse applications, ranging from
monitoring land use, assessing the environmental impacts of human activity and managing natural
disasters. The primary difficulty is significant visual heterogeneity. Satellite images of the same
location can differ dramatically due to variations in time of day, season, and weather conditions, while
even objects within a single class (e.g. buildings) exhibit substantial diversity in size, shape, height,
function and lighting conditions.
A recent paper 22 introduces the “dense labeling remote sensing dataset” (DLRSD) for advanced
remote sensing tasks, including multi-label classification, image retrieval, and pixel-based applications
like semantic segmentation. This dataset is a densely labeled version of the UC Merced Land Use
Dataset 40 , a widely-used benchmark for image-level land use classification, whereby individual pixels
of each image are labeled with 17 class labels.
We prompted our method to train a model to classify pixels into the land cover classes and
provided a pre-specified, reproducible 80/20 train/test split of imagery in the DLRSD dataset. For
each experiment, we validated model performance on the held out test set of 420 randomly selected
images using a standard “mean intersection over union” (mIoU) metric.
The three top performing solutions generated by tree search significantly outperformed reported
results in recent academic papers on the DLRSD benchmark, achieving mIoU greater than 0.80
(Table 1, Supplementary Fig. 14). All three solutions build upon existing models, libraries and
strategies. Solutions 1 and 3 leverage standard UNet++ and U-Net models but paired with powerful
encoders (efficientnet-b7 and se-resnext101-32x4d) pre-trained on ImageNet 41 . Solution 2 uses
SegFormer, a state of the art Transformer-based architecture. Key differentiators among the models
included their data augmentation and prediction strategies. The U-Net++ and U-Net models leveraged
extensive augmentation from the Albumentations library, whereas the Segformer model used a more
basic set of transforms. All three solutions employ extensive Test-Time Augmentation (TTA) 42 by
predicting masks for multiple augmented versions of a single test image (e.g., horizontal flips, vertical
flips, rotations) which are then reverse-transformed and averaged to produce a final, more robust mask
which smooths out prediction errors and boosts performance. A representative tree and breakthrough
plot for Solution 3 is shown in Supplementary Fig. 15.

Neuroscience: Whole-Brain Neural Activity Prediction

We now consider the Zebrafish Activity Prediction Benchmark (ZAPBench), a recent dataset designed
to test predictions of cellular-resolution neural activity in an entire vertebrate brain 23 . The benchmark
uses a novel dataset capturing brain activity of a larval zebrafish over a two-hour session using
light-sheet fluorescent microscopy, resulting in 3D brain volumes recorded over time. Throughout
the recording, the animal was exposed to distinct visual stimulus conditions designed to elicit a
range of different behaviors. The raw volumetric video data was extensively processed to align,
motion-stabilize, and segment into activity traces, resulting in a final data matrix of activity traces for
71,721 neurons across 7,879 time steps.
Several state-of-the-art forecasting methods were evaluated on the benchmark 23 , including time-
series forecasting methods that operate on the extracted activity traces per neuron, as well as a
volumetric video prediction model (a Unet variant) that directly processes the 3D brain volumes
over time 48 . The video-based approach exploits spatial information that is lost when converting the
data to time series, but is computationally expensive. Among the different methods evaluated on
the benchmark, the video-based Unet model achieved the best overall performance, especially in the
setting where only a short window of past context is available.
We prompted our method to solve the multivariate time-series forecasting problem, predicting

10
 An AI system to help scientists write expert-level empirical software

Table 1 | Comparison of model performance on the DLRSD benchmark. The table shows the publication
year, architecture, key features, and reported mean Intersection over Union (mIoU) for tree search
solutions and the methods from the referenced papers.

Method Year Architecture Key Features / Tech- mIoU

Type niques
Solution 1 (TS) 2025 CNN (UNet++) ‘efficientnet-b7’ 0.81
encoder, 8-fold TTA
Solution 2 (TS) 2025 Transformer(SegFormer) ‘mit-b1’ encoder, 4- 0.82
fold TTA
Solution 3 (TS) 2025 CNN (U-Net) ‘se_resnext101_32x4d’ 0.80
encoder, 7-fold TTA
RE-Net 43 2021 CNN (Region- Region Context 0.762
based) Learning
FURSformer 44 2023 CNN+Transformer Custom fusion mod- 0.753
ule
SCGLU-Net 45 2024 CNN+Attention Spatial-Channel- 0.666
Global-Local block
MA-UNet 46 2022 Attention+U- Residual encoder 0.619
Net with simAM
W13 Net 47 2025 CNN Multi-stage encoding- 0.580
(Lightweight) decoding

the output activity of all neurons for up to 32 time steps ahead in the time-series domain, given their
past 4 time steps of activity as context, using the dataset splits provided by ZAPBench 23 which split
each stimulus condition into 70% for training, 10% for validation, and 20% for testing per stimulus
condition. We used the validation set for model selection, including hyperparameter tuning and early
stopping, and to obtain a score to guide the tree search. We score solutions using mean absolute error
(MAE) averaged across the prediction horizon, and compare solutions found by tree search against
the methods included in ZAPBench: These include a linear model 49 , TiDE 50 , TSMixer 51 , Time-Mix (a
variant of TSMixer where feature mixing is ablated), and a custom Unet architecture 48 .
Our initial experiment using tree search led to a best-performing model that uses a rich feature
set from the input window, combining temporal convolutions, a learned “global brain state”, and
neuron-specific embeddings. The model then processes these features through a series of weight-
shared residual blocks and a final dense layer to generate the multi-step prediction in one shot.
Figure 4 shows the result of this model, compared to other baselines. In that figure, the mean baseline
predicts the average over the context window, while the stimulus baseline predicts the average for
each stimulus phase. Remarkably, the model produced by tree search outperformed all other baselines,
including the best-performing video model, except for 1-step-ahead predictions. A representative
example of the breakthrough plot and tree is shown in Supplementary Fig. 16.
We then developed a separate model tuned specifically for 1-step-ahead predictions with another
tree search. The resulting solution is conceptually similar to the first in that both architectures
generate a learned global context vector to inform their per-feature predictions. However, this model
computes its global context using a dynamic attention mechanism for weighted aggregation and
modulates feature representations through a FiLM-like layer 52 for interactive conditioning. This
model achieved leading performance on 1-step-ahead predictions (Fig. 4).

11
 An AI system to help scientists write expert-level empirical software

1 step ahead 4 steps ahead 8 steps ahead 16 steps ahead 32 steps ahead
0.035

0.030
short context
MAE

0.025

0.020

0.015
Tide

Time-Mix
Unet
TS-1-Step
TS
TS-Jaxley

Tide

Time-Mix
Unet
TS
TS-Jaxley

Tide

Time-Mix
Unet
TS
TS-Jaxley

Tide

Time-Mix
Unet
TS
TS-Jaxley

Tide

Time-Mix
Unet
TS
TS-Jaxley
Linear

Tsmixer

Linear

Tsmixer

Linear

Tsmixer

Linear

Tsmixer

Linear

Tsmixer
Figure 4 | Comparison of the best tree search solutions to time-series and video forecasting methods
in terms of grand average mean absolute error (MAE) across conditions on ZAPBench (lower is
better). For our method, we report the performance of three different solutions (blue), and compare
them against baselines (red). Alongside our best general solution (TS), we include results from two
specialized runs: a tree search that was optimized for 1-step ahead forecasting as well as a solution
prompted to use Jaxley, a differentiable biophysical neuron simulator. The dotted and solid lines
represent the mean and stimulus baselines, respectively. To account for variability due to random
number generator seeding, each method was run three times. We report the mean, with error bars
indicating 95% confidence intervals.

Both of these solutions are orders of magnitude faster to train than the best-performing video
model–less than two hours on a single T4 GPU, as compared to 36 hours on 16 A100 GPUs for the Unet
model. In addition, our solutions effectively use cross-neuron information to generate predictions, a
major challenge highlighted in previous work 23 .
A key future direction is the development of models that incorporate biophysical information and
are more interpretable. The forthcoming synaptic-level structural reconstruction of the larval zebrafish
brain used for ZAPBench provides a unique opportunity to develop such models by integrating
anatomical wiring diagrams. As an initial exploratory step, we prompted our method to use Jaxley 53 ,
a JAX-based library for differentiable simulation of biophysically detailed neuron models, for the
tree search. The resulting best-performing solution simulates each neuron independently using
single-compartment Hodgkin-Huxley models. Crucially, it dynamically modulates each neuron’s
biophysical parameters based on its recent activity history. To account for inter-neuronal interactions
without the computational cost of direct synaptic simulation, the model then processes the outputs of
these independent simulations through a latent autoencoder. This learns a system-wide corrective
signal, effectively modeling a functional connectome–a reasonable hybrid approach in the absence of
the structural connectome. While this model did not outperform the top-performing video model, it
was competitive with time-series baselines (Fig. 4).

Time Series Forecasting: GIFT-Eval

General Time Series Forecasting Model Evaluation (GIFT-Eval) 24 is a benchmark for time series
forecasting, derived from 28 datasets from seven diverse domains, with 10 different frequencies,
from seconds to years, receiving ∼4 new submissions per month, from black box deep learning to
foundation models. Submissions are scored on official train/validation/test splits using a normalized
Mean Absolute Scaled Error (MASE) metric, calculated relative to a seasonal naive baseline.
We applied our method in two phases. We began with a per-dataset solution whereby the
search discovers an independent solution for each. The second unified solution created a single

12
 An AI system to help scientists write expert-level empirical software

a general-purpose forecasting model using only basic libraries by hill climbing against the average
score for the entire GIFT-Eval.

Per-dataset solution Here we allowed our method to use a full suite of Python libraries, including
scikit-learn, statsmodels, and xgboost. The results in Supplementary Table 10 are better than
the results in the May 18, 2025 leaderboard, outperforming foundation models 54–56 , deep learning
models 49,50,57 and standard time series methods like ARIMA 58 . The discovered solutions showed
strong convergence towards gradient boosting and ensemble/decomposition models (Supplemen-
tary Fig. 17).

Unified solution We wondered whether the code mutation system could create a unified, general-
purpose forecasting library from scratch, by hill climbing with a single code on the average MASE on
the entire GIFT-Eval dataset. To manage the benchmark’s diversity, we allowed library to have an
adaptive configuration system, whereby it could generate up to 8 preset hyperparameter configurations
to adapt to the diversity of datasets, with a validation step selecting the best performing configuration
for each dataset. As the search progressed, date and trend-related features often led to performance
breakthroughs leading to a model that sequentially forecasts and subtracts individual time series
components, including a base level, trend, seasonality, datetime-based features, and a final residual
correction. Supplementary Fig. 18 shows the breakthrough plot and tree structure for the search. The
configurations (Supplementary Table 11) include date-specific features, including one that featurizes
holidays in a specific set of countries ([‘US’, ‘DE’, ‘CN’, ‘GB’, ‘CA’, ‘AU’]) . The resulting unified
solution is highly competitive on the leaderboard (Supplementary Table 10).

Numerical Analysis: Library for numerical evaluation of difficult integrals

Finally, we turn to a problem in numerical analysis, the numerical evaluation of difficult integrals
using Gaussian quadratures. The gold standard was developed 59 by U.S. government research
laboratories in the 1980s, widely used as the core library underlying the popular Python function
[Link](). Nonetheless, this function can fail in multiple ways, among them: the
underlying algorithm can fail to converge; the algorithm samples its integrand, and the sampling may
miss important features; the algorithm loses precision when the problem exhibits precise cancellations.
While standard techniques exist to address these problems, we asked whether our method could
build a general-purpose method superior to quad(), by hill climbing on a benchmark set of integrals
where the standard algorithm fails but where the analytic answer to the integral is known. We
constructed this set of integrals from a standard applied mathematical reference book 60 , focusing on
oscillatory integrals with infinite upper limits and without other pathologies, but where the standard
quad() library returned an incorrect answer. This led to a set of 38 integrals (Supplementary Fig. 19).
We split these in half at random, using 19 for scoring the search and holding out the other 19 for
evaluation. We then initialized our method with a simple invocation of quad() and prompted the
system to improve it, scoring solutions with the logarithm of the absolute fractional error, where
the logarithm prevented the search from over-weighting outliers. Supplementary Fig. 20 shows the
resulting breakthrough plot and tree structure for the search.
The best solution builds on quad() by partitioning the infinite domain into a sequence of contiguous,
finite subintervals whose lengths may increase geometrically to cover the domain’s tail more efficiently.
The definite integral is thus transformed into an infinite series, where each term is the numerical
integral of the integrand over one of these finite segments, calculated using quad(). For integrals that
converge slowly, such as those with oscillatory integrands, direct summation of this series is impractical.

13
 An AI system to help scientists write expert-level empirical software

The algorithm therefore applies Euler’s transformation, a powerful series acceleration technique,
to this sequence of segment integrals. By repeatedly averaging adjacent terms, the transformation
extrapolates the limit of the slowly converging series from a finite number of its initial terms, providing
an accurate estimate of the integral’s true value.
Whereas [Link]() fails on every problem in the held-out set, the evolved code
correctly evaluated 17 out of 19 of held-out integrals to within a fractional error of less than 3 percent
(Supplementary Fig. 21).
The evolved code always applies [Link]() first. It only falls back to its more
specialized methods if quad() returns a large error estimate, returns NaN or Inf, or raises an exception.
This means the evolved code is as accurate as quad() in less pathological cases and so could reasonably
be used as a drop-in replacement.

Discussion
Our work introduces an AI-based system that drives a Tree Search (TS) with a Large Language Model
(LLM) to systematically create and improve software for scientific tasks. By defining the problem
of creating scientific software as a search for a program whose output maximizes a quality score,
we convert software creation into a “scorable task”, producing empirical software. Our method
is novel in its LLM-driven rewriting approach, which allows for the flexible integration of domain
knowledge and external research ideas. The ability of frontier LLMs to closely follow instructions
enables efficient exploration of research ideas. Our method builds upon ideas from several distinct
but related areas of research: Genetic Programming, Generative Programming, the application of
LLMs to code, Automated Machine Learning (AutoML), and agents for scientific discovery.
Genetic Programming — The idea of automatically evolving computer programs to solve a problem
is not new. Genetic Programming (GP) provides a foundation to our work. In GP, a population of
programs is iteratively improved using evolutionary principles like selection, crossover, and mutation.
The fitness of each program is determined by a "fitness function," which is directly analogous to our
"quality score" 61 . While GP has been successful, it traditionally relies on random mutations and
structured recombination of code fragments (e.g., swapping sub-trees in an abstract syntax tree). A
key difference in our system is the use of an LLM to perform intelligent, semantic-aware "mutations"
by rewriting the code, which can produce more complex and meaningful variations than the random
changes typical in GP.
Generative Programming — Our system can be viewed as a modern, AI-driven realization of this
concept. In traditional generative programming, a developer creates a program generator (using
techniques like templates, domain-specific languages 62 , or metaprogramming) that produces tailored
source code for a family of related problems 63 . In contrast, we employ an LLM guided by a tree search
as the generative engine. This approach offers greater flexibility, allowing the system to synthesize
novel programs by exploring a vast solution space and integrating diverse domain knowledge in ways
not easily achievable with more template-based methods.
LLMs for Code Generation — The advent of large language models pre-trained on vast code
corpora has revolutionized code generation. Systems like OpenAI’s Codex 64 and Google DeepMind’s
AlphaCode 65 have demonstrated the ability to generate correct and complex code from natural
language descriptions. These systems are typically used for "one-shot" generation from a prompt. Our
approach differs by using the LLM in an iterative refinement loop. Instead of generating code from
scratch, our LLM rewrites existing software candidates, guided by a search algorithm (TS) that uses
the quality score as a signal.

14
 An AI system to help scientists write expert-level empirical software

Combining LLMs and Search — The most closely related work involves combining LLMs with
search algorithms to overcome the limitations of one-shot generation. A recent example is Google
DeepMind’s FunSearch, which uses an LLM to search for new mathematical discoveries 16 . FunSearch
works by pairing a creative LLM with an automated evaluator. The LLM suggests improvements (new
code) to an existing program, and these improvements are only kept if they pass evaluation. This
creates an evolutionary feedback loop. This is conceptually very similar to our system’s use of an
LLM rewriter and a quality score. However, our system generalizes the search process using TS, a
robust algorithm for exploring large search trees. Our system also incorporates knowledge from the
literature.
AutoML — our work is conceptually related to Automated Machine Learning (AutoML). AutoML
systems aim to automate the process of building machine learning pipelines by searching for optimal
model architectures and hyperparameters. The goal is to maximize a performance metric (e.g.,
accuracy, F1-score) on a validation dataset 66 , which fits our definition of a scorable task. While
AutoML focuses specifically on finding the best model within a fixed set of ML frameworks, our system
is more general. It can rewrite any software, including pre-processing steps, complex simulations, or
mathematical heuristics—areas that fall outside the typical scope of AutoML.
Agents for science problems — This sub-field has seen remarkable, expert-exceeding performance
from highly specialized systems 67 . Much of the existing literature focuses on agents that either
automate standard workflows within a single domain, such as computational biology 68–72 , or act as
ideation assistants whose proposals require significant human validation 73,74 . Instead of specializing
in one domain, our system demonstrates a general problem-solving capability, achieving expert-
exceeding performance on public leaderboards and in academic literature across multiple fields.
To summarize, we have developed a method that combines a code mutation system based on Tree
Search 12,13 with the ability to integrate complex research ideas. Such research ideas could come
from the published literature, from research agents (e.g. 35,36,75 ) or from combining previous ideas
and solutions that the LLM has found itself. Because the system creates code that can follow a specific
idea, it can search over externally supplied research ideas. We demonstrate over a wide range of
scientific scorable tasks that reaches an expert-level when integrating and exploring complex research
ideas.
Our method created 40 methods that beat the best known method for scRNA-seq batch integration
and 14 methods that outperformed the CDC ensemble for epidemiological prediction. Additionally,
our method achieved state of the art performance on geospatial reasoning, neural activity prediction,
time series prediction and algorithms for computational mathematics. With minimal prompting, the
system invents and implement ideas for combining complex architectures (U-Nets, transformers) for
a fundamental task in geospatial reasoning, and in neural activity prediction it was not only able to
outperform all methods on the current benchmark, but easily incorporates a biophysical simulator
into a performant solution.
Trial and error is essential to scientific progress, both for humans and for the automated approaches
we outline here. The system generates expert-level solutions extraordinarily quickly, reducing explo-
ration of a set of ideas from weeks or months, to hours or days. Accelerating research in this way
has profound consequences for scientific advancement. Based on this work, we believe that progress
in scientific fields where solutions can be scored by machines is on the precipice of a revolutionary
acceleration.

15
 An AI system to help scientists write expert-level empirical software

Acknowledgements
We are grateful to our colleagues in Google Research and Google DeepMind for the incredible
environment within which to do this work. We would like to specifically thank Niv Efron, Viren Jain,
Anupam Pathak and Jamie Smith for many incisive discussions, and Nicholas Reich for comments on
the manuscript.

Author Contributions Statement

Code Mutation System (E.A., A.B., G.C., M.C., H.C., P.N., D.S., J.T., S.V., M.P., J.K., P.R., J.W., L. W.,
S.M. and M.P.B.) Single Cell RNA-seq Batch Integration ([Link]., C.Y.M., C.H., Y.Z., M.P.B.) COVID
Forecasting (Z.S., S.M., M.P., M.C., M.P.B.) Geospatial Analysis (R.J., Je.C., Q.Z., M.P.B.) ZAPBench
(B.P.W., J-M.L, Q.Z.) GIFT-Eval (J.G., M.P.B.) Integrals (A.K., R.K., M.P.B.) User Interfaces (E.A., G.C.,
P.N., A.K., M.K., M.P.B., J-M.L., D.L., J.K., C.C., S.E.) Graphical Design (G.J.) Program Management
(M.A., E.B.) Leadership (K.C., J.M., Y.M., J.C.P., L.D., S.M., M.P.B.)

Code Availability
We are open sourcing the best candidate solutions generated from each of the examples outlined in
this paper ([Link]/google-research/score). Additionally, we are providing a user interface to
examine the full tree search data for a representative example of each of the six scientific problems
discussed in the paper. The interface allows inspecting the solution progression and breakthrough
plot as the tree search proceeds, as well as highlighting the code diffs.

16
 An AI system to help scientists write expert-level empirical software

Methods

Code Mutation System

We prompt an LLM (Supplementary Fig. 22) providing a description, the evaluation metric and the
relevant data. The LLM produces Python code, which is then executed and scored on a sandbox.
Searching over strategies dramatically increases performance: The agent uses the score together with
output logs and other information to hill climb towards a better score. We used a tree search (TS)
strategy with an upper confidence bound (UCB) inspired by AlphaZero 13 . A critical difference from
AlphaZero is that our problems don’t allow exhaustive enumeration of all possible children of a node,
so every node is a candidate for expansion. We therefore modify the UCB algorithm to count visits
and compute mean values using the tree. However, when sampling a node to expand, we sample
directly from the whole set instead of recursing from the root like AlphaZero.
We also note that the algorithm differs from traditional TS, in that the scoring of the nodes do not
involve random rollouts (e.g. of a game) to estimate the value of a node. Yet there is still randomness
for scoring each node, caused by the sampling of the LLM itself, which produces a distribution of
different codes (scores) for each fixed prompt.
We use a PUCT tree search algorithm to explore the space of notebooks 12 . The PUCT (Predictor +
Upper Confidence bound applied to Trees) algorithm is described in Algorithm 1. For tree 𝑇 , and
executed candidate 𝑢, we define the flat prior 𝑃𝑇 ( 𝑢) = |𝑇1 | . To make it easier to tune the exploration
constant 𝑐 𝑝𝑢𝑐𝑡 across tasks, we convert task-specific scores TaskScore( 𝑢) to rank scores RankScore𝑇 ( 𝑢)
in the PUCT formula. We define RankScore𝑇 ( 𝑢) = Rank 𝑇 ( 𝑢 ) −1
| 𝑇 | −1 , when |𝑇 | > 1, and 1 otherwise, where
Rank𝑇 ( 𝑢) gives ascending-order ranks to the candidates.

Algorithm 1 UCB tree search (PUCT)

Input: GenerateAndExecute(), TaskScore() to define rank scores RankScore𝑇 ( 𝑢), exploration constant
𝑐 𝑝𝑢𝑐𝑡 , and a root node 𝑟 .
1: 𝑇 ← { 𝑟 } ⊲ Initialize the tree with a root node.
2: 𝑉 ( 𝑟 ) ← 1
3: for all iterations do
4:  ⊲ Get total visits across all nodes
Í
𝑁𝑡𝑜𝑡𝑎𝑙 ← 𝑢 ∈𝑇 𝑉 ( 𝑢)  √
5: Select 𝑢 ← argmax𝑢 ∈𝑇 RankScore𝑇 ( 𝑢) + 𝑐 𝑝𝑢𝑐𝑡 𝑃𝑇 ( 𝑢) 1+𝑉 (𝑢 ) ⊲ Select node with highest PUCT
∗ 𝑁𝑡𝑜𝑡𝑎𝑙

score
6: 𝑢𝑐 ← GenerateAndExecute( 𝑢∗ ) ⊲ Expand the selected node and Execute
7: 𝑇 ← 𝑇 ∪ {𝑢𝑐 }
8: 𝑉 ( 𝑢𝑐 ) ← 1
9: for all ancestors 𝑢𝑎 of 𝑢𝑐 (excluding 𝑢𝑐 ) do ⊲ Backpropagate results
10: 𝑉 ( 𝑢𝑎 ) ← 𝑉 ( 𝑢𝑎 ) + 1
11: end for
12: end for
13: return argmax𝑢 ∈𝑇 TaskScore( 𝑢) ⊲ Best solution found

Adding Research Ideas to the Code Mutation System

When an expert solves difficult scientific problems, they often search for prior work for ideas. Prior
work could be sourced from highly cited papers, specialized textbooks, or search engines. The search
for prior work can also be powered by LLMs 35,36,75–78 .

17
 An AI system to help scientists write expert-level empirical software

We emulate the expert behavior by injecting instructions for carrying out research ideas into the
prompt of our code mutation system (Figure 1). We applied the research instruction injection for
scRNA-seq batch integration, COVID prediction, segmenting remote sensing images, and whole-brain
neural activity prediction. While the most successful outcomes used top methods from the literature,
we also used two LLM driven search strategies: Deep Research from Gemini 2.5 Flash 35 and AI
co-scientist 36 .
For running these searches, we provided the tools with background information from the main
problem description, and instructed the models to create distinct ideas (Supplementary Table 12).
After manually filtering proposals and removing one proposed scRNA-seq batch integration method, we
prompted Gemini to format the ideas into a structure consistent with our baseline method descriptions
(Supplementary Table 13). Finally, we ran our method on these ideas to create empirical codes that
could be scored.

Recombination Experiments

For both scRNA-seq batch integration problem and COVID-19 forecasting, we combined ideas from
methods already generated using tree search. For the scRNA-seq batch integration problem, we used
the first versions of our 11 baseline methods. For the COVID-19 prediction problem, we used the eight
replications of models submitted to CovidHub. We first took the top-performing node from each tree
search run seeded with one of these methods, based on its score on the validation set (for COVID-19
prediction, this included six weeks of reference dates from 2025-02-22 to 2025-03-29). Then, for
every pair of these methods, we prompted Gemini 2.5 Flash to compare the two methods and explain
the core technical similarities and differences between the two parent models using a consistent
prompt (Supplementary Table 6). The explanatory response was then added to the the prompt, along
with a statement instructing tree search to recombine the ideas by combining the best parts of both
approaches (Supplementary Table 14). Subsequently, we ran our method to generate new hybrid
strategies. This process yielded 55 recombined methods for the scRNA-seq batch integration problem,
and 28 for the COVID-19 prediction problem (evaluated on the three-week holdout set 2025-04-05 to
2025-04-19, see Fig. 3d).

Gemini embeddings For each tree search implementation, we input the code snippets to the Gemini
text embedding model 79 , and the resulting 3,072-dimensional output vectors served as the semantic
representations of their respective implementations.

scRNA-seq batch integration

For all scRNA-seq experiments, we ran tree search with 500 nodes. Each experiment took roughly
seven hours to execute on our infrastructure.

Dataset We sourced a dataset from CZ CELLxGENE Discover 27 to use for hill climbing with tree
search. To identify datasets distinct from the six [Link] test datasets but that have
similar characteristics, we filtered to datasets that contain only healthy human cells, with primary
cell count ≥ 2,000, at least 10 unique cell types, at least seven unique donor ids (i.e. number of
batches), and contain at least two unique assays that are also present in the [Link]
datasets. This filtering process identified 22 candidate datasets. After manually investigating
the candidate datasets, we selected the dataset 364bd0c7-f7fd-48ed-99c1-ae26872b1042 version
ffdaa1f0-b1d1-4135-8774-9fed7bf039ba 19 .

18
 An AI system to help scientists write expert-level empirical software

Within the selected dataset, we applied quality control metrics and data processing steps identical
to the processing performed on the [Link] datasets 80,81 , yielding a processed dataset with
normalized expression values, highly variable genes, principal components, and 𝑘-nearest neighbors
all computed. For computational efficiency, we randomly selected two disjoint subsets of 𝑁 = 20, 000
cells each, attempting to match (batch, cell type) distributions of the entire processed dataset.
The “train” dataset was used for model training and selection of the highest-performing node in a
single tree search. The “validation” dataset was used to select the best tree search for methods in
which we ran multiple replicates of the same algorithm (Supplementary Fig. 1).

Evaluating scRNA-seq Batch Integration on the [Link] Benchmark We down-

loaded the OpenProblems v2.0.0 input and solution data from s3://openproblems-data/resources/
task_batch_integration/datasets/cellxgene_census/ and raw performance metrics from s3://
openproblems-data/resources/task_batch_integration/results/run_2025-01-23_18-03-16/score_
[Link]. We computed control-scaled metric results identically to the published OpenProblems
results. Briefly, for each (dataset, metric), lower and upper bounds on raw scores are defined
as the minimum and maximum values achieved by the seven “control” methods. Raw values were
linearly scaled between those extrema and clamped to be in [0, 1]. Overall score was computed as the
arithmetic mean over all 78 measurements (13 metrics computed for each of 6 datasets) with NaN
values replaced by 0 (i.e., failure to compute a metric causes it to be considered the worst possible
score).

Replication of Existing Methods for Batch Integration The [Link] benchmark profiles
the performance of several state-of-the-art existing methods. As of July 11, 2025 there were 19
different methods. Three methods have implementations in both R and Python: LIGER and pyliger,
Harmony and Harmonypy, and batchelor mnnCorrect and mnnpy. After grouping reimplementations of
the same method, there are 16 separate research ideas. From this list, we excluded all six foundation
model methods (UCE, SCimilarity, scGPT (zero shot), scGPT (fine-tuned), Geneformer, and
scPRINT) because they perform very poorly on the benchmark and use a much larger training set.
For example, only a single foundation model (UCE) performs better than the negative control of “No
integration” which simply performs PCA on the dataset. We further excluded scANVI, which is a
modification of scVI that is trained using cell type information. Since cell type information is used
to define the metrics, this represents data leakage and consequently we consider scANVI a control
method. This resulted in nine existing different research methods to optimize with tree search.
For each of the nine existing methods, we obtained the manuscript PDF corresponding to the
method. To obtain a short method description from the manuscript, we used Gemini 2.5 Pro Thinking
to summarize the paper (prompt in Supplementary Table 15, example output in Supplementary
Table 16). For batchelor fastMNN, which is a faster implementation of batchelor mnnCorrect, there
is no separate publication and thus we provided the paper PDF of batchelor mnnCorrect as well as
the docstring corresponding to batchelor fastMNN from [Link]
man/[Link] (Details section) with a slightly adjusted prompt. Finally, the method summary
is added to the tree search notebook, and is used to come up with better code solutions given the
method summary.
For each of the nine methods, we ran three replicates of tree search. For Fig. 2, we selected the
replicate that had the best performance based on the validation set score. We show the performance
of all replicates in Supplementary Fig. 3.

19
 An AI system to help scientists write expert-level empirical software

Hyperparameters To determine optimal hyperparameters for each base method, we employed

Optuna, an automated hyperparameter optimization framework 82 . Search spaces were defined across
integer, float, and categorical parameter types by experts. The optimization process ran for a total of
five times the number of parameters. In each trial, a model was trained using a sampled parameter
set and evaluated based on a performance metric that Optuna’s Tree-structured Parzen Estimator
(TPE) sampler aimed to maximize. All hyperparameter optimization was conducted solely on the
training dataset. The best identified hyperparameter set was then utilized to train the final base
methods and evaluate them on the held-out OpenProblems dataset.

COVID-19 prediction

Dataset Our primary data source was historical confirmed COVID-19 hospital admissions, which
corresponds to the target variable specified by CovidHub. These data are published weekly by the CDC
within the National Healthcare Safety Network (NHSN) Hospital Respiratory Data (HRD) dataset 83 .
Preprocessing was kept minimal–missing values in the dataset were replaced by zeros to enable tree
search to find executable code with the criterion score (WIS). The only additional data source used to
augment the target for our model was static jurisdiction-specific population values from the CovidHub
GitHub Repository 21 . For comparing model performance in Fig. 3c, we use all of the models submitted
to Forecast Hub which make predictions at a state by state level and have forecasts for at least 75
percent of the season and time horizons. We ran tree search with 2000 nodes for each reported run.

Replication of existing COVID-19 prediction models We selected eight models for replication from
those that had submitted to CovidHub based on the following inclusion criteria: (1) The method must
be reproducible solely using historical COVID-19 hospitalization data, without reliance on external
predictor variables, (2) The model submission must include predictions across all specified time
horizons, and (3) Model submissions must be available for over three months (12 weeks) to enable
meaningful comparison. Three models were excluded for failing these criteria: two were ensembles of
external forecasts, and one relied entirely on additional data. An additional five models were excluded
because they did not provide predictions for all forecast horizons. These five models originated
from the same forecasting team. As all our analysis involves aggregating model performance across
horizons, we have excluded these five models from all comparisons. Overall this gave a selection of
eight models for replication.
To instruct the search algorithm, we provided the method descriptions from the original authors’
official submission metadata. For example, the metadata for the UMASS-arc6-pooled model states:
“AR(6) model after fourth root data transform. AR coefficients are shared across all locations. A separate
variance parameter is estimated for each location.” We integrated these concise descriptions directly
into the tree search prompt as part of the model directions, transforming them into instructions by
prepending ‘Use a/an’ (see Methods, Supplementary Table 8).

GIFT-Eval Benchmark

We applied our tree search methodology to the General Time Series Forecasting Model Evaluation
(GIFT-Eval) benchmark 24 . The search begins from a root node defined by an initial code template
and proceeds via hill climbing, where new candidate solutions are generated and evaluated against
the GIFT-Eval validation folds. At the end of a tree search, we evaluated the solution on the held-out
test set using MASE point forecast as the scoring metric. Our results are based on a 5/18/2025
snapshot of the dataset, official leaderboard and scoring, all of which have been updated since. See
Supplementary Table 10 for a complete snapshot of the leaderboard.

20
 An AI system to help scientists write expert-level empirical software

We adhered to the benchmark’s framework, utilizing the official dataset source from Hugging
Face, its pre-defined training, validation, and test splits, as well as the scoring and evaluation code
commonly used in the existing submission notebooks.
Per-dataset Solution We conducted separate tree searches for 92 of the 97 GIFT-Eval datasets,
excluding the five largest due to computational constraints; for these, the naive baseline score was
used in order to produce the aggregated leaderboard score. For each dataset, we used a search of
300 nodes, with the agent permitted to use a broad suite of machine learning libraries, including
scikit-learn, XGBoost, and statsmodels. Supplementary Fig. 17 shows an analysis of the types of
models used across the 92 different solutions.
Unified Solution Here, we created a single, unified forecasting library that could generalize across
all 97 datasets. We used a tree search of over 1,000 nodes, guided by the geometric mean of the
normalized MASE scores across all datasets, providing a single objective function to optimize. To force
the model to reason from first principles, its access was restricted to basic libraries (numpy, pandas,
and holidays).
The resulting solution consists of two components: a single forecasting library and a list of
eight preset configurations. For each dataset, the best-performing configuration is identified on the
validation set. This selected configuration is then used with the unified library to produce the final
forecast on the test set, allowing the model to adapt its strategy without seeing test data.
The final solution was developed iteratively. An initial search yielded a base model with a MASE
of 0.82. A key breakthrough occurred in a subsequent run when the search space was expanded to
ten configurations and the agent was advised to use the holidays library, which improved the MASE
to 0.77 (Supplementary Fig. 18). A final 500 node refinement run pruned the configurations to an
optimized set of eight, achieving the final MASE of 0.734.
The final solution sequentially models and removes fundamental components of the series, with
the final forecast being the sum of the individual component forecasts. This approach allows the
model to be highly configurable while systematically accounting for different sources of variation in
the data. This process is outlined with the following steps:

1. Preprocessing: The input series first undergoes basic cleaning, including median imputation
for any missing values. An optional log-transform (log1p) can be applied to stabilize variance
in series with exponential growth patterns.
2. Base/Level Component: A base level is established using simple but robust methods like a
seasonal naive forecast or a rolling median of recent data points. This component captures the
basic magnitude of the series.
3. Trend Component: The residuals from the base component are then modeled to capture linear
or polynomial trends. This step includes a damping_factor to prevent unrealistic long-term
extrapolation by gradually flattening the trend.
4. Seasonality Component: The residuals from the trend component are analyzed to model
cyclical patterns (e.g., weekly, yearly). The model identifies the cycle length and forecasts
seasonality by averaging values at the same point in the cycle (e.g., the average value for all
Mondays).
5. Datetime and Holiday Features: To capture special events and non-seasonal cycles, features
are extracted from the timestamp (e.g., dayofweek, is_holiday_flag). The model calculates
the median effect of each feature category from the remaining residuals and adds it to the
forecast.
6. Residual Correction: As a final step, a correction is made by modeling the median of the most
recent unexplained errors. This autoregressive-like step helps correct for short-term biases in

21
 An AI system to help scientists write expert-level empirical software

the model. A decay_factor fades its impact over the forecast horizon.

To apply the unified solution to a new dataset, one would first split the historical data into training
and validation sets. Using the library’s adaptive configuration system, one can then find a suitable
forecasting strategy by evaluating the eight preset configurations on the validation data to select
the best-performing one. This provides a strong, data-driven starting point that can be used directly.
For more specialized applications, one can also create a custom configuration, allowing for manual
refinement of the model’s components and making the library both powerful out-of-the-box and
flexible enough for expert tuning.

Difficult Integrals

We carried out a tree search over 1000 nodes, using a list of integrals in Supplementary Fig. 21.
To build these lists of integrals, we started with a long list of integrals in LaTeX form from
Gradshteyn and Ryzhik 60 . We converted both the question and solution into a python expression
using SymPy 84 . Most expressions included free parameters, often with value constraints. To enable
numeric evaluations, we generated random values for all parameters consistent with the constraints.
Once an integral and its answer were in the form of SymPy expression objects, we evaluated
answers numerically by substituting our chosen parameter values using [Link]() and
evaluating via [Link](). We build integrand functions suitable for [Link]()
via [Link] for efficient evaluation. We compared each numerical answer to the number
returned by [Link]() and discarded cases where the numbers agreed within the
latter’s error estimate. We also discarded cases where that error estimate was greater than 2% of the
numbers’ magnitude.
All conversions from LaTeX to SymPy and all constrained parameter generations were performed
by Gemini using specialized prompts. The resulting SymPy expressions and parameter values were
examined manually for correctness. These manual steps were the limiting factor on the scale of our
dataset.
The scoring function we used during training used the absolute fractional error (discrepancy
between the generated solution’s number and the answer’s number) via a logarithm to prevent outliers
from dominating the result.
 response − answer 
score = − log 1 + (1)
answer

References
[1] Fortin, J. A., Cardille, J. A. & Perez, E. Multi-sensor detection of forest-cover change across 45
years in Mato Grosso, Brazil. Remote Sens. Environ. 238, 111266 (2020).

[2] Hohenberg, P. & Kohn, W. Inhomogeneous electron gas. Phys. Rev. 136, B864 (1964).

[3] Kohn, W. & Sham, L. J. Self-consistent equations including exchange and correlation effects.
Phys. Rev. 140, A1133 (1965).

[4] Warshel, A. & Levitt, M. Theoretical studies of enzymic reactions: dielectric, electrostatic and
steric stabilization of the carbonium ion in the reaction of lysozyme. J. Mol. Biol. 103, 227–249
(1976).

22
 An AI system to help scientists write expert-level empirical software

[5] Jumper, J. et al. Highly accurate protein structure prediction with AlphaFold. Nature 596,
583–589 (2021).

[6] Baek, M. et al. Accurate prediction of protein structures and interactions using a three-track
neural network. Science 373, 871–876 (2021).

[7] Hourdin, F. et al. The art and science of climate model tuning. Bull. Am. Meteorol. Soc. 98,
589–602 (2017).

[8] Anderson Jr., J. Basic philosophy of CFD. In Computational Fluid Dynamics, 3–14 (Springer,
2009).

[9] Silver, N. The signal and the noise: why so many predictions fail-but some don’t (Penguin, 2012).

[10] Farmer, J. D. Making sense of chaos: a better economics for a better world (Yale Univ. Press, 2024).

[11] Bernanke, B. & Blanchard, O. What caused the US pandemic-era inflation? Am. Econ. J.
Macroecon. 17, 1–35 (2025).

[12] Silver, D. et al. Mastering the game of Go with deep neural networks and tree search. Nature
529, 484–489 (2016).

[13] Silver, D. et al. Mastering the game of Go without human knowledge. Nature 550, 354–359
(2017).

[14] Jiang, Z. et al. AIDE: AI-driven exploration in the space of code. arXiv preprint arXiv:2502.13138
(2025).

[15] Novikov, A. et al. AlphaEvolve: A coding agent for scientific and algorithmic discovery. arXiv
preprint arXiv:2506.13131 (2025).

[16] Romera-Paredes, B. et al. Mathematical discoveries from program search with large language
models. Nature 625, 468–475 (2024).

[17] Wu, X., Wu, S.-h., Wu, J., Feng, L. & Tan, K. C. Evolutionary computation in the era of large
language model: survey and roadmap. IEEE Trans. Evol. Comput. (2024).

[18] Hu, S., Lu, C. & Clune, J. Automated design of agentic systems. arXiv preprint arXiv:2408.08435
(2024).

[19] Xu, C. et al. Automatic cell-type harmonization and integration across Human Cell Atlas datasets.
Cell 186, 5876–5891.e20 (2023).

[20] Regev, A. et al. The Human Cell Atlas. eLife 6, e27041 (2017).

[21] Centers for Disease Control and Prevention. COVID-19 forecast hub (2025). URL https:
//[Link]/cdcgov/covid19-forecast-hub?tab=readme-ov-file.

[22] Shao, Z., Yang, K. & Zhou, W. Performance evaluation of single-label and multi-label remote
sensing image retrieval using a dense labeling dataset. Remote Sens. 10, 964 (2018).

[23] Lueckmann, J.-M. et al. ZAPBench: a benchmark for whole-brain activity prediction in zebrafish.
arXiv preprint arXiv:2503.02618 (2025).

[24] Aksu, T. et al. GIFT-Eval: a benchmark for general time series forecasting model evaluation.
arXiv preprint arXiv:2410.10393 (2024). URL [Link]
GIFT-Eval.

23
 An AI system to help scientists write expert-level empirical software

[25] Jovic, D. et al. Single-cell RNA sequencing technologies and applications: a brief overview. Clin.
and Transl. Med. 12, e694 (2022).

[26] Svensson, V., Vento-Tormo, R. & Teichmann, S. A. Exponential scaling of single-cell RNA-seq in
the past decade. Nat. Protoc. 13, 599–604 (2018).

[27] CZI Cell Science Program et al. CZ CELLxGENE Discover: a single-cell data platform for scalable
exploration, analysis and modeling of aggregated data. Nucleic Acids Res. 53, D886–D900
(2025).

[28] Zhang, J. et al. Tahoe-100M: a giga-scale single-cell perturbation atlas for context-dependent
gene function and cellular modeling. bioRxiv 2025–02 (2025).

[29] Stuart, T. & Satija, R. Integrative single-cell analysis. Nat. Rev. Genet. 20, 257–272 (2019).

[30] Zappia, L., Phipson, B. & Oshlack, A. Exploring the single-cell RNA-seq analysis landscape with
the scRNA-tools database. PLoS Comput. Biol. 14, e1006245 (2018).

[31] Tran, H. T. N. et al. A benchmark of batch-effect correction methods for single-cell RNA
sequencing data. Genome Biol. 21, 1–32 (2020).

[32] Chazarra-Gil, R., van Dongen, S., Kiselev, V. Y. & Hemberg, M. Flexible comparison of batch
correction methods for single-cell RNA-seq using BatchBench. Nucleic Acids Res. 49, e42 (2021).

[33] Luecken, M. D. et al. Benchmarking atlas-level data integration in single-cell genomics. Nat.
Methods 19, 41–50 (2022).

[34] Luecken, M. D. et al. Defining and benchmarking open problems in single-cell analysis. Nat.
Biotechnol. 43, 1035–1040 (2025).

[35] Google. Gemini Deep Research (2025). URL [Link]

deep-research/?hl=en.

[36] Gottweis, J. et al. Towards an AI co-scientist. arXiv preprint arXiv:2502.18864 (2025).

[37] Johnson, W. E., Li, C. & Rabinovic, A. Adjusting batch effects in microarray expression data
using empirical Bayes methods. Biostatistics 8, 118–127 (2007).

[38] Polański, K. et al. BBKNN: fast batch alignment of single cell transcriptomes. Bioinformatics 36,
964–965 (2019).

[39] Chandrashekar, A. et al. TabVI: leveraging lightweight transformer architectures to learn

biologically meaningful cellular representations. bioRxiv 2025–02 (2025).

[40] Yang, Y. & Newsam, S. Bag-of-visual-words and spatial extensions for land-use classification. In
Proc. 18th SIGSPATIAL Int. Conf. on Adv. in Geogr. Inf. Syst., 270–279 (Association for Computing
Machinery, 2010).

[41] Russakovsky, O. et al. ImageNet large scale visual recognition challenge. Int. J. Comput. Vis.
115, 211–252 (2015).

[42] Krizhevsky, A., Sutskever, I. & Hinton, G. E. ImageNet classification with deep convolutional
neural networks. Adv. Neural Inf. Process. Syst. 25 (2012).

[43] Zhong, B., Du, J., Liu, M., Yang, A. & Wu, J. Region-enhancing network for semantic segmenta-
tion of remote-sensing imagery. Sensors 21 (2021).

24
 An AI system to help scientists write expert-level empirical software

[44] Zhang, Z., Liu, B. & Li, Y. FURSformer: semantic segmentation network for remote sensing
images with fused heterogeneous features. Electronics 12 (2023).

[45] Atiampo, A. K. & Diédié, G. H. F. New fusion approach of spatial and channel attention for
semantic segmentation of very high spatial resolution remote sensing images. Open J. Appl. Sci.
14, 288–319 (2024).

[46] Sun, Y., Bi, F., Gao, Y., Chen, L. & Feng, S. A multi-attention UNet for semantic segmentation in
remote sensing images. Symmetry 14, 906 (2022).

[47] Elgamily, K. M., Mohamed, M. A., Abou-Taleb, A. M. & Ata, M. M. A novel W13 deep CNN
structure for improved semantic segmentation of multiple objects in remote sensing imagery.
Neural Comput. Appl. 37, 5397–5427 (2025).

[48] Immer, A. et al. Forecasting whole-brain neuronal activity from volumetric video. arXiv preprint
arXiv:2503.00073 (2025).

[49] Zeng, A., Chen, M., Zhang, L. & Xu, Q. Are transformers effective for time series forecasting? In
Proc AAAI Conf. Artif. Intell., vol. 37, 11121–11128 (2023).

[50] Das, A. et al. Long-term forecasting with TiDE: Time-series Dense Encoder. Trans. Mach. Learn.
Res. (2023).

[51] Chen, S.-A., Li, C.-L., Yoder, N., Arik, S. O. & Pfister, T. TSMixer: An All-MLP architecture for
time series forecasting. Trans. Mach. Learn. Res. (2023).

[52] Perez, E., Strub, F., De Vries, H., Dumoulin, V. & Courville, A. FiLM: Visual reasoning with a
general conditioning layer. In Proc AAAI Conf. Artif. Intell., vol. 32 (2018).

[53] Deistler, M. et al. Differentiable simulation enables large-scale training of detailed biophysical
models of neural dynamics. bioRxiv 2024–08 (2024).

[54] Hoo, S. B., Müller, S., Salinas, D. & Hutter, F. From tables to time: how TabPFN-v2 outperforms
specialized time series forecasting models. arXiv preprint arXiv:2501.02945 (2025).

[55] Liu, Y. et al. Sundial: A family of highly capable time series foundation models. arXiv preprint
arXiv:2502.00816 (2025).

[56] Ansari, A. F. et al. Chronos: learning the language of time series. Trans. Mach. Learn. Res.
(2024).

[57] Oreshkin, B. N., Carpov, D., Chapados, N. & Bengio, Y. N-BEATS: neural basis expansion analysis
for interpretable time series forecasting. arXiv preprint arXiv:1905.10437 (2019).

[58] Ho, S. L. & Xie, M. The use of ARIMA models for reliability forecasting and analysis. Comput.
Ind. Eng. 35, 213–216 (1998).

[59] Piessens, R., de Doncker-Kapenga, E., Überhuber, C. W. & Kahaner, D. QUADPACK: a subroutine
package for automatic integration (Springer-Verlag, 1983).

[60] Gradshteyn, I. & Ryzhik, I. Table of integrals, series, and products, 8th edn (Academic Press,
1994).

[61] Koza, J. R. Genetic programming as a means for programming computers by natural selection.
Stat. Comput. 4, 87–112 (1994).

25
 An AI system to help scientists write expert-level empirical software

[62] Mernik, M., Heering, J. & Sloane, A. M. When and how to develop domain-specific languages.
ACM computing surveys (CSUR) 37, 316–344 (2005).

[63] Czarnecki, K. Generative programming: Methods, techniques, and applications tutorial abstract.
In International Conference on Software Reuse, 351–352 (Springer, 2002).

[64] Chen, M. et al. Evaluating large language models trained on code. arXiv preprint
arXiv:2107.03374 (2021).

[65] Li, Y. et al. Competition-level code generation with AlphaCode. Science 378, 1092–1097 (2022).

[66] Hutter, F., Kotthoff, L. & Vanschoren, J. Automated machine learning: methods, systems, challenges
(Springer Nature, 2019).

[67] Merchant, A. et al. Scaling deep learning for materials discovery. Nature 624, 80–85 (2023).

[68] Xiao, Y. et al. CellAgent: An LLM-driven multi-agent framework for automated single-cell data
analysis. arXiv preprint arXiv:2407.09811 (2024).

[69] Zhang, H. et al. CompBioAgent: An LLM-powered agent for single-cell RNA-seq data exploration.
bioRxiv 2025–03 (2025).

[70] Zhou, J. et al. An AI agent for fully automated multi-omic analyses. Adv. Sci. 11, 2407094
(2024).

[71] Xin, Q. et al. BioInformatics Agent (BIA): unleashing the power of large language models to
reshape bioinformatics workflow. bioRxiv 2024–05 (2024).

[72] Alber, S. et al. CellVoyager: AI compbio agent generates new insights by autonomously analyzing
biological data. bioRxiv 2025–06 (2025).

[73] Baek, J., Jauhar, S. K., Cucerzan, S. & Hwang, S. J. ResearchAgent: iterative research idea
generation over scientific literature with large language models. arXiv preprint arXiv:2404.07738
(2024).

[74] Lu, C. et al. The AI Scientist: towards fully automated open-ended scientific discovery. arXiv
preprint arXiv:2408.06292 (2024).

[75] Du, M., Xu, B., Zhu, C., Wang, X. & Mao, Z. DeepResearch Bench: a comprehensive benchmark
for deep research agents. arXiv preprint arXiv:2506.11763 (2025).

[76] Perplexity. Perplexity Deep Research (2025). URL [Link]

introducing-perplexity-deep-research.

[77] Coelho, J. et al. DeepResearchGym: A free, transparent, and reproducible evaluation sandbox
for deep research. arXiv preprint arXiv:2505.19253 (2025).

[78] Xu, R. & Peng, J. A comprehensive survey of deep research: Systems, methodologies, and
applications. arXiv preprint arXiv:2506.12594 (2025).

[79] Lee, J. et al. Gemini Embedding: Generalizable embeddings from Gemini. arXiv preprint
arXiv:2503.07891 (2025).

[80] Gigante, S., Cannoodt, R. et al. openproblems (2025). URL [Link]

openproblems-bio/openproblems.

26
 An AI system to help scientists write expert-level empirical software

[81] Cannoodt, R., Zappia, L., Burkhardt, D. et al. task_batch_integration (2025). URL https:
//[Link]/openproblems-bio/task_batch_integration.

[82] Akiba, T., Sano, S., Yanase, T., Ohta, T. & Koyama, M. Optuna: A next-generation hyperparameter
optimization framework. In Proc. 25th ACM SIGKDD Int. Conf. Knowl. Discov. Data Min. (2019).

[83] Centers for Disease Control and Prevention. Weekly Hospital Respiratory Data (HRD)
Metrics by Jurisdiction (2024). URL [Link]
Weekly-Hospital-Respiratory-Data-HRD-Metrics-by-Ju/mpgq-jmmr. Dataset ID: mpgq-
jmmr. Last updated: June 14, 2024.

[84] Meurer, A. et al. SymPy: symbolic computing in Python. PeerJ Comput. Sci. 3, e103 (2017).

[85] McInnes, L., Healy, J. & Melville, J. UMAP: uniform manifold approximation and projection for
dimension reduction. arXiv preprint arXiv:1802.03426 (2018).

[86] Polański, K. et al. bbknn (2018). URL [Link]

27
 An AI system to help scientists write expert-level empirical software

Supplementary Figures

28
 An AI system to help scientists write expert-level empirical software

a b
Generate prompt with method description
CELLxGENE
Tree search
Base Method prompt with
Method Description method
description
Diabetic Kidney
GTEX v9 HypoMap
Disease

Seed Tree Search with the prompt

3 replicas
Mouse Pancreatic
Immune Cell Atlas Tabula Sapiens
Islet Atlas

Tree search hill climbing

[Link]
Holdout Datasets

… Training Split
n=20,000

Tree Search Replica 1 Tree Search Replica 2 Tree Search Replica 3

ffdaa1f0-b1d1-4135
-8774-9fed7bf039b
…
a

Identify a top node based on hill climbing score

Method script from top node:

def eliminate_batch_effect_fn(adata):
[Link].normalize_total(adata)
…

Training Split
Validation n=20,000
Training Split
Split
n=20,000
n=20,000

Tree Search Replica 1 Tree Search Replica 2 Tree Search Replica 3

Select the best implementation based on validation score

Validation
Split
n=20,000

Tree Search Replica 1 Tree Search Replica 2 Tree Search Replica 3

Evaluate the best implementation on the holdout datasets

[Link]
Holdout Datasets

Final score

Supplementary Fig. 1 | Experimental design for single-cell batch integration. a, We sourced our
tree search development dataset from CELLxGENE. After filtering and manually selecting the dataset
364bd0c7-f7fd-48ed-99c1-ae26872b1042 version ffdaa1f0-b1d1-4135-8774-9fed7bf039ba (see
Methods), which has a similar profile to the six datasets used in the [Link] Batch
Integration benchmark (distinct datasets also in CELLxGENE), we sampled 20,000 cells for the
training split and 20,000 for the validation split. b, For each of the 11 base methods, we generated a
detailed method description and inserted it into a prompt to initialize the tree search. We ran three
independent tree search replicas per method, using the training split for hill climbing. From each
tree, we selected the top-performing node based on its training score. We then evaluated each top
node’s script on the validation split and selected the best one based on validation performance. The
best implementation per method was finally evaluated on the [Link] holdout datasets,
and the corresponding scores are reported as final results.

29
 An AI system to help scientists write expert-level empirical software

Supplementary Fig. 2 | Uniform Manifold Approximation and Projection 85 of BBKNN (TS) on the
Immune Cell Atlas dataset. a, The UMAP projection colored by cell type shows cell-type-specific
clusters. b, The UMAP projection colored by data batch shows good batch mixing across the dataset.

30
 An AI system to help scientists write expert-level empirical software

a Base method
0.6
Tree search
0.5
Overall score

0.4

0.3

0.2

0.1

0.0 BBKNN ComBat Harmony LIGER No advice SCALEX Scanorama TabVI batchelor batchelor scVI
fastMNN mnnCorrect
Method
b Base method
0.8
Tree search
0.7
0.6
Overall score

0.5
0.4
0.3
0.2
0.1
0.0 BBKNN ComBat Harmony LIGER No advice SCALEX Scanorama TabVI batchelor batchelor scVI
fastMNN mnnCorrect
Method

Supplementary Fig. 3 | Relative performance of base methods and our method replicates. a,
Overall scores on the holdout OpenProblems datasets for all replicates of methods evaluated in Fig. 2.
For tree search implementations, three replicates of the full process were performed. Dots indicate
the overall score of the replicate on the holdout OpenProblems datasets. The bar shows the
performance of the replicate with highest performance in the validation dataset (identical values to
those shown in Fig. 2). The lowest performing tree search replicates for BBKNN, Scanorama, and
TabVI only successfully computed 30, 57, and 45 of the 78 metrics, respectively. We note that failures
due to out of memory or compute time issues were not explicitly selected against in our algorithm
since all optimization was performed on datasets of only 20k cells. b, Average scores for each method
when restricting to only (method, dataset, metric) combinations that have non-NaN values for the
base method and all three tree search replicates. No advice and TabVI are absent since they have no
base method comparator.

31
 An AI system to help scientists write expert-level empirical software

0.6 Base method (hyperparameter optimized)

Tree search

0.5

0.4
Overall score

0.3

0.2

0.1

0.0 BBKNN ComBat Harmony LIGER No advice SCALEX Scanorama TabVI batchelor batchelor scVI
fastMNN mnnCorrect
Method
Supplementary Fig. 4 | Relative performance of base methods with optimized hyperparameters
and tree search replicates. Overall scores on the holdout OpenProblems datasets for all replicates of
methods evaluated in Fig. 2. Hyperparameters for the base methods were optimized using the
training dataset. For tree search implementations, three replicates of the full process were performed.
Dots indicate the overall score of the replicate on the holdout OpenProblems datasets. The bar shows
the performance of the replicate with highest performance in the validation dataset (identical values
to those shown in Fig. 2). The No advice and TabVI methods have no base method code available.
The batchelor mnnCorrect hyperparameter-optimized base method code failed to compute
embeddings on every OpenProblems dataset owing to out-of-memory errors.

32
 An AI system to help scientists write expert-level empirical software

#0
nan

#1 #2 #3 #4 #5 #6 #7 #8 #9 #10
-inf 0.532 0.458 0.192 -inf -inf 0.405 -inf -inf -inf

#12 #27 #28 #21 #22 #24 #19 #16 #14 #26 #15 #11 #13
0.239 0.532 0.321 0.458 0.458 -inf 0.383 0.542 0.419 0.423 -inf 0.212 -inf

#20 #32 #29 #30 #31 #33 #34 #23 #25 #18 #17
0.391 -inf 0.629 0.616 0.541 0.577 0.57 0.419 0.418 -inf -inf

#35 #36 #38 #37 #39 #40 #44 #41

0.628 0.623 0.62 0.536 0.536 0.539 0.577 0.577

#42 #43 #48 #49 #45 #46 #47

0.629 0.62 0.624 0.619 0.629 0.624 0.619

#50 #51 #54 #56 #59 #57 #52 #53 #58 #55
0.609 0.628 0.624 0.628 0.611 0.624 0.628 0.628 0.629 0.629

#60 #63 #61 #62 #67 #68 #64 #65 #66 #69
0.624 0.639 0.629 0.628 0.6 0.629 0.624 0.624 0.628 0.62

#73 #74 #75 #76 #77 #70 #71 #72 #78 #79 #81
0.639 0.639 0.639 0.639 0.642 0.628 0.629 0.629 0.628 0.624 0.541

#82 #84 #85 #83 #86 #87 #88 #89 #80

0.642 0.639 0.641 0.639 0.579 0.644 0.639 0.639 0.628

#90 #91 #93 #94 #92 #95 #97 #103 #96 #98
0.642 0.639 0.639 0.642 0.592 0.644 0.632 0.642 0.639 0.626

#99 #102 #100 #101 #104 #109 #105 #106 #107 #108 #110 #112 #113 #114
0.639 0.641 0.639 0.639 0.638 0.639 0.644 0.643 0.643 0.632 0.632 0.589 0.639 -inf

#111 #115 #118 #121 #123 #117 #120 #124 #116 #119 #122
0.639 0.643 0.644 0.643 0.644 0.59 -inf 0.642 0.644 0.632 0.644

#125 #127 #128 #130 #151 #152 #270 #271 #272 #273 #274 #275 #276 #277 #278 #281 #282 #283 #308 #309 #310 #311 #313 #314 #315 #316 #317 #318 #319 #320 #321 #322 #323 #324 #325 #326 #327 #328 #329 #330 #331 #332 #333 #335 #336 #337 #338 #339 #340 #341 #342 #343 #344 #345 #346 #347 #348 #349 #350 #351 #352 #353 #360 #361 #362 #363 #364 #366 #367 #368 #369 #370 #371 #372 #378 #379 #132 #126 #129 #131 #136 #175 #176 #225 #228 #240 #242 #377
0.644 0.644 0.644 0.643 0.644 0.643 0.644 0.644 0.644 0.644 -inf 0.643 0.614 0.644 0.644 0.643 0.644 0.643 0.639 0.521 0.644 0.639 0.632 0.644 0.644 0.632 -inf 0.596 0.643 -inf 0.644 -inf 0.644 0.623 0.644 0.644 0.644 0.643 0.604 0.643 0.561 0.643 0.643 0.644 0.644 0.643 0.642 0.642 0.63 0.644 0.643 0.632 0.644 0.643 0.644 0.644 0.644 0.643 0.644 0.612 0.626 0.644 0.644 0.639 0.644 0.602 0.641 0.643 0.623 0.643 -inf 0.644 0.644 0.542 0.644 0.644 0.643 0.644 0.643 0.644 0.643 0.643 0.643 0.644 0.644 0.643 0.639 0.644

#133 #135 #139 #146 #153 #137 #148 #162 #163 #166 #174 #179 #279 #280 #284 #285 #286 #287 #288 #289 #293 #294 #295 #298 #299 #302 #303 #307 #312 #334 #359 #391 #354 #355 #358 #356 #357 #373 #374 #375 #381 #384 #388 #380 #386 #389 #134 #138 #144 #149 #157 #178 #140 #141 #142 #145 #199 #224 #226 #236 #238 #241
0.644 0.643 0.643 0.644 0.644 0.614 0.631 0.643 0.643 0.644 0.644 0.644 0.641 0.644 0.554 0.643 0.644 0.644 0.644 0.643 0.643 0.643 0.614 0.644 0.644 0.644 0.639 0.643 0.641 0.643 0.644 0.644 0.644 0.574 0.644 0.643 0.644 0.643 0.644 0.644 0.643 0.644 0.643 0.632 0.643 0.644 0.588 0.643 0.643 0.639 0.643 0.588 0.644 0.644 0.632 0.644 0.537 0.643 0.637 0.641 0.644 0.644

#143 #155 #158 #159 #160 #165 #167 #161 #180 #181 #195 #290 #291 #292 #297 #296 #301 #398 #399 #401 #402 #405 #408 #412 #418 #419 #420 #423 #426 #365 #382 #383 #385 #387 #390 #147 #150 #154 #243 #244 #246 #247 #248 #249 #250 #251
0.643 0.643 0.643 0.644 0.644 0.644 0.644 0.644 0.537 0.643 0.643 0.632 0.644 0.644 0.632 0.643 0.619 0.643 0.644 0.643 0.644 0.549 0.639 0.643 0.644 0.643 0.644 0.643 0.639 0.643 0.644 0.644 0.644 0.644 0.643 0.644 0.571 0.613 0.644 0.632 0.644 0.644 0.644 0.644 0.641 0.644

#168 #169 #172 #173 #177 #170 #171 #300 #304 #305 #306 #428 429 #430 #431 #432 #433 #434 #435 #392 #393 #394 #395 #404 #406 #409 #413 #415 #421 #422 #427 #396 #397 #400 #403 #407 #156 #252 #253 #255 #256 #258 #259 #260 #262 #263 #264 #265 #266 #267 #268 #254 #261 #257
0.644 0.643 0.644 0.644 0.644 0.643 0.644 0.643 0.644 0.639 0.644 -inf 0.654 0.466 0.643 0.613 0.641 -inf 0.634 0.643 0.644 0.643 0.641 0.643 0.644 0.638 0.643 0.644 0.644 0.643 0.632 0.643 0.643 0.644 0.644 0.643 0.644 0.576 0.621 0.629 -inf 0.644 0.639 0.632 0.612 0.556 0.628 0.643 0.537 0.643 0.632 0.638 0.641 0.644

#183 #184 #186 #182 #185 #196 #187 #188 #189 #191 #193 #194 #197 #440 #441 #442 #443 #444 #447 #448 #449 #416 #417 #424 #425 #410 #411 #414 #164 #269
0.644 -inf 0.643 0.643 0.644 0.643 0.643 0.573 0.644 0.644 0.644 0.644 0.643 0.557 0.649 0.577 0.652 0.557 0.649 0.649 0.602 0.644 0.643 0.644 0.644 0.643 0.644 -inf 0.643 0.596

#190 #192 #198 #202 #208 #227 #201 #203 #206 #209 #213 #215 #229 #239 #376 #204 #205 #207 #214 #220 #223 #452 #453 #456 #465 #470 #450 #451 #454 #457 #466 #469 #458 #461 #463 #467 #472 #459 #460 #462 #464 #468 #471 #436 #437 #438 #439
0.643 0.644 0.626 0.643 -inf 0.643 0.638 0.575 0.643 0.644 0.644 0.643 0.643 0.644 0.644 0.557 0.591 0.56 0.644 0.564 0.427 0.553 0.553 0.552 0.639 0.649 -inf 0.593 0.55 0.642 0.652 0.546 0.552 -inf 0.64 -inf 0.541 0.552 -inf 0.649 0.649 0.653 0.552 0.644 0.643 0.644 0.644

#200 #218 #219 #221 #230 #231 #232 #233 #234 #235 #237 #480 #483 #478 #482 #473 #474 #475 #476 #477 #479 #481 #484 #445 #446
0.644 0.614 0.632 0.644 0.643 0.643 0.627 0.644 0.618 0.644 0.637 -inf 0.65 0.651 0.645 0.587 0.649 0.552 0.649 0.649 0.554 0.653 0.654 0.643 0.644

#210 #211 #212 #216 #222 #245 #498 #503 #486 #487 #493 #488 #490 #496 #485 #489 #491 #492 #505 #494 #495 #497 #506 #455
0.644 0.624 0.643 0.643 0.644 0.643 0.552 0.555 0.651 -inf 0.564 0.642 0.649 0.555 0.553 0.649 -inf 0.533 0.621 0.545 0.623 0.642 0.539 0.637

#499 #502 #507 #501 #500 #504

0.602 -inf 0.579 0.614 0.649 0.649

Supplementary Fig. 5 | Top Figure Breakthrough plot for the BBKNN (TS) tree search, showing the
evolution of the maximum score as a function of the number of nodes. The green dots label places
where the score abruptly increases due to an improvement in the code, and the label describes the
change in the code that resulted in the score increase. Bottom Figure Structure of the tree for this
same search. The color range consists of orange (lower scores) to green (higher scores) with the
highest score denoted by a diamond node.

33
 An AI system to help scientists write expert-level empirical software

0.6 BBKNN implementation

Tree search
Package

0.5

0.4
Overall score

0.3

0.2

0.1

0.0 BBKNN (TS) ComBat+PCA+BBKNN Only BBKNN Standardize+ComBat+PCA ComBat+PCA

Standardize
ComBat+PCA
BBKNN
BBKNN (TS) ComBat+PCA+BBKNN Only BBKNN Standardize+ComBat+PCA ComBat+PCA

Supplementary Fig. 6 | Ablation analysis of the top-performing BBKNN (TS) method. The BBKNN
(TS) method performed standard linear expression scaling to 104 total counts followed by log1p
transformation. It then applied three additional transforms: “Standardize” called [Link] to
further scale the data to mean 0 and unit variance, “ComBat+PCA” called [Link] followed by
[Link] to generate the expression embedding, and “BBKNN” applied an implementation of
batch-balanced 𝑘-nearest neighbors writted by our method. Bars here show the overall performance
in the OpenProblems datasets for ablations that include one or more of these components. For each
ablation that includes the “BBKNN” component, comparison of the written BBKNN implementation
(“Tree search”) and the bbknn package implementation 86 (“Package”) is shown. Black dots show
individual performance of three replicates of each method.

34
 An AI system to help scientists write expert-level empirical software

0.8 n=69/78 0.8 n=69/78 0.8 n=69/78 0.8 n=69/78 0.8 n=69/78
0.6 0.6 0.6 0.6 0.6

Overall score
0.4 0.4 0.4 0.4 0.4
0.2 0.2 0.2 0.2 0.2
0.0 0.0 0.0 0.0 0.0
Harmony batchelor Recomb LIGER TabVI Recomb batchelor No advice Recomb scVI TabVI Recomb scVI No advice Recomb
fastMNN mnnCorrect
0.8 n=69/78 0.8 n=69/78 0.8 n=69/78 0.8 n=69/78 0.8 n=69/78
0.6 0.6 0.6 0.6 0.6
Overall score

0.4 0.4 0.4 0.4 0.4

0.2 0.2 0.2 0.2 0.2
0.0 0.0 0.0 0.0 0.0
ComBat TabVI Recomb batchelor TabVI Recomb ComBat batchelor Recomb Harmony scVI Recomb Harmony batchelor Recomb
mnnCorrect mnnCorrect mnnCorrect
0.8 n=69/78 0.8 n=69/78 0.8 n=69/78 0.8 n=69/78 0.8 n=69/78
0.6 0.6 0.6 0.6 0.6
Overall score

0.4 0.4 0.4 0.4 0.4

0.2 0.2 0.2 0.2 0.2
0.0 0.0 0.0 0.0 0.0
Harmony No advice Recomb TabVI No advice Recomb ComBat Harmony Recomb LIGER No advice Recomb Harmony LIGER Recomb
0.8 n=69/78 0.8 n=69/78 0.8 n=69/78 0.8 n=69/78 0.8 n=69/78
0.6 0.6 0.6 0.6 0.6
Overall score

0.4 0.4 0.4 0.4 0.4

0.2 0.2 0.2 0.2 0.2
0.0 0.0 0.0 0.0 0.0
ComBat No advice Recomb ComBat LIGER Recomb scVI batchelor Recomb batchelor No advice Recomb ComBat batchelor Recomb
fastMNN fastMNN fastMNN
0.8 n=69/78 0.8 n=69/78 0.8 n=69/78 0.8 n=69/78 0.8 n=57/78
0.6 0.6 0.6 0.6 0.6
Overall score

0.4 0.4 0.4 0.4 0.4

0.2 0.2 0.2 0.2 0.2
0.0 0.0 0.0 0.0 0.0
batchelor LIGER Recomb batchelor batchelor Recomb LIGER batchelor Recomb LIGER scVI Recomb LIGER SCALEX Recomb
fastMNN fastMNN mnnCorrect mnnCorrect
0.8 n=57/78 0.8 n=57/78 0.8 n=57/78 0.8 n=57/78 0.8 n=57/78
0.6 0.6 0.6 0.6 0.6
Overall score

0.4 0.4 0.4 0.4 0.4

0.2 0.2 0.2 0.2 0.2
0.0 0.0 0.0 0.0 0.0
SCALEX batchelor Recomb SCALEX TabVI Recomb Harmony SCALEX Recomb SCALEX No advice Recomb No advice Scanorama Recomb
mnnCorrect
0.8 n=57/78 0.8 n=57/78 0.8 n=57/78 0.8 n=57/78 0.8 n=57/78
0.6 0.6 0.6 0.6 0.6
Overall score

0.4 0.4 0.4 0.4 0.4

0.2 0.2 0.2 0.2 0.2
0.0 0.0 0.0 0.0 0.0
ComBat SCALEX Recomb ComBat Scanorama Recomb Harmony Scanorama Recomb scVI Scanorama Recomb SCALEX Scanorama Recomb
0.8 n=57/78 0.8 n=57/78 0.8 n=57/78 0.8 n=55/78 0.8 n=45/78
0.6 0.6 0.6 0.6 0.6
Overall score

0.4 0.4 0.4 0.4 0.4

0.2 0.2 0.2 0.2 0.2
0.0 0.0 0.0 0.0 0.0
LIGER Scanorama Recomb batchelor SCALEX Recomb TabVI Scanorama Recomb batchelor Scanorama Recomb batchelor Scanorama Recomb
fastMNN fastMNN mnnCorrect
0.8 n=33/78 0.8 n=33/78 0.8 n=30/78 0.8 n=30/78 0.8 n=30/78
0.6 0.6 0.6 0.6 0.6
Overall score

0.4 0.4 0.4 0.4 0.4

0.2 0.2 0.2 0.2 0.2
0.0 0.0 0.0 0.0 0.0
ComBat scVI Recomb scVI batchelor Recomb BBKNN TabVI Recomb LIGER BBKNN Recomb SCALEX BBKNN Recomb
mnnCorrect
0.8 n=30/78 0.8 n=30/78 0.8 n=30/78 0.8 n=20/78 0.8 n=10/78
0.6 0.6 0.6 0.6 0.6
Overall score

0.4 0.4 0.4 0.4 0.4

0.2 0.2 0.2 0.2 0.2
0.0 0.0 0.0 0.0 0.0
Harmony BBKNN Recomb BBKNN batchelor Recomb ComBat BBKNN Recomb batchelor BBKNN Recomb batchelor TabVI Recomb
mnnCorrect fastMNN fastMNN
0.8 n=10/78 0.8 n=10/78 0.8 n=0/78 0.8 n=0/78 0.8 n=0/78
0.6 0.6 0.6 0.6 0.6
Overall score

0.4 0.4 0.4 0.4 0.4

0.2 0.2 0.2 0.2 0.2
0.0 0.0 0.0 0.0 0.0
BBKNN No advice Recomb BBKNN Scanorama Recomb Harmony TabVI Recomb scVI BBKNN Recomb scVI SCALEX Recomb

Supplementary Fig. 7 | Comparison of tree search performance on base methods and their
“recombination” over an intersection of successfully calculated metrics. We ran “recombination”
experiments by seeding tree search with the top variants from two base method runs (see Methods).
We compare the performance of two base methods and “recombination” on the OpenProblems test
dataset for all 55 pairwise combinations of the 11 base methods. Since sometimes methods may fail
getting a score for certain evaluation metrics due to errors like out of memory, we compare the
performance on a subset of metrics that were successfully computed for all three methods. “n=X/78”
on each subplot shows the number of successfully computed metrics, X, that we averaged over. For
each subplot, we show the base methods on the left in light blue, and the recombination method on
the right (labeled as “Recomb”), where a green bar means the recombination method outperforms
both of its base methods, dark blue means the recombination method outperforms one of the base
methods, and red means the recombination method does not outperform either of the base methods.

35
 An AI system to help scientists write expert-level empirical software

1.00 Method
Base method (TS)
0.98 Recombination (TS)
0.96 Deep Research (TS)
0.94
0.92
0.90

ComBat
ComBat x No advice
No advice(TS)
ComBat (TS)
(TS)
No advice
ComBat (TS)
(TS)
LIGER (TS)
Harmony xx TabVI SCALEX
Harmony
Harmony (TS)
Harmony
HarmonyxxxHarmony BBKNN
Scanorama
ComBat
Harmony x(TS)
No advice
Harmony
Harmony (TS)
Harmony xx batchelor
Harmony LIGER mnnCorrect
BBKNN
Scanorama (TS) (TS)
batchelor mnnCorrect
Scanorama (TS) (TS)(TS)
batchelor
No advice fastMNN
BBKNN
LIGER (TS)(TS)
(TS)
ComBat
LIGER x LIGER
BBKNN(TS)
ComBat (TS)
x BBKNN
LIGER
No x BBKNN
advice fastMNN
x Scanorama
batchelor
batchelor fastMNN x No advice
LIGER
LIGER xx NoscVIadvice x LIGER
Harmony x batchelor
ComBat x mnnCorrect fastMNN
batchelor fastMNN
batchelor
batchelor mnnCorrect x(TS)
Scanorama
Scanorama
batchelorx fastMNN (TS) (TS)
ComBat
batchelor Scanorama
mnnCorrect (TS)
batchelor
ComBat x fastMNN batchelorx(TS)
BBKNN
mnnCorrect
batchelor
BBKNN x fastMNN
Scanorama
batchelor
batchelor mnnCorrect
fastMNN x No advice
batchelor
BBKNN fastMNN xxmnnCorrect
xx batchelor
Scanorama
batchelor mnnCorrect
BBKNN
LIGER No advice
xx Scanorama
LIGER(TS)
TabVI batchelor mnnCorrect
CVAE Cycle VAE
Hierarchical
Multi-Discriminator
Contrastive Learning
Pseudo-Label
Ensemble
SCALEX (TS)
SCALEX (TS)
ComBat
SCALEX x TabVI
TabVI
scVI
BBKNN (TS)xxTabVI
scVI
SCALEXx SCALEX
(TS)
TabVI x(TS)
scVI (TS) TabVI
scVI
TabVI (TS)
scVI
ComBat(TS)x scVI
Harmony
scVI x scVI mnnCorrect
xxbatchelor
TabVI
batchelor Scanorama
fastMNN x TabVI
batchelor
SCALEX x mnnCorrect
Scanorama x TabVI
batchelor
LIGER x fastMNN
TabVI x SCALEX
SCALEX xx BBKNN
SCALEX batchelor mnnCorrect
SCALEX
TabVI x No advice
LIGERxxxBBKNN
scVI
No advice
SCALEX
Diffusion
GNN
Hybrid OTx SCALEX
ComBat
scVI
scVI x No advice
scVI xx Scanorama
batchelor fastMNN
Method

Hierarchical VAE
(TS)
(TS)
(TS)
x No advice
advice (TS) (TS)
advice (TS) (TS)
(TS)
(TS)

advice (TS) (TS)

(TS)
(TS)

(TS)
x LIGER(TS)
xx BBKNN (TS)
Scanorama
BBKNN
x NoxadviceLIGER
x No advicex scVI
fastMNN

x Scanorama (TS)
BBKNN
BBKNN x xScanorama (TS)
Scanorama
xScanorama
No advice
TabVICycle(TS)
Multi-Discriminator
Pseudo-Label
Ensemble (TS)
(TS)
(TS)
x SCALEX (TS)
(TS)
(TS)

x Scanorama

OT
GNN
scVIx xScanorama
x SCALEX
advice
scVI x batchelor fastMNN
x SCALEX
(TS)
ComBatx xScanorama
(TS)

(TS)

TabVI
xx TabVI
TabVI
x TabVI
(TS)
(TS)
scVIx scVI
x Scanorama TabVI
SCALEX
x TabVI
x BBKNN
advice
SCALEX
advice
scVIxxDiffusion
BBKNN
TabVI
Harmony x BBKNN
advice
Harmony
(TS)
LIGER
(TS)

fastMNN
x Scanorama (TS)

mnnCorrect
No advice
mnnCorrect
BBKNNx mnnCorrect
mnnCorrect

mnnCorrect x scVI
xx TabVI
mnnCorrect
Harmony x batchelor mnnCorrect

Learning

Hybrid
ComBatComBat
NoComBat
NoComBat
LIGER

Scanorama
BBKNN
Scanorama
batchelorNofastMNN
LIGER
BBKNN
LIGER
BBKNN

Scanorama
fastMNN
batchelorfastMNN

SCALEX
SCALEX
SCALEXscVI
SCALEX
TabVI
scVI
TabVI
Harmony

Harmony
Harmony

batchelor mnnCorrect

mnnCorrect
mnnCorrect

x
x

CVAE
x

LIGER

ComBat
Harmony

No
x No

No
xNo

ComBat
BBKNN
scVI

fastMNN
LIGER
Harmony

mnnCorrect
ComBat

fastMNN
Harmony

x
batchelor
batchelor

Contrastive
LIGER
ComBat

fastMNN

SCALEX
scVI

fastMNN

LIGER

ComBat
Harmony

x
x

x
fastMNN
LIGER

SCALEX
TabVI
Harmony

mnnCorrect
BBKNN x batchelor
batchelor
LIGER x batchelor

scVI x batchelor

SCALEX x batchelor
advice

ComBat

fastMNN
LIGER

TabVI
SCALEX

scVI
Harmony

mnnCorrect
batchelor
ComBat xbatchelor

batchelor
batchelorbatchelor

batchelor
batchelor

batchelor
x
x

batchelor
batchelor
No

ComBat
Harmony
batchelor

batchelor
batchelor fastMNN x
batchelor

Supplementary Fig. 8 | Heatmap of text embedding cosine similarities among tree

search-generated methods. The similarity matrix was hierarchically clustered along rows and
columns and reordered to group similar methods together. Three distinct color bars denote major
method categories. The pairwise cosine similarities between tree search-generated solutions were
greater than 0.85. For context, the lower bound of cosine similarity, established by averaging the
similarities between GIFT-Eval’s methods (a completely different benchmark) and batch integration
methods, was 0.74.

36
 An AI system to help scientists write expert-level empirical software

Supplementary Fig. 9 | Performance of the best retrospective COVID-19 hospitalization forecast

replicates. This figure presents WIS by reference date for the single best-performing replicate of each
validation window in our retrospective COVID-19 forecasting study. The best models are selected
based on their performance on the validation dates. The plot shows how finding optimum models on
a handfull of validation dates (6 weeks) generalizes on the next two weeks of unseen reference dates.

37
 An AI system to help scientists write expert-level empirical software

Supplementary Fig. 10 | Performance of retrospective COVID-19 hospitalization forecasts

across all replicates. Each panel displays the average WIS by reference date for individual replicates
of our proposed models, for all rolling validation dates. Lower WIS values indicate superior
forecasting accuracy and calibration. The consistent trends across replicates demonstrate the
robustness and reproducibility of tree search’s ability to generate high-performing probabilistic
forecasts.

38
 An AI system to help scientists write expert-level empirical software

Node #349 Node #1908

Corrected inverse transformations and added robustness epsilon. Removed forced minimum spread between integer quantiles.
0 Node #371 Node #1875
Added lagged differences, rolling std, and model ensembling. Removed unstable XGBoost ensemble, retaining LightGBM only.

-50
Node #770
Node #317 Ensembled LightGBM, XGBoost; robust prediction clipping.
Recursive test feature generation using median predictions.
Node #717
-100 Incorporated rolling median features for outlier robustness.
Node #290
Unioned train/test horizon categorical feature categories.
Maximum Score

Node #645
-150 Node #201 Removed XGBoost ensemble, relying solely on LightGBM.
Improved group-wise imputation and rolling window logic.
Node #606
Increased XGBoost stability and median ensemble aggregation.
Node #152
-200 Expanded time-series features and robust lag imputation.
Node #543
Node #28 Incorporated XGBoost models into LightGBM ensemble.
Population-normalized, log-transformed
Node #512
-250 target and robust lagged features.
Added squared weeks since start feature.

-300

0 500 1000 1500 2000

Number of Nodes
#0 #1 #2 #3 #5 #8 #9 #11
-inf -inf -inf -inf -inf -332 -inf -inf

#34 #4 #6 #16 #21 #7 #10 #12 #13 #14 #15 #17 #28 #38 #62
-109 -inf -inf -inf -inf -inf -inf -inf -inf -inf -inf -338 -167 -64.5 -64.5

#47 #37 #27 #29 #31 #32 #33 #35 #44 #45 #59 #60 #67 #68 #71
-67.5 -inf -inf -inf -inf -inf -inf -inf -inf -192 -inf -inf -inf -inf -inf

#84 #85 #86 #88 #89 #90 #91 #94 #98

-inf -53.6 -inf -55.8 -51.9 -65.9 -63.1 -64.2 -inf

#87 #131 #92 #93 #95 #96 #110

-inf -149 -inf -52.5 -51.9 -inf -inf

#126 #134 #152 #156 #97 #135

-inf -53.2 -47.7 -47.7 -51.2 -51.2

#154 #155 #158 #159 #160 #162 #163 #164 #165 #166 #167 #172 #173 #174 #177 #185 #123 #124 #125 #127 #128 #129 #130 #133 #147
-inf -inf -47.7 -47.7 -48.1 -46.9 -47.7 -47.7 -inf -47.7 -48.4 -47.7 -47.7 -47.7 -47.7 -47.7 -inf -inf -inf -inf -50.4 -inf -50.9 -51.2 -51.6

#168 #169 #170 #171 #175 #176 #178 #181 #183 #201 #222 #226 #139 #146 #157
-inf -inf -inf -46.9 -46.9 -47.1 -58.3 -46.9 -46.9 -46.4 -46.9 -47.1 -50.7 -50.5 -inf

#182 #184 #179 #180 #213 #203 #193 #195 #197 #198 #200 #212
-47.5 -46.9 -inf -inf -46.9 -47.1 -inf -inf -47.2 -46.9 -48.4 -46.3

#192 #194 #196 #199 #227 #228 #229 #230 #235 #236 #237 #241 #255 #268 #300
-inf -inf -55.9 -inf -inf -70.6 -inf -46.3 -46.3 -46.3 -46.3 -inf -46.3 -46.3 -46.3

#257 #259 #260 #265 #293 #294 #297 #299 #256 #258 #263 #267 #272 #279 #284 #253 #262 #264 #269 #273 #275 #282 #308 #252 #254
-inf -46.3 -inf -46.3 -46.9 -48.3 -46.3 -47.2 -46.3 -46.3 -46.8 -46.3 -46.3 -46.5 -46.3 -inf -46.4 -46.3 -46.3 -46.8 -46.3 -46.1 -46.3 -inf -inf

#281 #288 #289 #292 #310 #270 #271 #276 #261 #277 #280 #285 #290 #311 #328 #266 #274 #283 #286 #296 #291 #278 #295
-46.3 -46.2 -46.3 -46.1 -46.3 -inf -inf -inf -inf -47.1 -46.3 -46.3 -45.7 -45.7 -45.7 -46.3 -46.3 -46.3 -46.3 -46.3 -inf -inf -54.9

#302 #304 #305 #306 #307 #309 #312 #313 #314 #315 #316 #317 #321 #333 #337 #339 #352 #355 #287 #319 #301
-inf -45.7 -45.7 -45.7 -inf -58.4 -45.7 -47.6 -45.7 -45.7 -47.2 -45.4 -45.9 -45.7 -inf -46.5 -45.4 -45.4 -46.3 -46.3 -50.7

#318 #325 #320 #324 #335 #336 #340 #341 #327 #323 #338 #342 #343 #344 #345 #349 #350 #351 #354 #356 #359 #360 #361 #362 #363 #364 #365
-45.7 -45.7 -45.5 -45.4 -45.6 -45.7 -45.7 -45.6 -45.6 -47.3 -45.7 -inf -inf -45.4 -45.5 -42.6 -45 -inf -45.4 -45.4 -45.5 -45.4 -46.9 -51.7 -45.4 -45.5 -45.4

#368 #369 #370 #371 #372 #373 #374 #375 #376 #377 #378 #379 #381 #388
-inf -43.3 -42.6 -42.4 -42.6 -43.7 -42.6 -43.3 -42.6 -inf -44.3 -43.2 -42.6 -42.5

#380 #382 #383 #387 #394 #395

-inf -42.4 -42.4 -42.4 -inf -42.4

#405 #406 #407 #408 #409 #413 #414 #415 #416 #418 #421 #440 #443 #444 #446 #478 #480 #488
-42.4 -42.4 -42.4 -42.4 -42.4 -42.4 -43.2 -42.4 -42.4 -42.6 -42.4 -42.4 -42.4 -42.4 -42.4 -42.4 -42.4 -42.4

#436 #438 #447 #435 #439 #465 #431 #432 #433 #468 #470 #434 #442 #445 #430 #437
-42.4 -42.9 -50.7 -42.4 -42.7 -42.4 -inf -inf -inf -43.3 -43.3 -inf -42.4 -43.5 -inf -inf

#441 #467 #469 #473 #476 #472 #474 #477 #501 #515
-42.4 -42.4 -42.4 -42.4 -42.9 -42.4 -42.9 -42.4 -43.7 -42.4

#460 #464 #479 #489 #499 #517 #525 #527 #514 #503 #504 #508 #513
-42.4 -inf -inf -42.4 -42.4 -42.4 -42.4 -42.4 -43 -43.1 -42.4 -42.4 -43.5

#471 #484 #519 #521 #516 #526 #533 #529 #537

-42.4 -42.4 -42.4 -43.7 -43.6 -42.4 -43.2 -42.4 -42.4

#483 #485 #486 #500 #498 #512 #534 #539 #561 #566 #567 #562 #552
-42.5 -42.6 -43.4 -42.9 -inf -42 -42.7 -42 -42 -42 -42 -42.4 -42.4

#542 #543 #544 #573 #592

-43.3 -41.9 -42 -41.9 -42

#569 #570 #571 #572 #574 #609 #612 #619 #624

-41.9 -41.9 -43.9 -41.9 -62.5 -41.9 -41.9 -41.9 -41.9

#597 #600 #608 #575 #595 #596 #599 #603

-inf -41.9 -41.9 -42.1 -41.9 -41.9 -41.9 -41.9

#663 #604 #606 #605 #621 #631 #641

-41.7 -41.9 -41.6 -41.9 -41.9 -41.9 -41.8

#637 #626 #627 #628 #629 #630 #633 #634 #635 #636 #638 #639 #640 #645
-42.2 -inf -inf -inf -inf -inf -41.6 -42.1 -41.6 -41.6 -41.6 -41.6 -41.6 -41.5

#648 #650 #661 #643 #644 #656 #664 #647 #649 #679 #651
-41.7 -41.6 -41.6 -41.6 -41.6 -42.2 -41.6 -41.6 -41.6 -41.6 -41.6

#682 #671 #673 #700 #672 #674 #676 #678 #709 #680 #681 #720 #739 #742 #698 #675 #677 #710 #713
-inf -41.6 -42 -42.3 -41.6 -42.2 -41.6 -41.7 -41.6 -41.6 -41.6 -41.6 -41.6 -41.6 -41.6 -41.6 -41.6 -41.6 -41.6

#702 #701 #699 #717 #718 #879 #705 #703 #712 #683 #847 #851 #852
-41.7 -41.6 -inf -41 -41.6 -41.6 -inf -41.6 -41.6 -41.6 -inf -42.2 -41.8

#727 #732 #748 #853 #859 #890 #715 #728 #716

-41.6 -41 -41 -41.6 -41.6 -41.6 -42.8 -41.6 -41.6

#855 #744 #769 #773 #856 #857 #871 #854 #785

-41.6 -41 -41 -41 -inf -inf -41.6 -42.1 -41.7

#882 #883 #895 #770 #771 #878

-41.7 -41.4 -41.6 -40.9 -41 -inf

#898 #906 #908 #916 #910 #911 #912 #917 #921 #795 #796 #798 #799 #837
-41.8 -41.4 -41.4 -41.4 -41.6 -41.7 -41.6 -42 -41.4 -inf -inf -40.9 -inf -40.9

#914 #929 #930 #923 #820 #823

-41.4 -41.4 -41.4 -41.4 -56.5 -101

#931 #933 #938 #950 #941 #944 #947 #953

-inf -41.4 -41.5 -41.4 -inf -41.4 -42.3 -41.4

#946 #949 #994 #966 #954 #973

-inf -41.4 -41.4 -41.4 -41.4 -41.4

#977 #981 #1015 #1034 #987 #979

-41.4 -41.3 -41.3 -41.3 -41.7 -41.4

#1023 #1024 #1030

-41.4 -42 -41.4

#1031 #1035 #1044 #1052 #1057 #1065

-41.4 -41.4 -41.4 -41.4 -inf -41.4

#1038 #1063 #1050 #1054 #1061 #1067 #1086 #1087

-41.4 -41.7 -41.4 -41.4 -41.4 -41.9 -41.4 -41.4

#1053 #1056 #1069 #1071 #1080 #1103 #1092 #1095

-41.4 -41.4 -41.4 -42.1 -41.4 -41.4 -41.4 -42.2

#1064 #1068 #1072 #1093 #1098 #1110 #1091 #1122 #1115

-41.4 -44.1 -179 -42.1 -41.4 -41.4 -41.4 -41.4 -41.6

#1094 #1109 #1107 #1113 #1142 #1100 #1149 #1151 #1155

-42.4 -41.4 -41.4 -41.4 -41.4 -inf -41.4 -41.4 -41.4

#1119 #1140 #1138 #1148 #1160 #1180 #1190 #1201 #1212 #1189
-41.4 -42.3 -41.4 -41.4 -41.4 -41.4 -41.4 -41.4 -41.4 -42.2

#1136 #1156 #1172 #1166 #1176 #1204 #1207 #1213 #1214 #1210 #1222 #1231 #1236 #1224
-41.4 -41.4 -41.4 -41.4 -41.4 -41.4 -41.4 -41.7 -41.4 -41.4 -41.4 -41.4 -41.4 -41.4

#1152 #1153 #1163 #1164 #1171 #1230 #1244 #1219 #1186 #1191 #1216 #1220 #1232 #1233 #1240 #1253 #1242 #1256 #1257 #1243
-41.4 -41.4 -41.4 -inf -41.4 -41.4 -41.4 -41.4 -inf -41.4 -41.4 -41.4 -41.4 -41.4 -41.6 -41.4 -41.4 -41.4 -41.4 -41.4

#1183 #1184 #1173 #1188 #1248 #1196 #1252 #1260 #1228 #1234 #1262 #1251
-41.4 -41.4 -41.6 -41.4 -41.4 -41.8 -inf -41.4 -41.4 -41.4 -41.4 -41.4

#1206 #1203 #1200 #1239 #1249 #1258

-41.4 -41.4 -41.6 -45.7 -41.4 -41.4

#1246 #1254 #1264

-41.3 -41.7 -41.4

#1259 #1261 #1263 #1269 #1272 #1274 #1276 #1278

-41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3

#1267 #1273 #1275 #1277 #1302 #1307 #1282 #1297 #1289 #1301 #1316 #1325
-41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -42.4 -42.1 -41.3 -42

#1279 #1303 #1284 #1292 #1334 #1331 #1300 #1305 #1312 #1317 #1314 #1323
-inf -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3

#1326 #1342 #1344 #1376 #1355 #1358 #1321 #1335 #1336 #1318 #1332 #1348 #1352
-44 -41.3 -42 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -inf -41.3 -41.7 -41.3

#1389 #1378 #1384 #1374 #1383 #1409 #1351 #1390 #1393 #1401 #1367
-41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -42.5 -325 -41.3 -inf

#1392 #1394 #1416 #1399 #1377 #1379 #1385 #1400 #1413 #1414
-41.3 -41.3 -42 -41.3 -41.7 -41.3 -41.3 -41.3 -86.8 -41.3

#1397 #1402 #1453 #1421 #1436 #1420 #1426 #1437 #1440 #1442 #1450
-inf -41.5 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.7 -41.5

#1471 #1478 #1482 #1454 #1467 #1448 #1452 #1457 #1444 #1460 #1466 #1475 #1480
-41.3 -41.5 -41.3 -41.3 -41.3 -41.3 -41.3 -41.9 -41.3 -41.3 -41.3 -41.3 -41.3

#1506 #1464 #1456 #1493 #1463 #1481 #1483 #1515 #1533 #1473 #1489 #1490 #1512
-41.5 -41.3 -inf -41.3 -inf -41.3 -41.3 -41.3 -41.3 -inf -41.9 -41.3 -41.3

#1525 #1523 #1502 #1526 #1530

-41.6 -41.4 -41.3 -41.3 -41.3

#1517 #1518 #1537 #1538 #1544 #1548 #1547

-41.3 -41.5 -41.3 -41.3 -41.5 -41.3 -41.3

#1559 #1585 #1550 #1571 #1573 #1575

-41.5 -41.3 -41.3 -41.3 -41.3 -42.5

#1609 #1556 #1587 #1605 #1592

-42.6 -41.3 -inf -139 -41.3

#1568 #1604
-41.3 -41.3

#1574 #1627 #1630 #1651

-41.3 -41.3 -41.3 -41.3

#1595 #1601 #1644 #1656 #1671 #1677

-41.3 -41.3 -41.3 -41.3 -41.3 -41.3

#1611 #1622 #1612 #1613 #1649 #1668 #1700 #1697 #1698

-41.3 -41.3 -inf -41.3 -inf -41.3 -41.8 -41.3 -41.3

#1652 #1655 #1639 #1643 #1629 #1688 #1689 #1696 #1732 #1754 #1708 #1726
-41.3 -41.3 -41.3 -41.3 -42.4 -41.9 -41.3 -41.3 -42.2 -42.1 -41.7 -41.3

#1682 #1693 #1647 #1664 #1707 #1721 #1758

-41.3 -41.3 -41.3 -41.7 -41.3 -41.3 -41.3

#1763 #1719 #1670 #1672 #1747 #1761 #1770 #1774

-41.3 -41.3 -41.3 -41.3 -41.3 -inf -41.3 -41.3

#1782 #1701 #1711 #1714 #1783 #1786 #1794 #1792 #1811 #1784 #1797
-41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -inf -41.3 -41.3 -41.3

#1810 #1718 #1730 #1743 #1731 #1737 #1738 #1741 #1723 #1802 #1812 #1817 #1818 #1804 #1813 #1855 #1857 #1862 #1849 #1815
-inf -41.9 -41.3 -41.5 -inf -41.3 -41.3 -41.3 -inf -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3

#1751 #1746 #1750 #1745 #1852 #1844 #1845 #1858 #1841 #1861 #1824 #1856 #1823
-41.3 -41.3 -inf -inf -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3

#1785 #1788 #1796 #1798 #1799 #1762 #1868 #1860 #1871 #1875 #1887 #1854 #1872 #1864 #1873
-41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -41.3 -40.6 -41.3 -41.3 -41.3 -41.3 -41.3

#1814 #1842 #1846 #1829 #1834 #1821 #1776 #1890 #1892 #1893 #1895 #1897 #1901 #1905 #1909 #1917 #1888 #1884
-149 -41.3 -41.3 -41.3 -41.3 -41.9 -inf -40.6 -40.6 -40.6 -40.6 -40.6 -40.6 -40.6 -40.6 -40.6 -41.3 -41.3

#1880 #1866 #1908 #1945 #1949 #1910 #1911 #1915 #1921 #1900 #1912 #1925 #1967 #1973
-41.6 -41.3 -40.5 -40.5 -40.5 -40.6 -40.6 -40.6 -41.4 -41 -40.6 -40.6 -40.6 -40.6

#1926 #1927 #1930 1937 #1939 #1940 1960 1975 #1935 #1938 #1969 #1989
-inf -101 -40.5 -40.5 -41.1 -40.7 -40.5 -40.5 -40.6 -40.6 -40.6 -inf

#1941 #1942 #1976 #1943 #1984

-40.5 -40.7 -40.5 -inf -40.6

#1956 #1957 #1968 #1971 #1974 #1997 #1998

-inf -328 -40.5 -41.1 -40.6 -40.5 -40.5

#1987 #1993 #1995

-inf -40.7 -40.9

Supplementary Fig. 11 | Top Figure Breakthrough plot for the retrospective COVID-19 prediction,
showing the evolution of the maximum score as a function of the number of nodes. The green dots
label places where the score abruptly increases due to an improvement in the code, and the label
describes the change in the code that resulted in the score increase. Bottom Figure Structure of the
tree for this same search. The color range consists of orange (lower scores) to green (higher scores)
with the highest score denoted by a diamond node.

39
 An AI system to help scientists write expert-level empirical software

Supplementary Fig. 12 | Performance of recombination experiments for COVID-19 forecasting.

This series of bar plots illustrates the average WIS achieved by various hybrid models (right bar,
labeled "Recomb") compared to their constituent baseline models (left bars, typically light blue) from
the CovidHub competition. Each subplot represents a recombination experiment, demonstrating the
success of our system in synthesizing novel forecasting strategies. Green bars indicate that the
recombination outperformed both parent models, dark blue indicates it outperformed one, and red
indicates it outperformed neither. These results emphasize the search system’s ability to combine the
strengths of existing methodologies to achieve superior predictive performance.

40
 An AI system to help scientists write expert-level empirical software

Supplementary Fig. 13 | Heatmap of conceptual similarities among COVID-19 forecasting

generated codes for methods. This figure displays the pairwise cosine similarities between text
embeddings of all forecasting models generated by tree search for the COVID-19 prediction task. Text
embeddings were produced using a Gemini model 79 . The similarity matrix was then hierarchically
clustered and reordered to group conceptually related strategies. The color-coded sidebar
categorizes each method by its origin illustrating the composition of the emergent conceptual
clusters. The No Advice methods are from the Google Retrospective study.

41
 An AI system to help scientists write expert-level empirical software

Supplementary Fig. 14 | Example output segmenting DLRSD image pixels from our method
Solution 1 (U-Net++).

42
 An AI system to help scientists write expert-level empirical software

Node #190 Node #549

Adjusted decoder dropout and learning rate factor. Upgraded U-Net encoder to ResNeXt101.

Node #233 Node #627

Reduced batch size to 16 for performance. Increased training epochs from 175 to 200
0.9
Node #284 Node #704
Reverted batch size to 16 for better generalization. Reverted `rotate_limit` to recover performance.

0.8
Node #160 Node #529
Reverted to adaptive learning rate scheduler. Upgraded U-Net encoder to se_resnext101_32x4d.
0.7 Node #130 Node #515
Implemented Test-Time Augmentation for robust inference. Added 90, 180, 270-degree Test-Time Augmentation.
Maximum Score

Node #102 Node #485

0.6 Extended training epochs, added L2 regularization, increased LR patience. Restored decoder dropout to 0.025.

Node #88 Node #472

Upgraded U-Net encoder to SE-ResNeXt50 backbone. Increased rotation limit in geometric data augmentation.
0.5 Node #77 Node #396
Increased Dice loss weight, reduced LR patience, more epochs. Reduced decoder dropout to 0.025, increased data augmentation limits.
Node #49
Node #334
0.4 Upgraded U-Net encoder from ResNet34 to EfficientNet-B4. Adjusted LR scheduler factor to 0.7.
Node #28
Node #298
Dynamic learning rate adjustment using scheduler.
Added HueSaturationValue data augmentation.
0.3 Node #15
Weighted Dice and Cross-Entropy segmentation loss.

0 200 400 600 800 1000

Number of Nodes
#0
nan

#1 #2 #3 #4 #5 #6 #7 #8 #9 #10
-inf 0.226 -inf 0.421 0.421 -inf -inf 0.0028 0.485 -inf

#15 #14 #18 #23 #21 #12 #11 #16 #17 #19 #20 #22 #24 #25 #26 #13
0.632 -inf 0.661 -inf 0.641 -inf -inf 0.0106 0.599 -inf 0.62 0.645 0.559 0.678 0.663 0.0067

#28 #33 #29 #30 #32 #36 #37 #34 #35 #38 #27 #31 #39 #40 #42 #43 #45 #46 #41 #44
0.704 0.709 -inf -inf -inf -inf -inf -inf 0.406 -inf -inf -inf -inf 0.686 -inf -inf 0.714 0.667 0.655 0.651

#47 #48 #49 #50 #51 #52 #54 #53 #55 #56 #57 #59 #61
0.735 0.697 0.756 0.715 0.69 -inf 0.703 0.693 -inf -inf 0.716 0.713 0.686

#62 #66 #72 #58 #64 #65 #67 #70 #63 #60 #68 #69 #71
-inf 0.68 -inf -inf 0.755 0.742 0.682 0.752 0.738 0.732 0.532 0.715 0.726

#77 #79 #81 #84 #76 #82 #78 #80 #83 #74 #73 #75
0.769 0.771 0.76 0.763 0.752 0.763 0.755 0.76 -inf 0.76 0.695 -inf

#89 #91 #93 #96 #97 #98 #88 #95 #99 #90 #92 #85 #86 #87
0.774 -inf 0.763 0.762 -inf 0.773 0.778 -inf 0.748 0.771 0.779 0.75 0.763 0.765

#106 #108 #109 #101 #104 #103 #100 #102 #107 #94 #105
0.781 0.782 0.775 0.779 0.782 0.771 -inf 0.793 0.791 0.756 0.768

#116 #117 #122 #118 #110 #111 #114 #115 #112 #113 #119 #121 #124
0.789 -inf 0.785 0.774 0.789 0.764 0.775 0.782 0.788 0.785 0.786 0.792 0.781

#131 #137 #135 #123 #127 #126 #120 #128 #125 #130 #133 #134 #136 #138 #164
0.788 0.774 0.791 0.779 0.78 0.782 0.778 0.78 0.79 0.794 0.785 0.784 -inf 0.783 0.787

#154 #152 #153 #155 #156 #139 #129 #132 #141 #142 #143 #145 #146 #147 #151 #157 #148
-inf 0.784 -inf 0.788 -inf 0.786 0.787 0.781 0.789 0.776 0.79 -inf 0.782 0.764 0.782 0.783 0.789

#150 #140 #144 #149 #162 #163 #178

0.779 0.777 -inf 0.792 0.79 0.787 0.767

#158 #159 #160 #161 #177 #179

-inf 0.787 0.797 0.771 0.793 0.794

#165 #166 #167 #168 #169 #170 #171 #197 #198

0.778 -inf 0.795 0.762 -inf -inf 0.795 0.787 0.787

#172 #173 #174 #175 #176 #180 #181 #182

-inf 0.8 -inf -inf 0.797 -inf 0.796 0.795

#183 #184 #185 #233 #245 #186 #187 #188 #189 #191 #190 #192
-inf 0.787 0.79 0.803 0.796 -inf 0.801 0.796 0.788 -inf 0.801 0.791

#248 #249 #253 #256 #260 #193 #194 #196 #195 #199 #200 #201 #202
0.802 0.795 0.797 -inf 0.792 -inf 0.8 0.792 0.799 0.799 0.801 0.798 0.8

#262 #263 #265 #267 #203 #205 #229 #236 #204 #216 #207 #214 #223 #206 #209 #213 #217 #247 #208 #220 #210 #212 #215 #221
0.797 0.793 0.799 0.8 0.797 0.793 0.799 0.799 0.795 0.798 0.762 0.797 0.796 0.799 0.796 -inf 0.797 0.8 -inf 0.795 -inf 0.799 -inf 0.796

#277 #279 #211 #240 #241 #242 #246 #252 #226 #231 #227 #218 #219 #222 #228 #261 #266 #224 #225 #237 #232
0.801 0.801 0.775 -inf 0.798 -inf -inf 0.8 0.798 0.791 -inf 0.798 0.796 -inf 0.796 0.792 -inf 0.789 0.797 0.792 -inf

#287 #290 #305 #292 #293 #259 #264 #276 #238 #243 #234 #235 #230 #244 #239
-inf 0.794 -inf -inf 0.802 0.802 0.801 0.79 -inf 0.795 0.793 0.802 -inf -inf 0.8

#309 #310 #312 #269 #271 #273 #274 #281 #278 #280 #282 #250 #251 #254 #258 #255 #257
0.795 0.802 0.789 0.798 -inf -inf 0.794 0.795 0.796 0.795 0.8 -inf -inf 0.799 -inf 0.797 0.801

#320 #321 #323 #328 #296 #268 #270 #272 #275

0.804 0.792 0.797 0.796 0.799 0.797 0.8 0.801 0.803

#329 #330 #331 #332 #333 #334 #335 #337 #340 #344 #283 #284 #285 #295 #286 #288 #289 #291 #294
-inf 0.794 0.793 0.798 0.801 0.804 0.802 0.801 0.8 -inf -inf 0.804 0.798 0.801 0.8 -inf 0.8 0.801 0.797

#345 #346 #348 #349 #350 #354 #360 #362 #368 #370 #372 #376 #377 #378 #383 #384 #387 #388 #389 #401 #347 #351 #363 #297 #298 #299 #300 #302 #304 #307 #311 #319 #308 #301 #303 #306
0.802 0.8 0.787 0.783 0.793 0.798 0.8 0.796 0.797 0.798 0.797 -inf 0.798 -inf 0.799 0.794 0.787 0.8 0.795 0.803 0.804 0.799 0.8 0.788 0.804 -inf 0.799 0.796 -inf 0.797 0.79 0.795 0.799 -inf -inf 0.801

#355 #356 #364 #413 #416 #421 #438 #459 #357 #358 #359 #361 #369 #385 #313 #314 #315 #317 #318 #322 #324 #325 #326 #336 #341 #342 #316
0.803 0.797 0.8 0.8 0.798 0.795 -inf 0.802 0.799 0.789 0.8 0.802 0.797 0.8 0.797 0.8 0.796 0.801 0.8 0.796 0.797 0.796 0.795 0.781 0.802 0.8 -inf

#365 #366 #367 #371 #379 #471 #473 #373 #374 #375 #386 #339 #327 #338 #343 #352 #353 #382
0.801 0.796 -inf -inf 0.803 -inf 0.801 -inf 0.8 0.799 0.8 -inf 0.796 -inf 0.792 0.8 0.791 0.799

#380 #381 #390 #392 #394 #396 #397 #399 #403 #402
-inf 0.802 0.796 -inf 0.803 0.805 0.797 0.804 0.8 -inf

#391 #393 #395 #398 #400 #404 #405 #407 #408 #409 #410 #419 #427 #429 #434 #435 #437 #440 #445 #446 #460 #462 #463 #411 #414 #417 #420
-inf 0.802 0.798 0.799 0.802 0.803 0.796 0.779 0.8 -inf 0.8 0.794 0.8 0.801 -inf 0.799 0.802 0.802 0.798 0.794 -inf 0.798 0.802 0.805 0.795 -inf 0.801

#406 #412 #433 #415 #418 #430 #455 #449 #452 #450 #451 #457 #475 #422 #423 #424 #425 #426 #428 #431 #436 #439 #441 #461 #432
0.794 0.801 0.803 0.8 0.799 0.795 0.8 0.796 -inf 0.799 0.799 0.798 0.803 -inf 0.799 0.798 -inf 0.802 0.801 0.789 -inf -inf 0.796 0.799 -inf

#447 #448 #453 #484 #487 #442 #443 #444

0.793 0.801 0.805 0.793 0.796 0.799 0.802 0.798

#458 #464 #465 #466 #467 #470 #472 #474 #478 #479 #480 #481 #454 #456
0.8 -inf 0.801 0.802 0.801 0.8 0.805 0.801 0.801 0.801 -inf 0.801 0.797 0.803

#477 #482 #483 #485 #486 #493 #494 #468 #469 #476
0.798 0.802 0.802 0.806 0.799 0.801 0.799 -inf 0.8 0.804

#495 #491 #502 #504 #506 #512 #517 #488 #489 #490 #492
0.798 0.777 0.798 0.798 0.802 0.803 0.8 0.805 0.802 0.797 0.805

#515 #523 #548 #496 #497 #499 #500 #501 #498 #503 #505 #510 #518
0.807 -inf -inf 0.805 -inf 0.803 0.8 0.802 0.798 0.804 0.799 0.801 0.795

#526 #528 #531 #534 #537 #540 #554 #507 #508 #511 #516 #509 #513 #514
0.801 0.793 0.8 0.802 0.803 0.801 0.798 0.804 0.796 0.804 -inf 0.802 0.806 0.805

#550 #519 #520 #524 #521 #522 #525 #529 #532 #536 #539 #527 #530 #533 #538 #551
0.8 0.8 0.803 0.799 0.797 0.797 0.797 0.807 0.802 0.794 0.802 -inf -inf 0.8 0.798 0.802

#535 #541 #542 #543 #544 #545 #546 #549 #552 #547 #553
0.796 -inf 0.805 0.803 -inf -inf 0.806 0.811 0.808 0.805 0.775

#555 #556 #558 #557 #561 #564 #569 #574 #562 #563 #567 #570 #572 #565 #566 #568 #571 #573 #559 #560
-inf -inf 0.798 0.801 0.802 0.804 0.806 0.806 0.806 0.81 -inf 0.809 0.788 0.804 0.803 -inf -inf 0.803 0.799 0.8

#581 #584 #585 #590 #594 #597 #599 #577 #578 #586 #579 #580 #582 #591 #593 #587 #588 #589 #592 #575 #576 #583
0.804 0.802 0.807 -inf 0.806 0.802 0.804 0.807 -inf -inf 0.805 -inf 0.791 0.804 0.798 -inf -inf -inf -inf 0.799 -inf 0.789

#601 #604 #605 #606 #608 #609 #610 #611 #595 #596 #598 #603 #600 #602 #607
-inf 0.806 -inf -inf 0.809 0.811 -inf 0.805 -inf -inf 0.802 0.801 0.807 0.809 0.806

#620 #621 #623 #628 #637 #624 #625 #630 #642 #612 #613 #618 #614 #615 #616 #617 #619 #622
0.808 0.805 0.803 0.803 -inf 0.807 0.806 0.805 0.804 0.805 0.809 -inf 0.798 -inf 0.804 -inf -inf 0.805

#631 #632 #639 #640 #634 #635 #638 #643 #629 #626 #627 #636 #644 #633
0.806 0.804 0.804 -inf 0.807 -inf 0.809 -inf 0.805 -inf 0.811 -inf 0.804 0.806

#648 #658 #651 #652 #653 #654 #655 #656 #641 #645 #646 #647 #649 #657 #650
0.808 0.807 0.803 -inf 0.808 0.806 -inf 0.809 0.803 -inf -inf 0.808 0.81 0.808 0.805

#659 #664 #669 #665 #666 #667 #668 #673 #678 #660 #663 #661 #662 #677 #686 #671 #674
0.807 0.803 0.807 -inf 0.806 0.807 0.806 0.808 0.805 0.807 0.809 -inf 0.805 0.806 0.8 0.804 -inf

#670 #687 #684 #685 #679 #680 #682 #688 #689 #672 #683 #675 #676 #681 #693
0.805 0.805 -inf 0.808 -inf 0.807 -inf 0.808 0.799 -inf 0.808 -inf 0.809 0.809 0.805

#694 #703 #691 #705 #700 #701 #717 #696 #699 #690 #692 #695 #704 #697 #698 #702 #711
-inf 0.803 0.805 0.808 0.804 0.803 0.81 0.805 0.801 -inf 0.8 0.81 0.812 0.804 0.808 0.805 0.808

#714 #716 #731 #732 #706 #707 #708 #709 #710 #718 #719 #722 #723 #724 #729 #712 #713 #725 #726 #730
0.807 -inf 0.808 0.806 0.807 0.808 0.807 -inf -inf 0.81 0.811 -inf -inf 0.806 -inf 0.808 0.811 -inf 0.807 0.803

#727 #741 #763 #715 #721 #728 #720 #738 #742 #744 #739 #740 #743 #733 #734 #735 #737 #745
-inf -inf -inf 0.805 -inf -inf 0.808 0.805 0.812 0.802 0.808 0.81 0.808 0.809 0.81 -inf 0.809 0.807

#736 #754 #755 #757 #759 #762 #768 #752 #769 #751 #753 #760 #767 #756 #747 #750 #764 #746 #748 #761 #749 #758
-inf -inf -inf -inf -inf -inf 0.806 0.808 0.81 0.807 0.808 -inf -inf 0.809 0.808 0.807 0.813 -inf 0.806 0.804 0.805 -inf

#772 #774 #780 #783 #791 771 #775 #776 #765 #766 #773 #777 #778 #785 #790 #793
-inf 0.806 0.807 0.807 0.806 0.815 0.803 0.808 0.809 -inf 0.81 0.808 0.809 0.811 0.811 -inf

#781 #782 #784 #788 #795 #789 #779 #786 #787 #794 #797 #800 #801 #803 #805 #809 #810 #813 #820 #822
-inf 0.809 0.808 0.811 0.809 -inf 0.807 0.809 0.81 0.808 -inf 0.807 0.811 -inf 0.81 -inf 0.807 0.812 0.81 0.809

#792 #799 #806 #807 #811 #819 #808 #796 #798 #802 #804 #818 #815 #816 #817 #826 #831 #835 #837 #839 #846 #836 #845 #843
-inf -inf 0.808 0.813 0.809 0.805 0.804 0.81 0.807 -inf 0.804 -inf -inf -inf 0.81 0.808 0.808 0.809 0.806 0.806 0.809 0.811 0.811 -inf

#821 #823 #824 #825 #830 #832 #827 #812 #814 #828 #829 #849 #847 #857 #851 #854 #856 #858 #859 #862
-inf -inf 0.811 0.807 -inf 0.81 0.809 -inf -inf 0.81 -inf -inf 0.808 0.81 0.81 0.809 0.809 0.807 0.81 0.809

#833 #834 #840 #848 #838 #841 #842 #850 #870 #871 #865 #866 #867 #868 #875 #876 #878 #877
0.809 0.805 0.811 -inf 0.806 -inf 0.806 0.81 0.809 0.81 -inf 0.812 0.809 -inf -inf 0.806 0.81 -inf

#844 #852 #853 #855 #860 #863 #889 #900 #880 #881 #882 #883 #884 #987 #885 #894
-inf 0.81 0.809 0.808 0.807 0.81 0.811 0.812 0.808 -inf -inf -inf 0.809 -inf 0.811 0.81

#861 #869 #864 #874 #879 #904 #905 #907 #913 #914 #916 #924 #895 #897 #912
-inf 0.805 0.81 0.806 0.811 0.81 0.81 0.809 0.811 0.809 0.812 0.813 -inf -inf 0.805

#872 #873 #890 #891 #919 #927 #921 #925 #926 #932 #933 #945 #938 #940 #942 #946 #955
0.811 0.803 0.812 0.809 -inf 0.805 0.809 -inf -inf 0.812 0.809 0.808 0.812 -inf -inf 0.811 0.81

#886 #887 #888 #902 #903 #906 #909 #947 #948 #950 #954 #957 #958 #959 #962 #964 #960 #961 #969
0.813 0.81 0.81 0.81 0.807 0.809 0.811 0.807 -inf 0.809 0.812 0.81 0.808 0.808 -inf 0.809 0.808 0.812 -inf

#892 #893 #896 #898 #965 #899 #901 #922 #917 #918 #967 #968 #970 #971 #973 #972 #978 #979 #980 #981
0.808 0.808 0.81 0.809 0.811 -inf 0.813 0.81 0.813 0.808 0.81 0.809 -inf 0.81 0.814 0.811 -inf 0.813 0.811 0.809

#908 #974 #975 #976 #910 #911 #915 #920 #923 #952 #929 #930 #936 #982 #992 #996 #997 #998 #1007 #984 #985 #986 #999 #1000 #1005 #1003
-inf 0.813 -inf 0.812 -inf 0.807 0.809 0.811 0.812 -inf 0.81 0.812 0.808 0.815 0.808 -inf -inf 0.812 0.811 -inf -inf -inf 0.809 0.812 -inf 0.81

#989 #990 #991 #993 #994 #995 #928 #931 #956 #934 #935 #937 #953 #966 #983 #944 #939 #943 #949 #951 #963 #977 #988 #1001 #1002 #1004
0.809 -inf -inf 0.81 -inf 0.809 0.807 0.807 0.807 0.803 0.806 0.808 0.806 0.804 0.807 -inf -inf 0.807 -inf -inf -inf -inf -inf 0.811 0.813 -inf

#1008
0.806

Supplementary Fig. 15 | Top Figure Breakthrough plot for the U-Net Geospatial DLRSD solution
(solution 3), showing the evolution of the maximum score as a function of the number of nodes. The
green dots label places where the score abruptly increases due to an improvement in the code, and
the label describes the change in the code that resulted in the score increase. Bottom Figure
Structure of the tree for this same search. The color range consists of orange (lower scores) to green
(higher scores) with the highest score denoted by a diamond node.

43
 An AI system to help scientists write expert-level empirical software

-0.0260

Node #390
-0.0265 Increased feature embedding and global state dimensions.
Maximum Score

Node #242
Added explicit delta features to feature extractor.

-0.0270
Node #168
Consolidated temporal 1D CNN within feature extractor.

-0.0275

-0.0280

0 200 400 600 800 1000

Number of Nodes

Supplementary Fig. 16 | Top Figure Breakthrough plot for the ZAPBench tree search, showing the
evolution of the maximum score as a function of the number of nodes. The green dots label places
where the score abruptly increases due to an improvement in the code, and the label describes the
change in the code that resulted in the score increase. Bottom Figure Structure of the tree for this
same search. The color range consists of orange (lower scores) to green (higher scores) with the
highest score denoted by a diamond node.

44
 An AI system to help scientists write expert-level empirical software

Supplementary Fig. 17 | Categories of solutions on the GIFT-Eval benchmark on the per-dataset

solution (v1). We prompted an LLM (Gemini 2.5 Pro) to categorize the code from each of the
solutions into a class of methods. The figure shows the percentage of the best codes for each of the
92 competitions in the specified categories: Gradient Boosted Method (GBM); Ensemble; Seasonal;
Error, Trend and Seasonality (ETS); Arima 58 ; Linear; Averaging; Decomposition and Unobserved
components model (UCM).

45
 An AI system to help scientists write expert-level empirical software

-0.60 Node #482

Damped linear trend projection prevents overshooting.

-0.65 Node #14

Fixed trend extrapolation and residual correction damping bugs.
Node #812
Dynamically inferred primary seasonality for `season_length=1`.
-0.70
Node #279
Maximum Score

Combined Day-Hour seasonality to capture interactions Node #613

Stabilized trend, optimized residual correction damping.

-0.75 Node #169

Residual correction component uses exponential decay.

Node #149
Residual correction decay changed to exponential.
-0.80
Node #81
Applied residual correction damping to forecast component.

-0.85
Node #4
Reduced redundant configs for enhanced generalization and efficiency.

-0.90
0 200 400 600 800 1000
Number of Nodes
#0
nan

#1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20
-0.906 -0.873 -0.899 -0.794 -0.858 -0.73 -0.853 -0.851 -0.899 -0.835 -0.887 -0.912 -0.835 -0.69 -0.884 -0.754 -0.876 -0.892 -0.85 -0.734

#22 #23 #32 #33 #34 #35 #38 #21 #30 #27 #40 #41 #42 #43 #44 #45 #46 #47 #49 #51 #52 #56 #58 #59 #60 #24 #25 #26 #28 #29 #31 #36 #37 #39
-0.831 -0.806 -0.814 -0.736 -0.905 -0.73 -0.731 -1.06 -0.862 -0.751 -0.69 -0.77 -0.76 -0.838 -0.723 -inf -0.765 -0.834 -0.939 -0.707 -0.69 -0.702 -0.69 -0.705 -0.713 -0.863 -0.882 -0.996 -0.887 -0.964 -1.07 -0.858 -0.92 -0.895

#50 #53 #54 #55 #57 #65 #48 #70 #71 #72 #83 #87 #90 #97 #98 #61 #62 #63 #64 #66 #67 #68 #69 #75 #76 #79 #81 #86 #91 #96 #101 #78 #84 #73 #74 #77 #82 #85 #89 #80 #88
-0.734 -0.782 -0.73 -0.731 -0.75 -0.731 -0.751 -0.767 -0.739 -0.739 -0.772 -0.885 -0.703 -0.712 -0.699 -0.724 -0.726 -0.883 -0.724 -0.721 -0.771 -0.827 -0.744 -0.729 -0.741 -0.78 -0.688 -0.705 -0.69 -0.757 -0.692 -0.756 -0.746 -0.69 -0.69 -0.69 -0.69 -0.699 -0.69 -0.91 -0.867

#107 #108 #109 #111 #112 #114 #117 #118 #119 #120 #115 #116 #102 #103 #104 #105 #106 #92 #95 #93 #94 #99 #100 #110 #113
-0.688 -0.696 -0.74 -0.767 -0.725 -0.722 -0.688 -0.773 -0.727 -0.696 -0.703 -0.912 -0.69 -0.69 -0.69 -0.69 -0.69 -0.756 -0.719 -10.3 -0.695 -0.69 -0.752 -0.76 -0.69

#121 #122 #123 #124 #125 #126 #128 #129 #131 #132 #135 #136 #137 #140 #141 #142 #143 #144 #145 #146 #147 #148 #149 #151 #152 #153 #154 #155 #127 #130 #133 #139 #134 #138
-0.747 -0.878 -0.738 -0.688 -0.713 -0.738 -0.695 -0.715 -0.724 -0.855 -0.912 -0.847 -0.717 -0.688 -0.711 -0.694 -0.735 -0.758 -0.779 -0.73 -0.79 -0.95 -0.688 -0.739 -0.821 -0.688 -0.749 -0.907 -0.997 -0.726 -0.9 -0.69 -0.69 -0.69

#156 #157 #158 #159 #160 #162 #163 #170 #174 #164 #165 #166 #167 #168 #169 #171 #172 #173 #175 #176 #180 #184 #185 #186 #187 #188 #189 #177 #178 #179 #183 #150 #161
-0.868 -0.843 -0.833 -0.861 -0.722 -0.723 -0.691 -0.742 -0.688 -0.747 -0.738 -0.812 -0.849 -0.688 -0.685 -0.767 -0.853 -0.712 -0.786 -0.828 -0.688 -0.718 -0.734 -0.718 -0.694 -0.822 -0.768 -0.813 -0.799 -0.827 -0.697 -0.69 -0.689

#192 #202 #193 #194 #196 #200 #204 #207 #190 #191 #195 #197 #199 #201 #203 #205 #206 #214 #220 #208 #209 #210 #211 #213 #181 #182
-0.704 -0.773 -0.688 -0.815 -0.689 -0.688 -0.728 -0.691 -0.816 -0.786 -0.81 -0.685 -0.685 -0.831 -0.708 -0.748 -0.736 -0.717 -0.793 -0.922 -0.742 -0.859 -0.726 -0.714 -0.689 -0.689

#215 #216 #217 #218 #219 #230 #231 #235 #244 #221 #223 #226 #228 #229 #233 #237 #239 #241 #243 #246 #247 #222 #224 #225 #227 #232 #234 #236 #238 #240 #242 #245 #248 #249 #212 #198
-0.728 -0.828 -0.953 -0.849 -0.785 -0.767 -0.741 -0.759 -0.688 -0.734 -0.844 -0.685 -0.708 -0.844 -0.75 -0.822 -0.705 -0.827 -0.708 -0.688 -0.763 -0.783 -0.793 -0.733 -0.804 -0.685 -0.762 -0.724 -0.803 -0.751 -0.704 -0.69 -0.701 -0.766 -0.689 -0.694

#268 #269 #270 #282 #250 #251 #252 #253 #259 #260 #263 #266 #267 #272 #274 #276 #279 #281 #284 #286 #287 #289 #262 #254 #255 #256 #257 #258 #261 #264 #265 #271 #273 #275 #277 #278 #280 #283 #285 #288
-0.688 -0.688 -0.7 -0.695 -0.756 -0.708 -0.741 -0.737 -1.04 -0.791 -0.695 -0.789 -0.72 -0.684 -0.773 -0.706 -0.684 -0.685 -0.684 -0.738 -0.784 -0.781 -0.726 -0.937 -0.981 -0.846 -0.732 -0.775 -0.761 -0.944 -0.773 -0.77 -0.771 -0.751 -0.903 -0.887 -0.786 -0.703 -0.819 -0.859

#295 #299 #292 #300 #290 #291 #293 #294 #296 #297 #298 #301 #302 #303 #304 #305 #308 #309 #311 #314 #316 #318 #326 #334 #337 #338 #340 #344 #345 #347 #351 #307 #313 #329 #393 #306 #310 #312 #315 #317 #327 #335 #339 #343 #346 #350 #373 #384 #387 #392
-0.688 -0.689 -0.878 -0.687 -0.817 -0.712 -0.684 -0.699 -0.713 -0.72 -0.987 -0.783 -0.684 -0.774 -0.764 -0.762 -0.761 -inf -0.718 -0.745 -0.775 -0.702 -0.684 -0.715 -0.754 -0.719 -0.879 -0.73 -0.724 -0.71 -0.736 -0.758 -0.808 -0.77 -0.803 -0.707 -0.843 -0.746 -0.736 -0.817 -0.804 -0.736 -0.775 -0.828 -0.796 -0.807 -0.713 -0.77 -0.786 -0.745

#324 #319 #320 #321 #322 #323 #325 #328 #342 #348 #330 #331 #332 #333 #336 #341 #349 #352 #353 #354 #355 #357 #358 #359 #360 #362 #363 #364 #365 #366 #367 #368 #369 #370 #371 #372 #381 #382 #383 #385 #386 #388 #390 #391
-0.738 -0.695 -0.862 -0.735 -0.79 -0.747 -0.767 -0.771 -0.709 -0.684 -0.874 -0.702 -0.788 -0.774 -0.686 -0.712 -0.749 -0.69 -0.98 -0.745 -0.746 -0.705 -0.751 -0.707 -0.715 -0.72 -0.69 -0.69 -0.736 -0.715 -0.773 -0.785 -0.684 -0.813 -0.732 -0.691 -0.684 -0.759 -0.706 -0.738 -0.712 -0.714 -0.684 -0.725

#374 #375 #376 #377 #378 #379 #380 #389 #356 #361 #394 #395 #399 #402 #404 #407 #409 #412 #396 #398 #400 #403 #405 #408 #410 #411 #413 #414 #415 #416 #417 #418 #419
-0.717 -0.718 -0.691 -inf -0.684 -0.751 -0.773 -0.743 -0.689 -0.714 -0.684 -0.71 -0.717 -0.788 -0.713 -0.687 -0.731 -0.726 -0.718 -0.782 -0.75 -0.684 -0.726 -0.684 -0.741 -0.685 -0.684 -0.754 -0.736 -0.793 -0.684 -0.859 -0.7

#397 #401 #406 #420 #421 #422 #423 #430 #431 #435 #428 #424 #425 #426 #427 #429 #433 #434 #441 #442 #447 #448 #453 #468 #473 #476 #490 #432 #438 #439 #443 #450 #436 #437 #440 #451 #465 #470 #475 #488 #491
-0.818 -0.727 -0.716 -0.684 -0.854 -0.74 -0.687 -0.706 -0.871 -0.706 -0.736 -0.714 -0.836 -0.739 -0.783 -0.701 -0.758 -0.684 -0.788 -0.784 -0.852 -0.727 -0.78 -0.788 -0.757 -0.706 -0.829 -0.708 -0.764 -0.747 -0.684 -0.7 -0.711 -0.701 -0.727 -0.857 -0.755 -0.759 -0.826 -0.753 -0.841

#444 #445 #446 #449 #463 #466 #471 #478 #452 #454 #455 #458 #461 #464 #469 #472 #479 #456 #457 #459 #460 #462 #467 #474 #477
-0.784 -0.824 -1.04 -0.736 -0.872 -0.684 -0.729 -0.684 -0.758 -0.717 -0.685 -0.684 -0.754 -0.778 -0.742 -0.737 -0.713 -0.726 -0.694 -0.732 -0.739 -0.709 -0.706 -0.707 -0.796

#492 #493 #494 #495 #498 #499 #500 #502 #503 #497 #501 #480 #481 #482 #483 #484 #485 #486 #487 #489 #504
-1.05 -0.801 -0.756 -0.684 -0.842 -0.784 -0.693 -0.862 -1.75 -0.747 -0.807 -0.686 -0.783 -0.646 -1.04 -0.873 -0.697 -0.754 -0.81 -0.684 -0.795

#525 #526 #527 #530 #496 #506 #507 #508 #510 #511 #512 #514 #515 #516 #517 #518 #520 #528 #529 #531 #532 #533 #534 #535 #536 #505 #509 #513 #519
-0.684 -0.856 -0.88 -0.684 -0.738 -0.715 -0.696 -0.815 -0.672 -0.779 -0.723 -0.652 -0.72 -0.664 -0.691 -0.768 -0.715 -0.646 -0.657 -0.646 -0.782 -0.858 -0.692 -0.914 -0.787 -0.684 -0.685 -0.711 -0.704

#540 #547 #544 #545 #548 #537 #539 #542 #549 #538 #541 #543 #565 #550 #552 #556 #559 #563 #566 #551 #555 #558 #564 #553 #554 #557 #560 #561 #521 #522 #523 #524
-0.684 -0.684 -0.817 -0.683 -0.985 -0.716 -0.696 -0.856 -0.652 -0.896 -0.705 -0.786 -0.738 -0.668 -0.688 -0.646 -0.658 -0.74 -0.646 -0.657 -0.702 -0.747 -0.736 -0.688 -0.713 -0.668 -0.726 -0.703 -0.684 -0.742 -0.771 -0.898

#562 #570 #572 #573 #575 #577 #588 #567 #571 #581 #582 #583 #584 #586 #594 #578 #579 #587 #589 #590 #591 #592 #593 #568 #569 #585 #574 #576 #546
-0.684 -0.879 -0.755 -0.672 -0.684 -0.652 -0.761 -0.694 -0.72 -0.72 -0.646 -0.646 -0.681 -0.646 -0.649 -0.668 -0.668 -0.668 -0.79 -0.649 -0.721 -0.668 -0.663 -0.704 -0.657 -0.715 -1.21 -0.669 -0.72

#580 #600 #601 #602 #604 #605 #613 #621 #624 #607 #611 #619 #603 #606 #614 #620 #626 #610 #615 #599 #598 #622 #608 #618 #634 #623 #609 #595 #597 #596
-0.73 -0.824 -0.771 -0.652 -0.689 -0.739 -0.645 -0.646 -0.679 -0.648 -0.699 -0.68 -0.678 -0.675 -0.66 -0.693 -0.686 -0.663 -0.729 -0.76 -0.658 -0.658 -0.798 -0.725 -0.751 -0.755 -0.663 -0.664 -0.657 -0.782

#627 #629 #635 #636 #638 #640 #641 #643 #644 #825 #832 #833 #834 #646 #647 #648 #650 #652 #653 #631 #632 #633 #639 #628 #612 #649 #630 #616 #617 #625
-0.693 -0.753 -0.646 -0.666 -0.837 -0.83 -0.693 -0.674 -0.689 -0.761 -0.65 -0.77 -0.733 -0.655 -0.667 -0.646 -0.646 -0.646 -0.67 -0.648 -0.769 -0.648 -0.741 -0.678 -0.681 -0.757 -0.663 -0.687 -0.657 -0.657

#655 #656 #658 #662 #663 #665 #667 #668 #669 #670 #671 #673 #674 #675 #679 #680 #688 #676 #677 #678 #681 #682 #654 #661 #657 #659 #664 #651 #637 #645 #642
-0.657 -0.851 -0.721 -0.677 -0.646 -0.695 -0.661 -0.659 -0.649 -0.646 -0.713 -0.67 -0.671 -0.646 -0.653 -0.758 -0.686 -0.646 -0.659 -0.677 -0.677 -0.646 -0.703 -0.759 -0.649 -0.795 -0.796 -0.655 -0.674 -0.657 -0.657

#684 #685 #687 #691 #692 #693 #694 #695 #699 #701 #703 #711 #705 #700 #702 #704 #722 #729 #696 #697 #698 #683 #690 #672 #666 #660
-0.727 -0.853 -0.649 -0.649 -0.646 -0.654 -0.646 -0.667 -0.791 -0.668 -0.647 -0.695 -0.675 -0.67 -0.691 -0.764 -0.678 -0.667 -0.651 -0.646 -0.646 -0.655 -0.66 -0.655 -0.657 -0.659

#707 #716 #712 #715 #717 #726 #708 #709 #710 #727 #721 #720 #723 #724 #725 #728 #713 #714 #718 #719 #730 #706 #689 #686
-0.657 -0.649 -0.646 -0.671 -0.667 -0.662 -0.779 -0.663 -0.646 -0.742 -0.804 -0.675 -0.661 -0.651 -0.661 -0.646 -0.671 -0.646 -0.646 -0.646 -0.646 -0.767 -0.668 -0.669

#732 #731 #733 #734 #737 #745 #748 #752 #758 #738 #739 #740 #756 #742 #743 #744 #754 #735 #736 #741 #746 #747 #749 #757 #760
-0.679 -0.662 -0.698 -0.664 -0.671 -0.646 -0.759 -0.661 -0.682 -0.651 -0.672 -0.646 -0.716 -0.668 -0.646 -0.667 -0.646 -0.646 -0.65 -0.664 -0.648 -inf -0.669 -0.646 -0.682

#769 #770 #761 #764 #767 #762 #763 #766 #771 #773 #750 #751 #753 #755 #759 #765 #777 #782
-0.646 -0.651 -0.647 -0.646 -0.646 -0.646 -0.646 -0.646 -0.646 -0.646 -0.646 -0.646 -0.646 -0.666 -0.65 -0.657 -0.646 -0.651

#783 #790 #803 #778 #779 #791 #793 #786 #789 #804 #787 #788 #807 #780 #781 #794 #796 #795 #799 #811 #768 #772 #776 #774 #775 #798 #801 #814
-0.646 -0.646 -0.673 -0.671 -0.651 -0.659 -0.646 -0.731 -0.658 -0.646 -0.664 -0.659 -0.646 -0.646 -0.646 -0.646 -0.656 -0.683 -0.646 -0.672 -0.646 -0.646 -0.77 -0.65 -0.694 -0.692 -0.646 -0.731

#805 #806 #810 #812 #813 #815 #816 #826 #828 #827 #829 #802 #797 #800 #817 #818 #823 #824 #784 #785 #809 #792 #819 #821
-inf -0.675 -0.646 -0.645 -0.682 -0.65 -0.732 -0.646 -0.66 -0.646 -0.676 -0.646 -0.646 -0.654 -0.683 -0.683 -0.646 -0.718 -0.652 -0.667 -0.667 -0.646 -0.676 -0.68

#830 #831 #835 #836 #837 #838 #839 #840 #842 #843 #845 #848 851 #841 #844 #849 #850 #820 #822 #846 #847 #808
-0.666 -0.673 -0.645 -0.645 -0.664 -0.703 -0.665 -0.709 -0.645 -0.688 -0.666 -0.829 -0.639 -0.646 -0.648 -0.657 -0.673 -0.661 -0.671 -0.651 -0.646 -0.713

#858 #859 #861 #862 #864 #865 #852 #853 #854 #855 #856 #857 #872 #874 #866 #867 #868 #869 #870 #871 #873 #875 #876 #877 #878 #879 #883 885 #860 #863
-0.645 -0.756 -0.876 -0.673 -0.646 -0.66 -0.681 -0.691 -0.651 -0.67 -0.672 -0.668 -0.674 -0.645 -0.667 -0.658 -0.656 -0.645 -0.689 -0.682 -0.669 -0.647 -0.653 -0.644 -0.641 -0.643 -0.644 -0.639 -0.672 -0.67

#880 #881 #882 #884 #891 #892 #900 #886 #887 #888 #889 #890 #893 #894 #898 #912 #895 #896 #897 #899 #904 #905 #908 #911 #901 #906 #909 #902 #903 #907 #910 #916
-0.651 -0.684 -0.679 -0.667 -0.653 -0.685 -0.657 -0.656 -0.663 -0.658 -0.656 -0.644 -0.641 -0.646 -0.646 -0.642 -0.646 -0.641 -0.647 -0.657 -0.648 -0.65 -0.644 -0.642 -0.706 -0.641 -0.652 -0.639 -0.648 -0.641 -0.642 -0.666

#913 #914 #915 #917 #918 #919 #922 #933 #940 #920 #921 #946 #958 #983 #925 #926 #934 #937 #954 #928 #935 #951 #959 #984 #923 #924 #927 #930 #932 #942 #956 #979 #986 #931 #936 #952 #973
-0.659 -0.644 -0.679 -0.661 -0.657 -0.654 -0.642 -0.642 -0.643 -0.69 -0.754 -inf -0.681 -0.674 -0.643 -0.657 -0.671 -0.644 -0.644 -0.646 -0.647 -0.658 -0.657 -0.645 -0.859 -0.639 -0.665 -0.651 -0.662 -0.649 -0.657 -0.655 -0.64 -0.658 -0.717 -0.642 -0.66

#929 #943 #944 #945 #955 #949 #950 #957 #989 #963 #966 #947 #948 #987 #961 #967 #977 #978 #993 #997 #938 #939 #941 #953 #960 #982 #988 #1001 #1003 #1009 #1013 #1017 #974 #975 #976 #991
-0.698 -0.644 -0.642 -0.643 -0.677 -0.663 -0.692 -0.668 -0.647 -0.644 -0.662 -0.647 -0.741 -0.643 -0.664 -0.803 -0.644 -0.644 -0.657 -0.648 -0.656 -0.669 -0.646 -0.698 -0.659 -0.64 -0.642 -0.672 -0.673 -0.722 -0.708 -0.641 -0.669 -0.708 -0.644 -0.665

#962 #968 #969 #970 #971 #985 #964 #965 #972 #980 #981 #990 #1010 #1015 #998 #1004 #999 #1002 #1008 #1012 #1016 #1005 #1006 #1014 #1000
-0.674 -0.676 -0.664 -0.69 -0.642 -0.642 -0.665 -0.646 -0.776 -0.67 -0.657 -0.65 -0.662 -0.645 -0.644 -0.644 -0.656 -0.659 -0.692 -0.66 -0.644 -0.643 -0.645 -0.658 -0.741

#992 #994 #995 #996 #1007 #1011

-0.645 -0.673 -0.701 -0.663 -0.642 -0.647

Supplementary Fig. 18 | Top Figure Breakthrough plot for the GIFT-Eval tree search, showing the
evolution of the maximum score as a function of the number of nodes. The green dots label places
where the score abruptly increases due to an improvement in the code, and the label describes the
change in the code that resulted in the score increase. Bottom Figure Structure of the tree for this
same search. The color range consists of orange (lower scores) to green (higher scores) with the
highest score denoted by a diamond node.

46
 An AI system to help scientists write expert-level empirical software

train set test set

∫∞ ∫∞ 

445.001 sin 𝑥 2 446.021 sin4 𝑎𝑥 2 − sin4 𝑏𝑥 2 𝑑𝑥



𝑑𝑥
0 0
∫∞ ∫∞
445.017 sin 𝑎𝑥 2 cos (2𝑏𝑥 ) 𝑑𝑥 446.045 𝑥 cos 𝑎𝑥 2 cos (2𝑏𝑥 ) 𝑑𝑥



0 0
∫∞ 
𝑏2
 ∫∞
447.012 sin 𝑎𝑥 2 + 𝑎
cos (2𝑏𝑥 ) 𝑑𝑥 449.013 𝑥 𝜇 −1 sin ( 𝑎𝑥 ) cos ( 𝑏𝑥 ) 𝑑𝑥
0 0
∫∞  𝛾+𝑥 𝛾−𝑥
 ∫∞ ( 3−4 sin2 ( 𝑎𝑥 ) ) sin2 ( 𝑎𝑥 )
458.031 − sin ( 𝑎𝑥 ) 𝑑𝑥 465.002 𝑑𝑥
𝛽 2 +( 𝛾 + 𝑥 ) 2 𝛽 2 +( 𝛾 − 𝑥 ) 2 𝑥
0 0
∫∞ 𝑥 sin ( 𝑎𝑥 ) cos ( 𝑏𝑥 )
∫∞ sin2𝑚+1 ( 𝑥 ) sin ( 𝑥 (6𝑚+3) )
462.034 𝑐2 + 𝑥 2
𝑑𝑥 465.013 𝑎2 + 𝑥 2
𝑑𝑥
0 0
∫∞ 𝑥 sin ( 𝑎𝑥 )+cos ( 𝑎𝑥 )
∫∞ sin ( 𝑥 ) cos ( 𝑥 )
477.049 𝑥 2 +1
𝑑𝑥 467.025 √ 𝑑𝑥
0 0 𝑥 sin2 ( 𝑥 )+1
∫∞ (cos ( 𝑎 ) −cos ( 𝑎𝑛𝑥 ) ) sin ( 𝑚𝑥 )
∫∞
478.036 𝑥
𝑑𝑥 478.031 sin ( 𝑎𝑥 𝑝 ) 𝑑𝑥
0 0
∫∞ 1 sin ( 𝑥 )
∫∞ cos ( 𝑎𝑥 )
487.011 2 2 𝑑𝑥 478.050 √
− 𝑢+𝑥
𝑑𝑥
𝑎2 cos2 ( 𝑥 )+𝑏2 sin ( 𝑥 ) )
𝑥
0 ( 𝑢
∫∞ 1 sin ( 𝑥 ) cos2 ( 𝑥 )
∫∞
487.026 484.059 sin 𝑎 − 𝑥 2 + cos 𝑎 − 𝑥 2 𝑑𝑥




2 2 𝑑𝑥
𝑎2 cos2 ( 𝑥 )+𝑏2 sin ( 𝑥 ) )
𝑥
0 ( 0
∫∞ 1
3
sin ( 𝑥 ) cos ( 𝑥 )
∫∞ cos ( 𝑥 ) cos ( 𝑎 cos ( 𝑥 ) ) cos (2𝑛𝑥 ) sinh ( 𝑎 sin ( 𝑥 ) )
488.014 4 𝑑𝑥 487.068 𝑑𝑥
0
𝑥
( 𝑎2 cos2 (2𝑥 )+𝑏2 sin2 (2𝑥 ) ) 0
𝑥

∫∞ cos2𝑚 ( 𝑥 )
∫∞
491.004 494.006 𝑥 sin (2𝑏𝑥 ) cos 𝑎𝑥 2 𝑑𝑥


𝑎2 + 𝑥 2
𝑑𝑥
0 0
∫∞ cos2𝑚+1 ( 𝑥 )
∫∞ sin3 ( 𝑥 ) 1
491.006 𝑎2 + 𝑥 2
𝑑𝑥 496.037 2 2 2 2 3 𝑥 𝑑𝑥
0 0 ( 𝑎 cos ( 𝑥 )+𝑏 sin ( 𝑥 ) )
∫∞ 𝑥 sin (2𝑎𝑥 ) cos2 ( 𝑏𝑥 )
∫∞ sin ( 𝑎𝑥 𝑝 )
491.014 𝛽 2 +𝑥 2
𝑑𝑥 504.025 𝑥
𝑑𝑥
0 0
∫∞ sin (2𝑎𝑥 ) cos2 ( 𝑏𝑥 )
∫∞ sin3 ( 𝑥 ) cos ( 𝑥 )
493.056 𝑑𝑥 504.061 √ 𝑑𝑥
𝑥 sin2 (2 𝑥 )+1
𝑥
0 0 √
∫∞ sin3 ( 𝑎𝑥 ) sin2 ( 𝑏𝑥 )
∫ − 𝑏+√𝑏2 +𝑥 2 sin ( 𝑎𝑥 )
∞
495.029 𝑑𝑥 505.006 √ 𝑑𝑥
𝑥 𝑏2 + 𝑥 2
0 0
∫∞ sin3 ( 𝑥 ) cos ( 𝑥 )
∫∞ sin ( 𝑥 )
504.057 √ 𝑑𝑥 505.008 𝑑𝑥
𝑥 cos2 (2 𝑥 )+1 𝑥 ( 𝑎2 sin2 ( 𝑥 )+𝑏2 cos2 ( 𝑥 ) )
0  √  0
∞
∫ cos ( 𝑏𝑥 ) cos 𝑝 𝑎2 +𝑥 2 ∫∞ (cos ( 𝑎 ) −cos ( 𝑎𝑛𝑥 ) ) sin ( 𝑚𝑥 )
512.029 𝑐2 + 𝑥 2
𝑑𝑥 505.023 𝑥
𝑑𝑥
0  √  0
∫∞ cos ( 𝑏𝑥 ) cos 𝑝 𝑎2 +𝑥 2 ∫∞ sin3 ( 𝑎𝑥 ) cos (3𝑏𝑥 )
512.037 𝑎2 + 𝑥 2
𝑑𝑥 513.033 𝑥2
𝑑𝑥
0 0
∫∞ sin ( 𝑎𝑥 ) coth ( 𝜋𝑥 ∫∞ sin3 ( 𝑎2 𝑥 2 )
2 )
550.003 𝑥 2 +1
𝑑𝑥 551.027 𝑥2
𝑑𝑥
0 0

Supplementary Fig. 19 | The dataset of 38 definite integrals with oscillatory integrands on

semi-infinite domains 60 , none of which were solved correctly by [Link]().
Parameters like 𝑎, 𝑏, 𝑐 were chosen randomly between 0 and 5 with exponents constrained to be
integers.

47
 An AI system to help scientists write expert-level empirical software

Node #361 Node #961

Prioritized `x=1/t` transform; adaptive Euler segments Increased numerical limits, earlier Euler acceleration.
Node #342
5 Eliminated tangent transformation's harmful endpoint heuristic. Node #411
Removed problematic integral transformations, prioritized Euler acceleration
Node #309
Added tangent transformation for infinite integrals. Node #478
Removed problematic transformations; prioritized Euler acceleration.
0 Node #282
Aggressively increased integration
effort and precision checks. Node #764
Increased initial `quad` limits and reduced Euler segments
-5
Node #234 Node #747
Euler acceleration applied to negative infinite lower limits. Relaxed `quad` tolerance, optimized segment limits.
Node #687
Maximum Score

Node #161
-10 Transformed infinite limits to finite intervals via tangent.
Stronger `[Link]` attempts, refined Euler convergence.

Node #617
Node #119
Accelerated segment growth; earlier Euler engagement.
Maximized `quad` limits and tightened tolerances.
-15 Node #59
Node #606
Faster segment growth, earlier Euler convergence.
Prioritized infinite-to-finite domain transformation.
Node #510
Node #47
Faster segment growth, fewer segments, looser segment tolerances.
-20 Aggressively tuned `quad` parameters and smarter Euler termination.

Node #34
Expanded `quad` acceptance, more robust Euler method.
-25 Node #7
Ensured series alternation for Euler acceleration.

0 200 400 600 800 1000

Number of Nodes
#0
nan

#1 #2 #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13 #14 #15 #16 #17 #18 #19 #20
-27.8 -inf -40 -64.5 -60.2 -inf -19 -66.2 -inf -16.4 -19 -17 -59 -68.9 -20.4 -66.2 -77.4 -19 -82.8 -68.9

#22 #27 #26 #38 #30 #31 #33 #34 #35 #40 #41 #42 #45 #32 #36 #37 #39 #44 #21 #23 #25 #24 #28 #29
-inf -68.8 -76.8 -19 -23.7 -inf -19 -14.9 -67.9 -inf -inf -22.2 -54.6 -61.2 -19 -19 -17 -19 -83 -24.1 -79.2 -68.9 -19 -19

#47 #48 #49 #50 #51 #54 #55 #56 #57 #59 #60 #61 #52 #53 #58 #43 #46
-14.6 -30.1 -19 -inf -inf -15.9 -15.8 -26 -15.5 -6.72 -9.19 -17 -19 -19 -69.5 -19 -19

#62 #63 #64 #65 #66 #67 #68 #69 #71 #79 #75 #72 #73 #74 #78 #85 #76 #77 #80
-28.5 -29 -14.6 -16.2 -14 -8.76 -14.4 -14.7 -19.4 -16.5 -17.9 -18.7 -inf -6.72 -inf -8.01 -11.1 -16.7 -10

#86 #70 #81 #83 #89 #91 #94 #98 #82 #87 #84 #88 #90 #92 #93 #95 #96 #97 #99 #100 #102 #103
-24 -12.5 -19.7 -20.6 -8.76 -8.76 -8.76 -11 -20.9 -15.5 -14.7 -6.72 -6.72 -6.72 -6.72 -6.72 -10.1 -6.72 -6.72 -6.72 -10 -10

#104 #105 #101 #108 #106 #116 #135 #107 #109 #110 #112 #129 #142 #148 #111 #117 #113 #114 #136 #146 #115
-12.5 -12.5 -25.7 -6.72 -18.3 -24.1 -14.8 -6.72 -6.72 -6.72 -inf -14.5 -10.3 -11.5 -6.72 -12.9 -18.1 -inf -17.2 -77.4 -6.72

#119 #128 #133 #122 #130 #145 #121 #127 #143 #118 #126 #132 #150 #123 #124 #134 #120 #125 #131 #151
-6.52 -11.5 -6.72 -inf -15.7 -9.02 -18.8 -76.6 -15.7 -13.8 -inf -67.7 -inf -inf -72.9 -6.72 -28.7 -15.3 -9.16 -inf

#152 #153 #159 #162 #164 #167 #170 #171 #172 #140 #141 #144 #147 #137 #138 #139 #149
-11.3 -14.5 -15.4 -12.5 -17.7 -4.34 -inf -22.1 -6.72 -16.7 -6.72 -6.72 -6.64 -6.72 -inf -6.58 -6.55

#184 #186 #187 #194 #198 #196 #156 #160 #166 #155 #158 #163 #168 #154 #157 #161 #165 #169
-4.4 -inf -inf -6.65 -6.62 -9.03 -17 -35.2 -6.64 -5.75 -18.8 -15.3 -6.72 -11.1 -6.55 -4.3 -6.55 -6.55

#201 #203 #207 #177 #173 #176 #179 #182 #183 #178 #174 #181 #188 #189 #190 #193 #195 #197 #175 #180 #185 #191
-4.4 -4.4 -12.5 -6.64 -14.5 -inf -11.1 -10.5 -inf -6.81 -6.4 -16.7 -4.99 -inf -4.3 -4.3 -16.8 -4.3 -inf -13.6 -11.1 -9.95

#227 #229 #192 #199 #205 #213 #212 #217 #232 #200 #202 #204 #206 #208 #209 #210 #211 #223
-34.9 -8.48 -11.4 -16 -6.4 -32.9 -inf -4.3 -4.3 -inf -4.3 -4.3 -4.3 -inf -4.3 -4.3 -6.39 -4.3

#221 #241 #267 #245 #256 #215 #220 #230 #216 #219 #214 #218 #222 #224 #228 #231 #225 #226 #233 #237
-inf -15.4 -4.3 -16.9 -16.5 -4.3 -4.3 -4.3 -4.97 -16.2 -6.4 -4.3 -4.3 -inf -4.3 -4.3 -4.3 -4.3 -4.3 -4.3

#277 #238 #242 #234 #243 #244 #259 #236 #239 #247 #255 #248 #250 #240 #264 #235 #249 #246 #262 #251 #254
-12.5 -4.3 -4.3 -4.28 -11.8 -6.98 -17 -4.3 -71.6 -4.3 -12.5 -4.3 -22.1 -14.7 -4.3 -16.5 -16.9 -11.2 -4.3 -9.13 -18.1

#252 #260 #265 #268 #269 #270 #271 #274 #275 #276 #281 #282 #283 #253 #258 #257 #266 #261 #263 #280 #273
-19 -25 -4.3 -13.8 -20.6 -17.5 -inf -4.28 -15.9 -4.28 -inf -4.28 -27 -4.3 -inf -15.4 -inf -15.2 -4.3 -17.5 -4.3

#278 #289 #292 #297 #311 #314 #300 #301 #308 #312 #298 #299 #305 #307 #310 #313 #316 #279 #284 #285 #286 #287 #288
-6.35 -5.61 -6.95 -17.1 -20.3 -19 -4.28 -14.2 -16.8 -46.2 -14.7 -6.56 -4.28 -12.5 -4.28 -4.28 -7.08 -4.3 -4.3 -inf -16.5 -4.3 -4.3

#320 #330 #319 #321 #340 #323 #327 #329 #344 #324 #295 #296 #290 #293 #291 #294 #317
-inf -4.28 -15.4 -4.28 -4.28 -4.28 -5.31 -6.36 -19 -4.28 -4.3 -6.96 -4.3 -18 -33.2 -19 -4.3

#360 #369 #350 #359 #364 #368 #353 #337 #343 #302 #304 #309 #315 #303 #306 #318 #322
-4.28 -12.3 -6.95 -17.8 -17.4 -16.8 -4.28 -23.4 -6.53 -17.5 -4.98 -3.5 -3.51 -12.4 -14.7 -18 -4.3

#379 #373 #374 #331 #332 #333 #335 #342 #345 #325 #326 #328 #334
-4.28 -6.58 -inf -4.29 -4.31 -25.4 -3.5 -2.8 -6.95 -3.51 -71 -16.2 -inf

#397 #354 #348 #349 #352 #355 #357 #358 #361 #362 #363 #365 #366 #367 #370 #372 #336 #338 #339 #341 #346
-4.28 -6.36 -4.47 -23.5 -inf -30.8 -4.29 -inf -2.1 -2.8 -18.6 -inf -9.79 -12.5 -2.79 -6.96 -inf -3.51 -inf -inf -inf

#409 #375 #380 #382 #387 #394 #381 #383 #395 #388 #389 #390 #398 #347 #351 #356 #371
-4.28 -6.95 -17.5 -14 -2.1 -3.14 -2.82 -13 -inf -2.79 -6.96 -inf -11.1 -9.57 -inf -inf -3.51

#439 #402 #403 #404 #406 #408 #416 #418 #412 #413 #415 #410 #411 #414 #421 #426 #376 #377 #378
-4.28 -2.1 -inf -inf -2.1 -inf -2.1 -4.89 -7.98 -3.14 -15.2 -6.27 -2.09 -inf -2.79 -16.9 -11.2 -3.51 -3.51

#420 #422 #425 #436 #440 #419 #423 #433 #429 #430 #432 #454 #424 #427 #428 #431 #435 #438 #450 #451 #385 #399 #384 #386
-11.1 -inf -2.1 -27.6 -19 -22.3 -3.34 -19 -inf -9.03 -8.33 -7.07 -3.14 -2.11 -14.7 -21.9 -inf -12 -3.49 -19 -12.5 -20.2 -2.12 -inf

#441 #443 #446 #447 #453 #434 #437 #442 #445 #448 #391 #392 #393 #396
-inf -2.1 -64.2 -3.36 -19.4 -2.8 -inf -3.47 -inf -2.11 -inf -8.25 -15.4 -2.12

#458 #460 #461 #463 #467 #468 #474 #457 #444 #449 #452 #455 #456 #400 #401 #405 #407 #417
-18 -13.1 -inf -11.8 -19.4 -12.5 -inf -3.36 -12.5 -34.3 -61.3 -2.11 -2.92 -4.3 -6.3 -11.1 -17.1 -6.3

#464 #459 #462 #472

-6.45 -2.11 -2.11 -2.92

#469 #470 #471 #465 #466 #473

-11.4 -2.11 -2.11 -2.11 -2.11 -2.11

#478 #482 #479 #484 #476 #483 #475 #477 #480 #481
-2.09 -23.6 -2.11 -2.11 -inf -2.11 -2.11 -2.11 -inf -inf

#486 #487 #490 #491 #494 #495 #485 #492 #493 #488 #489
-2.09 -61 -3.47 -64.6 -14.5 -3.47 -2.11 -4.87 -2.11 -2.11 -2.8

#496 #497 #500 #501 #503 #505 #508 #499 #502 #498
-21 -26.8 -2.09 -11.1 -23.8 -19 -2.09 -2.11 -2.11 -2.11

#510 #511 #513 #514 #515 #517 #522 #523 #524 #504 #507 #506
-2.09 -inf -11.1 -2.09 -2.09 -2.09 -13.4 -2.1 -2.1 -2.11 -61 -2.11

#518 #519 #520 #521 #528 #529 #530 #536 #538 #547 #552 #539 #541 #548 #551 #540 #544 #550 #542 #525 #526 #527 #531 #512 #509
-20.3 -27.7 -19 -2.09 -22.2 -6.1 -inf -23.6 -2.09 -11.1 -2.09 -3.69 -17.9 -21.5 -2.09 -inf -2.09 -inf -2.1 -2.1 -11.4 -2.1 -11.8 -2.11 -2.83

#535 #537 #543 #546 #549 #553 #554 #560 #568 #571 #589 #566 #575 #561 #590 #582 #533 #545 #532 #534 #555 #516
-11.8 -inf -12.8 -2.09 -inf -2.11 -2.09 -2.09 -22.7 -22.7 -inf -2.09 -20.6 -11.2 -11.1 -15.9 -2.09 -14.9 -inf -2.12 -11.2 -inf

#557 #558 #563 #587 #559 #562 #572 #576 #591 #569 #581 #577 #578 #556 #588
-2.2 -2.09 -2.09 -3.48 -inf -2.09 -11.1 -23.4 -2.09 -22.7 -2.09 -2.09 -68.8 -11.8 -11.8

#564 #565 #573 #579 #593 #594 #567 #570 #574 #580 #598 #599 #600 #602 #603 #604 #605 #606 #607 #608 #595 #596 #610 #614
-inf -2.78 -inf -2.1 -12.7 -12.2 -inf -2.09 -20.7 -22 -17.2 -inf -24.5 -14.9 -13.2 -inf -63 -2.09 -inf -23.6 -2.09 -2.09 -24.2 -2.09

#601 #583 #584 #585 #586 #592 #597 #609 #611 #612 #613 #615 #617 #616 #619
-2.1 -inf -61.7 -24.6 -12.8 -15.6 -17.7 -2.09 -2.09 -2.09 -17.1 -2.09 -2.07 -2.09 -2.09

#618 #620 #626 #623 #624 #621 #627 #622 #625 #628 #630 #638 #643 #642 #639
-inf -2.09 -12.4 -2.09 -2.09 -12 -2.24 -2.09 -11.8 -inf -62.4 -2.08 -2.08 -2.09 -2.09

#629 #631 #633 #637 #634 #636 #632 #635 #644 #647 #650 #662 #653 #658 #660
-2.78 -2.09 -inf -17.1 -12.9 -12.5 -11.8 -2.09 -2.09 -28.2 -inf -11.1 -2.08 -11.1 -2.08

#640 #641 #645 #646 #654 #665 #666 #667 #670 #675 #679
-2.09 -2.09 -2.09 -2.09 -2.09 -17.6 -2.08 -80.3 -2.08 -2.08 -11.2

#648 #651 #649 #652 #655 #668 #656 #669 #663 #671 #676 #681 #687 #684 #689
-2.09 -11.4 -2.09 -14.9 -11.1 -2.09 -11.2 -2.09 -61 -11.9 -2.08 -11.3 -0.884 -2.08 -2.76

#657 #659 #664 #661 #680 #682 #685 #690 #696 #702 #705 #712 #697 #703 #711
-2.09 -27 -2.09 -2.09 -2.09 -2.09 -12.5 -2.08 -0.884 -0.884 -11.3 -18 -11.2 -inf -2.1

#672 #678 #673 #677 #674 #695 #693 #698 #704 #714 #716 #723 #713 #715 #722
-2.09 -2.09 -2.08 -2.09 -2.09 -2.09 -11.4 -2.1 -inf -0.884 -11.8 -0.884 -0.884 -14.6 -11.2

#683 #692 #686 #694 #688 #691 #709 #727 #730 #733 #738 #742 #744 #748 #728 #729 #734 #747
-2.09 -18.4 -2.08 -2.08 -2.09 -2.09 -2.09 -18 -0.888 -11.3 -1.56 -11.3 -0.884 -0.884 -inf -inf -0.884 -0.874

#701 #699 #707 #706 #708 #700 #710 #739 #743 #745 #759 #753 #755 #757 #778 #780 #782 #760 #761 #749 #754 #756 #758 #764 #765 #772 #775
-2.09 -2.08 -2.08 -2.08 -2.08 -2.09 -2.09 -11.4 -8.51 -2.14 -0.888 -1.56 -1.56 -12.5 -11.1 -0.884 -11.3 -0.884 -0.884 -0.884 -inf -0.884 -19 -0.868 -0.874 -inf -inf

#719 #718 #737 #720 #726 #721 #724 #725 #736 #717 #785 #795 #766 #773 #769 #774 #815 #816 #818 #776 #781 #777 #779 #763 #770 #767 #768 #783 #786 #788 #796 #784 #787 #789
-2.09 -12.5 -11.2 -2.08 -13.2 -2.08 -2.08 -11.1 -2.08 -2.78 -0.888 -12 -inf -11.2 -1.56 -inf -inf -0.884 -0.884 -11.1 -0.884 -11.3 -1.4 -65.3 -0.884 -11.8 -inf -11.7 -0.868 -11.3 -19.6 -11.3 -inf -0.874

#732 #731 #735 #741 #740 #752 #810 #811 #792 #859 #835 #844 #797 #798 #793 #813 #791 #794 #817 #799 #801 #803 #805 #812 #800 #802 #804 #809
-2.09 -2.08 -2.08 -12.5 -14.1 -2.08 -inf -0.888 -1.56 -66.3 -inf -inf -0.884 -0.884 -11.8 -12.7 -11.8 -2.13 -inf -19 -0.868 -11.3 -18.8 -inf -0.874 -0.874 -0.874 -0.874

#746 #751 #750 #762 #839 #807 #806 #808 #814 #820 #823 #819 #821 #822 #824 #870 #830 #833 #843 #825 #826 #828 #829 #831 #832 #842 #834 #838 #841
-2.09 -12.5 -2.08 -2.08 -0.888 -1.56 -2.08 -inf -11.1 -0.884 -0.884 -inf -inf -11.8 -inf -11.3 -11.1 -0.88 -19 -11.5 -63.7 -2.08 -11.3 -0.874 -11.1 -0.874 -0.874 -12.5 -21.9

#771 #865 #827 #837 #851 #836 #850 #846 #848 #852 #845 #847 #849 #855 #862 #866 #856 #861 #867
-2.08 -6.7 -1.56 -11.3 -0.884 -0.884 -11.3 -12 -0.87 -12.8 -0.874 -inf -2.09 -0.874 -0.874 -inf -11.3 -0.874 -0.874

#790 #840 #860 #853 #854 #858 #864 #868 #857 #863 #873 #884 #890 #883 #888 #878 #887 #877 #886
-2.08 -1.56 -0.884 -0.884 -0.87 -0.87 -0.87 -11.1 -11.1 -0.874 -2.09 -17.9 -0.874 -0.874 -34.3 -15.9 -18.8 -12.1 -0.874

#871 #872 #869 #876 #882 #874 #881 #875 #880 #879 #912 #906 #908 #902 #904
-1.56 -0.884 -0.884 -0.87 -0.87 -1.57 -0.87 -65.3 -0.87 -0.874 -0.874 -11.3 -inf -12.8 -17.7

#889 #885 #891 #894 #898 #899 #892 #893 #901 #895 #896 #900 #903 #929
-0.884 -0.884 -0.87 -0.87 -2.11 -0.87 -inf -0.87 -11.3 -13.2 -0.868 -0.87 -0.874 -0.874

#909 #897 #905 #907 #916 #919 #918 #923 #911 #913 #928 #915 #917 #925 #946 #948 #910 #914 #922 #966
-0.884 -11.3 -0.871 -0.87 -0.87 -0.87 -0.87 -17.7 -11.2 -0.877 -0.87 -11.3 -inf -2.1 -11.3 -0.868 -9.01 -11.3 -0.874 -14.7

#921 #926 #920 #924 #933 #940 #934 #939 #930 #938 #945 #931 #936 #942 #959 #961 #965 #967 #941
-13.2 -0.871 -0.87 -0.87 -0.87 -0.87 -0.87 -0.87 -19 -0.87 -0.877 -0.87 -11.3 -0.87 -12 -0.867 -11.8 -11.1 -inf

#935 #947 #943 #944 #932 #937 #951 #952 #949 #953 #960 #955 #963 #950 #962 #979 #981 #983 #984
-11.3 -0.871 -0.87 -0.87 -0.87 -inf -11.3 -0.87 -0.87 -11.6 -0.87 -19 -0.87 -0.87 -0.87 -0.867 -11.1 -12.5 -0.867

#964 #957 #958 #956 #972 #954 #971 #980 #982 #969 #976 #968 #985 #987 #993 #994 #999 #1006 #1011
-0.871 -0.87 -0.87 -0.87 -0.87 -11.3 -0.87 -0.87 -0.87 -0.87 -11.1 -11.3 -0.867 -11.3 -0.867 -inf -11.3 -0.867 -1.75

#977 #974 #970 #973 #988 #989 #995 #996 #998 #1001 #986 #992 #1000 #1005 #1013 #1002 #1004 #1010
-0.871 -35.6 -0.87 -0.87 -17 -0.87 -0.87 -0.87 -0.87 -12.8 -11.1 -2.69 -0.867 -2.08 -12.1 -11.9 -0.867 -0.867

#997 #991 #990 #1007 #1003 #1012 1014 #1015 #1016

-0.871 -0.87 -0.87 -0.87 -0.867 -19 -0.851 -18.7 -11.3

#1009 #1008
-0.869 -0.87

Supplementary Fig. 20 | Top Figure Breakthrough plot for the Integral tree search, showing the
evolution of the maximum score as a function of the number of nodes. The green dots label places
where the score abruptly increases due to an improvement in the code, and the label describes the
change in the code that resulted in the score increase. Bottom Figure Structure of the tree for this
same search. The color range consists of orange (lower scores) to green (higher scores) with the
highest score denoted by a diamond node.

48
 An AI system to help scientists write expert-level empirical software

Supplementary Fig. 21 | Scores of the best numerical integration routine applied to the held-out set
of 19 integrals. Zero is a perfect score. The generated function solved 17 of 19 integrals to within 3
percent. The standard function, [Link]() failed in all these cases.

49
 An AI system to help scientists write expert-level empirical software

Scorable Scientific Task

sandbox
Description Data Agent
In this competition Train, Validation
you will be using … sample_submission.csv Plan
Model exploration submission
Scoring Metric: Root Mean Square Error Debug
Train Evaluation
+ Research ideas Validate Metric Leaderboard
Scientific Papers
Create Submission
LLM Deep Research Execution feedback
Self-reflection Score: 0.336
Improve
Recombination
68.3 %

Supplementary Fig. 22 | Schematic of Algorithm, consisting of a code mutation system, where the
prompt is augmented with research ideas. Research ideas can be sourced from the primary literature,
or from a search algorithm.

50
 An AI system to help scientists write expert-level empirical software

Supplementary Tables

51
 An AI system to help scientists write expert-level empirical software

Supplementary Table 1 | Basic Prompt Playground Competitions. The prompt is used for the TS
on the Kaggle Playground Benchmark. This example is for Season 3 Episode 17.

Prompt for Kaggle Playground Competitions

Please write the python code to work on a Kaggle competition. Use any model you like.
Kaggle competition name: Binary Classification of Machine Failures
The competition is evaluated as follows: Submissions are evaluated on area under the ROC
curve between the predicted probability and the observed target.

Submission File
For each `id` in the test set, you must predict the probability of a `Machine failure`.
The file should contain a header and have the following format:

id,Machine failure
136429,0.5
136430,0.1
136431,0.9
etc.

Here are a few lines of each of the files:

file_name : sample_submission.csv

file_contents:

id,Machine failure
79996,0
100009,0
etc.

====
file_name : [Link]

file_contents:
etc.
====
file_name : [Link]

file_contents:
etc.
====
Please provide complete code that will generate the submission file in the
format below:

```python
YOUR CODE
```

52
 An AI system to help scientists write expert-level empirical software

Supplementary Table 2 | Expert Advice for Playground Competitions. The prompt is used for the
TS with Expert Advice on the Kaggle Playground Benchmark.

Expert Advice Prompt for Kaggle Playground Competitions

Here is high level advice: Instead of putting all your effort into a single model, experiment with
combining two or more models. Start with simple averaging of predictions and then explore
more advanced techniques like stacking.
Try out several different types of models (e.g., gradient boosting machines, linear models, and
even simpler models like logistic regression) to see how they perform.
Look for opportunities to go beyond standard preprocessing. Investigate the data for potential
leaks, and consider using optimization libraries to find the best way to combine your models’
predictions.
While feature engineering is a crucial skill, it’s also important to recognize when it might not be
the most important factor. Sometimes, the choice of model and ensembling strategy can have
a bigger impact. Don’t be afraid to try a more "brute-force" approach with powerful models
that can handle raw data effectively.

Supplementary Table 3 | Boosted Decision trees for Playground Competitions. The prompt is
used for the TS with Boosted Decision Tree on the Kaggle Playground Benchmark.

Expert Advice Prompt for Kaggle Playground Competitions

Given the code you are given please rewrite any library code (such as XGBoost, LightGBM, and
CatBoost) by making internal algorithmic choices that produce performant training code and
models that generalize well in many situations. Things you can try are alternative representa-
tions of data, using different step size algorithms, using the output of a strong learner as input
to the next weak learner. If the code contains such libraries, please extract the raw code that is
being used in the library and rewrite it to improve performance.

53
 An AI system to help scientists write expert-level empirical software

Supplementary Table 4 | Example code generated by tree search. We prompted tree search to
generate a solution for single-cell batch integration. We refer to this solution as BBKNN (TS).

Example code generated by our method.

1 # [ rewrite_cell ]
2 from typing import Any
3 from sklearn . decomposition import TruncatedSVD
4 from sklearn . neighbors import NearestNeighbors
5 from scipy . sparse import lil_matrix , csr_matrix
6 import numpy as np
7 import scanpy as sc
8 import anndata as ad
9 import heapq # For efficiently getting top K elements from merged lists
10
11 # Define parameters for the config .
12 # These values are chosen to balance computational cost and integration performance
13 # for datasets with up to ~300 k cells and 2 k genes .
14 config = {
15 ' n_pca_components ': 100 , # Number of PCA components . Recommended : 50 -200.
16 # Captures sufficient variance while reducing dimensionality .
17 ' n_neighbors_per_batch ': 10 , # Number of neighbors to find within each batch . Recommended : 5 -15.
18 # This defines the local batch context for each cell .
19 ' total_k_neighbors ': 50 , # Total number of nearest neighbors to retain for the final graph . Recommended : 15 -100.
20 # This forms the global batch - integrated graph .
21 }
22
23
24 def eliminate_batch_effect_fn (
25 adata : ad . AnnData , config : dict [ str , Any ]
26 ) -> ad . AnnData :
27 # Create a copy to ensure the original input adata remains unchanged .
28 adata_integrated = adata . copy ()
29
30 # --- Preprocessing : Normalize , log - transform , scale ---
31 # These are standard initial steps for scRNA - seq data .
32 # Use adata . X which contains raw counts .
33 sc . pp . normalize_total ( adata_integrated , target_sum =1 e4 )
34 sc . pp . log1p ( adata_integrated )
35 sc . pp . scale ( adata_integrated , max_value =10) # Clip values to avoid extreme outliers
36
37 # --- Batch Correction : ComBat on the gene expression matrix ---
38 # This step applies a more robust linear model - based batch correction
39 # directly on the gene expression data before dimensionality reduction .
40 # ComBat modifies adata_integrated . X in place .
41 sc . pp . combat ( adata_integrated , key = ' batch ')
42
43 # --- Dimensionality Reduction : PCA on the ComBat - corrected data ---
44 # n_comps cannot exceed min ( n_obs - 1 , n_vars ) . Robustly handle small datasets .
45 n_pca_components = config . get ( ' n_pca_components ' , 100)
46 actual_n_pca_components = min ( n_pca_components , adata_integrated . n_vars , adata_integrated . n_obs - 1)
47
48 # Handle edge cases for PCA and graph construction where data is too small .
49 # If PCA cannot be run meaningfully , return a minimal AnnData object to avoid errors .
50 if actual_n_pca_components <= 0 or adata_integrated . n_obs <= 1:
51 print ( f " Warning : Too few observations ({ adata_integrated . n_obs }) or dimensions ({ adata_integrated . n_vars }) for PCA / graph construction .
Returning trivial embedding . " )
52 # Provide a placeholder embedding and empty graph structure .
53 adata_integrated . obsm [ ' X_emb '] = np . zeros (( adata_integrated . n_obs , 1) )
54 adata_integrated . obsp [ ' connectivities '] = csr_matrix (( adata_integrated . n_obs , adata_integrated . n_obs ) )
55 adata_integrated . obsp [ ' distances '] = csr_matrix (( adata_integrated . n_obs , adata_integrated . n_obs ) )
56 adata_integrated . uns [ ' neighbors '] = {
57 ' params ': {
58 ' n_neighbors ': 0 ,
59 ' method ': ' degenerate ' ,
60 ' n_pcs ': 0 ,
61 ' n_neighbors_per_batch ': 0 ,
62 ' pca_batch_correction ': ' none ' ,
63 },
64 ' connectivities_key ': ' connectivities ' ,
65 ' distances_key ': ' distances ' ,
66 }
67 return adata_integrated
68
69 sc . tl . pca ( adata_integrated , n_comps = actual_n_pca_components , svd_solver = ' arpack ')
70
71 # Set the ComBat - corrected PCA embedding as the integrated output embedding .
72 # This ' X_emb ' will be directly evaluated by metrics like ASW , LISI , PCR .
73 adata_integrated . obsm [ ' X_emb '] = adata_integrated . obsm [ ' X_pca ']
74
75
76 # --- Custom Batch - Aware Nearest Neighbors Graph Construction ---
77 # This implements the expert advice : find neighbors independently within batches , then merge .
78 # This part of the code remains largely the same , but now operates on the
79 # ComBat - corrected PCA embedding ( adata_integrated . obsm [ ' X_emb ']) .
80 k_batch_neighbors = config . get ( ' n_neighbors_per_batch ' , 10)
81 total_k_neighbors = config . get ( ' total_k_neighbors ' , 50)
82
83 # A list of dictionaries to store unique neighbors and their minimum distances for each cell .
84 # Using dictionaries allows efficient updating if a cell is found as a neighbor from multiple batches .
85 merged_neighbors_per_cell = [{} for _ in range ( adata_integrated . n_obs ) ]
86
87 # Group cell indices by batch for efficient querying .
88 batches = adata_integrated . obs [ ' batch ' ]. values
89 unique_batches = np . unique ( batches )
90 batch_to_indices = { b : np . where ( batches == b ) [0] for b in unique_batches }
91
92 # Pre - fit NearestNeighbors models for each batch 's data using the corrected PCA embedding .
93 # This avoids refitting the model for every query .
94 batch_nn_models = {}
95 for b_id in unique_batches :
96 batch_cell_indices = batch_to_indices [ b_id ]
97 # Ensure there are enough cells to fit a NearestNeighbors model ( at least k_batch_neighbors + 1 for self - exclusion , or just > 0 for min k =1)
98 if len ( batch_cell_indices ) > 0:
99 # Fit with a k that is at most the batch size to avoid errors if k_batch_neighbors is too high for a small batch .
100 k_fit_effective = min ( k_batch_neighbors + 1 , len ( batch_cell_indices ) ) # +1 to ensure self - loop can be found and excluded
101 if k_fit_effective > 0: # Only fit if there are points available
102 nn_model = NearestNeighbors ( n_neighbors = k_fit_effective , metric = ' euclidean ' , algorithm = ' auto ')
103 nn_model . fit ( adata_integrated . obsm [ ' X_emb ' ][ batch_cell_indices ])
104 batch_nn_models [ b_id ] = nn_model

54
 An AI system to help scientists write expert-level empirical software

Example code generated by our method (continued).

1
2 # Iterate through all possible query batches and target batches to find neighbors .
3 for query_batch_id in unique_batches :
4 query_global_indices = batch_to_indices [ query_batch_id ]
5 if len ( query_global_indices ) == 0:
6 continue # Skip empty query batches
7
8 query_data = adata_integrated . obsm [ ' X_emb ' ][ query_global_indices ]
9
10 for target_batch_id in unique_batches :
11 if target_batch_id not in batch_nn_models :
12 continue # Skip target batches that were too small to fit an NN model
13
14 nn_model = batch_nn_models [ target_batch_id ]
15 target_global_indices = batch_to_indices [ target_batch_id ]
16
17 # Ensure n_neighbors does not exceed the number of points in the target batch .
18 k_for_query = min ( k_batch_neighbors , len ( target_global_indices ) -1) # -1 to avoid finding self as neighbor if batch is query batch
19 if k_for_query <= 0: # No valid neighbors can be found in this target batch
20 continue
21
22 # Query neighbors for all cells in the current query batch against the target batch 's data .
23 distances , indices_in_target_batch = nn_model . kneighbors ( query_data , n_neighbors = k_for_query , return_distance = True )
24
25 for i_query_local in range ( len ( query_global_indices ) ) :
26 current_cell_global_idx = query_global_indices [ i_query_local ]
27
28 dists_for_cell = distances [ i_query_local ]
29 global_neighbors_for_cell = target_global_indices [ indices_in_target_batch [ i_query_local ]]
30
31 for k_idx in range ( len ( global_neighbors_for_cell ) ) :
32 neighbor_global_idx = global_neighbors_for_cell [ k_idx ]
33 dist = dists_for_cell [ k_idx ]
34
35 # Exclude self - loops : a cell should not be its own neighbor in graph construction .
36 if neighbor_global_idx == current_cell_global_idx :
37 continue
38
39 # Store neighbor and its distance . If already present , keep the minimum distance ( closest connection ) .
40 if ( neighbor_global_idx not in merged_neighbors_per_cell [ current_cell_global_idx ] or
41 dist < merged_neighbors_per_cell [ current_cell_global_idx ][ neighbor_global_idx ]) :
42 merged_neighbors_per_cell [ current_cell_global_idx ][ neighbor_global_idx ] = dist
43
44 # Convert collected neighbors and distances into sparse matrices .
45 rows = []
46 cols = []
47 data_distances = []
48
49 for i in range ( adata_integrated . n_obs ) :
50 # Retrieve all candidate neighbors for cell 'i ', sort by distance , and take the top ' total_k_neighbors '.
51 current_cell_candidates = list ( merged_neighbors_per_cell [ i ]. items () )
52
53 if not current_cell_candidates : # If a cell has no valid neighbors after all filtering
54 continue
55
56 # Use heapq for efficient selection of the smallest distances .
57 selected_neighbors = heapq . nsmallest ( total_k_neighbors , current_cell_candidates , key = lambda item : item [1])
58
59 for neighbor_idx , dist in selected_neighbors :
60 rows . append ( i )
61 cols . append ( neighbor_idx )
62 data_distances . append ( dist )
63
64 # Create distance matrix . Handle case with no neighbors found at all for the entire dataset .
65 if not rows :
66 distances_matrix = csr_matrix (( adata_integrated . n_obs , adata_integrated . n_obs ) )
67 else :
68 distances_matrix = csr_matrix (( data_distances , ( rows , cols ) ) , shape =( adata_integrated . n_obs , adata_integrated . n_obs ) )
69
70 # Symmetrize the distance matrix : if A is a neighbor of B , then B is also a neighbor of A ,
71 # with the distance being the maximum of the two observed distances ( ensures undirected graph ) .
72 distances_matrix = distances_matrix . maximum ( distances_matrix . T )
73 distances_matrix . eliminate_zeros () # Remove any explicit zeros created by max operation
74
75 # Create connectivities matrix ( binary representation of connections ) .
76 connectivities_matrix = distances_matrix . copy ()
77 connectivities_matrix . data [:] = 1.0 # All non - zero entries become 1.0 ( connected ) .
78 connectivities_matrix . eliminate_zeros ()
79 connectivities_matrix = connectivities_matrix . astype ( float )
80
81 # Store the custom graph in adata . obsp . These keys are used by scib metrics .
82 adata_integrated . obsp [ ' connectivities '] = connectivities_matrix
83 adata_integrated . obsp [ ' distances '] = distances_matrix
84
85 # Store parameters in adata . uns [ ' neighbors '] for completeness and scanpy / scib compatibility .
86 adata_integrated . uns [ ' neighbors '] = {
87 ' params ': {
88 ' n_neighbors ': total_k_neighbors ,
89 ' method ': ' custom_batch_aware_combat_pca ' , # Reflects the integration strategy
90 ' metric ': ' euclidean ' ,
91 ' n_pcs ': actual_n_pca_components ,
92 ' n_neighbors_per_batch ': k_batch_neighbors ,
93 ' pca_batch_correction ': ' combat ' , # Indicates ComBat was applied before PCA
94 },
95 ' connectivities_key ': ' connectivities ' ,
96 ' distances_key ': ' distances ' ,
97 }
98
99 return adata_integrated

55
 An AI system to help scientists write expert-level empirical software

Supplementary Table 5 | Expert manual inspection of adherence of tree search implementation

to method.

Method Replicate Judgment Notes

batchelor fastMNN 0 Follow
batchelor fastMNN 1 Follow
batchelor fastMNN 2 Follow
batchelor mnnCorrect 0 Follow
batchelor mnnCorrect 1 Follow
batchelor mnnCorrect 2 Follow
BBKNN 0 Follow Adds distances between batches, performs spectral
clustering on the graph. Does not compute connectiv-
ities.
BBKNN 1 Follow + Innovative Standardize + ComBat + PCA for embedding. BBKNN
implemented on that embedding.
BBKNN 2 Follow Corrects the data, computes neighbors, final embed-
ding is UMAP supposedly based on neighbors.
ComBat 0 Follow
ComBat 1 Follow
ComBat 2 Follow
Harmony 0 Follow Entropy-based diversity penalty.
Harmony 1 Follow Linear diversity penalty.
Harmony 2 Follow Linear diversity penalty.
LIGER 0 Follow Uses [Link] with multiplicative update solver.
LIGER 1 Follow Writes NMF function from scratch. Builds single
global KNN graph rather than by batch.
LIGER 2 Not relevant Uses ComBat + SVD.
No advice 0 Follow Uses batch-specific mean+std for all genes to rescale.
Then PCA.
No advice 1 Follow ComBat + SVD
No advice 2 Follow ComBat + PCA
SCALEX 0 Follow Adds log_var clipping and weight normalization.
SCALEX 1 Follow Learns batch embedding. Learns gamma and beta
conditioned on batch index. Batch index not supplied
to first layer of decoder.
SCALEX 2 Follow Uses min_delta for robust early stopping. batch_index
not supplied to the first layer of the decoder.
Scanorama 0 Follow
Scanorama 1 Not relevant Implements mnnpy via
[Link].mnn_correct.
Scanorama 2 Follow
scVI 0 Follow Applies log1p scaling with ZINB loss. Fits global dis-
persion theta rather than batch-specific.
scVI 1 Follow Applies optional log1p scaling with ZINB loss. Fits
global dispersion theta rather than batch-specific.
scVI 2 Follow Expression frequency exponentiated rather than soft-
maxed. Applies log1p scaling with ZINB loss. Fits
global dispersion theta rather than batch-specific.
TabVI 0 Follow
TabVI 1 Follow
TabVI 2 Follow
56
 An AI system to help scientists write expert-level empirical software

Supplementary Table 6 | Prompt for recombination of baseline method ideas. The prompt instructs
Gemini to identify the main differences in the principles of top-performing solutions, obtained from
tree search runs seeded with baseline methods. This generated summary then serves as part of an
explicit instruction for tree search to create hybrid strategies.

Prompt for summarizing differences between two baseline methods.

Compare these two code solutions to the same problem of integrating single-cell batch effects.
Explain the main principles that differ between the codes:

CODE 1: [CODE FROM BASELINE 1]

CODE 2: [CODE FROM BASELINE 2]

57
 An AI system to help scientists write expert-level empirical software

Supplementary Table 7 | Method descriptions used for replicating COVID-19 models submitted
to the CDC’s CovidHub.

CEPH-Rtrend_covid
“Use a renewal equation method based on Bayesian estimation of Rt from hospitalization data.
Model forecasts should be obtained by using a renewal equation based on the estimated net
reproduction number Rt. Apply a lowpass filter to the time series of weekly hospitalizations,
then interpolate it to daily resolution. Then use MCMC Metropolis-Hastings sampling to
estimate the posterior distribution of Rt based on the filtered data, considering an informed
prior on Rt based on COVID-19 literature. The estimated Rt in the last weeks of available data
is used to forecast Rt in the upcoming weeks, with a drift term proportional to the current
incidence. Finally, use the renewal equation with the posterior distribution and trend of the
estimated Rt in the most recent weeks of hospitalization data."

CMU-TimeSeries
“Use an ensemble of AR-based time-series models, involving a basic quantile autoregression fit
using lagged values of covid-related hospitalization counts (normalized by population). The
data should be smoothed in time. Fit the model jointly across all jurisdictions using the most
recently available 21 days of training data. Learn each of the 23 quantiles using a separate
quantile regression with nonnegativity and quantile sorting constraints applied post hoc."

CMU-climate_baseline
“Use an ensemble of historically formed quantiles. Using data from 2022 onwards, this clima-
tological model should use samples from the 7 weeks centered around the target week and
reference week to form the quantiles for the target week, as one might use climate information
to form a meteorological forecast. To get more variation at some potential issue of general-
ization, one can form quantiles after aggregating across geographic values as well as years
(after converting to a rate based case count). This model should use a simple average of the
geo-specific quantiles and the geo-aggregated quantiles."

JHU_CSSE-CSSE_Ensemble
“Use a Multi-Pathogen Optimized Geo-Hierarchical Ensemble Framework (MPOG-Ensemble).
Forecast state-level COVID-19 hospitalizations using a combination of time series forecasting
methods, organized across three hierarchical levels. At the individual state level, forecasts are
generated using Holt-Winters Exponential Smoothing. For regional predictions, which group
states based on past 2 years covid-19 activity trends identified through the Louvain method,
Long Short-Term Memory (LSTM) models are employed. Additionally, a LSTM model that
covers all states is implemented. These three-tiered model outputs are integrated, selecting
weights based on their recent performance in terms of Mean Absolute Error (MAE) to produce
the final prediction."

58
 An AI system to help scientists write expert-level empirical software

OHT_JHU-nbxd
“Use a neural network that encodes the data inputs using a TCN (Bai et al. 2018) and decodes
the result into a forecast using N-BEATS (Oreshkin et al. 2000). This is a residual block type
architecture that generates point forecasts from univariate time series data. The network
accepts a fixed lookback window of time points as input, and has a set number of output
nodes corresponding to the length of the forecast horizon. Extend the network with additional
residual blocks that output error variance forecasts (evaluated using a likelihood loss function)
which allows generating quantile forecasts, assuming a parametric (gamma) error distribution.
Additional predictor variables are incorporated using a temporal convolutional network (TCN;
Bai et al. 2018). The TCN accepts one input channel for each predictor time series (or static
variable), including past values of the target variable, and outputs a single channel with the
same length as the lookback window. The TCN output channel is used as the input to the
extended N-BEATS network. Each value in the TCN output sequence is a non-linear combination
of the predictor variables at that point and all previous points in the lookback window, which
preserves the temporal structure of the input. Forecast is the median of an ensemble of such
models with varying lookback window sizes and random initializations."

UM-DeepOutbreak

“Use a deep neural network model with conformal predictions. The neural network architecture
is a sequence-to-sequence model based on recurrent units and self-attention modules. It
is trained in a multi-task setting where each region is considered a task. The uncertainty
quantification is conducted post hoc with conformal predictions that follows adaptive conformal
inference to adapt to distribution shifts. Spatial correlation is not considered."

UMass-ar6_pooled

“Use an autoregressive model with shared coefficients across locations: AR(6) model after
fourth root data transform. AR coefficients are shared across all locations. A separate variance
parameter is estimated for each location."

UMass-gbqr

“Use gradient boosting quantile regression. Do gradient boosting using features summarizing
signal activity, properties of the location, information about the timing of forecast creation, and
the forecast horizon."

59
 An AI system to help scientists write expert-level empirical software

Supplementary Table 8 | Prompt for replicating COVID-19 models submitted to CovidHub by

injecting method descriptions as {method} into existing tree search prompt.

Prompt for replicating models submitted to CovidHub.

Please write the python code to work on a competition.

{method}
I’ve already loaded the train / test files and split out the x and y parts.
Please provide a new definition for the function below, complete with imports, that will
generalize well. However, do not do any cross-validation in here. Your function should expect
options to be passed in via the config argument. I’ll use cross-validation myself to select which
of the options in the config_list generalizes best.
{method}

from typing import Any # Don't forget this!

import pandas as pd

def fit_and_predict_fn(
train_x: [Link],
train_y: [Link],
test_x: [Link],
config: dict[str, Any]) -> [Link]:
"""Make predictions for test_x by modeling train_x to train_y.
Do not do any cross-validation in here.
"""
mean_y = [Link](train_y)
return [Link]([mean_y] * len(test_x), index=test_x.index)

# These will get scored by code that I supply. You'll get back a summary
# of the performance of each of them.

config_list = [{}]

And format it like this:

# YOUR CODE
# YOUR config_list

60
 An AI system to help scientists write expert-level empirical software

Supplementary Table 9 | Expert manual inspection of adherence of tree search implementation

to COVID-19 modeling methods.

Method Judgment Notes

CEPH-Rtrend_covid x Follow
CMU-TimeSeries
CEPH-Rtrend_covid x Follow
CMU-climate_
baseline
CEPH-Rtrend_covid x Follow
JHU_CSSE-
CSSE_Ensemble
CEPH-Rtrend_covid x Follow Translates 𝑅𝑡 into engineered features (lagged differences,
OHT_JHU-nbxd ratios).
CEPH-Rtrend_covid Follow Feeds mechanistic-inspired features into GRU-based en-
and coder, predicts quantiles via pinball loss.
UM-DeepOutbreak
CEPH-Rtrend_covid x Follow + Simulates from normal distribution in transformed space
UMass-ar6_pooled Innovate then inverse transforms to derive quantiles.
CEPH-Rtrend_covid x Follow Implements mechanistic model components as input fea-
UMass-gbqr tures to ML model.
CMU-TimeSeries x Follow AR model with climatological features as predictors.
CMU-
climate_baseline
CMU-TimeSeries x Follow Hierarchical ensemble of QuantReg AR models with
JHU_CSSE- performance-based weighting.
CSSE_Ensemble
CMU-TimeSeries x Follow Ensemble of bagged QuantReg AR models.
OHT_JHU-nbxd
CMU-TimeSeries x Follow LightGBM quantile regression models with iterative fore-
UM-DeepOutbreak casting + conformal-like calibration.
CMU-TimeSeries x Follow Ensemble of AR QuantReg models on fourth-root trans-
UMass-ar6_pooled formed data.
CMU-TimeSeries x Follow LightGBM quantile models on population-normalized data
UMass-gbqr with (un)smoothed lags + direct multi-horizon prediction.
CMU- Follow Hierarchical ensemble of climatological models.
climate_baseline x
JHU_CSSE-
CSSE_Ensemble
CMU- Follow Feeds climatological quantiles into LightGBM to learn di-
climate_baseline x rectly from seasonal baseline.
OHT_JHU-nbxd
CMU- Follow LightGBM to predict central trend + climatological model
climate_baseline x for empirical quantile spreads.
UM-DeepOutbreak
CMU- Follow Seasonally-aware method for estimating uncertainty based
climate_baseline x on empirical quantiles of AR residuals.
UMass-ar6_pooled
Continued on next page

61
 An AI system to help scientists write expert-level empirical software

Supplementary Table 9 – continued from previous page

Method Judgment Notes
CMU- Follow Feeds climatological statistics as features into LightGBM.
climate_baseline x
UMass-gbqr
JHU_CSSE- Partially Hierarchical structure (state, regional, national models) +
CSSE_Ensemble x Follow adaptive MAE-weighting.
OHT_JHU-nbxd
JHU_CSSE- Follow + Secondary model to predict error magnitudes & find quan-
CSSE_Ensemble x Innovate tiles of normalized residuals.
UM-DeepOutbreak
JHU_CSSE- Follow
CSSE_Ensemble x
UMass-ar6_pooled
JHU_CSSE- Follow Combines predictions from ‘adaptive’ model trained on re-
CSSE_Ensemble x cent data & ‘stable’ model trained on longer history.
UMass-gbqr
OHT_JHU-nbxd x Follow
UM-DeepOutbreak
OHT_JHU-nbxd x Follow Feature engineering + ensembling + variance-stabilizing
UMass-ar6_pooled transformation _ recursive forecasting.
OHT_JHU-nbxd x Follow Uses LightGBM predicts parameters of Gamma distribution.
UMass-gbqr
UM-DeepOutbreak x Follow
UMass-ar6_pooled
UM-DeepOutbreak x Follow
UMass-gbqr
UMass-ar6_pooled x Follow LightGBM quantile regression on fourth-root transformed
UMass-gbqr target.
DEEP-RESEARCH- Follow Simplified static graph + synthetically generated policy
CSTGT feature.
DEEP-RESEARCH- Follow Meta-model to predict WIS.
MetaEnsembler
DEEP-RESEARCH- Follow Iterative re-weighting approximates composite fairness loss.
FairnessAwareOptimization
DEEP-RESEARCH- Follow
RegimeSwitchingDetection
CO-SCIENTIST- Does not AR quantile regression model using LightGBM. Omits AgACI
STGNN-AgACI Follow stage, replaces with simpler post-processing.
CO-SCIENTIST-MAPS Partially 3-stage ensemble: substitutes core models (GNN, TCN, GPR,
Follow MLP) with feature-engineered LightGBM proxies.
DEEP-RESEARCH- Follow Uses mock API calls.
GenomiWastewater
Fusion
DEEP-RESEARCH- Follow + Implements a post-hoc GAN structure. Composite loss func-
AdversarialRecalibration Innovate tion combining adversarial + pinball loss.
DEEP-RESEARCH- Follow Simulates external data.
BehavioralSensing
Continued on next page

62
 An AI system to help scientists write expert-level empirical software

Supplementary Table 9 – continued from previous page

Method Judgment Notes
DEEP-RESEARCH- Follow Three-level model: Negative Binomial observation layer,
HierarchicalBayesian Neural ODE for jurisdiction-level dynamics, global hyper-
NODE priors for partial pooling.
CO-SCIENTIST-HGPC Partially LightGBM quantile regression, uses feature engineering as
Follow proxy for complex stages.
DEEP-RESEARCH- Follow Implements conditional Denoising Diffusion Probabilistic
PIDM Model (DDPM) with U-Net backbone, with loss function a
weighted composite of standard diffusion loss and a physics-
based regularization term derived from an SEIR-H model’s
outputs. Probabilistic forecasts generated by sampling from
the learned reverse process.
CO-SCIENTIST-HQE Partially Trains multiple base models, feeds their predictions into
Follow a meta-learner, then applies a conformal prediction step
to adjust final quantiles. Uses multiple LightGBM models
instead of suggested Prophet/TBATS for diversity, manually
implements conformal prediction instead of using MAPIE.
DEEP-RESEARCH- Follow + Follows Monte Carlo structure: defines uncertain drivers
CounterfactualSimulation Innovate with distributions, simulates N trajectories by applying sam-
pled shocks to base median forecast, calculates empirical
quantiles. Introduces Poisson noise on top of scenario-
driven forecasts.
rep-OHT_JHU-nbxd Follow Implements TCN encoder and N-BEATS decoder architec-
ture, including extension of parallel residual blocks to fore-
cast mean and variance for Gamma distribution. The final
forecast is generated as a median of an ensemble with vary-
ing lookback windows and initializations.
rep-CMU-TimeSeries Follow Implements a quantile autoregression model fit jointly
across jurisdictions on smoothed, population-normalized
data.
rep-UMass- Follow Uses OLS on lagged, fourth-root transformed data to create
ar6_pooled a shared-coefficient AR model, then calculates separate
variance parameters for each location based on residuals.
rep-UM- Follow Implements sequence-to-sequence model using a GRU and
DeepOutbreak self-attention, with location embeddings. Uncertainty quan-
tified post hoc using split conformal prediction on a recent
time window.
rep-UMass-gbqr Follow Uses LightGBM with engineered features (lags for signal
activity, location and population for location properties,
date components for timing, and the horizon itself).
rep-JHU_CSSE- Follow + Implements three-tiered hierarchical ensemble, using Holt-
CSSE_Ensemble Innovate Winters, regional LSTMs with Louvain grouping, and a
national LSTM, combined with MAE-based weighting. Uses
scaled residuals to create prediction intervals that adapt to
the magnitude of the forecast to generate quantile predic-
tions.
Continued on next page

63
 An AI system to help scientists write expert-level empirical software

Supplementary Table 9 – continued from previous page

Method Judgment Notes
rep-CMU- Follow + Averages geo-specific and geo-aggregated quantiles within a
climate_baseline Innovate centered weekly window. Introduces a configurable ’smooth-
ing_factor’, which regularizes final predictions by pulling
them towards zero.
rep-CEPH- Follow Lowpass filtering, daily interpolation, MCMC for Bayesian
Rtrend_covid Rt estimation, and a renewal equation forecast. The Rt
forecast correctly incorporates a sophisticated drift term
that is modulated by the current incidence level.
retro_1 Follow

64
 An AI system to help scientists write expert-level empirical software

Supplementary Table 10 | Full GIFT-Eval leaderboard (05/18/2025 snapshot)

Model MASE Type

Per-dataset 0.671 tree-search
TTM-R2-Finetuned 0.679 fine-tuned
timesfm_2_0_500m 0.680 pretrained
TabPFN-TS 0.692 pretrained
chronos_bolt_base 0.725 pretrained
Unified 0.734 tree-search
chronos_bolt_small 0.738 pretrained
PatchTST 0.762 deep-learning
TEMPO_ensemble 0.773 fine-tuned
VisionTS 0.775 pretrained
Chronos_large 0.781 pretrained
Moirai_large 0.785 pretrained
Chronos_base 0.786 pretrained
Chronos_small 0.800 pretrained
Moirai_base 0.809 pretrained
TFT 0.822 deep-learning
N-BEATS 0.842 deep-learning
Moirai_small 0.849 pretrained
TTM-R2-Zeroshot 0.915 pretrained
DLinear 0.952 deep-learning
Auto_Arima 0.964 statistical
TimesFM 0.967 pretrained
TTM-R1-Zeroshot 0.969 pretrained
Auto_Theta 0.978 statistical
TIDE 0.980 deep-learning
Seasonal_Naive 1.000 statistical
Timer 1.019 pretrained
Auto_ETS 1.088 statistical
Lag-Llama 1.102 pretrained
DeepAR 1.206 deep-learning
Naive 1.260 statistical
Crossformer 2.310 deep-learning

65
 An AI system to help scientists write expert-level empirical software

Supplementary Table 11 | Three example configurations from the final unified solution. Each
dictionary defines a complete forecasting strategy discovered by the tree search, combining different
components of the Iterative Decomposition Model. The validation process selects the best configuration
for each dataset.

Unified Solution Example Configurations

config_list = [
{
'name': 'seasonal_naive_baseline',
'description': 'Robust baseline...',
'components': [{'type': 'base', 'method': 'seasonal_naive_adaptive'}],
'transform_log': False, 'non_negative': False, 'version': 4,
},
{
'name': 'additive_damped_linear_LogTransform',
'description': 'General-purpose additive model...',
'components': [
{'type': 'base', 'method': 'median_all'},
{'type': 'trend', 'method': 'polynomial', 'degree': 1, 'damping_factor': 0.90},
{'type': 'seasonal', 'method': 'average', 'window_multiplier': 5.0},
{'type': 'residual', 'method': 'median', 'window_size': 18, 'decay_factor': 0.90},
],
'transform_log': True, 'non_negative': True, 'version': 4,
},
{
'name': 'date_features_seasonal',
'description': 'Robust additive model with key cyclical and datetime features...',
'components': [
{'type': 'base', 'method': 'median_all'},
{'type': 'datetime', 'features': [
['dayofweek', 'hour'], 'month', 'is_month_start', 'weekofyear',
'is_weekend', 'is_quarter_start',
{'name': '_is_holiday_flag',
'country_codes': ['US', 'DE', 'CN', 'GB', 'CA', 'AU']}
]},
{'type': 'seasonal', 'method': 'average', 'window_multiplier': 4.0},
{'type': 'residual', 'method': 'median', 'window_size': 14, 'decay_factor': 0.92},
],
'transform_log': False, 'non_negative': False, 'version': 4,
},
% ... other configurations can be added here ...
]

66
 An AI system to help scientists write expert-level empirical software

Supplementary Table 12 | Prompt for Gemini Deep Research to generate ideas to integrate
single-cell batch effects.

Prompt for Gemini Deep Research.

I am developing new methods for winning single-cell batch integration competitions, as

proposed by the Kaggle and extensively researched in the single-cell genomics community.

Briefly: Modelers are asked to develop a function, eliminate_batch_effect_fn, that

transforms raw gene expression count data from multiple batches into a low-dimensional
embedding or feature matrix. This transformed output should effectively remove technical
variation (batch effects) while rigorously preserving biological information (e.g., cell type
identity). The performance of these methods is evaluated against a suite of metrics that
quantify both batch mixing and biological conservation.

The key problem is to develop a method that takes an AnnData object of raw gene ex-
pression counts with batch labels and returns an AnnData object with a batch-integrated
low-dimensional embedding in the .obsm['X_emb'] field. The method must excel across
a diverse set of evaluation metrics, including ASW Batch, ASW Label, ARI, NMI, Graph
Connectivity, Isolated Labels ASW, Isolated Labels F1, kBET, iLISI, cLISI, PCR, and Cell Cycle
Conservation Score, aiming to maximize their average.

The following principles should be obeyed when choosing models:

* **Batch Effect Removal**: Prioritize techniques that explicitly model and mitigate batch-
specific variations without collapsing biological signal.
* **Biological Conservation**: Ensure the integrated representation retains and accurately
reflects genuine biological differences, particularly cell type distinctions, as measured by
clustering and silhouette metrics.
* **Scalability and Efficiency**: Given the large dataset sizes (e.g., 329, 762 cells × 2, 000
genes), models must be computationally efficient and avoid out-of-memory errors.
* **Constraint Adherence**: The implementation must strictly avoid using cell_type
information during integration and should primarily leverage scanpy, sklearn, numpy, scipy,
tensorflow, torch, jax, or equivalent native implementations rather than specialized
single-cell packages.

This task aims to develop a SUPERHUMAN METHOD for solving this problem.

Please give me 10 highly novel and creative ideas with detailed implementation notes for the
set of methods I should explore for solving this task. I aim to create the best method for solving
this problem, preferably creating the best ever method.

67
 An AI system to help scientists write expert-level empirical software

Supplementary Table 13 | Prompt for formatting Deep Research ideas into a structure similar
to baseline method descriptions.

Prompt for formatting Deep Research ideas.

Structure the given idea into the following format:

<description>
Your description about the method goes here.
</description>

<steps>
Your list of steps to implement the method goes here.
</steps>

<notes>
Strengths and weaknesses of the idea goes here.
</notes>

68
 An AI system to help scientists write expert-level empirical software

Supplementary Table 14 | Prompt for guiding tree search to generate hybrid strategies.

Prompt for guiding tree search to generate hybrid strategies.

We have up until now done experiments with two major types of codes, that are described
in detail below. PLEASE CREATE AN ALGORITHM THAT USES THE BEST PARTS OF BOTH
STRATEGIES TO CREATE A HYBRID STRATEGY THAT IS TRULY WONDERFUL AND SCORES
HIGHER THAN EITHER OF THE INDIVIDUAL STRATEGIES.

[Output from Supplementary Table 6.]

69
 An AI system to help scientists write expert-level empirical software

Supplementary Table 15 | Prompt for summarizing existing batch integration methods. The
prompt instructs the LLM to write a brief description and step. This generated summary then serves
as part of an explicit instruction for tree search to create optimized version of the existing batch
integration method. Example output in shown in Supplementary Table 16.

Prompt for summarizing differences between two baseline methods.

Given the following paper, please identify the main method being proposed. Then write a very
short method description. This method description will be used to reproduce the method. DO
NOT mention the algorithm by name.

Your output must follow this format:

<description>
Your very short description goes here.
</description>

<steps>
Your short list of steps goes here.
</steps>

70
 An AI system to help scientists write expert-level empirical software

Supplementary Table 16 | Example of existing method description (BBKNN) given to tree search.
Tree search used this short method description to further optimize the method.

Example of existing method description (BBKNN) given to tree search.

<description>
This method performs batch correction by modifying the neighborhood graph construction
step. For each cell, its nearest neighbors are identified independently within each batch, rather
than across the entire combined dataset. The resulting batch-specific neighbor lists for each
cell are then merged to create a single, integrated graph. This approach assumes that shared
cell types exist across batches and that biological differences are greater than technical batch
effects.
</description>

<steps>

1. For each cell, iterate through every batch in the dataset.

2. Find the k-nearest neighbors for the cell from within the current batch, based on a given
distance metric (e.g., Euclidean distance in PCA space).
3. After iterating through all batches, merge the identified neighbor sets for the cell into a
single neighborhood.
4. Repeat for all cells to construct a batch-corrected neighborhood graph.

</steps>

71