Code and data accompanying:
A Systematic Approach to Batch Active Learning for Material Optimization Andrea Gardin et al. (journal / DOI to be added on acceptance)
Important
This repository is a frozen archive. It is deposited as the artifact of the paper above and is not developed further. It is self-contained: everything needed to reproduce the figures — the framework, the ground-truth landscapes, the raw campaign results and the analysis notebooks — is in this repository. No external download is required.
The framework itself (activereg) continues to be developed elsewhere; that
development is not reflected here, and the version archived here is the one
that produced the published results.
activereg/ The active-learning framework (installable package)
scripts/ Campaign drivers, configuration templates, analysis helpers
datasets/ Ground-truth landscapes (gzipped) and lab design spaces
benchmarks/ Raw results of the 65 in-silico campaigns
lab_experiments/ Raw data and analysis of the robotic-lab campaigns
figures/ Notebooks that generate the paper figures, plus the panels
requirements-paper.txt Exact package versions used to produce the results
The results were produced with Python 3.12.1. To recreate that environment:
conda create -n batchal python=3.12.1
conda activate batchal
pip install -r requirements-paper.txt
pip install -e .requirements-paper.txt pins the exact versions used for the published runs.
It is layered — only the first block is needed to regenerate the figures from
the archived results; the later blocks are needed to re-run campaigns from
scratch.
If you only want to read the data, no installation is needed: every result is a plain CSV.
The package installs without PyTorch. Heavier dependencies are optional extras:
| Extra | Provides | Needed for |
|---|---|---|
| (core) | GPR and k-NN surrogates, the AL loop, all plotting | Every figure; all 3D runs and the 6D GPR/KNN runs |
nn |
MLP, AnchoredEnsembleMLP, BayesianNN (PyTorch + Pyro) |
Re-running the 6D MLPA campaigns |
benchmarks |
Synthetic test functions, landscape generation (BoTorch) | Regenerating the ground-truth landscapes in datasets/ |
pip install -e '.[all]' # or '.[nn]' / '.[benchmarks]'Neural-network models are imported lazily, so a core install raises an
explanatory ImportError naming the missing extra rather than failing at
import time.
The archived campaign results are sufficient — you do not need to re-run any campaign to regenerate the paper figures.
cd figures
jupyter lab| Notebook | Produces | Reads from |
|---|---|---|
results_figure_1.ipynb |
FIG_1_panels/ |
benchmarks/3D/PURE_BATCHES/ |
results_figure_2.ipynb |
FIG_2_panels/ |
benchmarks/3D/ (all four groups) |
results_figure_3.ipynb |
FIG_3_panels/ |
benchmarks/6D/ (all three surrogates) |
lab_analysis.ipynb |
lab_analysis_panels/ |
lab_experiments/analysis/ |
The generated panels are committed alongside the notebooks, so the published
figures can be inspected without running anything. Notebooks locate data through
activereg.format, which resolves paths relative to the repository root — they
work from a fresh clone with no path editing.
figures/benchmark_landscape/ holds the ground-truth landscape illustrations.
65 campaigns, 5 independent repetitions each.
| Group | Runs | Focus |
|---|---|---|
PURE_BATCHES/ |
15 | Single acquisition mode per batch — Hartmann-3D |
MIXED_BATCHES_H/ |
17 | Mixed-mode batches — Hartmann-3D |
MIXED_BATCHES_A/ |
3 | Mixed-mode batches — Ackley-3D |
MIXED_BATCHES_ST/ |
3 | Mixed-mode batches — Styblinski-Tang-3D |
A complete 3 x 3 x 3 factorial:
- Functions: Hartmann-6D, Ackley-6D, Styblinski-Tang-6D
- Surrogates:
GPR/,KNN/,MLPA/(anchored-ensemble MLP) - Budgets: 100 cycles x 2 points, 40 x 5, 20 x 10 — 200 acquisitions each
Across all runs the acquisition modes exercised are expected_improvement,
exploration_mutual_info, uncertainty_landscape and random; batch selection
is highest_landscape throughout, varying the percentile and the within-batch
sampling method.
| File | Description |
|---|---|
benchmark_data.csv |
Per-cycle metrics — the primary scientific record |
train_points_data.csv |
Every acquired point, with cycle, repetition and acquisition source |
config/ |
The full configuration: benchmark_config.yaml, model_config.yaml, acquisition_mode_settings.yaml, plus target_function_config.yaml for the 6D runs |
pool_scaler.joblib |
Fitted feature scaler |
ml_model_rep*.joblib |
Final fitted surrogate per repetition |
config.original.yaml |
(3D only) the as-executed configuration — see below |
grid_search_fixed_params.yaml |
(18 runs) parameters held fixed during grid search |
benchmark_data.csv columns:
repetition, cycle, y_best_screened, y_best_predicted_pool,
y_best_predicted_val, rmse_vs_gt_pool, mae_vs_gt_pool, rmse_vs_gt_val,
mae_vs_gt_val, nll_val, picp95_val, mpiw95_val
Each run directory carries the complete configuration that produced it, so a
campaign is re-launched by pointing the driver at that run's config/.
The two dimensionalities used different drivers. The 3D campaigns sampled a
precomputed ground-truth landscape (benchmark_gtlandscape.py); the 6D
campaigns evaluated the analytic test function directly, using the landscape only
as a holdout validation set (benchmark_functions.py, which additionally takes
--target_function_config and therefore needs the benchmarks extra).
# 3D — core install is enough
RUN=benchmarks/3D/PURE_BATCHES/<run>
python scripts/benchmark_gtlandscape.py \
--benchmark_config $RUN/config/benchmark_config.yaml \
--model_config $RUN/config/model_config.yaml \
--acquisition_mode_settings $RUN/config/acquisition_mode_settings.yaml \
--repetitions 5
# 6D — requires the `benchmarks` extra (and `nn` for the MLPA runs)
RUN=benchmarks/6D/GPR/<run>
python scripts/benchmark_functions.py \
--benchmark_config $RUN/config/benchmark_config.yaml \
--model_config $RUN/config/model_config.yaml \
--acquisition_mode_settings $RUN/config/acquisition_mode_settings.yaml \
--target_function_config $RUN/config/target_function_config.yaml \
--repetitions 5Add --rerun to overwrite an existing output directory. scripts/run_benchmark_gtlandsc.sh
and scripts/run_benchmark_funcs.sh are the batch wrappers originally used.
Ground-truth landscapes are stored gzipped so that every file stays under common
git-host size limits. The configurations still name the plain .csv, because
they are provenance records of what actually ran and were deliberately not
rewritten; activereg.format.resolve_dataset_path resolves either variant
transparently.
Some notes on how reproducibility of the experiment should be handled.
Configuration schema. The 38 3D campaigns were executed under an earlier,
single-file configuration schema and were migrated to the current multi-document
layout so that all 65 runs can be read uniformly. The migration is
value-preserving; the untouched original is kept next to each migrated run as
config.original.yaml, and the migration script is scripts/migrate_3d_configs.py.
Run paths inside configs. The experiment_name field records the output path
at execution time, before runs were grouped into the subfolders used here (for
example 6D/styblinskitang_gpr_100cy_2pts_1stage_1_1 rather than
6D/GPR/styblinskitang_gpr_100cy_2pts_1stage_1_1). The grouping is presentational
only; no data was altered.
Derived files are not archived. Per-run candidate pools (pool_dataset.csv)
and adaptive-refinement point dumps are omitted. They are large and are
regenerated deterministically from the ground-truth landscapes in datasets/.
Multi-property optimization. activereg contains code for multi-property
(scalarized, ParEGO-style) acquisition. No result in this paper uses it. It
is present because it was developed in the same codebase and could not be
separated cleanly, and it is included here so that the archived package is
exactly the one that ran. It is neither exercised nor validated by anything in
this repository.
Please cite the paper. (BibTeX entry to be added on acceptance.)
MIT — see LICENSE.